CSIS 150 – Lab 7: Making a multi-touch applet using TUIO

August 2010

Objectives:

Learn about the TUIO protocol
Learn how to use a library
Practice working with objects
Practice making your own classes
Practice working with ArrayLists
Practice working with applets

1 Introduction to TUIO and its uses

TUIO (Tangible User Interface Objects) is a protocol used for sending and receiving information (like the location of a user’s finger) in multi-touch systems. It is similar to internet data in the way that it sends packets of information and therefore can be thought of a constant stream of data that your program can keep an eye on. The reason an entire protocol was invented for this seemingly small purpose is so that input devices like multi-touch tables can be developed independently of the software that makes use of them. For example let’s say that you’ve already made a cool program for our multi-touch table and then somebody comes up with a new multi-touch hardware platform that you want to run your program on. If the new device implements the TUIO protocol you wouldn’t have to do any work whatsoever!

2 Adding the TUIO library to an Eclipse project

Create a new Java Project in Eclipse and call it TUIO_Program.
Create a new Class file in the src folder and also name it TUIO_Program.
Download the TUIO library by going to http://www.tuio.org/?software and downloading TUIO_JAVA.zip.
Unzip the contents of this file to the folder where you have created this project.
Add the libTUIO.jar file to your project’s build path the following way:
1. Right click on your project and go to Properties.
2. Go to Java Build Path and click on the Libraries tab.
3. Click the button that says Add External JARs.
4. Locate the TUIO_Java folder that you unzipped and add the file named libTUIO.jar.
5. Press OK.

3 Setting up TUIO in your applet

Now that your project knows how to find TUIO when you reference it you can now use it without breaking anything. At this point we will be working with the TUIO library as an object so there are a couple of things we need to do to actually use TUIO in our program. First we must import the TUIO library into our program, then we have to declare and instantiate an object. Because of the unpredictable and real-time nature of TUIO we must also have our program listen for new events. Here is how we do all that:

Set up your main Java file as an applet (you need an init() method and a paint() method).
Change the size of your applet window to 640×480 using variables (don’t hard-code the values) that you can reference later on in the program. (Use the setSize() JApplet method.)
The very first line of your program should be an import statement for the TUIO library, like so:
import TUIO.*;

Next we should have our program listen for TUIO events by making it implement TuioListener.
Eclipse will tell you there is an error and underline the class name in your program. When you implement a class like TuioListener you sometimes need to add a couple methods required by that class. If you hover over the class name you can click on “Add unimplemented methods”, Eclipse will take care of everything for you!
Now we need to actually tell our program to begin actively listening for TUIO events by declaring and instantiating an object (called TuioClient) and calling a couple of methods that belong to that object.
1. Inside of your init() method use the following line to set up the object:
  TuioClient client = new TuioClient();
2. Now we must add a listener for our TuioClient object so it knows where actually send events. The tricky thing is that you have to actually obtain a reference to your program as an object so you can pass it as a parameter to the TuioClient object! Here’s how:
  TUIO_Program app = new TUIO_Program(); client.addTuioListener(app);
3. Finally we call the connect method on the client object to get the party started.
  client.connect();

4 Using TUIO in your applet

Understanding what TUIO messages represent

In the case of our multi-touch table TUIO messages represent fingers touching the surface of the table. These messages contain information such as location (X-Y coordinates), velocity and the path of the finger. One thing that may be confusing upon looking at the methods that you had to add when implementing TuioListener is that TUIO refers to two different kinds of objects: TUIOCursor and TUIOObject.

TUIOCursor – an entity representing a non-specific shape on the surface such as a finger.
TUIOObject – an entity representing a very specific shape, like a 2D barcode or well-defined shape.

In general you will only need to work within the methods dealing with TUIOCursors because we usually just work with finger-based input and not well-defined shapes.

Working with TUIOCursors

A TUIOCursor can be thought of as being very similar to a mouse cursor with one key difference; a program can use many TUIOCursors simultaneously (multi-touch) whereas programs can usually only use one mouse cursor at a time. When you use a mouse cursor you usually work with methods such as mousePressed() or mouseReleased() and access mouse coordinates within those methods. When you want to work with TUIOCursors you should instead work with the following methods:

addTuioCursor – invoked automatically when a new blob has been detected.
updateTuioCursor – invoked automatically when properties of an existing blob have changed (such as location).
removeTuioCursor – invoked automatically when a previously identified blob is no longer detected.

You will notice that each of these methods has the parameter TUIOCursor arg0 which is a reference to the actual TUIOCursor that the event is referring to.

You can read about other methods you can call using the TUIOCursor JavaDoc at http://www.tuio.org/?java.

5 Visualizing the TUIO stream

5.1 Plan of attack

What we want to ultimately do is draw a circle on the screen for every TUIOCursor that is found. To do this we will want to create a custom class called Circle to represent each TUIOCursor. We will keep track of all of the TUIOCursors as Circles within our main Java class using the ArrayList class. The main Java program will then continuously loop through this ArrayList and render each of the Circles that it contains. The actual maintenance of this ArrayList (adding, removing objects) will be taken care of through the TUIOCursor methods mentioned in the previous section. For example, each time the addTuioCursor method is called a new Circle object should be created using the X-Y coordinates of the TuioCursor passed into that method. This Circle should then be added to the ArrayList, which the paint() method will automatically see and try to render.

5.2 Creating a custom class called Circle

Create a new class file and call it Circle. It should have four member variables integer: x, y and size, and long: id.
1. Make size final and initialize it to a reasonable value like 10.
2. The constructor should accept and assign values for id, x and y.
3. Create a method called render that accepts an object called g of type Graphics (this will be called from the main Java class’s paint method later). This method should set the active pen color to white (look at the method setColor) and draw a circle using the x, y and size variables

Hint: The Color class contains member variables for various common colors that you can reference. Also you may have a hard time finding a way to draw a circle, but try looking for a method that draws an oval instead.

Create getters and setters for each of the variables (except for size, because it is final).

5.3 Creating an ArrayList of Circles

Create an ArrayList of Circles in your main Java class with a broad enough scope that it can be accessed by multiple methods within the main Java class. Don’t forget to initialize it in the init() method!
Modify each of the three *TuioCursor methods to perform maintenance operations on the ArrayList but be sure to read this first!

NOTE: The X and Y values sent through TUIO are normalizedfloat values that are always between 0 and 1, while your applet window has its own width and height. This means that X and Y values in TUIO are more like percentages than coordinates, so make sure you multiply the X and Y values you get from each TuioCursor by the width and height of the applet window! Also, because a Circle object using integer X and Y values, and the TUIO X and Y values are float values, you will need to properly cast these variables.

addTuioCursor – Create a new Circle using the SessionID and X-Y coordinates of the TuioCursor object, then add it to the ArrayList.
updateTuioCursor – Search through the ArrayList for a Circle with the same SessionID of the TuioCursor that was passed into the method. If it is already in the ArrayList then update the Circle’s X and Y coordinates using the coordinates from the TuioCursor (be careful with your datatypes).
removeTuioCursor – Search through the ArrayList for a Circle with the same SessionID of the TuioCursor that was passed into the method. If it was found in the ArrayList, remove it.
Set up your main paint() method to draw (render) each of the Circles from the ArrayList
1. Remember to call the super-class version of paint
2. Loop through the ArrayList of Circles and call each Circle’s render() method that we set up earlier. This method should have been set up to accept the variable g so it can draw things to the screen.
3. Call repaint() at the very end of your paint method to cause the program to continuously draw the Circles on the screen.

6 Troubleshooting your TUIO program

6.1 Absolutely nothing is happening

Make sure that CCV is open, configured correctly and transmitting TUIO messages. When you touch the MT Mini you should see white blobs show up in the Tracked Image area. These blobs must have ID numbers floating next to them. Go back to the Configuring CCV section to make sure things are set up correctly.
Save your project and restart Eclipse and then run your applet again. If you had CCV open before and during the making of your program the TUIO library may have not been paying attention.
Make sure that you don’t already have another applet window open. Only one applet can use TUIO on port 3333 at a time.