KinectDTW and XNA Game

Jun 22, 2012 at 8:36 AM

Hello All,

In February I started an internship with the goal to create an application that can be controlled by Kinect. So far I have created a library which contains my own algorithm to detect some gestures. A few days ago I started to implement the KinectDTW technique in my library, which was successful. Right now I can switch between two algorithm to detect some gestures, that will move a game object to the left, right, up, down or zoom in/out.

Here's a screenshot of my game, and an object (plane) moving according to gestures:

https://fbcdn-sphotos-a.akamaihd.net/hphotos-ak-ash4/288366_459038310791189_574767804_o.jpg

Now that this is all working, I have to say that KinectDTW works pretty fine. But I can't seem to find the right settings for it to work any better, since 'pretty fine' is not enough in my opinion. It happens a lot that when I perform a "SwipeUp" (move hand upwards), the algorithm thinks I'm still performing a "SwipeLeft" (move hand left, duh). I recorded the gestures about 3 times with the KinectDTW WPF application to make sure the fault wouldn't be there.

I have been searching through the discussions around here, but couldn't find any good settings. So here are the current settings I'm using:

MinimumFrames = 6
BufferSize = 24
Ignore = 1

dim = 12
threshold = 0.6
firstThreshold = 2
ms = 2
minLen = 10

I hope anyone of you could tell me something about improving the performance of the algorithm since nobody has done so far in the discussions.

Thanks in advance and my kind regards,

André

Jul 3, 2012 at 10:05 AM
Edited Jul 3, 2012 at 10:45 AM

Hello André,

Could you share how you were able to utilise the KinectDTW to recognise gestures in realtime? If possible could you share your source?

And as for your question, I think optimization can be done in recognising the gesture. Maybe you can perform an initial comparision between the first few frames of  the different gesture files and then select the top ones and continue analysing those with the DTW. Run this recursively and you should get the best match. Should lower performance head and also increase recognition speed.

Thanks

 

Also this is there-http://kinectdtw.codeplex.com/discussions/273744

Jul 5, 2012 at 8:47 AM

It's actually pretty easy. You have an update loop in XNA in which you can call DTW for a gesture. You could do something like this:

Gesture g = DetectGestures();

Given that "Gesture" is an enum, you would be able to switch over them and act differently on each Gesture, every update.

 

The only struggle is that KinectDTW itself is event based, which I didn't like to merge into my game. So I have done some things:

- copied the ProcessData method of the internal class Skeleton2DDataExtract, gave it a return type of Point[].

- copied and renamed the internal class Skeleton2DdataCoordEventArgs to Skeleton2DdataCoords, which needs to be instantiated for use with KinectDTW.

- copied the code from NuiSkeleton2DdataCoordReady to a method "public Gesture DetectGestures()".

- Added a little piece of code to DetectGestures that instantiates a Skeleton2DdataCoords object:

            Skeleton2DdataCoords dtwSkelData = new Skeleton2DdataCoords(ProcessData(getLatestSkeleton());

- getLatestSkeleton() is a new method that returns a Skeleton object from the Kinect.

 

These changes allow me to ask the algorithm for a recognized gesture instead of waiting for events. So now the "CoordReady" event is replaced by the game's update loop calling for gestures through "DetectGestures()". This method uses the internal class "Skeleton2DdataCoords" to get the data that was normally delivered by the event's arguments: Skeleton2DdataCoordEventArgs. The data can be created with the ProcessData method that was found in Skeleton2DDataExtract, returning a Point array instead of firing the event.

Thanks for your insight :) I'll see what I can do!