Summary:
This paper examines the question of whether gestures without a relationship to a specific meaning are speaker independent or whether gestures are determined by the speaker's view of a discourse topic. To answer the question, a new characterization of gestures in terms of spatiotemporal interest points is used to extract a low level representation of gestures from video without manual intervention. The interest points are grouped into categories of codewords. The distribution of these codewords across speakers and topics form the basis for analysis.
During the spatiotemporal interest point extraction from images, principal component analysis is used to reduce the dimensionality of the data to form the final feature vectors describing the gestures. A hierarchical Bayesian model was used to cluster gesture features and determine whether the gesture is related to the speaker or the discourse topic. To allow inference of the hidden variables in the Bayesian model, Gibbs sampling is used since it is guaranteed to converge to the true distribution over the hidden variables in the limit.
An experiment was conducted to analyze 33 short videos of fifteen different speakers covering five different topics. Four of the topics of discussion were chosen to be mechanical devices, since direct gestural metaphors would likely show up in the discussion of such items. The study concluded that gestures are topic dependent and the relationship overshadows specific speakers. With correct labeling of topics, 12% of gestures were related to the topic, but randomly labeled topics, less than 3% of gestures are considered topic-specific.
Discussion:
The interest point features that were extracted exist at a lower level of abstraction than most other attempts at describing gestures. It is interesting that such low-level attributes are sufficient to perform successful recognition.
I think the limitation to mainly mechanical devices as topics could have influenced the ability to spot topic-specific gestures. If general categories were chosen, I think the topic-specific nature of gestures would be reduced.
I was not familiar with all of the mathematics discussed in the paper, so I'm not sure how sound their technique is from a theoretical point of view. Looking at the pictures of identified interest points and the frames before and after the interest point convinced me that the feature extraction must work in some cases.
Friday, May 2, 2008
Tuesday, April 29, 2008
3D Visual Detection of Correct NGT Sign Production (Lichtenauer 2007)
Summary:
In this paper, a 3D visual detection system is proposed to aid children in learning Dutch sign language. Two video cameras with wide angle lenses capture 640x480 resolution images at 25 frames per second. The user's head and hands are tracked based on following skin-colored segments of the image from frame to frame. Adaptive skin color modeling determines how skin color appears under different lighting conditions, but must first be initialized by selecting some pixels within the face to and a square of pixels surrounding the head. In practice, the colors of pixels showing skin were distributed in a bimodal manner due to two different light sources. Each modality was modeled separately to reduce mis-classification of pixels as skin colored or not. Classification of gestures is done based on fifty hand features. Dynamic Time Warping is used to find the level of time-correspondence between an input gesture and a reference gesture.
A set of 120 different NGT signs performed by 70 individuals are used to test the sign classification. Cross validation was performed to effectively increase the amount of test data many times. Overall, the true positive classification rate was 95%. Dynamic time warping only improved recognition of signs with repetitious motion.
Discussion:
The classification algorithm relies on knowing the approximate start and end times of a sign. Some other kind of segmentation scheme could be applied so that the hands must not always come to rest on the table between signs. As a teaching tool, resting between signs limits the learning to single words. Multi-gesture phrases and sentences cannot be recognized without segmentation.
The left and right blobs identified during skin detection tracking are assigned to the left and right hands, regardless of which hand is on which side of the body. So, crossing hands will cause skin blobs to be mis-labeled, giving rise to the possibility of classifying two distinct gestures (both consisting of the same basic motion, but one in which hands cross and one in which they do not) as the same.
In this paper, a 3D visual detection system is proposed to aid children in learning Dutch sign language. Two video cameras with wide angle lenses capture 640x480 resolution images at 25 frames per second. The user's head and hands are tracked based on following skin-colored segments of the image from frame to frame. Adaptive skin color modeling determines how skin color appears under different lighting conditions, but must first be initialized by selecting some pixels within the face to and a square of pixels surrounding the head. In practice, the colors of pixels showing skin were distributed in a bimodal manner due to two different light sources. Each modality was modeled separately to reduce mis-classification of pixels as skin colored or not. Classification of gestures is done based on fifty hand features. Dynamic Time Warping is used to find the level of time-correspondence between an input gesture and a reference gesture.
A set of 120 different NGT signs performed by 70 individuals are used to test the sign classification. Cross validation was performed to effectively increase the amount of test data many times. Overall, the true positive classification rate was 95%. Dynamic time warping only improved recognition of signs with repetitious motion.
Discussion:
The classification algorithm relies on knowing the approximate start and end times of a sign. Some other kind of segmentation scheme could be applied so that the hands must not always come to rest on the table between signs. As a teaching tool, resting between signs limits the learning to single words. Multi-gesture phrases and sentences cannot be recognized without segmentation.
The left and right blobs identified during skin detection tracking are assigned to the left and right hands, regardless of which hand is on which side of the body. So, crossing hands will cause skin blobs to be mis-labeled, giving rise to the possibility of classifying two distinct gestures (both consisting of the same basic motion, but one in which hands cross and one in which they do not) as the same.
Television control by hand gestures (Freeman 1994)
Summary:
This paper described a hand movement recognition system for controlling a television. The movements of an open hand were tracked and displayed to the user via a hand icon imposed on a secondary display, but which could overlay a television display in a more commercially viable prototype. The system captures images from a video camera which are downsampled to a 320 x 240 resolution. The images are processed on an HP 735 workstation which is connected via serial port to an All-In-One-12 remote control which can send commands to the television at a rate of about once per second. The recognition of the open hand trigger gesture takes about half a second. Once recognized, the hand position can be tracked at a rate of about five times per second. The images in the appendix were helpful in showing what the interface looked like and illustrating how background removal and local orientation video processing worked.
Discussion:
One scenario not covered in the paper was two users attempting to control the television simultaneously. The slider bar control interface for changing channels would not work well for a large number of channels because very fine control would be needed. A possible improvement would be to use a number pad or one slider per channel digit. The paper mentions that the users of the prototype system were excited, but I think it was due to the novelty of the system instead of an inherent fun factor. No results of the user study were given besides the fact that people seemed to enjoy using the system.
Monday, April 28, 2008
Invariant features for 3-D gesture recognition (Campbell 1996)
Summary:
This paper creates a systme that uses video input from two cameras to track head and hand positions in three dimensions. This data is used to construct view-invariant features from the motions recorded by the cameras. The video tracking system is able to track head and hand position to an accuracy of two centimeters at thirty frames per second. The HTK2.0 HMM toolkit from Entropic Research was used to analyze the feature vectors created from the data captured by the video tracking system.
A set of eighteen T'ai Chi gestures were used in an experiment to test the system. Each gesture is modeled as a forward chaining continuous HMM, using some function of the coordinates of the head and hand positions as features. Several functions of the coordinates are examined, including Cartesian positions (x,y,z) and velocities (dx,dy,dz), polar positions (r,theta,z) and velocities (dr, dTheta, dz), and curvature (ds, log(rho), dz). The features expressed in Cartesian coordinates included sets with and without the head data.
The feature vector with the best overall recognition rate was the polar velocity feature (dr, dTheta, dz) at 95%, followed closely by the polar velocity with radius-scaled theta feature at 93% and the polar velocity feature with redundant Cartesian velocity data at 92%. Not surprisingly, the authors concluded that Cartesian velocity features perform better with data containing translational shifts and polar velocity features perform better when the data contain rotations.
Discussion:
T'ai Chi was an interesting choice of gestures to recognize. From the images of gestures in figure 1, it seems the head position remains relatively fixed throughout all the gestures. If the head remains fixed, head position does not seem like it would be a very useful feature to examine when recognizing gestures.
The T'ai Chi gestures were performed seated due to space constraints. This undoubtedly affected the normal head movement produced when performing the gestures while not seated. Performing the T'ai Chi gestures while not seated would most likely produce more translation and possibly allow the Cartesian velocity feature to achieve higher recognition rates.
The data was resampled to a rate of 30 Hz, but the actual sample rate varied from 15 to 30 Hz. A cubic spline was used for resampling; it might be interesting to see the effects on data and recognition rates if different data fitting curves would have been used.
Shifted and rotated versions of the gesture data were collected during the experiment, but some of the rotation was added synthetically. I wonder why the shifted and rotated data were collected at all, if they could have been generated synthetically as well.
This paper creates a systme that uses video input from two cameras to track head and hand positions in three dimensions. This data is used to construct view-invariant features from the motions recorded by the cameras. The video tracking system is able to track head and hand position to an accuracy of two centimeters at thirty frames per second. The HTK2.0 HMM toolkit from Entropic Research was used to analyze the feature vectors created from the data captured by the video tracking system.
A set of eighteen T'ai Chi gestures were used in an experiment to test the system. Each gesture is modeled as a forward chaining continuous HMM, using some function of the coordinates of the head and hand positions as features. Several functions of the coordinates are examined, including Cartesian positions (x,y,z) and velocities (dx,dy,dz), polar positions (r,theta,z) and velocities (dr, dTheta, dz), and curvature (ds, log(rho), dz). The features expressed in Cartesian coordinates included sets with and without the head data.
The feature vector with the best overall recognition rate was the polar velocity feature (dr, dTheta, dz) at 95%, followed closely by the polar velocity with radius-scaled theta feature at 93% and the polar velocity feature with redundant Cartesian velocity data at 92%. Not surprisingly, the authors concluded that Cartesian velocity features perform better with data containing translational shifts and polar velocity features perform better when the data contain rotations.
Discussion:
T'ai Chi was an interesting choice of gestures to recognize. From the images of gestures in figure 1, it seems the head position remains relatively fixed throughout all the gestures. If the head remains fixed, head position does not seem like it would be a very useful feature to examine when recognizing gestures.
The T'ai Chi gestures were performed seated due to space constraints. This undoubtedly affected the normal head movement produced when performing the gestures while not seated. Performing the T'ai Chi gestures while not seated would most likely produce more translation and possibly allow the Cartesian velocity feature to achieve higher recognition rates.
The data was resampled to a rate of 30 Hz, but the actual sample rate varied from 15 to 30 Hz. A cubic spline was used for resampling; it might be interesting to see the effects on data and recognition rates if different data fitting curves would have been used.
Shifted and rotated versions of the gesture data were collected during the experiment, but some of the rotation was added synthetically. I wonder why the shifted and rotated data were collected at all, if they could have been generated synthetically as well.
FreeDrawer - A Free-Form Sketching System on the Responsive Workbench (Wesche 2001)
Summary:
FreeDrawer is a collection of 3D curve drawing and deformation tools targeted towards product design. The Responsive Workbench program models objects using Catmull-Clark surfaces and B-spline curves as opposed to Constructive Solid Geometry or voxel-based virtual clay. Deformations allowed by B-spline curves and Catmull-Clark surfaces are beneficial for product design, in which original sketches need to be modified to produce the final shape.
The first curve of an object can be freely drawn anywhere in space, but subsequent curves are woven into the existing curve network - being altered if they do not intersect an existing curve. When a curve is modified, the rest of the model can be altered in two ways. In local mode, the modified curve and curves adjacent to it are altered. In global mode, all curves are altered recursively. Surfaces can be filled in with Catmull-Clark or Kuriyama surfaces when a closed loop of curves is selected.
The non-dominant hand is responsible for translating and rotating the model, while the dominant hand is responsible for editing and deforming the model. An original 3D widget with multiple pointers spread out in a fan shape support multiple editing tools - each pointer is mapped to a different tool. A sketch of a chair seat took an experienced user approximately 15 minutes to create.
Discussion:
The idea of the designer drawing the full sweep of a curve across both halves of a mirrored plane and the program averaging each to match symmetrically is new to me. Previously, I had only seen modeling programs where the shape being drawn is mirrored as it is being drawn, and the modeler is only required to draw half of the shape.
The subdivision level for surfaces are chosen so that each curve piece has the same number of segments. This makes creating the subdivided surfaces easier to compute, but is not the best option for models containing areas of small detail and large areas without much detail. In this case, the large areas of less detail are most likely over defined.
Neighboring surfaces are connected with only G1 continuity. For several real-world modeling applications, higher continuity connection is required. Usually, G2 continuity is the minimum smoothness demanded for surfaces which reflect or which need to have good air flow characteristics. If this system were to be used for real-world design, more work would have to be done so that higher continuity can be achieved. Unfortunately, the option of higher continuity may make the system slightly less easy to use.
While the fan-shaped, multi-function widget is novel and may reduce tool selection time for expert users, I think it would produce confusion and tool selection errors in novice users.
FreeDrawer is a collection of 3D curve drawing and deformation tools targeted towards product design. The Responsive Workbench program models objects using Catmull-Clark surfaces and B-spline curves as opposed to Constructive Solid Geometry or voxel-based virtual clay. Deformations allowed by B-spline curves and Catmull-Clark surfaces are beneficial for product design, in which original sketches need to be modified to produce the final shape.
The first curve of an object can be freely drawn anywhere in space, but subsequent curves are woven into the existing curve network - being altered if they do not intersect an existing curve. When a curve is modified, the rest of the model can be altered in two ways. In local mode, the modified curve and curves adjacent to it are altered. In global mode, all curves are altered recursively. Surfaces can be filled in with Catmull-Clark or Kuriyama surfaces when a closed loop of curves is selected.
The non-dominant hand is responsible for translating and rotating the model, while the dominant hand is responsible for editing and deforming the model. An original 3D widget with multiple pointers spread out in a fan shape support multiple editing tools - each pointer is mapped to a different tool. A sketch of a chair seat took an experienced user approximately 15 minutes to create.
Discussion:
The idea of the designer drawing the full sweep of a curve across both halves of a mirrored plane and the program averaging each to match symmetrically is new to me. Previously, I had only seen modeling programs where the shape being drawn is mirrored as it is being drawn, and the modeler is only required to draw half of the shape.
The subdivision level for surfaces are chosen so that each curve piece has the same number of segments. This makes creating the subdivided surfaces easier to compute, but is not the best option for models containing areas of small detail and large areas without much detail. In this case, the large areas of less detail are most likely over defined.
Neighboring surfaces are connected with only G1 continuity. For several real-world modeling applications, higher continuity connection is required. Usually, G2 continuity is the minimum smoothness demanded for surfaces which reflect or which need to have good air flow characteristics. If this system were to be used for real-world design, more work would have to be done so that higher continuity can be achieved. Unfortunately, the option of higher continuity may make the system slightly less easy to use.
While the fan-shaped, multi-function widget is novel and may reduce tool selection time for expert users, I think it would produce confusion and tool selection errors in novice users.
Shape Your Imagination: Iconic Gestural-Based Interaction (Marsh 1998)
Summary:
Iconic gestures are a natural, intuitive, and effective way to convey spatial information. This paper separates iconic gestures into virtual and substitutive categories. Virtual depiction is defined as using the spatial relationship between the hands as an outline or tracing the picture of an object or shape. Substitutive depiction is defined as shaping the hands to match the form of an object as if it were being held or grasped in the hands.
A user study is performed to determine if humans use iconic hand gestures during non-verbal communication of shapes and objects, and if so, what kind of gestures are used with what frequency, and how many hands are involved in the gestures. Twelve volunteers were seated in a room and asked to non-verbally describe fifteen shapes and objects using hand gestures. Care was taken not to influence the volunteers' descriptions or otherwise influence their gestures by providing images of the objects.
Subjects used two-handed gestures 88.1% of the time, preferring virtual depiction. 75% of gestures were performed using purely virtual depiction, 17.9% were performed using purely substitutive depiction, and 7.1% used a combination of both. Only one object, the circle, was performed using primarily one-handed virtual depiction gestures.
Discussion:
The results of this paper were not surprising. I suppose it was still good to confirm the ideas experimentally, although a larger group of subjects could have been used to obtain more statistically significant results. Also, since the goal was to see how gestures are used naturally, an experiment could be designed to observe when gestures accompany certain spoken words or phrases. Telling the subjects that they must communicate completely non-verbally does not seem like a situation in which the most natural gestures would be made.
In general, users found describing 2D shapes easier than 3D shapes. I think an analogy to this insight exists in the recognition realm. Sketch recognition in two dimensions is an easier problem than gesture recognition in three dimensions.
Iconic gestures are a natural, intuitive, and effective way to convey spatial information. This paper separates iconic gestures into virtual and substitutive categories. Virtual depiction is defined as using the spatial relationship between the hands as an outline or tracing the picture of an object or shape. Substitutive depiction is defined as shaping the hands to match the form of an object as if it were being held or grasped in the hands.
A user study is performed to determine if humans use iconic hand gestures during non-verbal communication of shapes and objects, and if so, what kind of gestures are used with what frequency, and how many hands are involved in the gestures. Twelve volunteers were seated in a room and asked to non-verbally describe fifteen shapes and objects using hand gestures. Care was taken not to influence the volunteers' descriptions or otherwise influence their gestures by providing images of the objects.
Subjects used two-handed gestures 88.1% of the time, preferring virtual depiction. 75% of gestures were performed using purely virtual depiction, 17.9% were performed using purely substitutive depiction, and 7.1% used a combination of both. Only one object, the circle, was performed using primarily one-handed virtual depiction gestures.
Discussion:
The results of this paper were not surprising. I suppose it was still good to confirm the ideas experimentally, although a larger group of subjects could have been used to obtain more statistically significant results. Also, since the goal was to see how gestures are used naturally, an experiment could be designed to observe when gestures accompany certain spoken words or phrases. Telling the subjects that they must communicate completely non-verbally does not seem like a situation in which the most natural gestures would be made.
In general, users found describing 2D shapes easier than 3D shapes. I think an analogy to this insight exists in the recognition realm. Sketch recognition in two dimensions is an easier problem than gesture recognition in three dimensions.
A Dynamic Gesture Recognition System for the Korean Sign Language (Kim 1996)
Summary:
The paper describes a system for recognizing Korean Sign Language (KSL) gestures and converting them into text. The system employs two ten-sensor VPL Data-Gloves to measure the bend in the joints of each digit on both hands. The gloves also sense the position and orientation of each hand relative to a fixed source. The hand gestures need to be recognized regardless of their position relative to an initial position which varies, the position data is recorded as the difference between the previous position and the current position. For the paper, 25 of around 6000 KSL gestures were analyzed and partitioned into ten sets based on their general direction. To determine which of the ten direction categories the motion will be classified as during recognition, the change over the five most recent readings of region data are examined. Hand postures are recognized by applying the technique of Fuzzy Min-Max Neural (FMMN) Networks. Input from the data glove are identified as one of the direction classes and then recognized by the FMMN network. The system classifies gestures correctly almost 85% of the time.
Discussion:
Figure 8 shows a diagram of the min-max neural network used for classification. There are ten input nodes at the bottom, one for each of the flex angles measured from the fingers. However, there should be fourteen class noes at the top of the diagram instead of ten since there are fourteen posture classes.
When motion data is expressed in its compressed form, the order of region data is preserved, but data relating to the length in time of each region is lost.
Although the paper states that the FMMN network requires no pre-learning about posture class and has on-line adaptability, I'd say the basic idea of neural networks does require learning since that is how the weights of the network are adjusted.
The mis-classifications are partially blamed on abnormal motions in gestures and postures, but dealing with data that exhibits less than ideal characteristics seems to be part of the point of applying complex solutions to gesture recognition.
The paper describes a system for recognizing Korean Sign Language (KSL) gestures and converting them into text. The system employs two ten-sensor VPL Data-Gloves to measure the bend in the joints of each digit on both hands. The gloves also sense the position and orientation of each hand relative to a fixed source. The hand gestures need to be recognized regardless of their position relative to an initial position which varies, the position data is recorded as the difference between the previous position and the current position. For the paper, 25 of around 6000 KSL gestures were analyzed and partitioned into ten sets based on their general direction. To determine which of the ten direction categories the motion will be classified as during recognition, the change over the five most recent readings of region data are examined. Hand postures are recognized by applying the technique of Fuzzy Min-Max Neural (FMMN) Networks. Input from the data glove are identified as one of the direction classes and then recognized by the FMMN network. The system classifies gestures correctly almost 85% of the time.
Discussion:
Figure 8 shows a diagram of the min-max neural network used for classification. There are ten input nodes at the bottom, one for each of the flex angles measured from the fingers. However, there should be fourteen class noes at the top of the diagram instead of ten since there are fourteen posture classes.
When motion data is expressed in its compressed form, the order of region data is preserved, but data relating to the length in time of each region is lost.
Although the paper states that the FMMN network requires no pre-learning about posture class and has on-line adaptability, I'd say the basic idea of neural networks does require learning since that is how the weights of the network are adjusted.
The mis-classifications are partially blamed on abnormal motions in gestures and postures, but dealing with data that exhibits less than ideal characteristics seems to be part of the point of applying complex solutions to gesture recognition.
Subscribe to:
Posts (Atom)