Summary:
The paper focuses on a method of mapping the hand motion of a user to locomotion of virtual characters in real time. The proposed method allows control of a multi-joint character through the motion of fingers whose joint angles are detected by a sensing glove. Animations are recorded via motion capture or generated using 3D modeling software and played back to a user. The user mimics the animation by moving his hands and the motion is used to generate a function that maps the hand motion to the standard animation. Then, new locomotion animations can be generated in real-time when the user's hand moves in different ways not necessarily within the domain of the motion used to generate the mapping function.
When calculating the correspondence between finger and animation motion, a cyclic locomotion is assumed in the calibration cycle. The period of each degree of freedom is calculated by finding the auto-correlation of the trajectories of that degree of freedom's signal. During play, the period of each joint in the animation is compared to the period of finger motion and the joints are classified as full cycle, half cycle, or exceptional. The hand's general coordinates are matched with the character's. Relative velocities of the character's end effectors to it's root are compared with relative velocities of fingertips to wrist to match each end effector with a finger. The motion of a joint is determined by the motion of finger(s) associated with the end effector(s) which are its descendants.
To test the method, mapping functions were generated for both human and dog walking animations. The animations were successfully extrapolated into running animations in real time. An experiment involving four subjects was conducted to compare the speed and number of collisions of a game character being navigated through a maze. On average, keyboard controlled locomotion took 9% less time, but resulted in more than twice as many collisions as sensing glove controlled locomotion.
Discussion:
An extension that would improve the realism of the motion would be animation blending. Currently, the system uses a single animation played at different speeds controlled by finger speed. However, a walking animation played back at double the intended rate does not look the same as a running animation. Animation blending uses some function to combine the weights of joint angles in an animation so that the resulting frames of animation transition naturally between animations, depending on speed.
During the control stage, when finger motion exceeds the original bounds of the mapping function generated in the calibration stage, extrapolation of the mapping function is performed for a limited space outside of the original domain. Instead of only considering the tangent (just the first derivative) to the mapping function at its endpoint, more derivatives could be considered to obtain a more accurate estimate of motion outside the original domain.
Controlling jumping motion of a character with the hand seems like a strange gameplay choice. If the hand position was mapped to character height, then holding the hand in the air would allow the player to hover, which is not realistic. If jumping were simulated in a physically realistic way, then mapping hand position to character height becomes pointless since all hand-height information would be discarded, except for the start time of the jump.
Thursday, May 8, 2008
The 3D Tractus: A Three-Dimensional Drawing Board (Lapides 2006)
Summary:
The 3D Tractus is a system for creating three dimensional sketches whose interface consists of a 2D drawing surface that slides up and down to incorporate the third dimension. A user sitting at the Tractus draws on the tabletop surface with one hand and slides the tabletop up and down with the other. The goal of directly mapping virtual and physical space with the tabletop height is achieved with a string potentiometer-based sensor. A Toshiba Portege M200 Tablet PC was used as the interactive top of the 3D Tractus table. A counterweight system was used to ease the act of adjusting table height.
The software interface for the Tractus incorporates a 2D drawing area and a 3D overview window. The drawing area accepts pen based input and the 3D view can be rotated by dragging the pen in the desired direction. Multiple attempts at expressing depth on the 2D drawing surface were attempted. Changing color and color intensity in response to line depth did not provide intuitive feedback with enough contrast to communicate distance accurately. The current implementation uses varying line width and a perspective projection to convey depth. Lines above the drawing plane are not rendered.
Three students with art backgrounds evaluated the system for 30 minutes and asked for feedback. The most requested feature was the ability to tilt the drawing surface to allow work to be done from any angle, similar to how the 3D view can be rotated.
Discussion:
One of the programs discussed in the related work section allows a user to select a slice of their 3D object, pull it out of the stack, and edit it. This approach would be unmanageably tedious if working in th direction normal to the surface of the slices. Even if the Tractus doesn't completely remove the problem of working in a direction along the normal of slice surfaces, it greatly reduces the time required to do so by removing the steps of selecting to remove, and replacing, slices in the stack.
Directly mapping physical and virtual space limits the detail of sketches that could be more easily accomplished if zoom functionality was available. The ability to scale the mapping would allow the user to work at multiple levels of detail. Another improvement could be the use of stereovision to give the impression of depth on the 2D drawing area. Also, the addition of a button to toggle the rendering of lines above the drawing plane would aid sketch construction in some cases.
The 3D Tractus is a system for creating three dimensional sketches whose interface consists of a 2D drawing surface that slides up and down to incorporate the third dimension. A user sitting at the Tractus draws on the tabletop surface with one hand and slides the tabletop up and down with the other. The goal of directly mapping virtual and physical space with the tabletop height is achieved with a string potentiometer-based sensor. A Toshiba Portege M200 Tablet PC was used as the interactive top of the 3D Tractus table. A counterweight system was used to ease the act of adjusting table height.
The software interface for the Tractus incorporates a 2D drawing area and a 3D overview window. The drawing area accepts pen based input and the 3D view can be rotated by dragging the pen in the desired direction. Multiple attempts at expressing depth on the 2D drawing surface were attempted. Changing color and color intensity in response to line depth did not provide intuitive feedback with enough contrast to communicate distance accurately. The current implementation uses varying line width and a perspective projection to convey depth. Lines above the drawing plane are not rendered.
Three students with art backgrounds evaluated the system for 30 minutes and asked for feedback. The most requested feature was the ability to tilt the drawing surface to allow work to be done from any angle, similar to how the 3D view can be rotated.
Discussion:
One of the programs discussed in the related work section allows a user to select a slice of their 3D object, pull it out of the stack, and edit it. This approach would be unmanageably tedious if working in th direction normal to the surface of the slices. Even if the Tractus doesn't completely remove the problem of working in a direction along the normal of slice surfaces, it greatly reduces the time required to do so by removing the steps of selecting to remove, and replacing, slices in the stack.
Directly mapping physical and virtual space limits the detail of sketches that could be more easily accomplished if zoom functionality was available. The ability to scale the mapping would allow the user to work at multiple levels of detail. Another improvement could be the use of stereovision to give the impression of depth on the 2D drawing area. Also, the addition of a button to toggle the rendering of lines above the drawing plane would aid sketch construction in some cases.
TIKL: Development of a Wearable Vibrotactile Feedback Suit for Improved Human Motor Learning (Lieberman 2007)
Summary:
The TIKL system is proposed as an additional channel for learning motor skills, providing low-latency feedback through a vibrotactile feedback suit. The Vicon tracking system uses twelve high speed infrared cameras which record positions of markers so that the 3d position of the markers can be reconstructed and joint angles can be calculated. Differences in joint angle between the teacher and learner are communicated to the learner through timed tactile pulses that utilize the sensory saltation effect. Corrective rotation can be communicated by a sequence of pulses around actuators that are attached in a ring around the arm.
An experiment was run to test how users' ability to learn motor tasks was affected by the tactile feedback provided by TIKL. Using deviations in joint angle as the metric for error, forty subjects' reactions during training sequences were measured. Real-time errors were reduced by up to 27% and learning was improved by up to 23%. Subjects reported no significant loss of comfort due to the wearable system.
Discussion:
One downside of the TIKL system is its reliance on the expensive and bulky Vicon system. Currently, only angle errors are used in determining feedback signal strength, which may not be the best measurement of task fulfillment. Other measures of error such as end effector position probably describe error better for some tasks. The problem of correcting end effector position alone can be hard since there would be multiple joint positions that could result in the same end position.
The users participating in the study were told to mimic joint angles as opposed to another metric. The error reporting assumes that people easily grasp the concept of mimicking angles and remembered to follow only that metric throughout the study. I don't think people naturally mimic only joint angles and that some error was probably introduced due to using only joint angle as the error metric. In regard to the results section, I thought polling users seems like a somewhat subjective way to measure qualities of the system. A more quantitative study could have been designed.
The TIKL system is proposed as an additional channel for learning motor skills, providing low-latency feedback through a vibrotactile feedback suit. The Vicon tracking system uses twelve high speed infrared cameras which record positions of markers so that the 3d position of the markers can be reconstructed and joint angles can be calculated. Differences in joint angle between the teacher and learner are communicated to the learner through timed tactile pulses that utilize the sensory saltation effect. Corrective rotation can be communicated by a sequence of pulses around actuators that are attached in a ring around the arm.
An experiment was run to test how users' ability to learn motor tasks was affected by the tactile feedback provided by TIKL. Using deviations in joint angle as the metric for error, forty subjects' reactions during training sequences were measured. Real-time errors were reduced by up to 27% and learning was improved by up to 23%. Subjects reported no significant loss of comfort due to the wearable system.
Discussion:
One downside of the TIKL system is its reliance on the expensive and bulky Vicon system. Currently, only angle errors are used in determining feedback signal strength, which may not be the best measurement of task fulfillment. Other measures of error such as end effector position probably describe error better for some tasks. The problem of correcting end effector position alone can be hard since there would be multiple joint positions that could result in the same end position.
The users participating in the study were told to mimic joint angles as opposed to another metric. The error reporting assumes that people easily grasp the concept of mimicking angles and remembered to follow only that metric throughout the study. I don't think people naturally mimic only joint angles and that some error was probably introduced due to using only joint angle as the error metric. In regard to the results section, I thought polling users seems like a somewhat subjective way to measure qualities of the system. A more quantitative study could have been designed.
A Hidden Markov Model Based Sensor Fusion Approach for Recognizing Continuous Human Grasping Sequences (Bernardin 2005)
Summary:
This paper proposes a system that uses information about both hand shape and contact points obtained from a combination of a data glove and tactile sensors to recognize continuous human grasp sequences. The long term goal of the study would be to teach a robotic system to perform a task simply by observing a human teacher instead of explicitly programming the robot. For this kind of learning to take place, the robot must be able to infer what has been done and map that to a known skill that can be described symbolically. This paper analyzes grasping gestures in order to aid the construction of their symbolic descriptions.
An 18 sensor Cyberglove is used in conjunction with an array of 16 capacitive pressure sensitive sensors affixed to the fingers and palm. Classification of grasping gestures is mad according to Kamakura's grasp taxonomy, which identifies 14 different kinds of grasps. The regions of the hand which are covered by pressure sensors were chosen to maximize the detection of contact with a minimal number of sensors while also corresponding to the main regions in Kamakura's grasp types. An HMM was built for each type of grasp using 112 training gestures. The HMMs have a flat topology with 9 states and were trained offline.
The main benefit gleaned from the tactile sensors was the ability to perform segmentation easier.
Discussion:
The customization required of the capacitive pressure sensors indicates that there is not currently a mass-produced component to fill the demand for grasp detection hardware. In the description of the HMM recognizer, it is mentioned that a task grammar was used to reduce the search space of the recognizer. Since only grasp and release sequences are recognized, the segmentation problem is avoided.
If the end goal is to teach a robot to learn grasps by observation, I think an experiment that used both visual-based and glove-based inputs would be required to discern a link between the visual and tactile realms. The visual signal could be analyzed and possibly mapped to a tactile response.
This paper proposes a system that uses information about both hand shape and contact points obtained from a combination of a data glove and tactile sensors to recognize continuous human grasp sequences. The long term goal of the study would be to teach a robotic system to perform a task simply by observing a human teacher instead of explicitly programming the robot. For this kind of learning to take place, the robot must be able to infer what has been done and map that to a known skill that can be described symbolically. This paper analyzes grasping gestures in order to aid the construction of their symbolic descriptions.
An 18 sensor Cyberglove is used in conjunction with an array of 16 capacitive pressure sensitive sensors affixed to the fingers and palm. Classification of grasping gestures is mad according to Kamakura's grasp taxonomy, which identifies 14 different kinds of grasps. The regions of the hand which are covered by pressure sensors were chosen to maximize the detection of contact with a minimal number of sensors while also corresponding to the main regions in Kamakura's grasp types. An HMM was built for each type of grasp using 112 training gestures. The HMMs have a flat topology with 9 states and were trained offline.
The main benefit gleaned from the tactile sensors was the ability to perform segmentation easier.
Discussion:
The customization required of the capacitive pressure sensors indicates that there is not currently a mass-produced component to fill the demand for grasp detection hardware. In the description of the HMM recognizer, it is mentioned that a task grammar was used to reduce the search space of the recognizer. Since only grasp and release sequences are recognized, the segmentation problem is avoided.
If the end goal is to teach a robot to learn grasps by observation, I think an experiment that used both visual-based and glove-based inputs would be required to discern a link between the visual and tactile realms. The visual signal could be analyzed and possibly mapped to a tactile response.
Articulated Hand Tracking by PCA-ICA Approach (Kato 2006)
Summary:
The method proposed to track finger and hand motions in this paper begins by performing principle component analysis to reduce the number of feature dimensions that are considered. Then independent component analysis is performed on the lower dimensional result to get intrinsic feature vectors. The ICA technique finds a linear, non-orthogonal coordinate system in multivariate data with the goal of performing a linear transformation which makes the resulting variables as statistically independent as possible.
A hand was modeled in OpenGL and projected onto an image captured of a real hand to test the PCA-ICA method. The report of the results only says the effectiveness of the method used to track a hand in real image sequences was demonstrated. There were not any numerical statistics reported in the results.
Discussion:
The PCA-ICA method considers only finger bentness, not hand position or orientation. Recognizing hand gestures that rely on that kind of data could not solely depend n the PCA-ICA approach.
The pictures of hand motion based on ICA basis vectors look better than the pictures of hand motion based on PCA basis vectors. I think this may be due to the fact that five vectors were considered. What would ICA look like if some number other than five was used? I don't know if there would be a good mapping of ICA basis vectors to hand renderings for a number of vectors other than five (one for each finger).
I think one of the insights of this paper was its use of knowledge from a different field. The number of dimensions used in component analysis can be reduced by considering important observations from the field of biomechanics. In particular the bend of an end joint of a finger can typically be described as two thirds the bend of the next joint's bend along the same finger.
The method proposed to track finger and hand motions in this paper begins by performing principle component analysis to reduce the number of feature dimensions that are considered. Then independent component analysis is performed on the lower dimensional result to get intrinsic feature vectors. The ICA technique finds a linear, non-orthogonal coordinate system in multivariate data with the goal of performing a linear transformation which makes the resulting variables as statistically independent as possible.
A hand was modeled in OpenGL and projected onto an image captured of a real hand to test the PCA-ICA method. The report of the results only says the effectiveness of the method used to track a hand in real image sequences was demonstrated. There were not any numerical statistics reported in the results.
Discussion:
The PCA-ICA method considers only finger bentness, not hand position or orientation. Recognizing hand gestures that rely on that kind of data could not solely depend n the PCA-ICA approach.
The pictures of hand motion based on ICA basis vectors look better than the pictures of hand motion based on PCA basis vectors. I think this may be due to the fact that five vectors were considered. What would ICA look like if some number other than five was used? I don't know if there would be a good mapping of ICA basis vectors to hand renderings for a number of vectors other than five (one for each finger).
I think one of the insights of this paper was its use of knowledge from a different field. The number of dimensions used in component analysis can be reduced by considering important observations from the field of biomechanics. In particular the bend of an end joint of a finger can typically be described as two thirds the bend of the next joint's bend along the same finger.
Friday, May 2, 2008
A Survey of Hand Posture and Gesture Recognition Techniques and Technology (LaViola 1999)
Summary:
This aptly named paper presents a variety of gesture and posture recognition techniques. The techniques are loosely grouped as either feature extraction and modeling, learning algorithms, or miscellaneous. Template matching is one of the simplest techniques to implement and accurate for small sets of postures, but is not suited for gestures. Feature extraction and analysis uses a layered architecture that can handle both postures and gestures, but it can be computationally expensive if large numbers of features are extracted. Active shape models allow for real time recognition but only tracks the open hand. Principal component analysis can recognize around thirty postures, but it requires training by more than one person to achieve accurate results for multiple users. Linear fingertip models are only concerned with the starting and ending points of fingertips and only recognize a small set of postures. Causal analysis uses information about how humans interact with the world to identify gestures, and therefore, can only be applied to a limited set of gestures. Neural networks can recognize large posture or gesture sets with high accuracy given enough training data. However, the training can be very time consuming and the network must be retrained when items are added or removed from the set to be recognized. Hidden Markov models are well covered in the literature and can be used with either a vision or instrumented approach. Training HMMs can be time consuming and does not necessarily give good recognition rates. Instance-based learning techniques are relatively simple to implement, but require a large amount of memory and computation time and are not suited for real time recognition. Spatio-temporal vector analysis is a non-obtrusive, computationally intensive, vision based approach which has not reported any recognition accuracy results.
Discussion:
One aspect of this paper I liked was the summaries after each subsection which highlighted key points of the previous paragraphs. We have discussed HMM based techniques so often in class, that it was refreshing to see a wider variety of approaches. This paper was good for brainstorming what techniques to use, expand on, or combine for continuing work in gesture recognition. The experiments we did in class where a gesture was shown and described in words by each person gave some experience with the linguistic approach. Through some of the difficulties in class, I realize that it can be difficult to describe a posture accurately and universally understandable using words only. The linguistic approach in the paper only considered postures in which fingers were fully extended or contracted, which covers only a small set of all possible postures. The paper says the approach is simple, but I would say it is difficult to do when considering a set of postures that is not tightly constrained.
This aptly named paper presents a variety of gesture and posture recognition techniques. The techniques are loosely grouped as either feature extraction and modeling, learning algorithms, or miscellaneous. Template matching is one of the simplest techniques to implement and accurate for small sets of postures, but is not suited for gestures. Feature extraction and analysis uses a layered architecture that can handle both postures and gestures, but it can be computationally expensive if large numbers of features are extracted. Active shape models allow for real time recognition but only tracks the open hand. Principal component analysis can recognize around thirty postures, but it requires training by more than one person to achieve accurate results for multiple users. Linear fingertip models are only concerned with the starting and ending points of fingertips and only recognize a small set of postures. Causal analysis uses information about how humans interact with the world to identify gestures, and therefore, can only be applied to a limited set of gestures. Neural networks can recognize large posture or gesture sets with high accuracy given enough training data. However, the training can be very time consuming and the network must be retrained when items are added or removed from the set to be recognized. Hidden Markov models are well covered in the literature and can be used with either a vision or instrumented approach. Training HMMs can be time consuming and does not necessarily give good recognition rates. Instance-based learning techniques are relatively simple to implement, but require a large amount of memory and computation time and are not suited for real time recognition. Spatio-temporal vector analysis is a non-obtrusive, computationally intensive, vision based approach which has not reported any recognition accuracy results.
Discussion:
One aspect of this paper I liked was the summaries after each subsection which highlighted key points of the previous paragraphs. We have discussed HMM based techniques so often in class, that it was refreshing to see a wider variety of approaches. This paper was good for brainstorming what techniques to use, expand on, or combine for continuing work in gesture recognition. The experiments we did in class where a gesture was shown and described in words by each person gave some experience with the linguistic approach. Through some of the difficulties in class, I realize that it can be difficult to describe a posture accurately and universally understandable using words only. The linguistic approach in the paper only considered postures in which fingers were fully extended or contracted, which covers only a small set of all possible postures. The paper says the approach is simple, but I would say it is difficult to do when considering a set of postures that is not tightly constrained.
Discourse Topic and Gestural Form (Eisenstein 2008)
Summary:
This paper examines the question of whether gestures without a relationship to a specific meaning are speaker independent or whether gestures are determined by the speaker's view of a discourse topic. To answer the question, a new characterization of gestures in terms of spatiotemporal interest points is used to extract a low level representation of gestures from video without manual intervention. The interest points are grouped into categories of codewords. The distribution of these codewords across speakers and topics form the basis for analysis.
During the spatiotemporal interest point extraction from images, principal component analysis is used to reduce the dimensionality of the data to form the final feature vectors describing the gestures. A hierarchical Bayesian model was used to cluster gesture features and determine whether the gesture is related to the speaker or the discourse topic. To allow inference of the hidden variables in the Bayesian model, Gibbs sampling is used since it is guaranteed to converge to the true distribution over the hidden variables in the limit.
An experiment was conducted to analyze 33 short videos of fifteen different speakers covering five different topics. Four of the topics of discussion were chosen to be mechanical devices, since direct gestural metaphors would likely show up in the discussion of such items. The study concluded that gestures are topic dependent and the relationship overshadows specific speakers. With correct labeling of topics, 12% of gestures were related to the topic, but randomly labeled topics, less than 3% of gestures are considered topic-specific.
Discussion:
The interest point features that were extracted exist at a lower level of abstraction than most other attempts at describing gestures. It is interesting that such low-level attributes are sufficient to perform successful recognition.
I think the limitation to mainly mechanical devices as topics could have influenced the ability to spot topic-specific gestures. If general categories were chosen, I think the topic-specific nature of gestures would be reduced.
I was not familiar with all of the mathematics discussed in the paper, so I'm not sure how sound their technique is from a theoretical point of view. Looking at the pictures of identified interest points and the frames before and after the interest point convinced me that the feature extraction must work in some cases.
This paper examines the question of whether gestures without a relationship to a specific meaning are speaker independent or whether gestures are determined by the speaker's view of a discourse topic. To answer the question, a new characterization of gestures in terms of spatiotemporal interest points is used to extract a low level representation of gestures from video without manual intervention. The interest points are grouped into categories of codewords. The distribution of these codewords across speakers and topics form the basis for analysis.
During the spatiotemporal interest point extraction from images, principal component analysis is used to reduce the dimensionality of the data to form the final feature vectors describing the gestures. A hierarchical Bayesian model was used to cluster gesture features and determine whether the gesture is related to the speaker or the discourse topic. To allow inference of the hidden variables in the Bayesian model, Gibbs sampling is used since it is guaranteed to converge to the true distribution over the hidden variables in the limit.
An experiment was conducted to analyze 33 short videos of fifteen different speakers covering five different topics. Four of the topics of discussion were chosen to be mechanical devices, since direct gestural metaphors would likely show up in the discussion of such items. The study concluded that gestures are topic dependent and the relationship overshadows specific speakers. With correct labeling of topics, 12% of gestures were related to the topic, but randomly labeled topics, less than 3% of gestures are considered topic-specific.
Discussion:
The interest point features that were extracted exist at a lower level of abstraction than most other attempts at describing gestures. It is interesting that such low-level attributes are sufficient to perform successful recognition.
I think the limitation to mainly mechanical devices as topics could have influenced the ability to spot topic-specific gestures. If general categories were chosen, I think the topic-specific nature of gestures would be reduced.
I was not familiar with all of the mathematics discussed in the paper, so I'm not sure how sound their technique is from a theoretical point of view. Looking at the pictures of identified interest points and the frames before and after the interest point convinced me that the feature extraction must work in some cases.
Tuesday, April 29, 2008
3D Visual Detection of Correct NGT Sign Production (Lichtenauer 2007)
Summary:
In this paper, a 3D visual detection system is proposed to aid children in learning Dutch sign language. Two video cameras with wide angle lenses capture 640x480 resolution images at 25 frames per second. The user's head and hands are tracked based on following skin-colored segments of the image from frame to frame. Adaptive skin color modeling determines how skin color appears under different lighting conditions, but must first be initialized by selecting some pixels within the face to and a square of pixels surrounding the head. In practice, the colors of pixels showing skin were distributed in a bimodal manner due to two different light sources. Each modality was modeled separately to reduce mis-classification of pixels as skin colored or not. Classification of gestures is done based on fifty hand features. Dynamic Time Warping is used to find the level of time-correspondence between an input gesture and a reference gesture.
A set of 120 different NGT signs performed by 70 individuals are used to test the sign classification. Cross validation was performed to effectively increase the amount of test data many times. Overall, the true positive classification rate was 95%. Dynamic time warping only improved recognition of signs with repetitious motion.
Discussion:
The classification algorithm relies on knowing the approximate start and end times of a sign. Some other kind of segmentation scheme could be applied so that the hands must not always come to rest on the table between signs. As a teaching tool, resting between signs limits the learning to single words. Multi-gesture phrases and sentences cannot be recognized without segmentation.
The left and right blobs identified during skin detection tracking are assigned to the left and right hands, regardless of which hand is on which side of the body. So, crossing hands will cause skin blobs to be mis-labeled, giving rise to the possibility of classifying two distinct gestures (both consisting of the same basic motion, but one in which hands cross and one in which they do not) as the same.
In this paper, a 3D visual detection system is proposed to aid children in learning Dutch sign language. Two video cameras with wide angle lenses capture 640x480 resolution images at 25 frames per second. The user's head and hands are tracked based on following skin-colored segments of the image from frame to frame. Adaptive skin color modeling determines how skin color appears under different lighting conditions, but must first be initialized by selecting some pixels within the face to and a square of pixels surrounding the head. In practice, the colors of pixels showing skin were distributed in a bimodal manner due to two different light sources. Each modality was modeled separately to reduce mis-classification of pixels as skin colored or not. Classification of gestures is done based on fifty hand features. Dynamic Time Warping is used to find the level of time-correspondence between an input gesture and a reference gesture.
A set of 120 different NGT signs performed by 70 individuals are used to test the sign classification. Cross validation was performed to effectively increase the amount of test data many times. Overall, the true positive classification rate was 95%. Dynamic time warping only improved recognition of signs with repetitious motion.
Discussion:
The classification algorithm relies on knowing the approximate start and end times of a sign. Some other kind of segmentation scheme could be applied so that the hands must not always come to rest on the table between signs. As a teaching tool, resting between signs limits the learning to single words. Multi-gesture phrases and sentences cannot be recognized without segmentation.
The left and right blobs identified during skin detection tracking are assigned to the left and right hands, regardless of which hand is on which side of the body. So, crossing hands will cause skin blobs to be mis-labeled, giving rise to the possibility of classifying two distinct gestures (both consisting of the same basic motion, but one in which hands cross and one in which they do not) as the same.
Television control by hand gestures (Freeman 1994)
Summary:
This paper described a hand movement recognition system for controlling a television. The movements of an open hand were tracked and displayed to the user via a hand icon imposed on a secondary display, but which could overlay a television display in a more commercially viable prototype. The system captures images from a video camera which are downsampled to a 320 x 240 resolution. The images are processed on an HP 735 workstation which is connected via serial port to an All-In-One-12 remote control which can send commands to the television at a rate of about once per second. The recognition of the open hand trigger gesture takes about half a second. Once recognized, the hand position can be tracked at a rate of about five times per second. The images in the appendix were helpful in showing what the interface looked like and illustrating how background removal and local orientation video processing worked.
Discussion:
One scenario not covered in the paper was two users attempting to control the television simultaneously. The slider bar control interface for changing channels would not work well for a large number of channels because very fine control would be needed. A possible improvement would be to use a number pad or one slider per channel digit. The paper mentions that the users of the prototype system were excited, but I think it was due to the novelty of the system instead of an inherent fun factor. No results of the user study were given besides the fact that people seemed to enjoy using the system.
Monday, April 28, 2008
Invariant features for 3-D gesture recognition (Campbell 1996)
Summary:
This paper creates a systme that uses video input from two cameras to track head and hand positions in three dimensions. This data is used to construct view-invariant features from the motions recorded by the cameras. The video tracking system is able to track head and hand position to an accuracy of two centimeters at thirty frames per second. The HTK2.0 HMM toolkit from Entropic Research was used to analyze the feature vectors created from the data captured by the video tracking system.
A set of eighteen T'ai Chi gestures were used in an experiment to test the system. Each gesture is modeled as a forward chaining continuous HMM, using some function of the coordinates of the head and hand positions as features. Several functions of the coordinates are examined, including Cartesian positions (x,y,z) and velocities (dx,dy,dz), polar positions (r,theta,z) and velocities (dr, dTheta, dz), and curvature (ds, log(rho), dz). The features expressed in Cartesian coordinates included sets with and without the head data.
The feature vector with the best overall recognition rate was the polar velocity feature (dr, dTheta, dz) at 95%, followed closely by the polar velocity with radius-scaled theta feature at 93% and the polar velocity feature with redundant Cartesian velocity data at 92%. Not surprisingly, the authors concluded that Cartesian velocity features perform better with data containing translational shifts and polar velocity features perform better when the data contain rotations.
Discussion:
T'ai Chi was an interesting choice of gestures to recognize. From the images of gestures in figure 1, it seems the head position remains relatively fixed throughout all the gestures. If the head remains fixed, head position does not seem like it would be a very useful feature to examine when recognizing gestures.
The T'ai Chi gestures were performed seated due to space constraints. This undoubtedly affected the normal head movement produced when performing the gestures while not seated. Performing the T'ai Chi gestures while not seated would most likely produce more translation and possibly allow the Cartesian velocity feature to achieve higher recognition rates.
The data was resampled to a rate of 30 Hz, but the actual sample rate varied from 15 to 30 Hz. A cubic spline was used for resampling; it might be interesting to see the effects on data and recognition rates if different data fitting curves would have been used.
Shifted and rotated versions of the gesture data were collected during the experiment, but some of the rotation was added synthetically. I wonder why the shifted and rotated data were collected at all, if they could have been generated synthetically as well.
This paper creates a systme that uses video input from two cameras to track head and hand positions in three dimensions. This data is used to construct view-invariant features from the motions recorded by the cameras. The video tracking system is able to track head and hand position to an accuracy of two centimeters at thirty frames per second. The HTK2.0 HMM toolkit from Entropic Research was used to analyze the feature vectors created from the data captured by the video tracking system.
A set of eighteen T'ai Chi gestures were used in an experiment to test the system. Each gesture is modeled as a forward chaining continuous HMM, using some function of the coordinates of the head and hand positions as features. Several functions of the coordinates are examined, including Cartesian positions (x,y,z) and velocities (dx,dy,dz), polar positions (r,theta,z) and velocities (dr, dTheta, dz), and curvature (ds, log(rho), dz). The features expressed in Cartesian coordinates included sets with and without the head data.
The feature vector with the best overall recognition rate was the polar velocity feature (dr, dTheta, dz) at 95%, followed closely by the polar velocity with radius-scaled theta feature at 93% and the polar velocity feature with redundant Cartesian velocity data at 92%. Not surprisingly, the authors concluded that Cartesian velocity features perform better with data containing translational shifts and polar velocity features perform better when the data contain rotations.
Discussion:
T'ai Chi was an interesting choice of gestures to recognize. From the images of gestures in figure 1, it seems the head position remains relatively fixed throughout all the gestures. If the head remains fixed, head position does not seem like it would be a very useful feature to examine when recognizing gestures.
The T'ai Chi gestures were performed seated due to space constraints. This undoubtedly affected the normal head movement produced when performing the gestures while not seated. Performing the T'ai Chi gestures while not seated would most likely produce more translation and possibly allow the Cartesian velocity feature to achieve higher recognition rates.
The data was resampled to a rate of 30 Hz, but the actual sample rate varied from 15 to 30 Hz. A cubic spline was used for resampling; it might be interesting to see the effects on data and recognition rates if different data fitting curves would have been used.
Shifted and rotated versions of the gesture data were collected during the experiment, but some of the rotation was added synthetically. I wonder why the shifted and rotated data were collected at all, if they could have been generated synthetically as well.
FreeDrawer - A Free-Form Sketching System on the Responsive Workbench (Wesche 2001)
Summary:
FreeDrawer is a collection of 3D curve drawing and deformation tools targeted towards product design. The Responsive Workbench program models objects using Catmull-Clark surfaces and B-spline curves as opposed to Constructive Solid Geometry or voxel-based virtual clay. Deformations allowed by B-spline curves and Catmull-Clark surfaces are beneficial for product design, in which original sketches need to be modified to produce the final shape.
The first curve of an object can be freely drawn anywhere in space, but subsequent curves are woven into the existing curve network - being altered if they do not intersect an existing curve. When a curve is modified, the rest of the model can be altered in two ways. In local mode, the modified curve and curves adjacent to it are altered. In global mode, all curves are altered recursively. Surfaces can be filled in with Catmull-Clark or Kuriyama surfaces when a closed loop of curves is selected.
The non-dominant hand is responsible for translating and rotating the model, while the dominant hand is responsible for editing and deforming the model. An original 3D widget with multiple pointers spread out in a fan shape support multiple editing tools - each pointer is mapped to a different tool. A sketch of a chair seat took an experienced user approximately 15 minutes to create.
Discussion:
The idea of the designer drawing the full sweep of a curve across both halves of a mirrored plane and the program averaging each to match symmetrically is new to me. Previously, I had only seen modeling programs where the shape being drawn is mirrored as it is being drawn, and the modeler is only required to draw half of the shape.
The subdivision level for surfaces are chosen so that each curve piece has the same number of segments. This makes creating the subdivided surfaces easier to compute, but is not the best option for models containing areas of small detail and large areas without much detail. In this case, the large areas of less detail are most likely over defined.
Neighboring surfaces are connected with only G1 continuity. For several real-world modeling applications, higher continuity connection is required. Usually, G2 continuity is the minimum smoothness demanded for surfaces which reflect or which need to have good air flow characteristics. If this system were to be used for real-world design, more work would have to be done so that higher continuity can be achieved. Unfortunately, the option of higher continuity may make the system slightly less easy to use.
While the fan-shaped, multi-function widget is novel and may reduce tool selection time for expert users, I think it would produce confusion and tool selection errors in novice users.
FreeDrawer is a collection of 3D curve drawing and deformation tools targeted towards product design. The Responsive Workbench program models objects using Catmull-Clark surfaces and B-spline curves as opposed to Constructive Solid Geometry or voxel-based virtual clay. Deformations allowed by B-spline curves and Catmull-Clark surfaces are beneficial for product design, in which original sketches need to be modified to produce the final shape.
The first curve of an object can be freely drawn anywhere in space, but subsequent curves are woven into the existing curve network - being altered if they do not intersect an existing curve. When a curve is modified, the rest of the model can be altered in two ways. In local mode, the modified curve and curves adjacent to it are altered. In global mode, all curves are altered recursively. Surfaces can be filled in with Catmull-Clark or Kuriyama surfaces when a closed loop of curves is selected.
The non-dominant hand is responsible for translating and rotating the model, while the dominant hand is responsible for editing and deforming the model. An original 3D widget with multiple pointers spread out in a fan shape support multiple editing tools - each pointer is mapped to a different tool. A sketch of a chair seat took an experienced user approximately 15 minutes to create.
Discussion:
The idea of the designer drawing the full sweep of a curve across both halves of a mirrored plane and the program averaging each to match symmetrically is new to me. Previously, I had only seen modeling programs where the shape being drawn is mirrored as it is being drawn, and the modeler is only required to draw half of the shape.
The subdivision level for surfaces are chosen so that each curve piece has the same number of segments. This makes creating the subdivided surfaces easier to compute, but is not the best option for models containing areas of small detail and large areas without much detail. In this case, the large areas of less detail are most likely over defined.
Neighboring surfaces are connected with only G1 continuity. For several real-world modeling applications, higher continuity connection is required. Usually, G2 continuity is the minimum smoothness demanded for surfaces which reflect or which need to have good air flow characteristics. If this system were to be used for real-world design, more work would have to be done so that higher continuity can be achieved. Unfortunately, the option of higher continuity may make the system slightly less easy to use.
While the fan-shaped, multi-function widget is novel and may reduce tool selection time for expert users, I think it would produce confusion and tool selection errors in novice users.
Shape Your Imagination: Iconic Gestural-Based Interaction (Marsh 1998)
Summary:
Iconic gestures are a natural, intuitive, and effective way to convey spatial information. This paper separates iconic gestures into virtual and substitutive categories. Virtual depiction is defined as using the spatial relationship between the hands as an outline or tracing the picture of an object or shape. Substitutive depiction is defined as shaping the hands to match the form of an object as if it were being held or grasped in the hands.
A user study is performed to determine if humans use iconic hand gestures during non-verbal communication of shapes and objects, and if so, what kind of gestures are used with what frequency, and how many hands are involved in the gestures. Twelve volunteers were seated in a room and asked to non-verbally describe fifteen shapes and objects using hand gestures. Care was taken not to influence the volunteers' descriptions or otherwise influence their gestures by providing images of the objects.
Subjects used two-handed gestures 88.1% of the time, preferring virtual depiction. 75% of gestures were performed using purely virtual depiction, 17.9% were performed using purely substitutive depiction, and 7.1% used a combination of both. Only one object, the circle, was performed using primarily one-handed virtual depiction gestures.
Discussion:
The results of this paper were not surprising. I suppose it was still good to confirm the ideas experimentally, although a larger group of subjects could have been used to obtain more statistically significant results. Also, since the goal was to see how gestures are used naturally, an experiment could be designed to observe when gestures accompany certain spoken words or phrases. Telling the subjects that they must communicate completely non-verbally does not seem like a situation in which the most natural gestures would be made.
In general, users found describing 2D shapes easier than 3D shapes. I think an analogy to this insight exists in the recognition realm. Sketch recognition in two dimensions is an easier problem than gesture recognition in three dimensions.
Iconic gestures are a natural, intuitive, and effective way to convey spatial information. This paper separates iconic gestures into virtual and substitutive categories. Virtual depiction is defined as using the spatial relationship between the hands as an outline or tracing the picture of an object or shape. Substitutive depiction is defined as shaping the hands to match the form of an object as if it were being held or grasped in the hands.
A user study is performed to determine if humans use iconic hand gestures during non-verbal communication of shapes and objects, and if so, what kind of gestures are used with what frequency, and how many hands are involved in the gestures. Twelve volunteers were seated in a room and asked to non-verbally describe fifteen shapes and objects using hand gestures. Care was taken not to influence the volunteers' descriptions or otherwise influence their gestures by providing images of the objects.
Subjects used two-handed gestures 88.1% of the time, preferring virtual depiction. 75% of gestures were performed using purely virtual depiction, 17.9% were performed using purely substitutive depiction, and 7.1% used a combination of both. Only one object, the circle, was performed using primarily one-handed virtual depiction gestures.
Discussion:
The results of this paper were not surprising. I suppose it was still good to confirm the ideas experimentally, although a larger group of subjects could have been used to obtain more statistically significant results. Also, since the goal was to see how gestures are used naturally, an experiment could be designed to observe when gestures accompany certain spoken words or phrases. Telling the subjects that they must communicate completely non-verbally does not seem like a situation in which the most natural gestures would be made.
In general, users found describing 2D shapes easier than 3D shapes. I think an analogy to this insight exists in the recognition realm. Sketch recognition in two dimensions is an easier problem than gesture recognition in three dimensions.
A Dynamic Gesture Recognition System for the Korean Sign Language (Kim 1996)
Summary:
The paper describes a system for recognizing Korean Sign Language (KSL) gestures and converting them into text. The system employs two ten-sensor VPL Data-Gloves to measure the bend in the joints of each digit on both hands. The gloves also sense the position and orientation of each hand relative to a fixed source. The hand gestures need to be recognized regardless of their position relative to an initial position which varies, the position data is recorded as the difference between the previous position and the current position. For the paper, 25 of around 6000 KSL gestures were analyzed and partitioned into ten sets based on their general direction. To determine which of the ten direction categories the motion will be classified as during recognition, the change over the five most recent readings of region data are examined. Hand postures are recognized by applying the technique of Fuzzy Min-Max Neural (FMMN) Networks. Input from the data glove are identified as one of the direction classes and then recognized by the FMMN network. The system classifies gestures correctly almost 85% of the time.
Discussion:
Figure 8 shows a diagram of the min-max neural network used for classification. There are ten input nodes at the bottom, one for each of the flex angles measured from the fingers. However, there should be fourteen class noes at the top of the diagram instead of ten since there are fourteen posture classes.
When motion data is expressed in its compressed form, the order of region data is preserved, but data relating to the length in time of each region is lost.
Although the paper states that the FMMN network requires no pre-learning about posture class and has on-line adaptability, I'd say the basic idea of neural networks does require learning since that is how the weights of the network are adjusted.
The mis-classifications are partially blamed on abnormal motions in gestures and postures, but dealing with data that exhibits less than ideal characteristics seems to be part of the point of applying complex solutions to gesture recognition.
The paper describes a system for recognizing Korean Sign Language (KSL) gestures and converting them into text. The system employs two ten-sensor VPL Data-Gloves to measure the bend in the joints of each digit on both hands. The gloves also sense the position and orientation of each hand relative to a fixed source. The hand gestures need to be recognized regardless of their position relative to an initial position which varies, the position data is recorded as the difference between the previous position and the current position. For the paper, 25 of around 6000 KSL gestures were analyzed and partitioned into ten sets based on their general direction. To determine which of the ten direction categories the motion will be classified as during recognition, the change over the five most recent readings of region data are examined. Hand postures are recognized by applying the technique of Fuzzy Min-Max Neural (FMMN) Networks. Input from the data glove are identified as one of the direction classes and then recognized by the FMMN network. The system classifies gestures correctly almost 85% of the time.
Discussion:
Figure 8 shows a diagram of the min-max neural network used for classification. There are ten input nodes at the bottom, one for each of the flex angles measured from the fingers. However, there should be fourteen class noes at the top of the diagram instead of ten since there are fourteen posture classes.
When motion data is expressed in its compressed form, the order of region data is preserved, but data relating to the length in time of each region is lost.
Although the paper states that the FMMN network requires no pre-learning about posture class and has on-line adaptability, I'd say the basic idea of neural networks does require learning since that is how the weights of the network are adjusted.
The mis-classifications are partially blamed on abnormal motions in gestures and postures, but dealing with data that exhibits less than ideal characteristics seems to be part of the point of applying complex solutions to gesture recognition.
Sunday, April 27, 2008
Simultaneous Gesture Segmentation and Recognition based on Forward Spotting Accumulative HMMs (Song 2006)
Summary:
Most gesture segmentation and recognition systems examine gesture patterns for places to segment the data, which must be done before recognition. This requires that the gesture be complete before recognition begins, which delays the rate at which recognition can be performed. This work attempts to use a forward spotting scheme to perform gesture segmentation and recognition at the same time. A sliding window of observations is used to compute the probability of the observations within the window being a gesture or a non-gesture. The window averages affects of abruptly changing features. Accumulative HMMs are used in the gesture recognition portion of the system; they accept partial posture segments for which the most likely posture is determined. The final posture is determined by majority voting between the partial posture candidates.
An experiment using eight gestures to open and close curtains and turn lights on and off was performed to compare automatic and manual gesture segmentation and recognition. For segmentation, the automatic threshold method was reported to be 17.62% better than the manual threshold method. For recognition, the automatic method was reported to be 3.75% more accurate than the manual method.
Discussion:
The application of controlling curtains and lights in a home seemed a bit non-useful to me, since a gesture recognition system in the living room would probably receive a lot of signals that were not intended to be interpreted by the system. Installing such a system in a real home could cause some interesting scenarios for a family playing charades.
The results of the experiment depend on the manual recognition process, which was not described in the paper. Since there was no description of how the manual processes were performed, I don't know how to interpret the reported benefits of the automatic methods.
The gestures chosen for recognition are very simple and could be recognized by simply detecting whether a hand of the person passed some threshold of the camera space. HMMs are too complex of a solution for the type of gestures in the experiment.
Most gesture segmentation and recognition systems examine gesture patterns for places to segment the data, which must be done before recognition. This requires that the gesture be complete before recognition begins, which delays the rate at which recognition can be performed. This work attempts to use a forward spotting scheme to perform gesture segmentation and recognition at the same time. A sliding window of observations is used to compute the probability of the observations within the window being a gesture or a non-gesture. The window averages affects of abruptly changing features. Accumulative HMMs are used in the gesture recognition portion of the system; they accept partial posture segments for which the most likely posture is determined. The final posture is determined by majority voting between the partial posture candidates.
An experiment using eight gestures to open and close curtains and turn lights on and off was performed to compare automatic and manual gesture segmentation and recognition. For segmentation, the automatic threshold method was reported to be 17.62% better than the manual threshold method. For recognition, the automatic method was reported to be 3.75% more accurate than the manual method.
Discussion:
The application of controlling curtains and lights in a home seemed a bit non-useful to me, since a gesture recognition system in the living room would probably receive a lot of signals that were not intended to be interpreted by the system. Installing such a system in a real home could cause some interesting scenarios for a family playing charades.
The results of the experiment depend on the manual recognition process, which was not described in the paper. Since there was no description of how the manual processes were performed, I don't know how to interpret the reported benefits of the automatic methods.
The gestures chosen for recognition are very simple and could be recognized by simply detecting whether a hand of the person passed some threshold of the camera space. HMMs are too complex of a solution for the type of gestures in the experiment.
A Survey of POMDP Applications (Cassandra 1998)
Summary:
The partially observable Markov decision process (POMDP) model is heavily based on regular Markov decision process models which have been the focus of other papers we have read dealing with Hidden Markov models. The defining parts of a POMDP model are three finite sets and three functions. Similar to HMMs, there is a set of states, a set of actions, and a set of observations. The transition and observation functions are the familiar probability distributions of HMMs. The main difference is the addition of an immediate reward function which gives the immediate benefit for performing each action while in each state.
The paper gives an overview of several application areas for the POMDP model. In the area of industrial applications, the example of finding a policy for machine maintenance is given. This is one of the oldest applications of the POMDP model and it fits the problem very well since the observations are probabilistically related to internal states. Other industrial applications include structural inspection, elevator control policies, and the fishery industry. Autonomous robots are given as another possible application of the POMDP model. Currently, finding control rules for robots using the method is done at a higher level of abstraction than at the sensor and actuator level. The paper mentions that it might be possible to use a hierarchical arrangement of POMDP models to get the model functioning closer to the level of the hardware. Other scientific applications include behavioral ecology and machine vision. Business applications include network troubleshooting, distributed database queries, and marketing. Some military and social applications are also mentioned.
Discussion:
One observation from the machine vision application example is that POMDP models work best in special purpose visual systems where the domain has been restricted. One of the special systems mentioned was a gesture recognizer which attempts pattern recognition. Since this area is the most closely related to instrumented gesture recognition, I think it may be beneficial to read the paper referenced in this section (Active Gesture Recognition using Partially Observable Markov Decision Processes by Trevor Darrell and Alex Pentland).
One of the more interesting application areas to me was education where the internal mental state of an individual is the model. The reward function could even be applied at an individual level, taking into consideration a student's learning style. I think the limitation of discretizing a space of concepts makes the application to learning not totally accurate. Concepts are inherently difficult to define. Some of the application areas seemed somewhat far-fetched in that too much information would be required to build an accurate model. In addition to these theoretical limitations, the practical limitations of representing and performing computation on the models is even more severe. Because of the difficulty of correct POMDP modeling, algorithms which consider characteristics of the problem area must be used to achieve timely results.
The partially observable Markov decision process (POMDP) model is heavily based on regular Markov decision process models which have been the focus of other papers we have read dealing with Hidden Markov models. The defining parts of a POMDP model are three finite sets and three functions. Similar to HMMs, there is a set of states, a set of actions, and a set of observations. The transition and observation functions are the familiar probability distributions of HMMs. The main difference is the addition of an immediate reward function which gives the immediate benefit for performing each action while in each state.
The paper gives an overview of several application areas for the POMDP model. In the area of industrial applications, the example of finding a policy for machine maintenance is given. This is one of the oldest applications of the POMDP model and it fits the problem very well since the observations are probabilistically related to internal states. Other industrial applications include structural inspection, elevator control policies, and the fishery industry. Autonomous robots are given as another possible application of the POMDP model. Currently, finding control rules for robots using the method is done at a higher level of abstraction than at the sensor and actuator level. The paper mentions that it might be possible to use a hierarchical arrangement of POMDP models to get the model functioning closer to the level of the hardware. Other scientific applications include behavioral ecology and machine vision. Business applications include network troubleshooting, distributed database queries, and marketing. Some military and social applications are also mentioned.
Discussion:
One observation from the machine vision application example is that POMDP models work best in special purpose visual systems where the domain has been restricted. One of the special systems mentioned was a gesture recognizer which attempts pattern recognition. Since this area is the most closely related to instrumented gesture recognition, I think it may be beneficial to read the paper referenced in this section (Active Gesture Recognition using Partially Observable Markov Decision Processes by Trevor Darrell and Alex Pentland).
One of the more interesting application areas to me was education where the internal mental state of an individual is the model. The reward function could even be applied at an individual level, taking into consideration a student's learning style. I think the limitation of discretizing a space of concepts makes the application to learning not totally accurate. Concepts are inherently difficult to define. Some of the application areas seemed somewhat far-fetched in that too much information would be required to build an accurate model. In addition to these theoretical limitations, the practical limitations of representing and performing computation on the models is even more severe. Because of the difficulty of correct POMDP modeling, algorithms which consider characteristics of the problem area must be used to achieve timely results.
A Similarity Measure for Motion Stream Segmentation and Recognition (Chuanjun 2005)
Summary:
The paper proposes a similarity measure to deal with problems in recognizing and segmenting motion streams. Specifically, the problems dealt with are differing lengths of motion duration, different variations in attributes of similar motions, and the fact that motion streams are continuous with no obvious "pause" at which to segment. The model presented represents a motion by a matrix in which columns correspond to joint angles and rows correspond to samples in time. Singular value decomposition is applied to two of these motion matrices of varying rows to determine a measurement of similarity. Eigenvalues and eigenvectors of the matrices representing both streaming motion input and known motion patterns are calculated and compared. The more similar two matrices are, the closer their eigenvectors are to being parallel and the closer their eigenvalues are to being proportional to each other.
The measurement of similarity is referred to as k Weighted Angular Similarity (kWAS). Data was collected from both a CyberGlove and a Vicon motion capture system to test the performance of the kWAS measurement. Although the first two eigenvalues (k = 2 in terms of kWAS) account for more than 95% of the sums of all the eigenvalues, considering the first two eigenvectors was not sufficient to recognize gestures at an acceptable rate. Increasing the value of k to six increased the accuracy of recognition for CyberGlove data to 94% and motion capture data to 94.6%. These results were higher than two other similarity measures called Eros and MAS.
Discussion:
Not only was the kWAS measure more accurate, it required about the same running time as the MAS measurement and was twice as fast as the Eros measurement. It might be interesting to see how the speed of kWAS is affected by changing the value of k - especially in the context of comparing it with the other measurements, if they have variables that can be adjusted which affect their running time.
Since the proposed measure of similarity does not take into account the order patterns were executed in, it is unable to distinguish gestures which pass through the same positions at different times. For instance, performing a gesture normally and exactly in reverse would be identical in terms of the kWAS similarity measure.
The paper proposes a similarity measure to deal with problems in recognizing and segmenting motion streams. Specifically, the problems dealt with are differing lengths of motion duration, different variations in attributes of similar motions, and the fact that motion streams are continuous with no obvious "pause" at which to segment. The model presented represents a motion by a matrix in which columns correspond to joint angles and rows correspond to samples in time. Singular value decomposition is applied to two of these motion matrices of varying rows to determine a measurement of similarity. Eigenvalues and eigenvectors of the matrices representing both streaming motion input and known motion patterns are calculated and compared. The more similar two matrices are, the closer their eigenvectors are to being parallel and the closer their eigenvalues are to being proportional to each other.
The measurement of similarity is referred to as k Weighted Angular Similarity (kWAS). Data was collected from both a CyberGlove and a Vicon motion capture system to test the performance of the kWAS measurement. Although the first two eigenvalues (k = 2 in terms of kWAS) account for more than 95% of the sums of all the eigenvalues, considering the first two eigenvectors was not sufficient to recognize gestures at an acceptable rate. Increasing the value of k to six increased the accuracy of recognition for CyberGlove data to 94% and motion capture data to 94.6%. These results were higher than two other similarity measures called Eros and MAS.
Discussion:
Not only was the kWAS measure more accurate, it required about the same running time as the MAS measurement and was twice as fast as the Eros measurement. It might be interesting to see how the speed of kWAS is affected by changing the value of k - especially in the context of comparing it with the other measurements, if they have variables that can be adjusted which affect their running time.
Since the proposed measure of similarity does not take into account the order patterns were executed in, it is unable to distinguish gestures which pass through the same positions at different times. For instance, performing a gesture normally and exactly in reverse would be identical in terms of the kWAS similarity measure.
Saturday, April 26, 2008
Cyber Composer: Hand Gesture-Driven Intelligent Music Composition and Generation (Ip 2005)
Summary:
Cyber Composer is a music generation system that combines capturing hand gesture commands with music theory to produce an interactive approach to compositions. The paper supplies much background information in music theory, most of which defines variables of music which need to be set in a certain way to achieve a certain musical style. Several rules were introduced that shape music production based on classical music theory. The system determines which diatomic scale to use based on what musical key is specified. Given an initial chord, the degree of affinity between chord pairs is considered when determining subsequent chords. Cadence is avoided in general, allowing the user to trigger it. Harmony notes are used for the melody note at the accented beat to serve as a balance between using too many or too few harmony notes.
The system itself uses two CyberGloves, each with a polhemus receiver, to track hand and finger motion, position, and orientation. Hand signals are translated into motion triggers by the main program module and sent to the melody-generation module where MIDI output is produced according to the style template and music-theory-based rules. The user defines tempo and key before the interactive portion of the composition to be used in background music generation. During the interactive composition, melody notes are triggered when the wrist is straightened. Pitch is controlled by the relative height of the right hand compared to its position during the previous notes. Force of a note is determined by finger extension, while volume is controlled by finger flexion. Closing the left hand fingers triggers cadence. There was no concrete measurement of the system's ability to follow through on claims of allowing laypeople to express music in an interactive, innovative, and intuitive way.
Discussion:
Waving the wrist at every melody note seems like it could get tiring over the duration of a long song. Perhaps fixing the rhythm to remain constant (until recognition of a certain finger indicating that change was desired) would make the composition experience less repetitious. This change makes sense since rhythm is somewhat temporally consistent, meaning that it doesn't change a lot from moment to moment, and a change is usually not followed quickly by another change.
There is no real user study to speak of, but the creative nature of the system does not lend itself well to analysis. The approach is fairly novel, and the gestures control a variety of musical aspects, even if the mapping from gestures to their effects are not the most intuitive. Perhaps a user study should be done to see what gestures people would perform when wanting to signal changes in variables used by Cyber Composer when generating music.
As an improvement to MIDI-based sounds, a system could be devised to play back sound samples of an instrument using a library such as FMOD. For holding longer notes, a loop point within each sound sample could be set to allow arbitrarily long, yet fluid, note samples to be generated.
Cyber Composer is a music generation system that combines capturing hand gesture commands with music theory to produce an interactive approach to compositions. The paper supplies much background information in music theory, most of which defines variables of music which need to be set in a certain way to achieve a certain musical style. Several rules were introduced that shape music production based on classical music theory. The system determines which diatomic scale to use based on what musical key is specified. Given an initial chord, the degree of affinity between chord pairs is considered when determining subsequent chords. Cadence is avoided in general, allowing the user to trigger it. Harmony notes are used for the melody note at the accented beat to serve as a balance between using too many or too few harmony notes.
The system itself uses two CyberGloves, each with a polhemus receiver, to track hand and finger motion, position, and orientation. Hand signals are translated into motion triggers by the main program module and sent to the melody-generation module where MIDI output is produced according to the style template and music-theory-based rules. The user defines tempo and key before the interactive portion of the composition to be used in background music generation. During the interactive composition, melody notes are triggered when the wrist is straightened. Pitch is controlled by the relative height of the right hand compared to its position during the previous notes. Force of a note is determined by finger extension, while volume is controlled by finger flexion. Closing the left hand fingers triggers cadence. There was no concrete measurement of the system's ability to follow through on claims of allowing laypeople to express music in an interactive, innovative, and intuitive way.
Discussion:
Waving the wrist at every melody note seems like it could get tiring over the duration of a long song. Perhaps fixing the rhythm to remain constant (until recognition of a certain finger indicating that change was desired) would make the composition experience less repetitious. This change makes sense since rhythm is somewhat temporally consistent, meaning that it doesn't change a lot from moment to moment, and a change is usually not followed quickly by another change.
There is no real user study to speak of, but the creative nature of the system does not lend itself well to analysis. The approach is fairly novel, and the gestures control a variety of musical aspects, even if the mapping from gestures to their effects are not the most intuitive. Perhaps a user study should be done to see what gestures people would perform when wanting to signal changes in variables used by Cyber Composer when generating music.
As an improvement to MIDI-based sounds, a system could be devised to play back sound samples of an instrument using a library such as FMOD. For holding longer notes, a loop point within each sound sample could be set to allow arbitrarily long, yet fluid, note samples to be generated.
A Multi-Class Pattern Recognition System for Practical Finger Spelling Translation (Hernandez-Rebollar 2002)
Summary:
This paper recognizes static postures from the sign language alphabet from data collected through dual-axis accelerometers mounted on each of the fingers. The ten dimensional data (five fingers with two directions of accelerometer sensing each) collected for each letter is reduced to 3 dimensions. The X-global and Y-global features are extracted from the 10 dimensional data as well as the index finger's height. Plotting distributions of these three features in 3D helps visualize clusters of data. The 3D data are projected onto planes to better decide dividing points for the classifiers. Gestures are identified by a hierarchical three level classifier.
Ten posture readings were collected for each letter from each of from five volunteers. The posture readings for letters 'J' and 'Z' were collected at the end of the motion for the letter. Once the data has been analyzed and boundaries for the classifiers have been set up, they are programmed onto a microcontroller which is connected to a speech synthesizer so that the letters formed by hand postures can be heard. Twenty one of the 26 letters had a recognition rate of 100%. The worst recognition rate was 78% for the letter 'U'.
Discussion:
The statistics reported about the accelerometers' resolution and the diagram describing the hierarchical classifier are sufficiently detailed. One unique aspect of this system is that it uses a computer only for off-line analyzation of data and not during the online recognition. The use of a microcontroller for recognition makes the prototype system portable and closer to a system that could actually be used in real life than most of the other systems covered in the literature.
This paper recognizes static postures from the sign language alphabet from data collected through dual-axis accelerometers mounted on each of the fingers. The ten dimensional data (five fingers with two directions of accelerometer sensing each) collected for each letter is reduced to 3 dimensions. The X-global and Y-global features are extracted from the 10 dimensional data as well as the index finger's height. Plotting distributions of these three features in 3D helps visualize clusters of data. The 3D data are projected onto planes to better decide dividing points for the classifiers. Gestures are identified by a hierarchical three level classifier.
Ten posture readings were collected for each letter from each of from five volunteers. The posture readings for letters 'J' and 'Z' were collected at the end of the motion for the letter. Once the data has been analyzed and boundaries for the classifiers have been set up, they are programmed onto a microcontroller which is connected to a speech synthesizer so that the letters formed by hand postures can be heard. Twenty one of the 26 letters had a recognition rate of 100%. The worst recognition rate was 78% for the letter 'U'.
Discussion:
The statistics reported about the accelerometers' resolution and the diagram describing the hierarchical classifier are sufficiently detailed. One unique aspect of this system is that it uses a computer only for off-line analyzation of data and not during the online recognition. The use of a microcontroller for recognition makes the prototype system portable and closer to a system that could actually be used in real life than most of the other systems covered in the literature.
Thursday, April 24, 2008
Hand Tension as a Gesture Segmentation Cue (Harling 1997)
Summary:
The premise of this paper is that hand tension can be used to aid segmentation of a sequence of gestures. The proposed idea is to look for places in time where the hand tension is minimized and treat these minimums as places to to segment the sequence of gestures. The model used to measure hand tension treats each finger as a rod of fixed length attached to two ideal springs whose tensions are calculated using Hooke's law and summed to produce an estimate of finger tension. Overall hand tension is computed by simply summing the tensions of each finger of the hand.
Gestures are split into four classifications, combinations of static and dynamic hand postures and hand locations. An experiment using a Power Glove to collect data from simple sequences of sign language gestures was performed. The Power Glove's limitation of only 4 levels of measurable tension and the study size of only one user contributed to the lack of robustness of the data used in the experiment.
Discussion:
The definition of posture and gesture are distinguished by the exclusion or inclusion of hand location and orientation information. I think these definitions are influenced by the hardware used for gathering data.
A solution to segmentation problem posed by a referenced paper is to hold a posture for one minute before it is recognized. This is quite a while in computer time, and seems unnecessarily long.
The hand tension segmentation algorithm does not attempt to recognize a posture until after a minimum value of hand tension that follows a possible gesture is identified. This enforced delay in recognition may seem a bit slow to the user and pushes the response time just outside the real-time range.
The hand tension model described cannot correctly segment gestures in which individual fingers increase and decrease tension at the same rate since the finger tensions are summed and there would be no drop in overall tension.
The calculation of hand tension doesn't account for tension in the palm of the hand, which is an important part of overall hand tension. The hand tension model could be improved by considering the direction of tension between adjacent fingers. Adjacent fingers whose tension lies in opposite directions should increase overall tension.
The premise of this paper is that hand tension can be used to aid segmentation of a sequence of gestures. The proposed idea is to look for places in time where the hand tension is minimized and treat these minimums as places to to segment the sequence of gestures. The model used to measure hand tension treats each finger as a rod of fixed length attached to two ideal springs whose tensions are calculated using Hooke's law and summed to produce an estimate of finger tension. Overall hand tension is computed by simply summing the tensions of each finger of the hand.
Gestures are split into four classifications, combinations of static and dynamic hand postures and hand locations. An experiment using a Power Glove to collect data from simple sequences of sign language gestures was performed. The Power Glove's limitation of only 4 levels of measurable tension and the study size of only one user contributed to the lack of robustness of the data used in the experiment.
Discussion:
The definition of posture and gesture are distinguished by the exclusion or inclusion of hand location and orientation information. I think these definitions are influenced by the hardware used for gathering data.
A solution to segmentation problem posed by a referenced paper is to hold a posture for one minute before it is recognized. This is quite a while in computer time, and seems unnecessarily long.
The hand tension segmentation algorithm does not attempt to recognize a posture until after a minimum value of hand tension that follows a possible gesture is identified. This enforced delay in recognition may seem a bit slow to the user and pushes the response time just outside the real-time range.
The hand tension model described cannot correctly segment gestures in which individual fingers increase and decrease tension at the same rate since the finger tensions are summed and there would be no drop in overall tension.
The calculation of hand tension doesn't account for tension in the palm of the hand, which is an important part of overall hand tension. The hand tension model could be improved by considering the direction of tension between adjacent fingers. Adjacent fingers whose tension lies in opposite directions should increase overall tension.
Toward Natural Gesture/Speech HCI: A Case Study of Weather Narration Poddar 1998)
Summary:
This paper examined gestures in the natural domain of weather newscasting. The authors describe a natural human computer interface as including a gesture recognition module, a speech recognition module, and a display for providing audio and visual feedback.
An HMM-based gesture recognition system was used to analyze video of five weather persons. The features used in the gesture recognition were extracted from the video using kalman filtering and color segmentation. Specifically, the distance, radial and angular velocities of each hand with respect to the head were used to describe hand motion. Gestures were classified into three main categories: pointing, area, and contour. Gestures were also separated into phases of preparation, retraction, and actual stroke. This led to the choice of using left-to-right causal models with 3 states for the preparation, retraction, and point HMMs and 4 states for the contour and rest HMMs.
A study was done to determine the co-occurrence of spoken words with specific gestures. When the results of the co-occurrence analysis were applied to the data, recognition rates were higher for three of the four video sequences examined. However, recognition rates remained somewhat low overall, the highest was 75% accuracy.
Discussion:
Since the data comes from a weather newscasting environment, the background filtering has the potential to be much simpler. Instead of using the video input of the composited image with the weather map displayed in the background, the raw video feed of the newscaster in front of the blue or green screen could be used. The color-based filtering algorithm would have a much easier job since the static, singly colored background can be easily filtered out.
The paper mentions a probability that could be interpreted as the weather person's handedness. I don't think handedness would affect the hand used for a gesture as much as where the weather person happened to be standing in relation to the portion of the map being discussed at the time of the gesture or which hand held the clicker that advances the background image to the next video feed.
This paper examined gestures in the natural domain of weather newscasting. The authors describe a natural human computer interface as including a gesture recognition module, a speech recognition module, and a display for providing audio and visual feedback.
An HMM-based gesture recognition system was used to analyze video of five weather persons. The features used in the gesture recognition were extracted from the video using kalman filtering and color segmentation. Specifically, the distance, radial and angular velocities of each hand with respect to the head were used to describe hand motion. Gestures were classified into three main categories: pointing, area, and contour. Gestures were also separated into phases of preparation, retraction, and actual stroke. This led to the choice of using left-to-right causal models with 3 states for the preparation, retraction, and point HMMs and 4 states for the contour and rest HMMs.
A study was done to determine the co-occurrence of spoken words with specific gestures. When the results of the co-occurrence analysis were applied to the data, recognition rates were higher for three of the four video sequences examined. However, recognition rates remained somewhat low overall, the highest was 75% accuracy.
Discussion:
Since the data comes from a weather newscasting environment, the background filtering has the potential to be much simpler. Instead of using the video input of the composited image with the weather map displayed in the background, the raw video feed of the newscaster in front of the blue or green screen could be used. The color-based filtering algorithm would have a much easier job since the static, singly colored background can be easily filtered out.
The paper mentions a probability that could be interpreted as the weather person's handedness. I don't think handedness would affect the hand used for a gesture as much as where the weather person happened to be standing in relation to the portion of the map being discussed at the time of the gesture or which hand held the clicker that advances the background image to the next video feed.
Subscribe to:
Posts (Atom)