Summary:
The paper suggests creating an "image externalization loop", which is just a fancy way of saying they provide visual feedback of a virtual object as it is being manipulated. Instead of representing and manipulating objects at the vertex or polygonal level, mathematical expressions of the 3D objects were created which take deformation parameters which can be altered through gestures. The system is implemented in C, using the OpenGL library for rendering graphics. Finger joint angles are read by two CyberGloves and fed into a static posture recognizer to classify hand shape. Position and orientation data for both hands are read by a polyhemus tracker and sent to a dynamic gesture recognizer. The recognized hand shape determines what operation to perform, while the movement is used to determine how to adjust the deformation parameters. Segmentation is performed based on the assumption that static hand posture remains generally fixed during motion of the hands. The left hand is used as a frame of reference for the motion of the right hand, which scales or rotates objects. It seems only three static gestures were recognized, an open hand for "deform", a closed fist for "grasp", and a pointing index finger for "point".
A gesture learning and recognition function, called the Two-Handed dynamic Gesture environment Shell (TGSH) utilizes a self-organizing feature map algorithm to allow users to specify their preferred gestures.
One experiment involved recreating virtual objects to match real ones. The dimensions of real objects were scanned using Minolta's Vivid 700 3D digitizer. Users were able to recreate the models using the system.
Discussion:
The amount of data required to store an object by its deformation parameters is greatly reduced when compared to polygonal representations (factors of 1:670 and 1:970 were given for two example objects). The idea of using the deformation parameters that describe the objects as searchable categories in a database of 3D objects is interesting.
The authors identified the trade-off between quality of the rendering and interactivity. The number of polygons was limited to allow sufficient drawing rates and interactive responsiveness. With today's technology (10 years of improvements), I doubt the restrictions of 8,000 and 32,000 polygons per object would be as strict.
Since implicit functions are used to model objects, the collision detection can be computed easily, by simply evaluating the function at a location in space and comparing the result to a constant. This is much easier than performing collision detection with polygonal based models.
The fact that the blending function used produces G2 continuous surfaces is important since it allows reflections off the surface, including highlights, to be displayed with G1 continuity. Curves with less than G1 continuity do not look smooth and are easily detectable by humans.
The system only implements one of the superquadrics, the ellipsoid. There are three other superquadrics, namely hyperboloids of one and two sheets, and the toroid. Adding these additional primitives would be a natural extension for the existing system.
Tuesday, April 22, 2008
Sunday, April 20, 2008
Feature selection for grasp recognition from optical markers (Chang 2007)
Summary:
The goal of this paper is to create a system which can learn how to automatically create grasps for a robot manipulator provided examples from a human demonstrator. The positions of a set of 3D markers placed on the back of the hand are used as an orientation-independant set of characteristics which are used to classify hand poses into one of six categories of grasps. Since a linear logistic regression classifier is sufficient to predict grasps, one goal of the paper is to reduce the number of markers while retaining recognition rates.
To bypass considering the exponential number of possible feature sets, two greedy sequential wrapper methods were used to evaluate the addition or removal of single features. The wrapper methods' goal was to achieve local optimization by considering which single feature to add or remove, while retaining the highest recognition rate. To achieve hand orientation independance, three of the 3D markers were attached to a rigid part of the back of the hand that would serve to orient all the other marker positions.
A data set pairing grasp type with marker positions for grasps of 46 objects was collected from a total of three subjects. Using all 30 markers resulted in an accuracy of 91.5%, while using only five resulted in an accuracy of 86%. The set of markers selected by the backward selection differed from the set of markers selected by forward selection, and had a higher recognition rate.
Discussion:
I thought using the ratio of the highest class probability to the second highest class probability as a measure of confidence was a good idea.
I think the choice to initially use 30 markers was artificially high. The experimenters probably expected a significant decrease in marker count without a great loss in grasp recognition accuracy, simply because they chose a high number of markers.
The goal of this paper is to create a system which can learn how to automatically create grasps for a robot manipulator provided examples from a human demonstrator. The positions of a set of 3D markers placed on the back of the hand are used as an orientation-independant set of characteristics which are used to classify hand poses into one of six categories of grasps. Since a linear logistic regression classifier is sufficient to predict grasps, one goal of the paper is to reduce the number of markers while retaining recognition rates.
To bypass considering the exponential number of possible feature sets, two greedy sequential wrapper methods were used to evaluate the addition or removal of single features. The wrapper methods' goal was to achieve local optimization by considering which single feature to add or remove, while retaining the highest recognition rate. To achieve hand orientation independance, three of the 3D markers were attached to a rigid part of the back of the hand that would serve to orient all the other marker positions.
A data set pairing grasp type with marker positions for grasps of 46 objects was collected from a total of three subjects. Using all 30 markers resulted in an accuracy of 91.5%, while using only five resulted in an accuracy of 86%. The set of markers selected by the backward selection differed from the set of markers selected by forward selection, and had a higher recognition rate.
Discussion:
I thought using the ratio of the highest class probability to the second highest class probability as a measure of confidence was a good idea.
I think the choice to initially use 30 markers was artificially high. The experimenters probably expected a significant decrease in marker count without a great loss in grasp recognition accuracy, simply because they chose a high number of markers.
Wednesday, April 16, 2008
Glove-TalkII - A Nerual-Network Interface which Maps Gestures to Parallel Format Speech Synthesizer Controls (Fels 1998)
Summary:
The Glove-TalkII system maps hand gestures to control parameters of a formant speech synthesizer using an adaptive interface based on neural networks. The right hand's position, orientation, and finger joints are sampled frequently to control the synthesizer parameters. When producing vowels, hand position determines what type of vowel sound to make. When producing consonants, the hand posture determines what consonant sound to make. The left hand wears a ContactGlove which measures nine points of contact and is used to signal stop consonants. A foot pedal controls overall volume of the system while hand height controls speech pitch.
Three neural networks were established to determine what sound to produce. The Vowel/Consonant network is a 10-5-1 feedforward network with sigmoid activations which determines whether the sound to be emitted should be a vowel or a consonant. The Vowel network is a 2-11-8 feedforward network which determines what vowel sound to produce, while the Consonant network is a 10-14-9 feedforward network that determines what consonant sound to produce. The 11 and 14 hidden units of these networks are normalized radial basis functions that correspond to one of 11 cardinal vowels and 14 static consonant phonemes, respectively. The 8 and 9 output units determine parameters to the formant synthesizer. The additional output of the consonant network determines the voicing parameter.
Discussion:
Only one person has been trained to use the system well enough to speak intelligibly. The subject was an accomplished piano player and trained for over 100 hours. Even so, the speed of speech is about 1.5 to 3 times slower than average. With such a time commitment and skill level as a barrier to using the system, I don't think it is likely to ever become practical.
The concept of stop consonants seemed to be introduced to help resolve the problem of sounds whose dynamics were too fast to be controlled by the user. This reminded me of a problem I am having with my final project which uses a Wii remote and a sensorbar to emulate a violin. Creating a smooth sounding transition between notes is not strait forward. I could try to introduce a special transition sound sample to be played in the midst of transitions that would create a more realistic sound.
The Glove-TalkII system maps hand gestures to control parameters of a formant speech synthesizer using an adaptive interface based on neural networks. The right hand's position, orientation, and finger joints are sampled frequently to control the synthesizer parameters. When producing vowels, hand position determines what type of vowel sound to make. When producing consonants, the hand posture determines what consonant sound to make. The left hand wears a ContactGlove which measures nine points of contact and is used to signal stop consonants. A foot pedal controls overall volume of the system while hand height controls speech pitch.
Three neural networks were established to determine what sound to produce. The Vowel/Consonant network is a 10-5-1 feedforward network with sigmoid activations which determines whether the sound to be emitted should be a vowel or a consonant. The Vowel network is a 2-11-8 feedforward network which determines what vowel sound to produce, while the Consonant network is a 10-14-9 feedforward network that determines what consonant sound to produce. The 11 and 14 hidden units of these networks are normalized radial basis functions that correspond to one of 11 cardinal vowels and 14 static consonant phonemes, respectively. The 8 and 9 output units determine parameters to the formant synthesizer. The additional output of the consonant network determines the voicing parameter.
Discussion:
Only one person has been trained to use the system well enough to speak intelligibly. The subject was an accomplished piano player and trained for over 100 hours. Even so, the speed of speech is about 1.5 to 3 times slower than average. With such a time commitment and skill level as a barrier to using the system, I don't think it is likely to ever become practical.
The concept of stop consonants seemed to be introduced to help resolve the problem of sounds whose dynamics were too fast to be controlled by the user. This reminded me of a problem I am having with my final project which uses a Wii remote and a sensorbar to emulate a violin. Creating a smooth sounding transition between notes is not strait forward. I could try to introduce a special transition sound sample to be played in the midst of transitions that would create a more realistic sound.
Activity Recognition using Visual Tracking and RFID (Krahnstoever 2005)
Summary:
The paper combines vision based tracking with RFID sensors to gain more information about tasks being performed. A minimum of two color cameras track the position of the head and hands of a subject by using skin color matching techniques. The subject's state is modeled by three ellipsoids whose centers are represented in spherical coordinates. The volume in which tracking takes place is divided into three parts, one for the head and one for each hand, to reduce the number of particles needed for tracking. The presence and orientation of key objects are tracked using RFID tags. To fully determine an object's orientation, three antennas must be attached to an RFID tag.
The information from visual tracking and RFID tags works together to identify activities of the subject. For instance, the presence of a user's hand in the emitter field of an object and the movement of an RFID-tagged object can imply that the user has picked up and is holding that object.
Discussion:
RFID-tagged objects are much easier to recognize than they would be using visual appearance alone. The system is able to track objects that would be occluded using visual information alone. One of the interesting applications suggested involved multiple RFID tags per item. If one were placed on the lid and one on the container, the action of opening the container could be detected.
When computing the likelihood of a pixel belonging either to a body part in the foreground of an image or to the background, each pixel in the projection of the union of the ellipsoids tracking the head and hands must be evaluated. Instead of performing collision detection on ellipsoidal volumes, rectangular bounding box volumes are used since their collisions are simpler to compute. This technique of reducing the computational complexity by simplifying collision detection boundaries is frequently used in the field of physically based modeling, and quite appropriate to use in this case.
The paper combines vision based tracking with RFID sensors to gain more information about tasks being performed. A minimum of two color cameras track the position of the head and hands of a subject by using skin color matching techniques. The subject's state is modeled by three ellipsoids whose centers are represented in spherical coordinates. The volume in which tracking takes place is divided into three parts, one for the head and one for each hand, to reduce the number of particles needed for tracking. The presence and orientation of key objects are tracked using RFID tags. To fully determine an object's orientation, three antennas must be attached to an RFID tag.
The information from visual tracking and RFID tags works together to identify activities of the subject. For instance, the presence of a user's hand in the emitter field of an object and the movement of an RFID-tagged object can imply that the user has picked up and is holding that object.
Discussion:
RFID-tagged objects are much easier to recognize than they would be using visual appearance alone. The system is able to track objects that would be occluded using visual information alone. One of the interesting applications suggested involved multiple RFID tags per item. If one were placed on the lid and one on the container, the action of opening the container could be detected.
When computing the likelihood of a pixel belonging either to a body part in the foreground of an image or to the background, each pixel in the projection of the union of the ellipsoids tracking the head and hands must be evaluated. Instead of performing collision detection on ellipsoidal volumes, rectangular bounding box volumes are used since their collisions are simpler to compute. This technique of reducing the computational complexity by simplifying collision detection boundaries is frequently used in the field of physically based modeling, and quite appropriate to use in this case.
Tuesday, April 15, 2008
Online, Interactive Learning of Gestures for Human-Robot Interfaces (Lee 1996)
Summary:
This paper's goal was online teaching of new gestures with few examples utilizing discrete HMMs to model each gesture. The system uses a CyberGlove to recognize letters from the sign language alphabet. Input data is sampled from 18 sensors at a rate of 10 Hz and then reduced to a one-dimensional sequence of symbols using vector quantization. The vectors are encoded as a symbol from a codebook, which is a set of vectors representative of the domain of vectors to be encoded. The encoded vectors are sent to a Bakis HMM where the gesture, whose probability is above a certain threshold and not too near other gestures, is chosen as the one recognized. Segmentation is handled simply, by requiring the operator to hold the hand still for a short time between gestures. Only 14 letters were used, and the letters that were chosen are relatively easy to distinguish from each other compared to all possible sets of 14 letters. After two training examples, two experiments yielded 1% and 2.4 % classification errors.
Discussion:
In the section describing the interactive training, the condition of the system being "certain" about a classification of a gesture is described as being handled by performing the associated action. The specifics of being "certain" are not given, but could have been reported as a threshold probability of observed data matching the sequence of observations describing a gesture.
At the end of the interactive training section, the scenario of specifying a new "halt" action is outlined. The details of how the user specifies the label and action associated with a gesture recognition are not given. This guidance required by the user seemed to raise the difficulty level for interacting with the system.
The choice of Bakis HMMs do work for the gesture set of basically static ASL letters, but would be poor for classifying repetitive motions such as waving. This system could not be generalized to recognize all types of gestures since the states of Bakis HMMs are restricted to moving forward.
Since the codebook is generated offline before recognition begins, is the correlation of the codebook vectors to observed vectors decreased when new gestures are being learned? Teaching new gestures will probably change the domain of vectors to be encoded. This should probably alter the codebook, but it does not.
This paper's goal was online teaching of new gestures with few examples utilizing discrete HMMs to model each gesture. The system uses a CyberGlove to recognize letters from the sign language alphabet. Input data is sampled from 18 sensors at a rate of 10 Hz and then reduced to a one-dimensional sequence of symbols using vector quantization. The vectors are encoded as a symbol from a codebook, which is a set of vectors representative of the domain of vectors to be encoded. The encoded vectors are sent to a Bakis HMM where the gesture, whose probability is above a certain threshold and not too near other gestures, is chosen as the one recognized. Segmentation is handled simply, by requiring the operator to hold the hand still for a short time between gestures. Only 14 letters were used, and the letters that were chosen are relatively easy to distinguish from each other compared to all possible sets of 14 letters. After two training examples, two experiments yielded 1% and 2.4 % classification errors.
Discussion:
In the section describing the interactive training, the condition of the system being "certain" about a classification of a gesture is described as being handled by performing the associated action. The specifics of being "certain" are not given, but could have been reported as a threshold probability of observed data matching the sequence of observations describing a gesture.
At the end of the interactive training section, the scenario of specifying a new "halt" action is outlined. The details of how the user specifies the label and action associated with a gesture recognition are not given. This guidance required by the user seemed to raise the difficulty level for interacting with the system.
The choice of Bakis HMMs do work for the gesture set of basically static ASL letters, but would be poor for classifying repetitive motions such as waving. This system could not be generalized to recognize all types of gestures since the states of Bakis HMMs are restricted to moving forward.
Since the codebook is generated offline before recognition begins, is the correlation of the codebook vectors to observed vectors decreased when new gestures are being learned? Teaching new gestures will probably change the domain of vectors to be encoded. This should probably alter the codebook, but it does not.
RFID-enabled Target Tracking and Following with a Mobile Robot Using Direction Finding Antennas (Kim 2007)
Summary:
The goal of the paper is to enable robots to track a target in a real environment containing obstacles without a predefined map. A direction finding RFID reader uses a dual-direction antenna which determines orientation to an RF emitter by comparing the strengths of RF signals on two orthogonal spiral antennas. The strength of the RF signal is proportional to the antenna's orientation to, and the distance from, the transponder. The RFID system is based on commercial active sensor nodes produced by Ymatic Limited and attached to a Pioneer-3DX mobile robot.
A couple of different experiments were run, mapping the ratio of the strength of the RF signals to relative positions of the RF reader and transponder for different paths with different obstacles in the environment. The general trend of the signal ratio strength could be predicted using equations from classical electromagnetic theory, but the graphs were fairly noisy in practice. A different set of experiments involved having the robot with the dual sensing antennas follow a target robot which could move about. The pictures from these experiments show that the tracking works, although the path of the following robot does not adhere closely to the path of the leading robot.
Discussion:
One major advantage of the RFID technology is its lack of dependence on line of sight. This attribute is important for gesture recognition as well as robot tracking. Supposing accurate location information with sufficient granularity could be obtained from RFID sensors, I think they could be used in collecting gesture data. The small size of RFID tags make them easier to attach and more convenient to wear in a glove than ultrasonic sensors.
The fact that the directional antenna was rotated independantly from the robot suggests that the accuracy of measurements obtained from the RFID sensor would not be great enough for accurate gesture data collection. Since the magnitude of the ratio of the RFID signals varies quite a bit with environmental conditions, it would be difficult to get the kind of quality data needed for gesture recognition.
The goal of the paper is to enable robots to track a target in a real environment containing obstacles without a predefined map. A direction finding RFID reader uses a dual-direction antenna which determines orientation to an RF emitter by comparing the strengths of RF signals on two orthogonal spiral antennas. The strength of the RF signal is proportional to the antenna's orientation to, and the distance from, the transponder. The RFID system is based on commercial active sensor nodes produced by Ymatic Limited and attached to a Pioneer-3DX mobile robot.
A couple of different experiments were run, mapping the ratio of the strength of the RF signals to relative positions of the RF reader and transponder for different paths with different obstacles in the environment. The general trend of the signal ratio strength could be predicted using equations from classical electromagnetic theory, but the graphs were fairly noisy in practice. A different set of experiments involved having the robot with the dual sensing antennas follow a target robot which could move about. The pictures from these experiments show that the tracking works, although the path of the following robot does not adhere closely to the path of the leading robot.
Discussion:
One major advantage of the RFID technology is its lack of dependence on line of sight. This attribute is important for gesture recognition as well as robot tracking. Supposing accurate location information with sufficient granularity could be obtained from RFID sensors, I think they could be used in collecting gesture data. The small size of RFID tags make them easier to attach and more convenient to wear in a glove than ultrasonic sensors.
The fact that the directional antenna was rotated independantly from the robot suggests that the accuracy of measurements obtained from the RFID sensor would not be great enough for accurate gesture data collection. Since the magnitude of the ratio of the RFID signals varies quite a bit with environmental conditions, it would be difficult to get the kind of quality data needed for gesture recognition.
A Dynamic Gesture Interface for Virtual Environments Based on Hidden Markov Models (Qing 2005)
Summary:
This paper points out that dynamic gestures are stochastic and while repeating the same gesture will result in different measurements per trial, there are statistical properties which can describe the motion. The paper suggests using HMMs to extract defining characteristics from gesture data in order to classify the gestures. An Expectation-Modification algorithm (another name for Baum-Welch) is used to solve the HMMs. The characteristics examined by their HMM algorithm are the standard deviation of the angle variation of each finger.
An experiment examined the recognition of three gestures, named "Great", "Quote", and "Trigger", which were used to control the rotation of a cube about three axes. The GLUT framework was used to create a simple colored cube which could be rendered and rotated using the OpenGL library. Gesture data was sampled at 10 Hz from a CyberGlove and then normalized. The data are quantized and their standard deviation is calculated before being sent to the HMM to be recognized. Each gesture HMM was trained with 10 data sets each. An observed gesture sequence is compared to each of the trained gestures, and the one with the highest probability is reported as the most likely match.
Discussion:
The authors use several abbreviations throughout the paper, and their overuse just adds another step required for translating the meaning of the paper. The overview of HMMs is a bit brief, but does refer the reader to a paper by Rabiner covering a tutorial of HMMs. The choice to use HMMs was supposedly made to be able to recognize dynamic gestures instead of static postures, but the three gestures used in the study can be distinguished without the use of HMMs. The motion of the hands was unnecessary to distinguish the gestures since the position of the hands is different enough to separate them. Also, there was no user study or reporting of recognition accuracy rates.
This paper points out that dynamic gestures are stochastic and while repeating the same gesture will result in different measurements per trial, there are statistical properties which can describe the motion. The paper suggests using HMMs to extract defining characteristics from gesture data in order to classify the gestures. An Expectation-Modification algorithm (another name for Baum-Welch) is used to solve the HMMs. The characteristics examined by their HMM algorithm are the standard deviation of the angle variation of each finger.
An experiment examined the recognition of three gestures, named "Great", "Quote", and "Trigger", which were used to control the rotation of a cube about three axes. The GLUT framework was used to create a simple colored cube which could be rendered and rotated using the OpenGL library. Gesture data was sampled at 10 Hz from a CyberGlove and then normalized. The data are quantized and their standard deviation is calculated before being sent to the HMM to be recognized. Each gesture HMM was trained with 10 data sets each. An observed gesture sequence is compared to each of the trained gestures, and the one with the highest probability is reported as the most likely match.
Discussion:
The authors use several abbreviations throughout the paper, and their overuse just adds another step required for translating the meaning of the paper. The overview of HMMs is a bit brief, but does refer the reader to a paper by Rabiner covering a tutorial of HMMs. The choice to use HMMs was supposedly made to be able to recognize dynamic gestures instead of static postures, but the three gestures used in the study can be distinguished without the use of HMMs. The motion of the hands was unnecessary to distinguish the gestures since the position of the hands is different enough to separate them. Also, there was no user study or reporting of recognition accuracy rates.
Subscribe to:
Posts (Atom)