Summary:
The motivation for this paper comes from the idea that humans performing gestures can be transferred to an electronic instrument to produce an emotional musical performance without incurring the difficulties of the human learning the technique for controlling a specific instrument. The assumption that emotion is better portrayed by the force applied to an object than by the position of the hand on an object is also crucial to the work, which uses three-dimensional acceleration sensors to capture gesture motion. The acceleration sensor data has poor quantitative reproducibility; so instead of using simple pattern matching, global features of the motions are extracted. In the study, the magnitude of the acceleration change in eight principal directions, rotational direction, intensity of motion, and z direction of motion are used as characteristics for gesture recognition.
A user must go through training the system, which builds up a standard pattern of data, before recognition takes place so thresholds can be set on an individual level. Gesture segmentation is triggered by the intensity of acceleration. Averages and standard deviations of acceleration data are calculated and used as a basis for comparison during gesture recognition. Ten types of gestures are recognized and used to control the performance of MIDI music. The user whose gesture data trained the system was able to perform gestures with a 100% recognition rate, while another user's gestures were misclassified for particular gestures.
A tempo prediction model based on the previous two tempos is used to allow for smoother performance of the music than could be achieved by the system waiting for the recognition of a human determined tempo.
Discussion:
The tempo recognition based on change in acceleration magnitude is more accurate than image based methods. This reduction of phase delay in the detection of tempo is important when the control issue is specifically related to timing as it is in music control. Some of the gestures used in the study did not seem naturally related to music composition, such as the star, triangle, and heart shaped motions. The paper also did not describe how gestures were mapped to effects on the music, besides the downward motion being mapped to tempo.
Saturday, April 5, 2008
Wednesday, April 2, 2008
Enabling Fast and Effortless Customisation in Accelerometer Based Gesture Interaction (Mantyjarvi 2004)
Summary:
The motivation for this paper is that gesture recognizers should be customizable, easy and quick to train. In order to keep training examples low, original training gestures are augmented with noise-distorted training gestures. A SoapBox (Sensing, Operating, and Activating Peripheral Box) sensing device with a three-axis accelerometer is used for collecting motion data. In the preprocessing phase, gesture data is normalized to equal length and amplitude. The normalized data is converted to one dimensional prototype vectors using the k-means algorithm with a codebook size of eight. An ergodic HMM was used in lieu of a left-to-right model, despite the left-to-right model's grooming to fit sequential time-series data, because both models were reported to give similar results according to Hoffman. Also, the choice of five states for each HMM was chosen to reflect the work done by Hoffman. New training data was generated from original gesture data with random noise added according to either a uniform or Gaussian distribution. The beginning and end of each gesture was signaled with a button press and release, alleviating the problem of segmentation.
A single subject performed eight gestures for controlling a DVD player thirty times each. As expected, increasing the number of training vectors increased recognition accuracy, but four training vectors was determined sufficient, since performing six repetitions of eight gestures was considered to be a nuisance to train. In an experiment to test the effects of the signal to noise ratio of the random distributions on recognition accuracies, the best accuracy was achieved with the Gaussian distributed noise with an SNR of 3, while the best uniformly distributed noise was achieved with an SNR of 5. For either type of noise distribution, recognition accuracy rates of over 96% were obtained with just one original and three distorted training examples. The best performance (nearly 98%) requiring less than three original vectors came from using two original vectors and two or four Gaussian distributed noise distorted vectors.
Discussion:
The results comparing uniform and Gaussian noise are very close, and I do not see a distinct winner. There does not seem to be a clear pattern to which noise distribution performs better, but I do think the addition of noisy copies of training vectors help recognition slightly. Although the paper consistently refers to recognizing 3D gestures, all the example gestures in the study for controlling a DVD player could adequately be defined in two dimensions. In the preprocessing stage, the gesture data is linearly extrapolated or interpolated if the data sequence is too long or short. A different type of extrapolation or interpolation technique besides linear could be used to approximate a better fit to the data. I am not sure that the reliance on Hoffman for the type of HMM and number of states was well founded. The paper states that adding noise increases detectability in decision making under certain conditions, and cites a paper by Kay for justification. It might be interesting to see what those conditions are, and if this paper meets the criteria set by those conditions.
The motivation for this paper is that gesture recognizers should be customizable, easy and quick to train. In order to keep training examples low, original training gestures are augmented with noise-distorted training gestures. A SoapBox (Sensing, Operating, and Activating Peripheral Box) sensing device with a three-axis accelerometer is used for collecting motion data. In the preprocessing phase, gesture data is normalized to equal length and amplitude. The normalized data is converted to one dimensional prototype vectors using the k-means algorithm with a codebook size of eight. An ergodic HMM was used in lieu of a left-to-right model, despite the left-to-right model's grooming to fit sequential time-series data, because both models were reported to give similar results according to Hoffman. Also, the choice of five states for each HMM was chosen to reflect the work done by Hoffman. New training data was generated from original gesture data with random noise added according to either a uniform or Gaussian distribution. The beginning and end of each gesture was signaled with a button press and release, alleviating the problem of segmentation.
A single subject performed eight gestures for controlling a DVD player thirty times each. As expected, increasing the number of training vectors increased recognition accuracy, but four training vectors was determined sufficient, since performing six repetitions of eight gestures was considered to be a nuisance to train. In an experiment to test the effects of the signal to noise ratio of the random distributions on recognition accuracies, the best accuracy was achieved with the Gaussian distributed noise with an SNR of 3, while the best uniformly distributed noise was achieved with an SNR of 5. For either type of noise distribution, recognition accuracy rates of over 96% were obtained with just one original and three distorted training examples. The best performance (nearly 98%) requiring less than three original vectors came from using two original vectors and two or four Gaussian distributed noise distorted vectors.
Discussion:
The results comparing uniform and Gaussian noise are very close, and I do not see a distinct winner. There does not seem to be a clear pattern to which noise distribution performs better, but I do think the addition of noisy copies of training vectors help recognition slightly. Although the paper consistently refers to recognizing 3D gestures, all the example gestures in the study for controlling a DVD player could adequately be defined in two dimensions. In the preprocessing stage, the gesture data is linearly extrapolated or interpolated if the data sequence is too long or short. A different type of extrapolation or interpolation technique besides linear could be used to approximate a better fit to the data. I am not sure that the reliance on Hoffman for the type of HMM and number of states was well founded. The paper states that adding noise increases detectability in decision making under certain conditions, and cites a paper by Kay for justification. It might be interesting to see what those conditions are, and if this paper meets the criteria set by those conditions.
Monday, March 31, 2008
Gestures without Libraries, Toolkits or Training: A $1 Recognizer for User Interface Prototypes (Wobbrock 2007)
Summary:
The $1 recognizer provides a solution to interactive recognition for use in human-computer interfaces that does not require in-depth algorithmic knowledge and intense mathematics to be usable. The $1 recognizer algorithm assumes an input of a sequence of points, outputs a list of most likely template matches, and can be broken down into four steps. First, the input point path is resampled so that the path is represented by N points that are equally spaced apart. Secondly, the points are rotated so that the vector from the centroid of all points in the gesture to the first point is at an angle of zero degrees. Next, the gesture points are scaled to a square and translated so that the centroid lies at the origin. Finally, the candidate points are compared to each template to determine which is the most likely match. The comparison is made between corresponding pairs of points in the candidate and template gestures by examining their distance apart, only after ensuring the candidate gesture is optimally aligned with the template gesture by performing a Golden Section Search to find the best rotational adjustment to the previously calculated indicative angle.
Ten users performed a total of 4800 gestures to compare the $1 recognizer to Rubine and Dynamic Time Warping (DTW) methods. $1 and DTW were significantly more accurate than Rubine, with recognition rates above 99%, compared to Rubine's 92%. The number of training examples and speed of gesture articulation did not produce significant effects across the three recognizers. DTW took considerably longer to run than either $1 or Rubine. Interestingly, $1 performed only 0.23% worse when the Golden Section Search portion of the algorithm was removed.
Discussion:
I think the recognition rates achieved by $1 are commendable, especially considering its ease of implementation. The simplicity of the concepts involved in its implementation are another plus for the $1 system. The algorithm presented in the appendix is the most clear and complete of any other description for implementing a gesture recognition system that I have encountered this semester. I see how the position data from an instrumented glove could be projected into two dimensions and used as input points for this system.
As presented in the paper, the $1 recognizer cannot distinguish gestures that depend on orientation. By recording the amount of rotation during the "indicative angle" rotation and the rotational optimization adjustment, the original orientation could be determined and used in the recognition process. For example, if the total rotation for each template gesture was recorded, and the total rotation for each candidate gesture was calculated, the values could be compared, and templates whose orientations were not within some tolerance level of the candidate could be eliminated.
The paper raises the question of whether the first point in a gesture is the best to use for finding the indicative angle. A possible alternative to the first point could be the centroid of the first n points. Experimentation could be done to find a suitable number for n, which would probably be related to N (possibly n = floor(N/10) or something similar?).
The $1 recognizer provides a solution to interactive recognition for use in human-computer interfaces that does not require in-depth algorithmic knowledge and intense mathematics to be usable. The $1 recognizer algorithm assumes an input of a sequence of points, outputs a list of most likely template matches, and can be broken down into four steps. First, the input point path is resampled so that the path is represented by N points that are equally spaced apart. Secondly, the points are rotated so that the vector from the centroid of all points in the gesture to the first point is at an angle of zero degrees. Next, the gesture points are scaled to a square and translated so that the centroid lies at the origin. Finally, the candidate points are compared to each template to determine which is the most likely match. The comparison is made between corresponding pairs of points in the candidate and template gestures by examining their distance apart, only after ensuring the candidate gesture is optimally aligned with the template gesture by performing a Golden Section Search to find the best rotational adjustment to the previously calculated indicative angle.
Ten users performed a total of 4800 gestures to compare the $1 recognizer to Rubine and Dynamic Time Warping (DTW) methods. $1 and DTW were significantly more accurate than Rubine, with recognition rates above 99%, compared to Rubine's 92%. The number of training examples and speed of gesture articulation did not produce significant effects across the three recognizers. DTW took considerably longer to run than either $1 or Rubine. Interestingly, $1 performed only 0.23% worse when the Golden Section Search portion of the algorithm was removed.
Discussion:
I think the recognition rates achieved by $1 are commendable, especially considering its ease of implementation. The simplicity of the concepts involved in its implementation are another plus for the $1 system. The algorithm presented in the appendix is the most clear and complete of any other description for implementing a gesture recognition system that I have encountered this semester. I see how the position data from an instrumented glove could be projected into two dimensions and used as input points for this system.
As presented in the paper, the $1 recognizer cannot distinguish gestures that depend on orientation. By recording the amount of rotation during the "indicative angle" rotation and the rotational optimization adjustment, the original orientation could be determined and used in the recognition process. For example, if the total rotation for each template gesture was recorded, and the total rotation for each candidate gesture was calculated, the values could be compared, and templates whose orientations were not within some tolerance level of the candidate could be eliminated.
The paper raises the question of whether the first point in a gesture is the best to use for finding the indicative angle. A possible alternative to the first point could be the centroid of the first n points. Experimentation could be done to find a suitable number for n, which would probably be related to N (possibly n = floor(N/10) or something similar?).
Sunday, February 17, 2008
An Architecture for Gesture-Based Control of Mobile Robots (Iba 1999)
Summary:
The paper looks at the use of hand gestures as a way for inexperienced users to direct robots. A difficulty arises in communicating the intention of a task to a robot instead of providing a set of actions that can be mimicked. The system described in the paper interprets gestures as either local or global instructions to a robot. Raw data captured from a CyberGlove is compacted into a 20-element vector describing 10 features and their first derivatives. The 20 dimensional data is reduced to one of 32 codewords. A 3-state HMM leverages temporal data from the gestures, turning sequences of codewords into one of six defined gestures, or an unspecified gesture. The HMM allows the classification of non-gestures (those whose probability is less than 0.5) due to the addition of an initial wait state in the HMM. The paper reports 96% accuracy in recognizing gestures using the HMM with the wait state.
Discussion:
The key insight in this paper's technique is the introduction of a "wait state" in the HMM that allows a gesture to be classified as not recognized. Adding more gestures to be recognized would reduce the effectiveness of a wait state, so the technique may not be well suited for distinguishing individual gestures from a large collection.
The distinction between gesture recognition and gesture spotting made at the beginning of the paper is important. The exclusion of non-gestures from an input sequence is a crucial facet of gesture spotting that distinguishes it from gesture recognition. This paper helped me form a better context for the possible applications of HMMs to gesture recognition. It was a good choice so soon after the Introduction to HMM paper by Rabiner and Juang.
The paper mentions that the length of an observation sequence should be limited to maintain a high probability that the observation sequence is produced by the model. However, no guidelines are given as how to select n, the number of most recent observations in a sequence.
It was not clear whether the waving motion to dodge obstacles temporarily placed the robot on a detour in the direction of the wave until the obstacle was passed, or permanently altered the destination of the robot. I suspect the latter was implemented, since it would be easier to divert the robot along a straight trajectory in th direction of the wave. Performing some kind of obstacle avoidance for a temporary detour could be a possible improvement for the system.
The paper looks at the use of hand gestures as a way for inexperienced users to direct robots. A difficulty arises in communicating the intention of a task to a robot instead of providing a set of actions that can be mimicked. The system described in the paper interprets gestures as either local or global instructions to a robot. Raw data captured from a CyberGlove is compacted into a 20-element vector describing 10 features and their first derivatives. The 20 dimensional data is reduced to one of 32 codewords. A 3-state HMM leverages temporal data from the gestures, turning sequences of codewords into one of six defined gestures, or an unspecified gesture. The HMM allows the classification of non-gestures (those whose probability is less than 0.5) due to the addition of an initial wait state in the HMM. The paper reports 96% accuracy in recognizing gestures using the HMM with the wait state.
Discussion:
The key insight in this paper's technique is the introduction of a "wait state" in the HMM that allows a gesture to be classified as not recognized. Adding more gestures to be recognized would reduce the effectiveness of a wait state, so the technique may not be well suited for distinguishing individual gestures from a large collection.
The distinction between gesture recognition and gesture spotting made at the beginning of the paper is important. The exclusion of non-gestures from an input sequence is a crucial facet of gesture spotting that distinguishes it from gesture recognition. This paper helped me form a better context for the possible applications of HMMs to gesture recognition. It was a good choice so soon after the Introduction to HMM paper by Rabiner and Juang.
The paper mentions that the length of an observation sequence should be limited to maintain a high probability that the observation sequence is produced by the model. However, no guidelines are given as how to select n, the number of most recent observations in a sequence.
It was not clear whether the waving motion to dodge obstacles temporarily placed the robot on a detour in the direction of the wave until the obstacle was passed, or permanently altered the destination of the robot. I suspect the latter was implemented, since it would be easier to divert the robot along a straight trajectory in th direction of the wave. Performing some kind of obstacle avoidance for a temporary detour could be a possible improvement for the system.
Sunday, February 3, 2008
The HoloSketch VR Sketching System (Deering 1996)
Summary:
The paper discusses a general 3D drawing program called HoloSketch which allows interaction using a specialized 3D input device. The display for the HoloSketch system is a 20" CRT monitor with a resolution of 960x680, relatively high compared to head mounted displays of that time. A pair of head-tracked shutter goggles allows the user to view a sequential stereo images as one 3D image, similar to a hologram. Although the HoloSketch system was optimized for the stereo CRT display, it could be used with other types of displays in a variety of applications.
The primary input device is a 3D wand that has six degrees of freedom. The menu system for HoloSketch takes advantage of the 3D positioning of the wand and is described as "fade-up", since the geometry fades away and the menu options fade into place just in front of where the geometry was located. Commands such as movement, rotation, and finer motion control require the wand input be combined with key presses to be interpreted. Elemental animation objects can be applied to primitives to achieve animation of position, orientation, scale, and color.
The paper mentions the possibility of interoperability with geometry from other 3D programs, but does not mention what specific packages or file formats are supported. The paper demonstrates that a general purpose drawing program can be extended into 3D, but suggests that the input devices required for 3D manipulation may not be accurate enough for mainstream users.
Discussion:
The extent to which accuracy was pursued in the project is commendable. I was impressed by the fact that the system corrects for changes in intraocular distance due to the rotation of the viewer's eyes in their sockets.
The design of the interface could be improved in two ways:
1. While an RGB cube is a uniquely 3D choice for color selection, I think the colors not on the exterior of the cube would be difficult to select for a novice user that doesn't have the intuition of the RGB colorspace. In particular, to get shades of gray colors, the user would have to move along the diagonal of the cube between the vertices corresponding to black and white. Looking at the exterior of the RGB color cube would not reveal any gray colors.
2. When selecting an object, the object flashes between it's intrinsic color and white. This visual cue does not work well for colors that are very near white and is imperceptible if the object is pure white. Instead, I think an improvement would flashing between the intrinsic color and its compliment. That way, a user will be able to notice the flashing regardless of its intrinsic color.
The ability to draw two-handed was added into the program because of the test user's request. The mouse motion was mapped to control the radius of the toothpaste primitive. Instead of requiring the mouse motion to control thickness, a possible extension would be to add a second wand-like input so that the interaction could take place more directly in 3D space. Using the mouse to vary the radius requires seeing the output to get feedback on the changes, whereas a second position-tracking device would provide one-to-one control.
The paper discusses a general 3D drawing program called HoloSketch which allows interaction using a specialized 3D input device. The display for the HoloSketch system is a 20" CRT monitor with a resolution of 960x680, relatively high compared to head mounted displays of that time. A pair of head-tracked shutter goggles allows the user to view a sequential stereo images as one 3D image, similar to a hologram. Although the HoloSketch system was optimized for the stereo CRT display, it could be used with other types of displays in a variety of applications.
The primary input device is a 3D wand that has six degrees of freedom. The menu system for HoloSketch takes advantage of the 3D positioning of the wand and is described as "fade-up", since the geometry fades away and the menu options fade into place just in front of where the geometry was located. Commands such as movement, rotation, and finer motion control require the wand input be combined with key presses to be interpreted. Elemental animation objects can be applied to primitives to achieve animation of position, orientation, scale, and color.
The paper mentions the possibility of interoperability with geometry from other 3D programs, but does not mention what specific packages or file formats are supported. The paper demonstrates that a general purpose drawing program can be extended into 3D, but suggests that the input devices required for 3D manipulation may not be accurate enough for mainstream users.
Discussion:
The extent to which accuracy was pursued in the project is commendable. I was impressed by the fact that the system corrects for changes in intraocular distance due to the rotation of the viewer's eyes in their sockets.
The design of the interface could be improved in two ways:
1. While an RGB cube is a uniquely 3D choice for color selection, I think the colors not on the exterior of the cube would be difficult to select for a novice user that doesn't have the intuition of the RGB colorspace. In particular, to get shades of gray colors, the user would have to move along the diagonal of the cube between the vertices corresponding to black and white. Looking at the exterior of the RGB color cube would not reveal any gray colors.
2. When selecting an object, the object flashes between it's intrinsic color and white. This visual cue does not work well for colors that are very near white and is imperceptible if the object is pure white. Instead, I think an improvement would flashing between the intrinsic color and its compliment. That way, a user will be able to notice the flashing regardless of its intrinsic color.
The ability to draw two-handed was added into the program because of the test user's request. The mouse motion was mapped to control the radius of the toothpaste primitive. Instead of requiring the mouse motion to control thickness, a possible extension would be to add a second wand-like input so that the interaction could take place more directly in 3D space. Using the mouse to vary the radius requires seeing the output to get feedback on the changes, whereas a second position-tracking device would provide one-to-one control.
An Introduction to Hidden Markov Models (Rabiner 1986)
Summary:
Markov chains try to solve the problem of recognizing sequences of observations by comparison to a signal model that is found to characterize some sequence of symbols. A first attempt suggests fitting linear system models over several short time segments to approximate the signal itself. This model is rejected in favor of the Hidden Markov Model.
A Hidden Markov Model consists of a number of states, each of which is associated with a probability distribution to decide what the next state to transition to should be. As each state is entered, an observation is recorded which is determined by a fixed, state-specific "observation probability distribution". In addition, there is an initial probability distribution which describes the likelihood of being in each state when observations begin.
Three types of problems associated with HMMs are outlined and their solutions are given. The first problem deals with finding the probability of an observation sequence given an observation sequence and a model. The second problem is concerned with finding the optimal sequence of states associated with a given observation sequence and is usually accomplished by considering each state individually. The third problem concentrates on how to optimize model parameters to best describe how the observed sequence occurs. The three problems can be solved using the forward-backward procedure, the Viterbi algorithm, and the Baum-Welch method, respectively.
The paper lists recognition results for three different HMM observation models (all of which are above 97%) when applied to single word recognition on vocabularies of 10 digits.
Discussion:
Hidden Markov Models are used in speech recognition, but the paper did not mention how a HMM could be applied to gesture recognition. I thought the coin tossing example was more confusing than simply defining the model. My introduction to Markov chains was in the context of weather prediction. Given a context for examples with inherent practical motivation seemed to make more sense than a non-motivated context, like the outcome of tossing coins.
Markov chains try to solve the problem of recognizing sequences of observations by comparison to a signal model that is found to characterize some sequence of symbols. A first attempt suggests fitting linear system models over several short time segments to approximate the signal itself. This model is rejected in favor of the Hidden Markov Model.
A Hidden Markov Model consists of a number of states, each of which is associated with a probability distribution to decide what the next state to transition to should be. As each state is entered, an observation is recorded which is determined by a fixed, state-specific "observation probability distribution". In addition, there is an initial probability distribution which describes the likelihood of being in each state when observations begin.
Three types of problems associated with HMMs are outlined and their solutions are given. The first problem deals with finding the probability of an observation sequence given an observation sequence and a model. The second problem is concerned with finding the optimal sequence of states associated with a given observation sequence and is usually accomplished by considering each state individually. The third problem concentrates on how to optimize model parameters to best describe how the observed sequence occurs. The three problems can be solved using the forward-backward procedure, the Viterbi algorithm, and the Baum-Welch method, respectively.
The paper lists recognition results for three different HMM observation models (all of which are above 97%) when applied to single word recognition on vocabularies of 10 digits.
Discussion:
Hidden Markov Models are used in speech recognition, but the paper did not mention how a HMM could be applied to gesture recognition. I thought the coin tossing example was more confusing than simply defining the model. My introduction to Markov chains was in the context of weather prediction. Given a context for examples with inherent practical motivation seemed to make more sense than a non-motivated context, like the outcome of tossing coins.
American Sign Language Fingerspelling Recognition System (Allen 2003)
Summary:
The paper proposes a system to identify the letters of the alphabet which are represented in American sign language by static gestures. The letters J and Z were not able to be recognized due to their signs involving moving gestures. The recognition system collected data from an 18 sensor CyberGlove using Labview, loaded the data into a MatLab program to train a perceptron network, and then used a second MatLab program to match glove input with the most closely corresponding letter. Only one user was selected to train the neural network, resulting in an "up to 90%" accuracy rate. MatLab's default configuration to not run in real-time was cited as a main obstacle in development.
Discussion:
The paper admittedly "took a more narrow initial focus", so to the fact that only static gestures were recognized is understandable. Still, translating static poses of one user into spoken letters is very far from the long term goal of translating moving gestures into spoken English. The single result of 90% reported was not very illuminating.
The paper proposes a system to identify the letters of the alphabet which are represented in American sign language by static gestures. The letters J and Z were not able to be recognized due to their signs involving moving gestures. The recognition system collected data from an 18 sensor CyberGlove using Labview, loaded the data into a MatLab program to train a perceptron network, and then used a second MatLab program to match glove input with the most closely corresponding letter. Only one user was selected to train the neural network, resulting in an "up to 90%" accuracy rate. MatLab's default configuration to not run in real-time was cited as a main obstacle in development.
Discussion:
The paper admittedly "took a more narrow initial focus", so to the fact that only static gestures were recognized is understandable. Still, translating static poses of one user into spoken letters is very far from the long term goal of translating moving gestures into spoken English. The single result of 90% reported was not very illuminating.
Subscribe to:
Posts (Atom)