Facial Gestures Recognition

A gesture is a specific sequence of face and head actions. Gesture recognition is basically the recognition of patterns in face and head actions, using hidden Markov models.

Hidden Markov models consist of the following:

- multiple hidden states

- multiple observations

- initial probability of each state

- transition probability from one state to other states

- emission probability: probability of an observation, given a specific state

For example, a hidden Markov model of the weather in London would be:

- states: rainy, cloudy, sunny

- observations: umbrella, jacket, unhappy, sunglasses

- initial probabilities: (rainy, 0.4), (cloudy, 0.4), (sunny, 0.2)

- transition probabilities: rainy: (cloudy, 0.9), (sunny, 0.1)

cloudy: (rainy, 0.7), (sunny, 0.3)

sunny: (cloudy, 0.5), (cloudy, 0.5)

- emission probability: rainy: (umbrella, 0.4), (jacket, 0.3), (unhappy, 0.3), (sunglasses, 0.0)

cloudy: (umbrella, 0.1), (jacket, 0.4), (unhappy, 0.4), (sunglasses, 0.1)

sunny: (umbrella, 0.1), (jacket, 0.1), (unhappy, 0.0), (sunglasses, 0.8)

Given a hidden Markov model (M) and a sequence of observations (O), the probability that (O) is generated by (M) can be evaluated. The evaluation is done by the forward/ backward algorithm.

Using the London model, and the observation sequence of (umbrella, unhappy, jacket), it can be evaluated that there is a high probability of being "London".

Conversely, if given the observation sequence of (sunglasses, jacket, unhappy), there is a lower probability of a match to the London model.

To recognise gestures, each gesture is represented by a hidden Markov Model, with its states being the sequence of face and head actions.

Examples of gestures with corresponding states:

Facial and Head Action Detection

Mental State Inference