AFFECTIVE COMPUTING

Mental State Inference

 

From a set of facial and head gesture observations at a given time instance, the mental state inference system is able to calculate the probability that the user is experiencing any of the supported mental states.

 

The system accepts multiple observed gestures as the input at any time instance, as a user can be smiling and nodding at the same time.

Each supported mental state is modelled as a Dynamic Bayesian Network (DBN) classifier, in which the hidden mental state is an event with two possible outcomes: it is true whenever the user is experiencing that mental state, and false otherwise.

The separate DBNs for each mental state allows the user to be modelled as experiencing multiple mental states at the same time, because some mental states are not mutually exclusive:

eg. One can be described as ‘concentrating’ and ‘thinking’ at the same time.

 

Structure of a mental state DBN:

  - a mental state event

  - nodes of observable gestures that relate to this mental state

  - arcs of conditional probability: P(gesture | mental state)

  - transition function which encodes temporal dependency between the mental state at time (t-1) and (t)

 

The probability of a user experiencing a mental state relates to the most recent gestures observed, and the previous inference of the user’s mental state.

More precisely, the probability of a mental state event at any time (t) is given by the joint probability of:

P(gesture | mental state) and P(mental state | previous mental state).

 

The gestures that relate to a mental state and the conditional probabilities are parameters that are generated by process of trial and error, testing the system against a library of labelled video clips.

Initially the system makes no assumptions as to which gestures positively or negatively relate to particular mental states, and all supported gestures are included as nodes in each mental state DBN.

The DBNs are then refined to only include gestures that relate or discriminate against the mental state.

 

Example of mental state DBN for 'disagreement'

 

 

 

 

 

 

 

 

 

 

 

 

 

     Gestures that relate to mental state of ‘disagreement’, and their conditional probabilities:

       - Probability of head shake given mental state of disagreement, P(head shake | disagreement): high

       - Probability of eyebrow raise given mental state of disagreement, P(eyebrow raise | disagreement): high

       - Probability of nod given mental state of disagreement, P(nod | disagreement): low

 

     Hence, if given this DBN, the previous mental state event of disagreement (true) at time (t-1), and the

     observed gestures set (head shake, eyebrow raise); the result would be a higher probability that the user is

     in the mental state of disagreement in time (t).

     Conversely, if given this DBN, the previous mental state event of not disagreement (false) at time (t-1),

     and the observed gestures set (nod, smile); the result would be a lower probability that the user is in the

     mental state of disagreement in time (t).

 

 

The overall mental state inference decision for a given video clip is obtained by integrating the probabilities of every mental state, over the entire duration of the video, and selecting the mental state with the greatest probability.