View on GitHub

thisemotiondoesnotexist

An interactive visualization of the relationship between facial expressions and their emotional perception

What is this app about?

The app showcases that high-dimensional data can be used to understand complex phenomena of the real world. The visualization is powered by a statistical model which captures the relationship of dynamic facial expressions and perceived emotion. Emotions and their perception remain difficult to research for numerous reasons, some of which are associated with the methodological approaches traditionally employed. This is a call to use modern means of data collection and analysis for empirical research.

What’s special about emotions?

Emotions are part of everyday life, yet they prove to be difficult to conceptualize, and hence to measure and study, because of their transient and incorporeal nature. The debate about the appropriate theoretical framework for the study of emotions is as old as the research on them itself. Popular theories used in emotion research are, for example, Ekman’s basic emotion theory and the theory of constructed emotion by Russell and Feldman Barrett. These theories exemplify the divide between categorical and dimensional and between biological deterministic and constructivistic views on emotions. Even though based on fundamentally different views, both theories provide frameworks that have advanced our understanding of the phenomenon tremendously. However, both feature strong abstractions that limit our ability to describe expressed and perceived emotion and which bias research towards their theoretical underpinning.

What’s your approach?

Where simplification was once a necessity, we now have the tools to capture emotion expression and perception in detail with computer-aided methods. For example, emotion perceptions can be described by dimensional ratings or freehand labeling, and face tracking software can quantify a facial expression as a set of landmarks over time. Using such techniques, however, produces high-dimensional data for which traditional analysis methods of psychology are not suitable. Such methods are designed for inferential analysis of the relationship between a few selected variables; not of hundreds or thousands. We present an approach which allows the usage of high-dimensional data for inferential analysis. Additionally, this method is largely independent of any specific emotion framework and can be used to visualize the investigated relationship in a graphic way, as illustrated by the app.

Is there deep learning involved?

No. Deep learning is a novel technology that has produced astonishing results. It can generate photorealistic faces, completely artificial accommodation ads and much more. By looking at these deep learning examples, it becomes quite clear that the essential structure of the subject was captured in the latent space of the models. However, assessing how that latent structure is laid out, i.e. how individual features relate to one another, is difficult because of the opacity of the model caused by the distribution of the computation across neurons. As a result, deep learning models are great for prediction and data generation, but lack interpretability.

So, how did you do this?

We combined emotion ratings and face tracking data obtained from facial expression videos with a technique called Partial Least Squares (PLS). PLS discovers interpretable latent variables which can also be tested for statistical significance. More details and results in a publication currently in preparation.

What did you use to build it?

What is it good for?

With this technique, perception, something that appears to be entirely subjective and inaccessible, can be compared objectively between groups or individuals. This is not limited to the emotional perception of facial expressions, but can be applied to any other perceptual process. The technique also allows mapping between different perceptions. For example, it could be used to capture how two individuals perceive emotions in facial expressions. Then, using this technique, it is possible to calculate how an expression that evokes a certain emotional response in one individual has to be modified to evoke the same response in the other individual.

Some facial expressions look unrealistic or distorted. What’s going on?

Several things are responsible here. First, tracking errors from the face tracking data will carry over into the model predictions. Open mouths in particular are notoriously hard to track. Second, prediction might fail when moving too far away from the original data, which might be the case for some combinations of the sliders. When sliders turn red they are set to negative values, which are also not present in the original data. This in particular might produce very unrealistic or comical faces, however, it might be useful to understand the effect of a particular emotion dimension, as the negative range will have the exact opposite effect of the positive range on the facial expression. Third, facial expressions might contain nonlinear information that cannot be sufficiently approximated by the linear PLS model.