Research indicates that we are best at recognizing which facial expression of emotion?

Facial expression and speech

As with gait and posture, a patient's facial expression and how they interact with you can provide clues to their physical and psychological wellbeing (Box 3.3). Reluctance to engage in the consultation may indicate underlying depression, anxiety, fear, anger or grief, and it is important to recognise these emotions to ensure that both the physical and the emotional needs of the patient are addressed effectively. Some people conceal anxieties and depression with inappropriate cheerfulness. Illness itself may alter demeanour: frontal lobe disease or bipolar disorders may lead to animated disinhibition, whereas poverty of expression may occur in depression or Parkinson's disease. Physical signs in the face that are associated with specific diagnoses are covered later (seeBox 3.9).

Be vigilant for abnormalities in the character of speech, such as slurring (due to alcohol, for example, or dysarthria caused by motor neurone disease;p. 125), hoarseness (which can represent recurrent laryngeal nerve damage;p. 186) or abnormality of speech cadence (which could be caused by pressure of speech in hyperthyroidism or slowing of speech in myxoedema;p. 197).

Facial expressions are one of the more important aspects of human communication. The face is responsible for communicating not only thoughts or ideas, but also emotions. What makes the communication of emotions interesting is that it appears as if some of these expressions of emotion (e.g., anger, disgust, fear, happy, sad, surprise, and to a lesser extent contempt, embarrassment, interest, pain, and shame) may be biologically hardwired, and are expressed the same way by all peoples of all cultures. This contrasts with other views that all facial expressions are a product of social learning and culture. Darwin was the first to propose that some of these facial expressions of emotion have their origins in the evolution of the human species. These expressions helped the organism survive because it appeared important to social animals like humans or chimpanzees to express these imminent behaviors implied by the emotions (running away in fear, attack in anger) so they could avoid conflict, danger, or allow approach, and so forth. However, these emotion expressions are not immune to modification by social learning; different cultures learn different display rules to manage their expressions of emotion. Although the current evidence supports Darwin's basic premise it is not without controversy. Future technological advances will allow facial expression research to expand to address many of the important issues that remain.

What Does the Face Show?

Facial expressions are the reflection of the experiences and messages of the endogroup (the group of belonging) and the exogroup (the comparison group). The terms we use to describe emotions are not unique or unequivocal. Once again the intercultural question comes into play. The message generated by an emotion has to be captured instantaneously. The start, duration, and end of the movement are the stages that have to be taken into account. Involuntary and momentary expressions activate a particular muscular contraction, giving indications on the type of associated emotion. Positive emotions share a particular expression (e.g., the smile), which can be observed in terms of time, intensity, and context. Negative emotions (e.g., sadness) also exhibit a particular morphology of expression (e.g., corners of the mouth, eyebrows) characteristic of unhappy states. Some emotions are given no expression on the human face. This means we are unable to identify a facial expression, vocal expression, or bodily behavior associated with them – no discernible pattern or sign of distinction exists. This is a legacy of ontogeny and phylogeny. There is a template at the CNS level which is unique for the emotions. The fact that an emotion is not linguistically coded does not mean that the emotion does not exist. There can be emotion without facial expression. Research using EMG has detected alterations in the pattern of facial activity. This would seem to contradict Tomkins, who asserts that facial expression is always part of emotion, even when it is not visible. Voice and posture (e.g., head and hand movements) also contribute to the configuration of the emotions. Isolated head movements do not by themselves communicate emotions. It may be possible that emotion can be characterized by vocal expression but not facial expression. Ekman agrees with Tomkins in contending that each emotion has its facial expression and its vocal expression too. The existence of emotions with no expressive or auditory signal has occupied many contemporary researchers. Ekman maintains that we must bear in mind certain discrepancies in the mode of facial expression of emotion (e.g., anger as a result of irritability), emotional state (anger as a result of hostility), and affective disorders (sadness as a result of depression).

Facial Expressions

Facial expressions are configurations of different micromotor (small muscle) movements in the face that are used to infer a person’s discrete emotional state (e.g., happiness, anger). Ekman and Friesen’s facial action coding system (FACS) was the first widely used and empirically validated approach to classifying a person’s emotional state from their facial expressions (Ekman, 1992; Ekman & Friesen, 1978).

An example of facial expressions being used by human coders to classify students’ nonbasic emotions comes from Craig, D’Mello, Witherspoon, and Graesser (2008) and D’Mello and Graesser (2010), who had two trained coders classify participants’ emotional states, while they viewed videos of participants interacting with AutoTutor, a CBLE designed to foster students’ comprehension of physics and computer literacy. They developed their coding scheme by reducing the set of action units from FACS (used to code facial expressions) to those that they judged relevant to classifying learner-centered emotions, such as boredom, confusion, and frustration. The interrater reliability of trained coders using this coding framework is good for time points selected by the coders for emotions that occurred with sufficient frequency (overall κ = 0.49; boredom, κ = 0.44; confusion, κ = 0.59; delight, κ = 0.58; frustration, κ = 0.37; neutral, κ = 0.31; D’Mello & Graesser, 2010). Judges interrater reliability scores were, however, much lower for evaluations of emotions at preselected, fixed points (for emotions that occurred with sufficient frequency; overall κ = 0.31; boredom, κ = 0.25; confusion, κ = 0.36; flow, κ = 0.30; frustration, κ = 0.27; D’Mello & Graesser, 2010). Although less than ideal, these kappa values are nonetheless common (Baker, D’Mello, Rodrigo, & Graesser, 2010; Porayska-Pomsta et al., 2013) and point to the difficulty of classifying participants’ learning-centered emotions, especially at preselected intervals where little affective information is available. For this reason, most emotional coding systems that use facial expressions classify either only facial features (e.g., eyebrow movement) or basic emotions (Calvo & D’Mello, 2010; Zeng et al., 2009), or combinations of facial features and vocalizations.

One of the relatively new and promising trends in using facial expressions to classify learners’ emotions is the development and use of software programs that automate the process of coding using advanced machine learning technologies (Grafsgaard, Wiggins, Boyer, Wiebe, & Lester, 2014; Harley, Bouchet, & Azevedo, 2013; Harley, Bouchet, Hussain, Azevedo, & Calvo, 2015). For example, FaceReader (5.0) is a commercially available facial recognition program that uses an active appearance model to model participant faces and identifies their facial expressions. The program further utilizes an artificial neural network, with seven outputs to classify learners’ emotional states according to six basic emotions, in addition to “neutral.” Harley et al. (2012, 2013, 2015) have conducted research with FaceReader to: (1) examine students’ emotions at different points in time over the learning session with MetaTutor (Azevedo et al., 2012; Azevedo et al., 2013); (2) investigate the occurrence of co-occurring or “mixed” emotional states; and (3) examine the degree of correspondence between facial expressions, skin conductance (i.e., electrodermal activity), and self-reports of emotional states, while learning with MetaTutor by aligning and comparing these methods.

Although automated facial recognition programs are able to analyze facial expressions much faster than human coders, they are not yet as accurate (Calvo & D’Mello, 2010; Terzis, Moridis, & Economides, 2010; Zeng et al., 2009). The accuracy of automatic facial expression programs varies, both by individual emotion and software program (similar to variance between studies of human coders). An important issue pertaining to automatic facial expression recognition software, especially commercial software, is its continuous evolution, including larger training databases and the inclusion of more naturalistic (non-posed or experimentally induced) emotion data (Zeng et al., 2009).

Less sophisticated, partial facial expression recognition programs are also used in research with CBLEs, such as the Blue Eyes camera system, that is able to detect specific facial features and motions (Arroyo et al., 2009; Burleson, 2011; Kapoor, Burleson, & Picard, 2007). These programs are used differently than fully developed automated or human facial coding programs because they do not provide discrete emotional labels. Instead, the facial features they provide are combined with other physiological (e.g., electrodermal activity, EDA) and behavioral data (e.g., posture) to create sets of predictive features (e.g., spike of arousal, leaning forward, eyebrows raised) that are correlated with other measures of emotions, such as self-report instruments, to validate their connection to different emotions or emotional dimensions. Studies that investigate different emotion detection methods and the conclusions we can draw from them regarding the value of individual methods are discussed later.

In summary, facial expressions have numerous advantages as a method for measuring emotional states. For one, they are the most traditional and remain one of the best measures of emotional states in terms of their widespread use and reliability (tested with multiple raters and with other methods of emotions) that is unmatched by most other methods that are more newly developed (see Calvo & D’Mello, 2010; Zeng et al., 2009). Furthermore, facial expressions can be analyzed in real-time using software programs, such as FaceReader and the Computer Expression Recognition Toolbox (CERT; Grafsgaard et al., 2014) or after the experimental session concludes, using human coders (Craig et al., 2008). Options to detect emotions in real-time, such as automatic facial recognition software, also make facial expressions a viable channel to provide information to the CBLE about the learners’ emotional state, which can in turn be used to provide emotionally supportive prompts (these environments are henceforth referred to as emotionally supportive CBLEs). Finally, and like most of the methods discussed in this chapter, facial expression recognition measures are online measures of emotion that capture the expression of an emotion as it occurs and therefore mitigates the shortcomings of offline self-report measures (discussed in detail below).

The disadvantages of using facial expressions to measure emotions are that most facial expression coding schemes rely on the FACS system traditionally used to classify only the six basic emotions, and are very labor-intensive if done by trained human coders rather than software (Calvo & D’Mello, 2010). Programs of research that use facial expressions to examine nonbasic emotions (e.g., D’Mello & Graesser, 2013; Grafsgaard et al., 2014; Rodrigo & Baker, 2011) require extensive cross-method validations to connect configurations of facial expressions with new emotional labels (e.g., engagement, frustration, boredom). Ultimately, facial expression research is currently best suited to examining basic emotions and most efficiently done when using automatic facial recognition programs, which are continuing to improve and approach levels of classification accuracy similar to human coders (Calvo & D’Mello, 2010; Zeng et al., 2009).

5.2.1.1.3.3 MMI Facial Expression Database

The MMI Facial Expression Database aspires to provide a large volume of visual data on facial expressions. A major issue hindering new developments in the field of automatic human behavior analysis in general, and recognition in particular, is the lack of databases with displays of behavior and affect. To address this problem, the MMI Facial Expression database was conceived in 2002 as a resource for building and evaluating facial expression recognition algorithms. The database addresses a number of key omissions in other databases of facial expressions. In particular, it contains recordings of the full temporal pattern of facial expressions, from neutral, through a series of onset, apex, and offset phases, and back again to a neutral face.

Secondly, whereas other databases focused on expressions of the six basic emotions, the MMI Facial Expression Database contains both these prototypical expressions and expressions with a single FACS AU activated, for all existing AUs and many other action descriptors. Recently, recordings of naturalistic expressions have also been added.

This database is a collection of 2,900 videos and high-resolution still images of 75 subjects, and is fully annotated.18

Facial Expressions

Facial expressions are an important visual cue in normal communication. There are some pointers to suggest that the ability to interpret expressions may be relatively selectively impaired. Bornstein (1963) reported two cases who had made a partial recovery from prosopagnosia. These patients however, complained of a persisting difficulty in recognising the expressions on faces. One patient complained, “It is clear that I have lost the ability to read a person's facial expression. This is apparent in situations such as seeing a film in which everyone can easily understand what is taking place from the facial expression.”

De Kosky, Heilman, Bowers, & Valenstein (1980) conducted a group study in which the ability to match faces with a neutral expression was compared with matching of faces with emotional expressions. Although three subjects were impaired on both tasks, three further cases were most impaired on matching faces for their emotional expressions. Etkoff (1984) reported a double dissociation between patients’ ability to discriminate facial identity and facial emotion. The patients were required to classify pictures into those which showed the same person or the same emotion (see Fig. 3.3). Two patients were selectively impaired on facial identity, and two on facial emotions. Etkoff commented that patients with difficulty in discriminating emotions adopted a feature-searching strategy when attempting to perform the task (e.g., whether the teeth were showing or whether a dimple was present).

The ability to interpret lip movements in a lip-reading task may dissociate from the interpretation of facial expressions. Campbell, Landis, & Regard (1986) studied one patient (D.) who was unable to identify the emotional expressions conveyed by faces. Remarkably, Campbell et al. were able to establish that she retained the ability to lip-read. For normal people, the information provided by lip movements provides information which can disambiguate the sounds of language. McGurk & MacDonald (1976) have shown that if normal subjects are shown the lip movements appropriate to the sound “ba” and are presented with the sound “ga,” they are strongly influenced by the visual information and have the illusion of hearing “da.” The prosopagnosic patient studied by Campbell et al. showed the normal illusion on this task, indicating that her lip reading was intact. She could also perform at a normal level in lip reading the sound being made on a silent film.

7.3 Measuring Emotion

Measuring emotion is difficult. Emotions are often fleeting, hidden, and conflicted. Asking a participant about what she is feeling through an interview or survey may not always be effective. Many participants tell us what they think we want to hear or simply have difficulty articulating what they are really feeling. Some are even hesitant or afraid to admit their true feelings to a perfect stranger.

Despite the difficulty in measuring emotions, it is still very important for the UX researcher to understand the emotional state of the participant. The participant’s emotional state while experiencing something is almost always a concern. Most UX researchers use a combination of probing questions, as well as interpretation of their facial expressions, and even body language to infer the participant’s emotional state. This may be acceptable for some products; however, it does not always suffice. Some products or experiences are relatively much more emotional and have a greater bearing on the overall user experience. Simply think about the range of emotions a participant might experience when calculating how much money he will have when he retires, reading about a health condition he has, or just playing an action game with friends.

There are essentially three different ways to measure emotions. Emotions can be inferred based on facial expressions, by skin conductance, or by use of EEG. This section highlights three different companies that used these three different approaches. All of these products and services are currently available commercially.

7.3.1 Affectiva and the Q-Sensor

Based on an interview with Daniel Bender, product manager, Affectiva (www.affectiva.com).

The Affective Computing Research group at MIT’s Media Lab was founded in 1998 by Professor Rosalind Picard Sc.D. in an effort to develop technologies that advance understanding of emotions. The aim of the research group is to restore a proper balance between emotion and cognition in the design of technologies for addressing human needs (http://affect.media.mit.edu/). Picard and coinvestigator Rana el Kaliouby, Ph.D., cofounded Affectiva in April 2009 to commercialize technologies developed at the MIT research group. The first product to come from Affectiva is called the Q Sensor (see Figure 7.12).

Figure 7.12. Affectiva’s Q Sensor, a wearable, wireless biosensor.

The Q Sensor is a device worn on the wrist that measures the electrical conductance of the skin known as electrodermal activity (EDA). EDA increases when you sweat—small increases in moisture are associated with increased sympathetic nervous system activity indicating emotional activation or arousal. Three types of activation can lead to increases in arousal: increases in cognitive load, affective state, and/or physical activity. Emotional states associated with EDA increases include fear, anger, and joy. Arousal increases are also associated with cognitive demands and may be seen when you are engaged in problem-solving activity. Our state of arousal—and hence the conductivity of our skin—is lower when we are in a relaxed state or bored.

Researchers in a number of fields are using the Q Sensor to measure sympathetic nervous system activity objectively. One of the initial use cases for the Q Sensor has been in understanding the emotional state of students on the autism spectrum. Individuals with autism spectrum disorders often present neutral facial expressions, despite feeling threatened, confused, or otherwise emotionally distressed. Researchers working with autistic students are reviewing EDA data captured with the Q Sensor to better understand the triggers for emotional outbursts. Eventually, the technology will make its way into the classroom where it will serve teachers by providing early warning signals that students are becoming stressed without outward displays of duress. This will enable teachers to respond to their students in a timely and appropriate way.

In the area of user experience research, the Q Sensor can be used to help pinpoint moments of excitement, frustration, or increased cognitive load experienced by the participant. The UX researcher establishes a baseline for each participant. Experiences are then compared to their baseline, with particular attention given to the peaks, or places where there was a peak level in arousal.

While it is helpful knowing what may have triggered an increased level of arousal, it does not tell the researcher whether the experience was positive or negative. This is known as valence. Picard recognized the need to measure valence objectively as she brought Affectiva cofounder el Kaliouby to MIT in January 2007. El Kaliouby’s research had been focused on measuring facial expressions using computer-vision and machine-learning techniques. This technology matured and was incorporated into Affectiva’s second product, the Affdex facial expression recognition system. Affdex is a passive web-based platform that can take streaming video as an input and predict the presence of facial expressions in close to real time. Affdex is being used to measure emotional response to media in online panels and in usability labs. Affdex facial-expression recognition provides an indication of the type of experience associated with the state of arousal.

Facial expressions are captured through a standard web camera on the participant’s computer and time synchronized with data from the Q Sensor. This provides a rich data set, as peaks in arousal can be associated with a positive or negative valence. With Affdex, Affectiva is building the largest database of spontaneously generated facial expressions in the world. This will allow Affectiva to develop more advanced classifiers of different emotions, which will be used to predict increases in sales or brand loyalty. This powerful technology will arm the UX researcher with an additional set of tools to better understand emotional engagement across a wide variety of experiences. Case study 10.5 highlights use of the Q Sensor in the context of using an onscreen and tablet-based textbook.

Relationship among Task Performance, Subjective Ratings, and Skin Conductance

In a study of participants playing a 3D video game (Super Mario 64), Lin, Hu, Omata, and Imamiya (2005) looked at the relationships among task performance, subjective ratings of stress, and skin conductance. Tasks involved playing three different parts of the game as quickly and accurately as possible. Participants played each part (task) for 10 minutes, during which period they could potentially complete the goal (succeed) multiple times. There was a strong correlation between participants’ ratings of how stressful each of the tasks was and their normalized skin conductance (change relative to the participant’s baseline) during the performance of each task. In addition, participants who had more successes during the performance of each task tended to have lower skin conductance levels, indicating that failure was associated with higher levels of stress (see Figure 7.13).

Figure 7.13. Data showing subjective ratings of stress (a) and normalized skin conductance (b) for three different tasks in a video game. Both show that Task 3 was the most stressful, followed by Task 2 and then Task 1.

Adapted from Lin et al. (2005).

7.3.2 Blue Bubble Lab and Emovision

Based on an interview with Ben van Dongen, CEO and founder, BlueBubbleLab (www.bluebubblelab.com)

Blue Bubble Lab is a media and technology company based in Palo Alto and Amsterdam that focuses on bringing more relevant messages to consumers based on their emotions and behavior. ThirdSight (www.thirdsight.com), a subsidiary of Blue Bubble Lab, has developed a suite of technology products that bring together computer vision, facial expression analysis, and eye tracking. One product, Emovision, is an application that allows the researcher to understand the participants’ emotional state while pinpointing what they are looking at. It is a powerful combination of technologies because the researcher can now draw a direct connection between visual stimuli and an emotional state at any moment in time. This will be invaluable in testing how different visual stimuli produce a range of emotional responses.

Emovision determines the emotional state based on the participants’ facial expressions. In the 1970s, Paul Ekman and Wallace Friesen (1975) developed taxonomy for characterizing every conceivable facial expression. They called it the Facial Action Coding System, which included 46 specific actions involving the facial muscles. From his research, Ekman identified six basic emotions: happiness, surprise, sadness, afraid, disgust, and anger. Each of these emotions exhibits a distinct set of facial expressions that can be reliably identified automatically through computer vision algorithms. Emovision uses a webcam to identify the facial expressions at any moment in time and then classifies it into one of seven unique emotions: neutral, happy, surprise, sad, scared, disgusted, and puzzled. At the same time, the webcam is used to detect eye movements.

Figure 7.14 shows how the Emovision application works. On the left side of Figure 7.14 the participant’s facial expressions are analyzed. Distinct facial muscles are identified and, depending on their shape and movement, an expression is identified. The right side of the application (Figure 7.14) shows the stimulus that is being viewed and the eye movements. In this case the participant is watching a TV commercial, and fixating (as represented by the red dot) in between the two women. The bottom of the screen shows the emotion (in this case it is happy) and assigns a percentage. The line graph depicts the change in emotion over time. When analyzing these data, the researcher can look at any moment in time and identify the associated emotion(s). Also, the researcher can view the overall mood of the experience by seeing the frequency distribution of all the emotions across the experiment. This might be valuable data when comparing different products.

Figure 7.14. Example of EmoVision application that incorporates webcam-based eye tracking and facial expression analysis in real time.

One of the more fascinating applications of this technology is being able to target messages to consumers based on their mood. Figure 7.15 is an example of how this technology can be used to capture facial expressions in the real world, determine the overall mood (positive or negative), as well as demographics such as age and gender, and then deliver a targeted message on a digital billboard or other platform.

Figure 7.15. Example of how ThirdSight products can be used to deliver tailored messages to consumers based on their mood and other demographics.

7.3.3 Seren and Emotiv

Based on an interview with Sven Krause, key account director, Seren (www.seren.com/)

Seren is a customer experience consultancy based in London. Sven Krause developed a way of measuring a user’s emotional engagement and behavior by combining electroencephalography and eye-tracking data. Seren is applying this technology to a wide variety of contexts, including branding, gaming, service, and website design. Researchers at Seren feel this new technology allows them to gain a more complete picture of the user experience as it measures participants’ unconscious responses to a stimulus.

Seren uses an EEG device developed by Emotiv (www.emotiv.com). EEG measures brain waves, specifically the amount of electrical activity on different parts of the participant’s scalp. Electrical activity is associated with cognitive and emotional states. There is a certain pattern of electrical activity when the participant is in a more excited state relative to a calm state. Also, specific patterns of electrical activity have been associated with other emotional states, such as frustration, boredom, and engagement. EEG technology has been used for many years, for example, helping diagnose patients with epilepsy, sleep disorders, strokes, and other neurological conditions. Only recently has it been applied within the field of marketing and customer experience.

Seren has worked with SMI (www.smivision.com) to integrate the Emotiv headset with their SMI eye tracker. This allows Seren’s researchers to determine what participants are looking at and what triggers their emotional and cognitive state. The integration of both EEG and eye-tracking data is critical, as all data will have a consistent time stamp, allowing the researcher to explore both eye movement and EEG data for a specific event.

Setting up and using their system is fairly straightforward. Participants wear the EEG device on their head, with a series of small conductive pads that contact the scalp and forehead. The EEG device is connected wirelessly to the eye tracker. Baseline measures are taken for a few minutes to allow the participant to get comfortable with the setting. After the researcher feels she has achieved an acceptable baseline, the study begins. Figure 7.16 shows a typical setup. The researcher is monitoring both eye movements and EEG feedback in real time (as shown in Figure 7.17).

Figure 7.16. Typical setup at Seren using EEG technology.

Figure 7.17. An SMI application that allows the researcher to observe EEG feedback and eye movements in real time.

Electroencephalography data are extremely useful in monitoring the emotional engagement of a participant throughout a session. Results can be used to prompt additional questions or to create “emotional heatmaps” that identify areas that led to a change of the emotional state.

Facial Expressions

Facial expressions of specific emotions can be used to identify infant responses to specific experiences. The Maximally Descriptive Facial Movements Code (Izard, 1979) identifies many of the emotions infants express. Expressions of distress have been used to observe the response of infants when their primary caregiver stops responding by holding a neutral, unresponsive “still face” (Weinberg and Tronick, 1994).

Researchers also measure looking patterns made by infants as a proxy for surprise and interest to make inferences about what infants notice about the world when shown possible and impossible events (Baillargeon and Wang, 2002). This violation of expectation method has been used to study what infants perceive and understand about causation, sequence, and characteristics of objects. Researchers using this method present participants with two events, one possible event and one impossible event. If participants look longer at impossible than possible events, researchers assume that infants understand that one event is not logically possible. Researchers have used this method to study collision of objects, object permanence, barriers to moving objects, and number of objects (Baillargeon, 2002).

The Social Meaning of Graded and Blended Signals

The facial expressions of nonhuman primates, particularly the great apes, are best described as graded signals because rarely are they observed in their peak-intensity prototypical form, as shown in Figure 2. Similar issues are known for human facial expressions, and it should be noted that most of the work on the meaning of human facial expressions has been done in a laboratory setting using either highly standardized, posed expressions or facial expressions represented only as peak-intensity displays. Their graded nature makes it extremely difficult to characterize facial expressions during ongoing social interaction in both chimpanzees and humans and challenges our ability to understand their form, preferred context of use, and social meaning or function. Moreover, there are two dimensions on which signals may be graded: They may vary quantitatively in terms of intensity level and/or qualitatively by forming intermediates with other signals. In an effort to understand the relationship between chimpanzee facial expressions and social behavior, observational studies have recorded in what social contexts specific expressions are made. Prototypical facial expressions were associated most often with a single social context; for example, the relaxed open-mouth face (or play face) was observed to occur almost exclusively within the social context of play. However, there was also considerable overlap in the contexts that gave rise to many of the other facial expressions present in the chimpanzee repertoire. Some facial expressions, for example, appear to share features in common with two or more prototypical categories; thus, they seem to be blended between two parent groups. Figure 3 illustrates one example, the stretch-pout whimper, which is an expression blend because it shares prominent features in common with both the bared-teeth display and the pout, and it was as likely to be expressed in the contexts associated with both of the parent expressions. Thus, it was both a morphological/structural blend and a functional blend. In most cases, however, the expression blends were only blended in visual appearance, not in terms of their context of use, because they were used in a manner similar to only one of the parent expressions that they were blended between.

Some variation in facial expression can also occur as a result of the accompanying vocalization. Although not all facial expressions have an accompanying vocalization, it is obvious that all vocalizations will have some changes in facial appearance. Whether these changes provide information in the visual domain, as opposed to being exclusively auditory signals, is not clear. A multimodal information-processing experiment was conducted in which videos of various facial expressions were presented with either their congruent vocalization (i.e., scream faces with scream vocalizations) or an incongruent vocalization (i.e., a scream face paired with a vocalization of another facial expression category, such as a pant-hoot). The chimpanzees were presented with the choice of matching the incongruent videos by selecting a facial expression photograph that depicted either the auditory cue (i.e., the pant-hoot) or the visual cue (i.e., the scream face). Results showed clear preferences for auditory modality for some expressions (pant-hoots and play faces (laughter)) but the visual modality for others (i.e., the scream). Other authors have shown that rhesus monkeys will orient to the visual display of one of two vocalizations, either a coo or a threat call, when they hear that corresponding vocalization, suggesting an important convergence between the auditory and visual components of multimodal facial signals.

Thus, the communication system among some primates, such as humans, appears to be highly varied in terms of signal appearance (graded and blended facial expressions), the ability to integrate multimodal cues, and in the varied use of facial expressions across a range of social contexts. This presents unique cognitive challenges for primates, particularly chimpanzees, that must know/learn how to use these varied signals as well as interpret their meaning in multiple contexts. The rules for this are unclear, but they may be simple in that specific facial expressions garner different meaning depending on their context of use, as has been suggested in some monkey species, although it was not studied how subtle differences in the form and/or temporal patterning of these signals may also have contributed to their differential interpretation. This type of redundancy may minimize the need to develop a completely new signal to express the same message in different contexts, but it raises the awareness that social context plays as important a role in signal meaning as signal type. This is also quite similar to how human facial expressions are interpreted, since both context and the identity of the performer help to interpret the meaning of expression.

Alternatively, the rules may be quite complex because the range of potential meanings that are possible when different forms of a signal are used in different social contexts, and perhaps even by different age/sex class individuals, grows exponentially large. In this latter case, the meaning conveyed by a specific signal may be a simple heuristic, as opposed to having a specific/referential meaning. Consider the bared-teeth face. This expression in the chimpanzee has been shown to be structurally homologous to the smile in humans: These expressions look very similar and probably share a similar evolutionary history. The bared-teeth expression is also used by chimpanzees in a wide range of social contexts and by all individuals in a social group. In contrast, this signal is highly ritualized in macaques in both form and context, being expressed unidirectionally from subordinate to dominant. Among rhesus monkeys, this has led to its label as a formal signal of dominance because it can be used to reliably predict dominance within a social group. It has also received a functional label drawing on its proposed emotional meaning, referred to by some as the ‘fear grin’ or ‘grimace.’ However, the relationship between this signal and the emotion ‘fear’ is unclear, and often this interpretation does not accurately describe the situation because the bared teeth may be used in unthreatening or nonconflict situations in which there is little risk to the sender. Among chimpanzees, the bared-teeth display appears to mean something much more general, and it is used across a wide variety of social contexts, including during play, with no predicted directionality between sender and receiver. It has been proposed that the bared-teeth display functions to indicate a general state of benign intent (i.e., “I mean you no harm”), which enables individuals to reduce proximity and potentially socially interact in a wide variety of situations. Its use in these situations can result in more grooming between individuals and improved quality of their social relationship. Further studies are needed before researchers will more fully understand how these facial expressions function during chimpanzee social interactions. The final section describes a new comparative research tool, ChimpFACS, which was developed from the human Facial Action Coding System (FACS) to aid in identifying homologous facial movements between chimpanzees and humans and to aid in the classification of graded signals.

23.5.1 Facial expression of emotions

While facial expression might be a very useful tool to measure the reaction of respondents to advertisements and other visual stimuli, as well as fragrances and aromas, its uses regarding food products are limited as a result of chewing, for example, which probably has a bigger effect on the facial displacement of respondents than the feelings experienced during consumption. In studying beverages this method is further hampered by the fact that, at the moment of most interest, i.e. the moment at which the flavour is first experienced, the mouth is at least partly obscured by a container from which the respondent is drinking.

As this method measures basic emotions (angry, disgusted, happy, sad, scared and surprised) and in the case of the Noldus Information Technology solution also the neutral facial expression (He, Boesveldt, de Graaf, & de Wijk, 2014) it is suitable to evaluate products with large differences in acceptability where we can expect products to be differentiated in terms of emotions such as disgust, surprise and happiness. As such it could be a suitable option for screening purposes.

Danner, Haindl, Joech, and Duerrschmid (2014) found significant differences in facial expressions between salted sauerkraut, grapefruit and mixed vegetable juices compared to orange juice and banana nectar. The juices were selected specifically for their differences in sensory characteristics and to span a hedonic range while being similar in familiarity to respondents. Respondents evaluated 2 cL of each sample in a semi-monadic design. After consuming the sample respondents had to wait 20 s to consider how much they liked the sample before answering on a 9-point scale. During this waiting time implicit emotion measurements were taken using Noldus FaceReader. This measurement started immediately after swallowing. Of the 99 respondents, the data from 10 were excluded because they wore glasses and this interfered with collecting facial expression data. Analysis of the implicit facial responses showed that samples that were disliked (salted sauerkraut, mixed vegetable and grapefruit juices) resulted in more intense negative emotions and less neutral expressions than liked samples (orange and banana juices) and liked samples resulted in only minor changes in spontaneous facial expressions with no differentiation between banana and orange juice. Interestingly, it was found that the emotion “happy” was expressed for the disliked salted sauerkraut juice. The researchers reported that upon asking respondents why they smiled after tasting this sample they were informed that respondents were surprised or did not expect the sample to taste like that. However, surprise was not identified as a differentiating emotion between samples in this study. Respondents’ liking scores differentiated between three groups of products. Banana juice was significantly more liked that orange juice, and both banana and orange juice were significantly more liked than the vegetable, salted sauerkraut and grapefruit juices. In this study, 9-point hedonic scores differentiated more among products than facial expression.

A study conducted with 495 UK respondents and 6 lager beers showed no significant difference in product differentiation in terms of facial expression (van Zyl, 2015). Respondents evaluated 6 products in a randomized design. In order to capture the emotions experienced as a result of the beverages, respondents had to follow a very specific procedure to avoid the obstruction of their facial expressions by the glass from which they tasted the product. This involved taking a sip of the beverage, keeping it in the mouth, putting down the glass, looking straight into the camera and then swallowing. Data was analyzed in two ways. First the facial point displacements were grouped into facial action clusters (Table 23.2) and a score between 0 and 1 assigned to each respondent for each cluster. The score is a measure of how clearly a respondent showed the facial expression related to the cluster. The data was then analyzed to determine whether there were differences between the products in terms of these clusters. The differences between the first three clusters were mostly related to head movement and cluster 4 consisted entirely of no movement at all. In the second analysis, the intensity of nine specific Facial Action Units which were found to be relevant to this study was estimated and data was again analyzed to determine if there were differences between the products in terms of the action units. Of the 495 respondents only 30 were found to show significant facial expression upon consumption of the beers and neither of the analyses resulted in significant differences among products (Fig. 23.3).

Table 23.2. Facial action clusters from a study with different lager beers.

For products which are expected to be very different in acceptability it is reasonable to expect to find differences in disgust. However, as indicated by the study of Danner et al. (2014) it is likely to find significant differences in hedonic scores between such products. If respondents are able to verbalize whether or not they like a product, studying facial expression of emotion does not necessarily add value when studying products that are liked. Danner, Sidorkina, Joechl, and Duerrschmid (2013) also found that in studying various types of orange juices, facial expression was only able to discriminate the least liked sample (diluted orange syrup) from the orange juice samples. Basic emotions, while sufficient to distinguish between liked and disliked products are not enough to distinguish in greater detail between products that are equally liked (Danner et al., 2014; Wendin, Allesen-Holm, & Bredie, 2011; Zeinstra, Koelen, Colindres, Kok, & de Graaf, 2009) and unpleasant tastes evoke more facial displays than pleasant tastes (Greimel, Macht, Krumhuber, & Ellring, 2006). To discriminate between products that are highly liked, combinations of hedonic ratings, feelings or affective terms and conceptual attributes are needed.

The reasons behind this low efficiency of facial methods on product discrimination could be due to (1) the fact that some of these tools were initially developed for clinical purposes and hence are not suitable to capture subtle differences, (2) the hedonic asymmetry on food evoked emotions as reported by Schifferstein and Desmet (2010) and (3) the fact that products consumers choose to consume are unlikely to evoke strong negative emotions which is what facial emotion measures mostly measure. Taking into consideration some of the feelings of interest in the context of beverages (such as relaxed and relieved) it is clear that they are not expressed in easy to recognize facial expressions. While the basic emotions of fear and anger result in facial expressions, it is not expected that beverages would elicit these types of emotions.

However, if facial expression methods can be used to study how negative emotions, such as feeling stressed and frustrated are reduced by beverages to result in a more positive emotional state, it would be of greater benefit in understanding the dynamics of emotions related to beverages.