By Alana Tran
Which is easier: realizing your friend is angry from their facial expression, or from the sound of their voice? Imagine your friend says, “I’m not upset,” but their voice sounds tense, and their face tells a different story. Which cue are you best able to use to figure out how they are feeling? Until now, we have often assumed that reading facial expressions is easier than interpreting their tone of voice, but our recent research suggests that this may be more complicated than we think.
We use many different cues in someone’s tone of voice, such as its acoustic characteristics–pitch, speech rate, and volume–to infer their emotions. The voice is inherently dynamic, meaning those key acoustic characteristics will fluctuate as we communicate verbally. Because of this, we may require more working memory skills to track changing vocal cues. Working memory refers to the cognitive systems that allow us to temporarily hold information in our minds and actively use it for day-to-day tasks.
Faces are different because they don’t require us to do this: researchers often ask participants to look at a static face that isn’t moving and try to guess what that person is feeling. Is that a fair comparison to the same task in the voice? We wanted to better understand whether vocal emotion recognition is truly more challenging than facial expressions, or whether there is an underlying characteristic of voices (such as their dynamism) that makes them more difficult to identify. As well, we wanted to understand whether working memory skills help people identify changing expressions in faces or voices.
How did we test these questions?
We asked 311 university students to identify the emotion portrayed by four distinct types of expressions. First, we included stimuli from two different modalities (facial vs. vocal) to explore how we perceive emotional expressions in faces versus voices. Second, the stimuli also varied in dynamism: faces and voices were either static or dynamic. In the static conditions, participants looked at static faces (photos of people expressing emotions) or heard short acoustic sounds (e.g., laughter, sobbing) that don’t fluctuate as much in acoustic cues. In the dynamic face conditions, participants watched a video of someone shifting from a neutral face to displaying an emotion or heard people saying short sentences in an emotional tone of voice; here, participants needed to actively track facial movements or acoustic cues as they shifted. Now, the playing field is much more even! We can better compare recognition of faces and voices when they are static vs. when they are dynamic. We can also now see whether participants with different levels of working memory are better at recognizing static or dynamic facial or vocal stimuli. To help us figure this out, participants also completed a variety of tasks to assess working memory; we hypothesized that those who demonstrated greater working memory skills would be more accurate at identifying the correct emotion displayed in faces and voices, particularly if they were highly dynamic.
What did we find?
Identifying emotions from vocal expressions is not always harder than from the face! Participants were equally accurate at recognizing emotions from static photos and vocal bursts (i.e., our static voice stimuli). However, participants struggled the most with identifying the correct emotion when it was spoken in sentences (i.e., our dynamic vocal stimuli). This differentiation–that dynamic vocal expressions are harder to recognize than static facial expressions, but that shorter/more static forms of vocal expressions are not–is important because it challenges a long-standing assumption in emotion recognition research that voices are more difficult to interpret. There is more nuance to the story!
We also observed a small effect of working memory skills on emotion recognition ability: participants who had greater visual working memory skills (on a specific task called the “N-back”, which requires people to retain visual shapes in their memory) were more accurate in emotion recognition for visual faces overall, regardless of dynamism. This finding suggests that our ability to process and retain visual information in our working memory may be important for inferring facial emotions.
What can we do with these findings?
Now that we understand that dynamism plays a large role in recognizing emotion in vocal expressions, future studies that compare visual and auditory modalities can improve their research methods by controlling for dynamism. This will reduce a large confound and ensure that differences in emotion recognition are due to different modalities rather than one stimulus being static. This research also highlights how working memory skills–particularly those related to retaining visual information in our minds–may be involved in inferring others’ emotions. Although we might think that understanding how others are feeling is a purely social task, we are likely also drawing on our cognitive skills during social interactions.