Looking at Sound: Sound Design and the Audiovisual Influences on Gaze

From the earliest films to the blockbusters of today, film has rarely been silent. Live musical accompaniment of silent movies progressed into the synchronized sound of ‘talkies’ and today film sound is a highly developed craft in which sound-designers believe they have the power to represent and accentuate aspects of the scene, focusing the viewer’s attention to specific events and conveying emotion (e.g. Bordwell & Thompson, 2013; Chion, 1994; Murch, 2001). This chapter will attempt to empirically validate some of these beliefs by exploring the separate and integrated influence of each of the primary auditory components of a film’s sound design (musical score, dialogue and sound effects; known as “sound stems”) on viewer behavior, specifically observing the role of sound in guiding a viewer’s gaze through a film. This chapter will approach these issues from the perspective of experimental cognitive psychology. For a review of sound design practice see Sonnenschein, (2001); for a review of the psychological impacts of music and sound see Cohen (2014) and for reviews of film theory on classic and modern sound design see Gorbman (1980) and Donnelly (2009), respectively.

This chapter considers the influence of sound design in two “found” experimental case studies in which filmmakers claim to have manipulated viewer behavior through sound design. Firstly, a highly dynamic and edited scene from How to Train Your Dragon (DeBlois & Sanders 2010) was viewed with the three sound stems independently (dialogue, sound effects, music and a silent condition), the attentional synchrony and affective responses between the sound conditions will be compared. Secondly, gaze behavior during the famous single long opening shot from The Conversation (Coppola 1974) compared the sound influences (the presence and absence of sound) during discrete sound events within the sequence and viewer gaze behavior via quantitative analysis of heat-maps. We conclude that the influence of sound on viewer gaze during film viewing is not as pronounced as often thought. Future studies are required to further our understanding of the nuanced influence of sound design and how it shapes our whole experience of a film including attention and affective responses.