Auditory icons: Design and physical characteristics

Applied Ergonomics 78 (2019) 224–239

Contents lists available at ScienceDirect

Applied Ergonomics journal homepage: www.elsevier.com/locate/apergo

Review article

Auditory icons: Design and physical characteristics a,∗

João Paulo Cabral , Gerard Bastiaan Remijn a b

b

T

Graduate School of Design, Department of Human Science, Kyushu University, 4-9-1 Shiobaru, Minamiku, Fukuoka, 815-8540, Japan Department of Human Science, Research Center for Applied Perceptual Science, Kyushu University, 4-9-1 Shiobaru, Minamiku, Fukuoka, 815-8540, Japan

ARTICLE INFO

ABSTRACT

Keywords: Auditory warning Auditory display Product design

Auditory icons are short sound messages that convey information about an object, event or situation. Originally, auditory icons have been used in computer interfaces, but are nowadays found in many other fields. In this review article, an overview is given of the main theoretical ideas behind the use and design of auditory icons. We identified the most common fields in which auditory icons have been used, and analyzed their acoustic characteristics. The review shows that few studies have provided a precise description of the physical characteristics of the sounds in auditory icons, e.g., their intensity level, duration, and frequency range. To improve the validity and replicability of research on auditory icons, and their universal design, precise descriptions of acoustic characteristics should thus be provided.

1. Introduction - what are auditory icons? Man-made auditory warnings are widely used in our daily life. They provide information about the status of a system, device, or event – from the beeps you receive from your alarm clock every morning to the vocal message saying that “you've got mail” – auditory warnings are practically everywhere. Many different types of auditory warnings exist, depending on the purpose for which they are used. The most common auditory warnings consist of pure-tone ‘beeps’, mostly repetitive, either with a steady-state or gliding pitch, or constituting a melody. Rather than this kind of auditory warnings, here we give an overview of the design and use of a very specific kind of auditory warnings, called “auditory icons”. Originally created in the field of informatics, auditory icons are short, one-shot sound messages, consisting of short snippets of sounds that are present in everyday life. Thus, contrary to most common auditory warnings, auditory icons consist of complex sounds. Gaver (1986) defined an auditory icon as “a unique sound from an event, providing a powerful resource for information about a situation.” Some of the earliest auditory icons were used in personal computer interfaces through SonicFinder (Gaver, 1989), where they were implemented to enhance the visual desktop metaphor with an auditory dimension. Auditory icons were presented to the user for actions such as selecting objects (a ‘hitting’ sound), opening files (a ‘whooshing’ sound), copying files (a ‘pouring’ sound) or putting files into the trashcan (a ‘metal trashcan’ sound or a ‘paper-crunching’ sound). Over the past 30+ years, auditory icons have been used in an

∗

increasing number of fields. Here we review the existing literature on auditory icons with the purpose of providing insight into how auditory icons have been designed and studied so far. The following research questions were raised: 1) What are the fields in which auditory icons are being studied? 2) Which sounds have been used to represent the object, event, or situation to which an auditory icon refers to (the “referent”), and which acoustic factors have been considered in the design of auditory icons? Throughout this article, the term “referent” will be used to indicate the object, event, or situation that is being referred to by an auditory icon. The term was introduced by Ogden and Richards in 1923. As an example, the referent of a ‘paper-crunching’ sound thus would be the event of putting files into the trashcan on a computer desktop. This review is structured as follows. After a general introduction of auditory icons and the method utilized for the review, we deal with the concepts and classifications of auditory icons. Also, we describe research on the advantages and disadvantages of using auditory icons and the key points for their usability. Following this, we describe the review results and detailed lists with examples of auditory icons that have been used so far. These lists include descriptions of acoustic characteristics of auditory icons, if available from the literature, as well as referent descriptions. This is followed by a general discussion and conclusion. 2. Method The literature described in this review was selected by means of a keyword search on the words “auditory icon”, performed in an

Corresponding author. E-mail addresses: [email protected], [email protected] (J.P. Cabral), [email protected] (G.B. Remijn).

https://doi.org/10.1016/j.apergo.2019.02.008 Received 16 March 2018; Received in revised form 14 January 2019; Accepted 25 February 2019 0003-6870/ © 2019 Published by Elsevier Ltd.

Applied Ergonomics 78 (2019) 224–239

J.P. Cabral and G.B. Remijn

academic citation database (“Google Scholar”). English, peer-reviewed journal articles or conference/symposium articles with these words either in the title, abstract, or keywords section were picked up for three time periods. The first period was from 1986 to 1996; 1986 was the year in which William Gaver introduced the concept of an auditory icon. The second period was from 1997 to 2007, and the third was from 2008 to May 2018. For the first two periods all articles that met the search criteria were considered. Since the initial search for the most recent period yielded 314 articles in total, we here included the 20 most cited articles, assuming that these would best represent the most important research issues. Besides the database search, the number of selected articles was further increased by a keyword search in five journals in the field of ergonomics/human factors, which had the highest SCImago Journal Ranking Indicator (SJR) in 2017. To include the most recent articles, this search concerned articles between 2008 to May 2018 that were not already picked up by the previous search. The literature search was conducted between May to July 2018. Tendencies in research fields of auditory icons over the three time periods are discussed in Section 5.1. The initial database and journal search resulted in 12 journal articles and 31 conference/symposium articles for the period in between 1986 and 1996. For the period in between 1997 and 2007, 44 journal articles and 104 conference/symposium articles were picked up, and for the 2008–2018 period there were 113 journal articles and 201 conference/symposium articles for potential review. From the five ergonomics/human factors journals 24 more articles were considered. Besides the criteria of having the words “auditory icon” in the title, the keywords section, or the abstract, our next criterion was that the journal/conference articles should have concrete descriptions of the sound(s) and referent(s) – or at least some related descriptions, e.g. about the duration, sound level in decibels (SPL, A-weighted, or Linearweighted), and frequency in Hertz. As a result, we here review a total of 32 articles: 3 from 1986 to 1996, 7 from 1997 to 2007, and 20 from 2008 to 2018, added with 2 articles from the ergonomics/human factors journals. From these articles, a total of 188 auditory icons was reviewed. Given that our search results showed a sharp increase in the number of research articles on auditory icons in recent years, we hope the present review has its purpose for the research community now and in future years.

people (Norman, 1988). According to Norman, mappings should be as instinctive as possible, reflecting a natural relationship between objects. For instance, when we select a file or folder on our computer desktop, the accompanying sound(s) (i.e., the auditory icon) could reflect the object size. Larger objects could be accompanied by lower-pitched sounds. This mapping between auditory icon and referent follows the rule of nature that large objects are more likely to be associated with lower frequencies than smaller objects (Gaver, 1989). Natural mapping thus takes advantage of physical analogies and learned associations and, ideally, leads to immediate understanding of the information the auditory icon is supposed to convey. If a strong, natural mapping exists between the sound(s) and the referent, auditory icons would be easier to learn and remember. 3.2. Classifications of auditory icons Based on their function in computer interfaces, Gaver (1986) classified auditory icons into three levels. Each level describes the degree of physical equivalence between the sound(s) of the auditory icon and the referent. The three levels are: I. Symbolic: the relation between the sound(s) of the auditory icon and the referent is essentially arbitrary and based on social convention, e.g., ‘sirens’ for an approaching ambulance, ‘hand-clapping’ sounds for approval (applause). II. Iconic or Nomic: the sound of the auditory icon is related to the physical source of the referent, e.g., a ‘wood-’ or ‘metal-hitting’ sound to represent wooden or metal objects. Additionally, sounds in an auditory icon can represent a series of physical events. A ‘mailbox’ auditory icon can consist of successive sounds representing the arrival of mail, starting with a sound that represents the physical opening of the mailbox, followed by a sound for the insertion of a letter, and finishing with a sound that represents the closing of the mailbox. III. Metaphorical: the relation between sounds in the auditory icon and the referent is not completely arbitrary, yet also not fully dependent on physical causation. The arbitrariness is based on some similarities between the sound and the referent, but not as strongly as in auditory icons at the iconic level (Buxton et al., 1994). An example of a metaphorical auditory icon is a ‘breaking glass’ sound on a distribution line to inform that there is a fragile product in a packaging box that should be managed carefully. Gaver (1989) argued that with regard to his classification, in general, iconic/ nomic mappings are more powerful than symbolic and metaphorical mappings, because iconic/nomic mappings show a direct relation between the auditory icon and the referent's physical source.

3. Auditory icons: concepts and classifications 3.1. Concepts in auditory icon creation According to Gaver (1986), an auditory icon can be interpreted from two types of information. First, the auditory icon can be interpreted according to its proximal stimulus qualities, i.e., the physical characteristics of the sound(s) in an auditory icon that are available to the listener's auditory system. Typical proximal stimulus qualities are sound duration, sound frequency, and intensity. Second, the auditory icon can be interpreted with regard to its distal stimulus qualities. These are related to the physical source of the sound (i.e., the referent) that exists in the world. Gaver highlighted the importance of distal stimulus qualities, which contain information about the characteristics of the referent, such as its material (e.g., wood, metal), or size (e.g., large, heavy). Related to this, Gibson (1979) introduced the ecological approach to perception, or the “theory of affordances”, which states that we do not experience the world around us directly, but through an internal mental image of the world. Applying Gibson's idea to the creation of auditory icons, we can say that the sound should afford the referent. Gaver et al. (1991) mentioned that a sound can afford the referent information in a collaborative way, complementing visual information. Another important concept in auditory icon creation is mapping. In vision, mapping refers to the relationship between two things (or two controls), and how their movements and disposition are interpreted by

A more complex classification of auditory warnings, including not only auditory icons but also speech and abstract sounds, was proposed by Keller and Stevens (2004). Using a structure adapted from vision research (Familant and Detweiler, 1993), their classification is based on three different levels of interaction. In the first level (“Signals”), auditory icons, speech and abstract sounds are represented. The next level (“Sign relations”) indicates the direction of the relation between a sound (or signal) and the target referent. This relation can be either iconic or non-iconic. An iconic relation is a relation between sound and referent as meant by Gaver (1986), i.e., the sound of the auditory icon is associated with the physical source of the referent. A non-iconic relation means that the relation between the sound and the referent is manufactured, i.e., the relation between the sound and the referent is metaphoric or symbolic, as meant by Gaver (1986). The last level in the classification of Keller and Stevens (2004) shows the “Referent relations.” It specifies the type of relations between a sound (auditory icon, speech, or abstract sound) and the referent. In an ecological relation, the referent and the sound are distinct – they are not necessarily physically similar yet are identifiable with one another 225

Applied Ergonomics 78 (2019) 224–239

J.P. Cabral and G.B. Remijn

because they tend to coexist in the target referent. Keller and Stevens (2004) used an example of a ‘seagull’ sound, which can be used to invoke a ship as the referent. In a metaphorical relation, the comparison between the sound and the referent is based on their affinity (e.g., appearance and function). The authors used an example of a ‘camera shutter’ sound to invoke an eye as the referent.

an auditory icon can convey information about age, size, speed, and other physical dimensions of the referent (Gaver, 1989). Physical features of the sound(s) used in auditory icons (see Section 4.3.3), such as pitch, reverberation, volume, and ramping, can be manipulated to convey, for example, the perceived location, distance, and size of the referent (Stevens et al., 2004). Auditory icons have been used this way in a number of fields. For example, in an office environment, the amount of email received (the referent) can be conveyed by an auditory icon with a varying number of ‘seagull sounds’ (Mynatt et al., 1998). In the automotive field, alerting drivers in real-time about the speed of their vehicle can be done by using auditory icons that mimic the wind when driving with the windows open. If the driver is speeding, the ‘wind’ icon is played more frequently (Cabral et al., 2015). Informing video surveillance operators in a control room about the trajectory and speed of eventual targets on the monitor can be accomplished by using various numbers of ‘footsteps’ icons (Höferlin et al., 2011).

4. Usability of auditory icons – advantages and disadvantages 4.1. Advantages of auditory icons Some research on the use of auditory icons so far has been performed with the end products being tested and evaluated by users. According to the literature, the main advantages of the use of auditory icons are the following. First, auditory icons can induce good reaction time, which is important when the user needs to manage a situation that requires a rapid response, for example, when driving. In general, compared with abstract warnings such as beep sounds, auditory icons not only induce better reaction times but better response accuracy as well. This has been shown in research with automotive interfaces (Graham, 1999; McKeown and Isherwood, 2007; Larsson et al., 2009; Bakowski et al., 2015), in an aviation environment (Perry et al., 2007), and in mobile phone interaction (Xu et al., 2010). Reaction times are influenced by the complexity of the referent and how the auditory icon is associated with it. Participants show a faster reaction time to auditory icons with a strong mapping with their referent than to auditory icons with an arbitrary relationship to their referent (Stephan et al., 2006). Reaction times to auditory icons also depend on whether the user is occupied by external matters other than the referent, for example, in situations requiring concurrent low and high demand tasks, as in aviation (Perry et al., 2007). Second, users can profit from the good learnability of auditory icons. Compared to abstract warnings, auditory icons are easier to learn (Graham, 1999; McKeown and Isherwood, 2007; Perry et al., 2007). Learnability, though, depends on a number of factors. Different from abstract warnings, the learnability of an auditory icon is directly related to the mapping it can provide. Iconic/nomic auditory icons, in which the sound of the auditory icon is directly related to the physical source of the referent, tend to provide a stronger mapping and are easier to learn than metaphorical or symbolic auditory icons. Auditory icons with a direct apparent relation provide better learnability than icons unrelated with their referents, even when the icons should be remembered after four weeks (Stephan et al., 2006). As a third advantage, auditory icons often induce a better user performance in complex environments or situations. For example, Gaver et al. (1991) implemented the use of several auditory icons in a complex soft-drink assembly line. The icons indicated the machine status over time and, following the rhythm of the assembly machinery, the sounds indicated the assembly line's speed. The study revealed that the auditory icons efficiently helped users to notice problems and to supervise the line. Although the sounds sometimes could cause misunderstanding related to the perceived urgency and the discrimination between auditory icons due to masking (see Section 4.2), the assembly line personnel learned quickly in a complex environment. Another case where auditory icons induced good user performance was in the interaction with different medical equipment, which also requires a perfect discrimination of the equipment and interpretation of the right urgency levels (Edworthy et al., 2014, 2017). A fourth advantage of the use of auditory icons is good user acceptance by listeners. Auditory icons can improve the interaction between the user and the device on which they are implemented (see also Section 4.3.4). Furthermore, auditory icons can also enhance the user experience – essentially, they can make the interaction more fun. This is particularly important for learning environments, e.g., virtual learning for children (Jacko, 1996) or adults (Alseid et al., 2014). Finally, as mentioned previously, by using dimensional information

4.2. Disadvantages of auditory icons The use of auditory icons has also some disadvantages compared to common auditory warnings. The first is misinterpretation. When auditory icons share the same (acoustic) characteristics, they can be misinterpreted as representing the same referent instead of different ones, and produce incorrect responses (Graham, 1999; Frimalm et al., 2014). Second, complex referents are difficult to express as auditory icons (Graham, 1999). One example of a complex auditory icon is the use of a sound from the animal kingdom, ‘elephant trumpeting’, to refer to an overweight aircraft (Perry et al., 2007; see also Section 5.3). An example of a complex auditory icon in the automotive industry is the use of a ‘rapid gunshots’ sound to signal that a driver's headway is closing fast (Mckeown et al., 2010). Complex auditory icons like these take time to learn. Another disadvantage of auditory icons, and potentially for auditory warnings in general, is that they are vulnerable to masking by other (environmental) sounds or auditory icons. Masking occurs when the listening threshold of the target sound, i.e., the auditory icon, is affected by background sounds in the environment. Masking can be the case when auditory icons are used in complex acoustic work environments such as assembly lines (see above; Gaver et al., 1991) or in the automotive industry. For instance, ‘rain with thunder roll’, to refer to bad weather conditions ahead, can be subject to masking in the presence of engine noise inside the cabin of a truck (Winters, 1998). Auditory icon sounds preferred by users during the evaluation in a quiet environment may thus not be ideal in an acoustically complex environment. Masking can also occur when multiple auditory icons are played at the same time, making it difficult for listeners to distinguish between them. To avoid masking, auditory icons can be diversified by time-varying frequency, amplitude, and timbre (Gaver et al., 1991). Furthermore, to avoid potential masking problems particularly in vehicles, the increased use of infotainment by audio and video should be considered in the design of auditory icons. The masking problem here is not just an acoustic masking problem; the act of driving is difficult per se because it requires shared attention among many tasks and sounds, including handling the vehicle, paying attention to sometimes very complex traffic scenarios, and to passengers, if any. Finally, the learnability of auditory icons depends on the user's experience with earlier auditory warnings. User experience, for example at the workplace, can affect learnability and acceptance of auditory icons – independent of the mapping strength of the icon with the referent. Stanton and Edworthy (1998) investigated the learnability and acceptance of 9 auditory icons designed to identify equipment in an intensive treatment unit (ITU). The participants were all staff members of the ITU, who were used to work with the equipment and their accompanying abstract auditory warnings. Their task was first to identify each new auditory icon and match it to a piece of equipment. Next, the participants were asked to rate the appropriateness and urgency of the 226

Applied Ergonomics 78 (2019) 224–239

J.P. Cabral and G.B. Remijn

sound(s) in the auditory icons, under the hypothesis that environmental sounds would be identified more quickly than abstract ones. The results showed that the staff members' interpretation regarding the purpose of an auditory icon depended on their experience at the ITU. Inexperienced staff associated the equipment more easily with the new auditory icons than the staff with vast ITU experience, who related the equipment still more easily to the old abstract warnings. As for the appropriateness ratings, both experienced and novice staff rated the new auditory icons as less appropriate for the referents compared to the old alarms. This shows that previous experience can hamper learnability. Inexperienced learners, however, can take up the relation between an auditory icon and its referent relatively quickly (see also Edworthy et al., 2014).

Fagerlönn, 2011; Nees et al., 2016). An example from the medical field is a ‘rattling pillbox’ to alert a patient to take medication (Edworthy et al., 2014, 2017). However, the referent cannot be conveyed easily to the user when the mapping of an auditory icon is associated with a difficult external referent, e.g., a ‘spaceship door’ sound to represent opening a file on a PC desktop (Petrie and Morley, 1998) or a ‘frog croaking’ sound to represent a network attack (Garcia-Ruiz et al., 2008). 4.3.3. Key point 3: physical characteristics of sounds in auditory icons The sound(s) used in auditory icons have been designed and manipulated in a number of ways. The simplest auditory icon consists of an everyday sound actually generated by a referent itself. A more complex way is to record the referent's sound(s) at different times, varying for example its duration or intensity. These types of auditory icons are called parametric auditory icons (Brazil and Fernström, 2011), and a typical example is that of a bouncing ball of different sizes recorded at different times. The number of variations in such icons should not be too large, however, in order to ensure sound recognition. A third way to design auditory icons is by synthesizing them through mathematical simulations applied in real time. For instance, the representation of a ‘marble rolling across a table’ sound can be manipulated by varying the size and speed of the marble. (Brazil and Fernström, 2011). The physical characteristics of the sound(s) in an auditory icon, such as sound duration, intensity, quality (e.g., sampling frequency), and frequency range directly influence its identification and, hence, its usability. When presented in a real or simulated situation, the physical characteristics of an auditory icon may be perceived differently according to the characteristics of the listening environment and equipment, or the listeners themselves. Stevens et al. (2004) investigated the acoustic characteristics of ‘bell’, ‘horn’, ‘dog bark’, and ‘footsteps’ sounds as auditory icons. Each auditory icon was modified to inform size (small/large), distance (near/far) and direction (away/toward), manipulated by pitch, reverberation, and volume ramping, respectively. The participants' task was to identify the acoustical manipulations across the auditory icons. The authors hypothesized that recognition accuracy would decrease as the number of acoustical manipulations increased. The results showed that up to three acoustical manipulations, the auditory icons continued to be well recognized. The manipulation of direction was usually identified better than the other acoustic manipulations. In addition, ‘dog bark’ was better identified compared to other icons under all acoustic characteristics investigated, which shows that iconic or nomic auditory icons might be more robust than others against acoustic manipulations.

4.3. Key points for the usability of auditory icons Even before any substantial research on the (dis)advantages of auditory icons was performed, Mynatt (1994) already discussed four key points that potentially could affect the usability of auditory icons. They are (1) the icon's identifiability, (2) its conceptual mapping, (3) the icon's physical parameters, and (4) user preferences. These key points seem to summarize previous lists of attributes for general interfaces and auditory displays. For example, according to Nielsen (1993), attributes important for the usability of general interfaces are learnability, memorability, efficiency, satisfaction, and error rate. For general auditory displays, Sanders and McCormick (1993) suggested to use sounds with natural relationships that could attract attention and give precise and sufficient information. Similar to Mynatt's (1994) key point about the mapping of auditory icons, Sanders and McCormick (1993) also mentioned that sounds in general auditory displays should be easily discernable and always label the same information. In view of the fact that Mynatt's (1994) was the first study that mentioned key points and guidelines specifically for the design of auditory icons, below we discuss each of her key points in further detail, along with relevant research. 4.3.1. Key point 1: identification of auditory icons When users are not able to identify the sound(s) of an auditory icon, its usefulness diminishes (Mynatt, 1994). Sounds in auditory icons are sounds that we typically hear around us every day. Regularity and familiarity between the sound(s) and the referent (the ecological frequency) are important for the identification of auditory icons. Furthermore, identification of auditory icons depends on the sensory processing of the sounds’ physical characteristics (see Section 4.3.3). According to Ballas et al. (1986), the identification of everyday sounds as used in auditory icons also depends on the context of where the sounds are heard, and the context in which the sounds are presented. If the sounds are heard in an unnatural, isolated context, e.g., when tested in a small room, identification could be more difficult than when they are presented in the actual user context. Besides the masking problem (see Section 4.2), if auditory icons are presented in the context of other sounds they may induce multiple semantic interpretations, and their recognition may become compromised as a consequence.

4.3.4. Key point 4: user preference for auditory icons The fourth key point of Mynatt (1994) that can influence the usability of auditory icons is user preference. Research has suggested that users prefer icons with a strong relation to their referents, that is, users generally prefer relations as in iconic or nomic icons where the sound of the icon is directly related to the source (Höferlin et al., 2011; Wang et al., 2012; Isherwood and Mckeown, 2016). User preference, however, is also determined by how the user responds emotionally to the auditory icon. Studies on informatics (see also Table 1) have shown that auditory icons can enhance the interaction between a computer and the user, making computer use more intuitive, efficient, and enjoyable. For instance, auditory icons can be used to relief the stress of users performing video surveillance (Höferlin et al., 2011; Sirkka et al., 2014). It needs to be noted, though, that McKeown and Isherwood (2007) demonstrated a strong correlation between auditory icons judged as conveying high urgency and their level of pleasantness: “urgent” auditory icons were judged as more unpleasant.

4.3.2. Key point 2: conceptual mapping of auditory icons The second factor that can affect the usability of an auditory icon is its conceptual mapping (Mynatt, 1994). Some studies have shown clues as to achieve better mapping between an auditory icon and the referent. First, as discussed in Section 3.1, the sound(s) used in an auditory icon ideally should map naturally or instinctively with the referent it characterizes. Petrie and Morley (1998) showed that the identification of an auditory icon increased when the mapping of the icon naturally corresponded with its graphical referent, e.g., a ‘typewriter keystroke’ auditory icon to represent typing. Examples in the automotive industry are a ‘bicycle bell’ auditory icon to represent a bicycle nearby or a ‘thunder’ icon to inform about rain ahead (Fricke and Thüring, 2009;

4.3.5. Design guidelines Considering the previous key points, Mynatt (1994) listed the following essential steps for the design of effective auditory icons. For a 227

Applied Ergonomics 78 (2019) 224–239

J.P. Cabral and G.B. Remijn

Table 1 Auditory icons used in informatics. Referent

Auditory icon

Dur. (ms)

Level a

Freq. (Hz)

Mynatt's key points

Reference

Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. – Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. – Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. – Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics.

Mynatt (1994)

Copying a file

‘Camera shutter’

Under 2000

–

–

Closing

‘Pouring sound’ ‘Closing car door’

– Under 2000

– –

– –

‘Zipping down’

Under 2000

–

–

A text field

‘Typing’

Under 2000

–

–

A slider

‘Whistle up’

Under 2000

–

–

Check box

‘Keystroke’

Under 2000

–

–

Window dragging

‘Whistle up’

Under 2000

–

–

‘Cars driving by’

Under 2000

–

–

‘Scraping sound’ ‘Open door’

– Under 2000

– –

– –

‘Whistle up’

Under 2000

–

–

‘Motorcycle’

Under 2000

–

–

‘Whooshing sound’ ‘Whip cracking’

– Under 2000

– –

– –

‘Short pop’

Under 2000

–

–

‘Keystroke’

Under 2000

–

–

‘Water drop’

Under 2000

–

–

‘Whistle down’

Under 2000

–

–

‘Flipping papers’

Under 2000

–

–

Opening

A push button

A menu

Gaver (1989) Mynatt (1994)

Mynatt (1994)

Mynatt (1994)

Mynatt (1994)

Mynatt (1994)

Mynatt (1994)

Mynatt (1994)

Gaver (1989) Mynatt (1994)

Mynatt (1994)

Mynatt (1994)

Gaver (1989) Mynatt (1994)

Mynatt (1994)

Mynatt (1994)

Mynatt (1994)

Mynatt (1994)

Mynatt (1994)

(continued on next page)

228

Applied Ergonomics 78 (2019) 224–239

J.P. Cabral and G.B. Remijn

Table 1 (continued) Referent

Auditory icon

Dur. (ms)

Level

Freq. (Hz)

Mynatt's key points

Reference

‘Opening door’

Under 2000

–

–

Mynatt (1994)

‘Pinball drop’

Under 2000

–

–

‘Short pop’

Under 2000

–

–

‘Long pop’

Under 2000

–

–

‘Winding’

Under 2000

–

–

‘Whistle up’

Under 2000

–

–

‘Whistle down’

Under 2000

–

–

‘Motorcycle’

Under 2000

–

–

‘Flipping papers’

Under 2000

–

–

‘Keystroke’

Under 2000

–a

–

‘Winding’

Under 2000

–

–

‘Whistle up’

Under 2000

–

–

‘Cars driving by’

Under 2000

–

–

Selection

‘Hit impact sound’ ‘Clink’ ‘Ripping papers’

– – –

– – –

– – –

“Drop” a file in a folder Window enhancing Window scrolling Trashcan drop-in Trashcan-empty Window appears

‘Noise of object landing’ ‘Clink’ ‘Tick sound’ ‘Crash’ ‘Crunch’ ‘Whistle up’

– – – – – –

– – – – – –

– – – – – –

Window disappears

‘Whistle down’

–

–

–

Pop-up window (focus)

‘Something popping up’

–

–

–

Switching between tasks

‘Paper flipping sound’

–

–

–

Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. – – Identifiability; Conceptual Mapping; User Preferences. – – – – – Identifiability; Conceptual Mapping; User Preferences. Identifiability; Conceptual Mapping; User Preferences. Identifiability; Conceptual Mapping; User Preferences. Identifiability; Conceptual Mapping; User Preferences.

Radio buttons

A scrolling list

Window appears when selecting from menu

Mynatt (1994)

Mynatt (1994)

Mynatt (1994)

Mynatt (1994)

Mynatt (1994)

Mynatt (1994)

Mynatt (1994)

Mynatt (1994)

Mynatt (1994)

Mynatt (1994)

Mynatt (1994)

Mynatt (1994)

Gaver (1989) Gaver (1989) Mynatt (1997) Gaver (1989) Gaver (1989) Gaver (1989) Gaver (1989) Gaver (1989) Mynatt (1997) Mynatt (1997) Mynatt (1997) Mynatt (1997)

(continued on next page)

229

Applied Ergonomics 78 (2019) 224–239

J.P. Cabral and G.B. Remijn

Table 1 (continued) Referent

Auditory icon

Dur. (ms)

Level

Freq. (Hz)

Mynatt's key points

Reference

Move between objects

‘Ball bouncing’

–

–

–

Mynatt (1997)

No new email

‘A single seagull cry’

–

–

–

1 to 5 new email(s)

‘A seagull calling a few times’

–

–

–

6 to 15 new emails

‘A few seagulls calling’

–

–

–

More than 15 new emails

‘Seagulls squabbling’

–

–

–

Low group activity

‘Distant surf’

–

–

–

Medium group activity

‘Closer waves’

–

–

–

High group activity

‘Closer, more active waves’

–

–

–

Error information

‘Glass breaking’

–

–

–

Comment information

‘Opening a bottle lid’

–

–

–

Giving an opinion to the user in eassessment Explanation of questions

‘Honking horn’

–

–

–

‘Closing window’

–

–

–

Suggestion period

‘Door opening’

–a

–

–

Informing an important note

‘Hand clapping’

–

–

–

Network attack ‘guess’

‘Frog croaking sound’

–

–

–

Network attack ‘rcp’

‘Cat mewing sound’

–

–

–

Network attack ‘rsh’

‘Horse whinnying sound’

–

–

–

Network attack ‘rlogin’

‘Rooster crowing ’

–

–

–

Network attack ‘port-scan’

‘Bird singing’

–

–

–

Allied soldier identified

‘Doorbell’

–

65.0 dB (Lin)

500–4000

Unknown individual identified

‘Two rapidly oscillating dissonant sweeps’

–

65.0 dB (Lin)

500–4000

The individual should be questioned

‘The sound of submarine sonar ping’

–

65.0 dB (Lin)

500–4000

Allied soldier in sector

‘Two sequential chords’

–

65.0 dB (Lin)

500–4000

Soldier not operational

‘Slow downward sweep’

–

65.0 dB (Lin)

500–4000

Military forces in danger

‘Siren sound’

–

–

–

New enemy identified

‘Gun reloading sound’

–

–

–

–

–

–

–

–

–

–

–

–

–

–

–

Identifiability; Conceptual Mapping; User Preferences. Identifiability; Conceptual Mapping; User Preferences. Identifiability; Conceptual Mapping; User Preferences. Identifiability; Conceptual Mapping; User Preferences. Identifiability; Conceptual Mapping; User Preferences. Identifiability; Conceptual Mapping; User Preferences. Identifiability; Conceptual Mapping; User Preferences. Identifiability; Conceptual Mapping; User Preferences. Identifiability; User Preferences. Identifiability; User Preferences. Identifiability; User Preferences. Identifiability; User Preferences. Identifiability; User Preferences. Identifiability; User Preferences. Identifiability; User Preferences. Identifiability; User Preferences. Identifiability; User Preferences. Identifiability; User Preferences. Identifiability; User Preferences. Conceptual Mapping; User Preferences; Physical Characteristics. Conceptual Mapping; User Preferences; Physical Characteristics. Conceptual Mapping; User Preferences; Physical Characteristics. Conceptual Mapping; User Preferences; Physical Characteristics. Conceptual Mapping; User Preferences; Physical Characteristics. Identifiability; User Preferences. Identifiability; User Preferences. Identifiability; User Preferences. Identifiability; User Preferences. Identifiability; User Preferences. Identifiability; Conceptual Mapping; User Preferences.

Enemy assigned Near to obstacles

‘Tire skid’

Approaching boundaries People identified on monitor

‘Footsteps sound’

Mynatt et al. (1998) Mynatt et al. (1998) Mynatt et al. (1998) Mynatt et al. (1998) Mynatt et al. (1998) Mynatt et al. (1998) Mynatt et al. (1998) Rigas and Algahtani (2014) Rigas and Algahtani (2014) Rigas and Algahtani (2014) Rigas and Algahtani (2014) Rigas and Algahtani (2014) Rigas and Algahtani (2014) Garcia-Ruiz et al. (2008) Garcia-Ruiz et al. (2008) Garcia-Ruiz et al. (2008) Garcia-Ruiz et al. (2008) Garcia-Ruiz et al. (2008) Haas and Schmidt (1995) Haas and Schmidt (1995) Haas and Schmidt (1995) Haas and Schmidt (1995) Haas and Schmidt (1995) Wei and Kenny (2009) Wei and Kenny (2009) Wei and Kenny (2009) Wei and Kenny (2009) Wei and Kenny (2009) Höferlin et al. (2011)

(continued on next page)

230

Applied Ergonomics 78 (2019) 224–239

J.P. Cabral and G.B. Remijn

Table 1 (continued) Referent

Auditory icon

Dur. (ms)

Level

Freq. (Hz)

Mynatt's key points

Reference

Vehicles identified on monitor

‘Engine sound’

–

–

–

Höferlin et al. (2011)

Problem in the washing unit

‘Water dropping sound’

∼1000

∼60.0 dB (A)

–

Problem in the shipping process

‘Breaking twig sound’

∼1000

∼61.0 dB (A)

–

Problem in the cooking section

‘Hissing steam kettle sound’

∼1000

∼57.0 dB (A)

–

Identifiability; Conceptual Mapping; User Preferences. Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics.

a

Sirkka et al. (2014)

Sirkka et al. (2014)

Sirkka et al. (2014)

Not described.

more recent overview of design guidelines of auditory alarms in general, please refer to Chi et al. (2017). Step I. Consider physical characteristics of sounds in auditory icons. Auditory icons should be short, so as to enable swift identification by experienced users. In addition, identification by inexperienced users will improve if all icons have an equal duration (Mynatt, 1994). Step II. Use free-form answers to evaluate auditory icon usability. According to Mynatt (1994), this will help the designer to recognize the different interpretations from listeners. Free-form answers are allowed in experiments when participants describe their interpretation of auditory icons presented to them without any constraint on the form. Step III. Consider learnability. Previous research has concentrated on the design of auditory icons that are easily learnable by expert and novice users. The evaluation of auditory icon recognition or identification in relation to learnability can be realized through periods of practice, followed by a recall test. Comparison with other auditory warnings is also often performed. Step IV. Test possible conceptual mappings for auditory icons by evaluating their use in the final interface (product). When auditory icons are tested outside the final interface or product, problems may not be correctly identified. Step V. Consider potential problems. A typical problem is a conflicting mapping between auditory icons, which occurs when different sounds are used in different auditory icons to represent the same referent (also see Section 3.1; see anti-collision referent in Table 2). Another problem is masking (see Section 4.2). Step VI. Apply usability experiments. According to Mynatt (1994), it is recommendable to analyze the users’ long-term performance when using the auditory icons in the final interface. Different from Step III, the evaluation here is related to how users experience the interface under certain conditions, for example, during long periods of time, or under certain emotional states.

different groups of computer users: visually impaired persons (Mynatt, 1994) and military personnel (Haas and Schmidt, 1995). Throughout 1997–2007, auditory icons were studied not only in informatics (2 studies), but also in the automotive industry (3 studies) and the aviation industry (2 studies). In informatics, families of auditory icons were introduced that conveyed dimensional information (i.e., the number of emails received, represented by different ‘seagull’ sounds, Table 1; Mynatt et al., 1998). Furthermore, the effects of the use of auditory icons on user responses, such as reaction time and accuracy, were evaluated and initial investigations were done on the correlation between auditory icons and visual icons in the automotive and aviation industry. In the most recent period (2008–2018), the 20 most-cited articles discussed auditory icons in informatics (5 studies), the automotive industry (10 studies), and aviation (1 study), but also in new fields such as mobile interaction (3 studies), medical devices (2 studies), and homecare systems (1 study). The interest in studies about auditory icons in the automotive industry especially regarded the use of auditory icons with visual displays, such as driving simulators. Auditory icons were investigated in smartphones and in personal digital assistants (PDAs) as well as in video surveillance. In this recent time period, researchers also started to study the use of auditory icons in comparison to other auditory warnings modalities, in homecare systems and in medical devices. The second research objective here was to determine which sounds had been used to represent a referent and which acoustic factors had been considered in the design of auditory icons. To show what kind of auditory icons have been use to represent a referent, we give an overview of auditory icons and their acoustic characteristics as used in informatics (Table 1), the automotive industry (Table 2), the aviation industry (Table 3), and mobile interaction (Table 4). Together, these fields concern the majority of the articles reviewed here. The tables show a) the referent, i.e., the object, event or situation to be represented by the auditory icon; b) the auditory icon, i.e., the sound (s) used to represent the referent; and c) physical characteristics of the sound(s) in the auditory icon, i.e., duration in milliseconds (“Dur.”), sound level (“Level”) in decibels (SPL, A-weighted, or Linear-weighted), and frequency range in Hertz (“Freq.”). We further indicate whether Mynatt's key point's (Section 4.3) were considered in the literature, provided the research was performed after Mynatt's (1994). Note that in many cases the exact acoustic characteristics of the sounds were not fully described in the article. We indicate an absence of data with a dash.

5. Review results 5.1. Research fields of auditory icons The first research objective addressed in this article was to examine the fields in which auditory icons are being studied. The 32 articles reviewed here dealt with auditory icons in informatics (10 studies), the automotive industry (13 studies), the aviation industry (3 studies), mobile interaction (3 studies), medical equipment (2 studies), and homecare systems (1 study). In the time period 1986–1996, auditory icons were only studied in the field of informatics. Besides Gaver (1986), two pioneering studies were the first to investigate the correlation between auditory icons and the referents they are supposed to represent, as investigated with

5.2. Auditory icons in informatics – Table 1 As mentioned in the introduction, one of the first uses of auditory 231

Applied Ergonomics 78 (2019) 224–239

J.P. Cabral and G.B. Remijn

Table 2 Auditory icons used in the automotive industry. Referent Anti-collision

Auditory icon ‘Tire-skid’

Freq. (Hz) a

62.8 dB (A)

–

700

81.0 dB (A)

–

700

81.0 dB (A)

–

700

59.4 dB (A)

355

500–1200

61.0 dB (A)

–

‘Car crashing’

– 3500

– 69.0–74.0 dB (SPL)

– –

‘Car screeching tires/crashing’

4270

75.0 dB (SPL)

127 -11110

‘Car screeching tires’

1000

71.0–75.0 dB (SPL)

–

–

–

–

‘Rapid gunshot from pistol’

1300

–

–

‘Pouring liquid’

3500

69.0–74.0 dB (SPL)

–

3910

75.0 dB (SPL)

52–11006

270–2200 M = 1130, SD = 72 3500

∼60 dB (SPL)

–

69.0–74.0 dB (SPL)

–

7630

75.0 dB (SPL)

122–10822

‘Water gurgling’ Low oil level

Level

700

‘Car horn’

Low fuel level

Dur. (ms)

‘Sound of steam and liquid’

Low oil pressure

‘Straw sucking in almost empty glass’

1300

–

–

Low tire pressure

‘Air release blast’

3500

69.0–74.0 dB (SPL)

–

Low tire pressure

‘Air release blast’

2400

75.0 dB (SPL)

119–11396

‘Air escaping from a balloon’

1300

–a

–

‘Air hiss’

270–2200 M=1130, SD=72

∼ 60.0 dB (SPL)

–

Low wiper fluid

‘Water squirt’

270–2200 M=1130, SD=72

∼ 60.0 dB (SPL)

–

Door is open

‘Car door closing’

3500

69.0–74.0 dB (SPL)

–

‘Car door opening and closing’

4010

75.0 dB (SPL)

76–11653

Mynatt's key points

Reference

Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. Identifiability; User Preferences; Physical Characteristics. Identifiability; User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; User Preferences; Physical Characteristics. Identifiability; User Preferences; Physical Characteristics. User Preferences. Identifiability; Conceptual Mapping; Physical Characteristics. Identifiability; User Preferences; Physical Characteristics. Identifiability; Physical Characteristics. User Preferences.

Graham (1999)

Identifiability; Conceptual Mapping; Physical Characteristics. Identifiability; Conceptual Mapping; Physical Characteristics. Identifiability; User Preferences; Physical Characteristics. Identifiability; User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; Physical Characteristics. Identifiability; User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; Physical Characteristics. Identifiability; Conceptual Mapping; Physical Characteristics. Identifiability; User Preferences; Physical Characteristics. Identifiability; User Preferences; Physical Characteristics. Identifiability; User Preferences; Physical Characteristics. Identifiability; User Preferences; Physical Characteristics. Identifiability; User Preferences; Physical Characteristics. Identifiability; User Preferences; Physical Characteristics.

Mckeown et al. (2010)

Belz et al. (1999) Belz et al. (1999) Graham (1999)

Fagerlönn (2011) Bakowski et al. (2015) McKeown and Isherwood (2007) Isherwood and Mckeown (2016) Fricke and Thüring (2009) Bakowski et al. (2015)

McKeown and Isherwood (2007) Isherwood and Mckeown (2016) Nees et al. (2016) McKeown and Isherwood (2007) Isherwood and Mckeown (2016) Mckeown et al. (2010) McKeown and Isherwood (2007) Isherwood and Mckeown (2016) Mckeown et al. (2010) Nees et al. (2016) Nees et al. (2016) Mckeown and Isherwood (2007) Isherwood and Mckeown (2016)

(continued on next page)

232

Applied Ergonomics 78 (2019) 224–239

J.P. Cabral and G.B. Remijn

Table 2 (continued) Referent

Auditory icon

Dur. (ms)

Level

Freq. (Hz)

Mynatt's key points

Reference

Over speed limit

‘Car quickly passing’

3500

69.0–74.0 dB (SPL)

–

McKeown and Isherwood (2007)

5330

75.0 dB (SPL)

219–11050

‘Fast wind’

∼ 1000

85.0 dB (A)

–

Under speed limit

‘Slow wind’

∼ 1000

85.0 dB (A)

–

Hand brake on

‘Squeaking sound’

3500

69.0–74.0 dB (SPL)

–

Car in the blind-spot

‘Car horn blast’

3500

69.0–74.0 dB (SPL)

–

‘Three brisk car horn blasts’

2110

75.0 dB (SPL)

97–11046

‘Driving over rumble strips’

3500

69.0–74.0 dB (SPL)

–

5230

75.0 dB (SPL)

89–11010

500–1200

61.0 dB (A)

–

1000

71.0-75.0 dB (SPL)

–

–

–

–

‘Crossing guard whistle’

270–2200 M=1130, SD=72

∼ 60.0 dB (SPL)

–

‘Footsteps’

–

–

–

Pedestrian nearby stopped

‘Human whistle’

–

–

–

Forward Collision Warning

‘Car horn continuous’b

1240

–

417

1240

–

417

Identifiability; User Preferences; Physical Characteristics. Identifiability; User Preferences; Physical Characteristics. User Preferences; Physical Characteristics. User Preferences; Physical Characteristics. User Preferences; Physical Characteristics. User Preferences; Physical Characteristics. User Preferences; Physical Characteristics. User Preferences; Physical Characteristics. User Preferences; Physical Characteristics. User Preferences; Physical Characteristics. User Preferences; Physical Characteristics. Identifiability; Conceptual Mapping; User Preferences. Identifiability; Conceptual Mapping; User Preferences. Identifiability; Conceptual Mapping; User Preferences. Identifiability; Conceptual Mapping; User Preferences. Identifiability; User Preferences; Physical Characteristics. Identifiability; User Preferences; Physical Characteristics. Identifiability; User Preferences; Physical Characteristics. Identifiability; User Preferences; Physical Characteristics. Identifiability; User Preferences; Physical Characteristics. Identifiability; User Preferences; Physical Characteristics. Identifiability; User Preferences; Physical Characteristics. Identifiability; User Preferences; Physical Characteristics. Identifiability; User Preferences; Physical Characteristics. Identifiability; User Preferences; Physical Characteristics. Identifiability; User Preferences; Physical Characteristics. Identifiability; User Preferences; Physical Characteristics. Identifiability; User Preferences; Physical Characteristics.

Drifting off the road

Bicycle nearby

Pedestrian walking nearby

‘Bicycle bell’

Autonomous Cruise Control Lane Change Support

‘2 car horn honks’

270

–

417

Need to contact service

‘Sound of a ratchet handle’b

–a

–

–

Lane departure

‘Low-frequency rumble noise’

500–1200

61.0 dB(A)

–

Slippery road

‘Ice scraping’

500–1200

61.0 dB(A)

–

Traffic queue

‘Horns playing simultaneously’

500–1200

61.0 dB(A)

–

Traffic reported ahead

‘Multiple car horns’

∼60.0 dB (SPL)

–

Construction reported ahead

‘Jackhammer’

∼60.0 dB (SPL)

–

Car's acceleration

‘Engine sound speeding’

270–2200 M = 1130, SD = 72 270–2200 M = 1130, SD = 72 3000

–

–

Braking the vehicle

‘Tire screeching’

3000

–

–

General indication

‘Speed up version of traditional beeps’

3000

–

–

Gear changing

‘Vehicle changing gears’

3000

–

–

Isherwood and Mckeown (2016) Cabral et al. (2015) Cabral et al. (2015) Mckeown and Isherwood (2007) McKeown and Isherwood (2007) Isherwood and Mckeown (2016) Mckeown and Isherwood (2007) Isherwood and Mckeown (2016) Fagerlönn (2011) Fricke and Thüring (2009) Wang et al. (2012) Nees et al. (2016) Wang et al. (2012) Wang et al. (2012) Larsson et al. (2009) Larsson et al. (2009) Larsson et al. (2009) Larsson et al. (2009) Fagerlönn (2011) Fagerlönn (2011) Fagerlönn (2011) Nees et al. (2016) Nees et al. (2016) Beattie et al. (2015) Beattie et al. (2015) Beattie et al. (2015) Beattie et al. (2015)

(continued on next page) 233

Applied Ergonomics 78 (2019) 224–239

J.P. Cabral and G.B. Remijn

Table 2 (continued) Referent

Auditory icon

Dur. (ms)

Level

Freq. (Hz)

Mynatt's key points

Reference

Motorcycle nearby

‘Motorcycle engine sound’

–

–

–

Identifiability; Conceptual Mapping; User Preferences.

Wang et al. (2012)

Vehicle parking nearby

‘Car engine sound’

–

–

–

Identifiability; Conceptual Mapping; User Preferences.

Wang et al. (2012)

Vehicle moving nearby

‘Car passing sound’

–

–

–

Identifiability; Conceptual Mapping; User Preferences.

Wang et al. (2012)

Rain reported ahead

‘Thunder’

∼60.0 dB (SPL)

–

‘Single car horn twice’

∼60.0 dB (SPL)

–

Front hazard

‘Tires screeching’

∼60.0 dB (SPL)

–

Identifiability; User Preferences; Physical Characteristics. Identifiability; User Preferences; Physical Characteristics. Identifiability; User Preferences; Physical Characteristics.

Nees et al. (2016)

Side hazard

270–2200 M = 1130, SD = 72 270–2200 M = 1130, SD = 72 270–2200 M = 1130, SD = 72

a b

Nees et al. (2016) Nees et al. (2016)

Not described. The auditory icon was preceded by two chime tones of 450 ms.

icons was in informatics (Gaver, 1986, 1989). Gaver suggested that an auditory icon could convey information about a file, such as the type of file and its size (see also Section 3.1). Auditory icons have also been included in computer interfaces to enhance accessibility for partially sighted or blind users (Mynatt, 1997), to give information to people who are on the move in their workplace and temporarily cannot watch their computer screen (Mynatt et al., 1998), or to inform about a network attack (Garcia-Ruiz et al., 2008). Another use has been the support of soldiers monitoring military operations on computer displays (Haas and Schmidt, 1995; Wei and Kenny, 2009). Finally, auditory

icons have also been applied to enhance visual information present on interfaces in control rooms for surveillance video (Höferlin et al., 2011) and for industrial processes (Sirkka et al., 2014). In some studies on auditory icons in informatics, users were encouraged to participate in the identification of the most appropriate sound(s) for a given referent (Mynatt, 1994; Haas and Schmidt, 1995; Wei and Kenny, 2009; Höferlin et al., 2011; Sirkka et al., 2014). The degree of mapping between the referent and the sound(s) used as an auditory icon was typically based on the complexity of the referent. Less complex referents were easier to convey, e.g., a ‘typing’ sound to refer

Table 3 Auditory icons used in aviation. Referent Low fuel

Auditory icon ‘Car failing to start’

Dur. (ms)

Level

Freq. (Hz) a

1000

65.0–70.0 dB (A)

–

1000

–

–

Conflicting air traffic

‘Car brakes screeching’

1000

65.0–70.0 dB (A)

–

Carbon monoxide

‘Coughing’

1000

–

–

1000

65.0–70.0 dB (A)

–

1000

–

–

1000

65.0–70.0 dB (A)

–

Ground proximity

‘Explosion’

Electrical failure

‘Zapping sound’

1000

65.0–70.0 dB (A)

–

Aircraft icing

‘Cold wind blowing’

1000

65.0–70.0 dB (A)

–

Aircraft overweight

‘Elephant trumpeting’

1000

65.0–70.0 dB (A)

–

Engine fire

‘Fire engine siren’

1000

–

–

Ship Air interceptor Anti-aircraft artillery Missile launch Surface to air missile Search radar Unknown threat Chaff dispensed

‘Fog horn’ ‘Bird of prey attack call’ ‘Machine gun firing’ ‘Monkey screech’ ‘Arrow’ ‘Sonar beep’ ‘Bird territorial call’ ‘Camera flash & windchimes’

– – – – – – – –

– – – – – – – –

– – – – – – – –

a

Not described. 234

Mynatt's key points

Reference

Identifiability; Physical Characteristics. Identifiability; Physical Characteristics. Identifiability; Physical Characteristics. Identifiability; Physical Characteristics. Identifiability; Physical Characteristics. Identifiability; Physical Characteristics. Identifiability; Physical Characteristics. Identifiability; Physical Characteristics. Identifiability; Physical Characteristics. Identifiability; Physical Characteristics. Identifiability; Physical Characteristics. Identifiability. Identifiability. Identifiability. Identifiability. Identifiability. Identifiability. Identifiability. Identifiability.

Perry et al. (2007) Stevens et al. (2009) Perry et al. (2007) Stevens et al. (2009) Perry et al. (2007) Stevens et al. (2009) Perry et al. (2007) Perry et al. (2007) Perry et al. (2007) Perry et al. (2007) Stevens et al. (2009) Leung Leung Leung Leung Leung Leung Leung Leung

et et et et et et et et

al. al. al. al. al. al. al. al.

(1997) (1997) (1997) (1997) (1997) (1997) (1997) (1997)

Applied Ergonomics 78 (2019) 224–239

J.P. Cabral and G.B. Remijn

Table 4 Auditory icons used in mobile interaction. Referent

Auditory icon

Dur. (ms) a

Level

Freq. (Hz)

Mynatt's key points

Reference

Identifiability; User Preferences. Identifiability; User Preferences. Identifiability; User Preferences. Identifiability; User Preferences. Identifiability; User Preferences. Identifiability; User Preferences. Identifiability; User Preferences; Physical Characteristics. Identifiability; User Preferences.

Garzonis (2009) Garzonis (2009) Garzonis (2009) Garzonis (2009) Garzonis (2009) Garzonis (2009) Garzonis (2008)

Receiving a news

‘BBC News ID’

–

–

–

Receiving sports news

‘Stadium crowd’

–

–

–

Breaking news

–

–

–

Entertainment - downloads

‘Public announcement at an airport’ ‘20th Century Fox theme’

–

–

–

Entertainment - live

‘Audience applauding’

–

–

–

Incoming calls

‘Old-fashioned phone ringing’

–

–

–

500–4210 SD = 0.99

–

–

et al. et al. et al. et al. et al. et al. et al.

Incoming SMS

‘Message transmitted in Morse code’

–

–

–

Self-reminders

‘Windows mobile reminder theme’

–

–

–

Identifiability; User Preferences.

Garzonis et al. (2009)

Backup reminders

‘Truck/lorry reversing’

–

–

–

Identifiability; User Preferences.

Garzonis et al. (2009)

Other services

‘Wind chimes’

–

–

–

Identifiability; User Preferences.

Garzonis et al. (2009)

To inform the proximity of a garbage bin (GPS map)

‘Sound of plastic’

–

–

–

Xu et al. (2010)

‘Sound of paper-crunching’

–

–

–

‘Sound of a metal’

–

–

–

‘Leaves rustling sound’

–

–

–

‘Bird singing’

–

–

–

‘Sawing’

–

–

–

‘Lighting sound’

–

–

–

‘Electricity sound’

–

–

–

‘Collision sound’

–

–

–

‘Liquid dropping sound’

–

–

–

‘Liquid splashing sound’

–

–

–

‘Stream sound’

–

–

–

‘Horn’

–

–

–

‘Engine sound’

–

–

–

‘Tire screening’

–

–

–

‘Laughing’

–

–

–

Identifiability; Conceptual Mapping; User Preferences. Identifiability; Conceptual Mapping; User Preferences. Identifiability; Conceptual Mapping; User Preferences. Identifiability; Conceptual Mapping; User Preferences. Identifiability; Conceptual Mapping; User Preferences. Identifiability; Conceptual Mapping; User Preferences. Identifiability; Conceptual Mapping; User Preferences. Identifiability; Conceptual Mapping; User Preferences. Identifiability; Conceptual Mapping; User Preferences. Identifiability; Conceptual Mapping; User Preferences. Identifiability; Conceptual mapping; User Preferences. Identifiability; Conceptual Mapping; User Preferences. Identifiability; Conceptual Mapping; User Preferences. Identifiability; Conceptual Mapping; User Preferences. Identifiability; Conceptual Mapping; User Preferences.

To inform the proximity of a tree (GPS map)

To inform the proximity of traffic poles (GPS map)

To inform the proximity of a puddle (GPS map)

Vehicles approaching

Travelers approaching

Garzonis et al. (2009)

Xu et al. (2010) Xu et al. (2010) Xu et al. (2010) Xu et al. (2010) Xu et al. (2010) Xu et al. (2010) Xu et al. (2010) Xu et al. (2010) Xu et al. (2010) Xu et al. (2010) Xu et al. (2010) Xu et al. (2010) Xu et al. (2010) Xu et al. (2010) Xu et al. (2010)

(continued on next page)

235

Applied Ergonomics 78 (2019) 224–239

J.P. Cabral and G.B. Remijn

Table 4 (continued) Referent

Turn on television

Stream of data

a

Auditory icon

Dur. (ms)

Level

Freq. (Hz)

Mynatt's key points

‘Footsteps’

–

–

–

‘Coughing’

–

–

–

‘Sound of a TV switching on to white noise’

500 - 4210 SD = 0.99

–

–

Identifiability; Conceptual Mapping; User Preferences. Identifiability; Conceptual Mapping; User Preferences. Identifiability; Conceptual Mapping; User Preferences. Identifiability;

‘Consistent dripping sound’

500 - 4210 SD = 0.99

–

–

User Preferences; Physical Characteristics. Identifiability; User Preferences; Physical Characteristics.

Reference

Xu et al. (2010) Xu et al. (2010) Garzonis et al. (2008) Garzonis et al. (2008)

Not described.

failing to start’ sound to represent a low-fuel referent, or an ‘elephant trumpeting’ sound to signal that an aircraft was overweight. Table 3 shows the acoustic characteristics of auditory icons used in aviation. The three studies from the aviation industry provided a total of 16 different referents, represented by 16 auditory icons. Two studies mentioned the sound duration and one of them also mentioned sound level (Table 3).

to a text field, or a ‘crunch’ paper sound to refer to emptying the trashcan (Table 1). Complex referents (object, events, or situations difficult to represent by an auditory icon) were often represented by less obvious sounds, e.g., a ‘keystroke’ sound to refer to selecting an option from the menu, or a ‘submarine radar ping’ sound to refer to a suspicious individual who should be questioned. Please note in Table 1 that different sounds have been used to inform the same referent across some studies, indicating that there sometimes was no clear design consensus. For example, a ‘whistle up’ sound, a ‘motorcycle’ sound, a ‘whooshing’ sound, or a ‘spaceship door opening’ sound have all been used to refer to the referent of opening a file on a computer. The 10 studies selected in the field of informatics described a total of 56 different referents, represented by 77 auditory icons. Just three studies mentioned the physical properties of the auditory icons. More specifically, two studies described only the icon's duration – yet imprecisely (“under 2000 ms”). The icons' sound levels were also mentioned by two studies, while one study mentioned the frequency ranges.

5.5. Auditory icons in mobile interaction – Table 4 Studies on mobile interaction have investigated the use of auditory icons as a means of notification on smartphones for text messages, news, or backup reminders (Garzonis et al., 2008; Garzonis et al., 2009; Xu et al., 2010). According to these studies, auditory icons positively contributed to learnability, memorability, and intuitiveness of the referent. The three studies on mobile interaction have covered 18 different referents, represented by 30 auditory icons. Just one study (Garzonis et al., 2008) mentioned the duration range of the auditory icons (between 500 and 4210 ms). Information about sound level and frequency range was not mentioned.

5.3. Auditory icons in the automotive industry – Table 2 In the automotive industry, auditory icons have been used to inform drivers about the condition of their vehicle, e.g., the tire pressure, or fuel level. Advanced driver-assistance systems (ADAS) can prevent drivers’ misbehaviors, such as forgetting to lock the door, drive without fastening the seatbelts, falling asleep behind the wheel, deviating from a straight route, or speeding. In addition, the use of ADAS can prevent accidents through anti-collision systems, e.g., by signaling the presence of pedestrians on the street or vehicles in the blind spot. Similar to certain cases in the informatics literature, also in the automotive industry different auditory icons have been designed to represent the same referent across different studies (Graham, 1999; Isherwood and Mckeown, 2016; McKeown and Isherwood, 2007; Mckeown et al., 2010). The 13 studies from the automotive industry described a total of 34 different referents, represented by 46 auditory icons. Eleven articles mentioned the sound duration of the auditory icons, and 8 of them mentioned also the sound level. Information about the frequency ranges of the auditory icons was mentioned in just 3 studies. The average auditory icon duration was 2624 ms (SD = 1655), over a range of 270–7630 ms.

5.6. Auditory icons in other fields Besides the four fields summarized in the tables, auditory icons have also been used in the operation of medical equipment and homecare systems. Most studies in these fields showed the positive contributions of auditory icons to user functioning. Studies on medical equipment have investigated the learnability, identification, and localization of different medical indices such as heart beat, respiration, temperature, or drug delivery alerts with the use of auditory icons. Compared to abstract alarms, speech messages, and traditional auditory alarms on medical equipment, auditory icons induced high learnability, even in users without a technical medical background (Edworthy et al., 2014, 2017). During dual tasks, the disruptive effect of auditory icons, speech messages, earcons, and visual, tactile, and olfactory warnings have been evaluated (Warnock et al., 2011). The results showed that when auditory icons and other warning types are unwanted by the participants, they induce mistakes during the execution of a secondary task. The two studies on medical equipment alarms described 17 different referents, represented by 17 auditory icons, such as ‘cardiac alert’ (Edworthy et al., 2014) and ‘patient's temperature’ referents (Edworthy et al., 2017). The average duration of these auditory icons was 2072 ms (SD = 778). Sound level was described as varying between 75.0 and 80.0 dB (A). The only study on homecare systems described 3 different referents, represented by 3 auditory icons, such as ‘heating’, ‘lights’, and

5.4. Auditory icons in aviation – Table 3 From the descriptions in the research so far, auditory icons in civil and military aviation typically had a metaphoric relationship with the referent. For example, Perry et al. (2007) used auditory icons like a ‘car 236

Applied Ergonomics 78 (2019) 224–239

J.P. Cabral and G.B. Remijn

‘telephone’ (Warnock et al., 2011). The 3 auditory icons each had a duration of 1000 ms. Information about the sound pressure level and the frequency ranges of the auditory icons was not mentioned.

1000–2000 ms should suffice. It needs to be noted, though, that urgent matters are often signaled with general auditory warnings consisting of repetitive simple sounds. Short inter-sound intervals are considered more urgent (Edworthy et al., 1995). The frequency range covered by the auditory icons, if described in the articles reviewed here, was roughly in between 50 and 12000 Hz. According to the International Organization for Standardization (ISO 7731, 2003), general auditory warnings for public and work areas should at least cover a range of 300–3000 Hz. Behavioral research has for example shown that lane departure warning systems with 1750 Hz and 3000 Hz sounds induce faster reaction times (Nees and Walker, 2011). Given that human hearing is most acute within this range (Suzuki and Takeshima, 2004), and hearing loss due to aging usually affects higher frequencies, auditory icon design also should focus on the 300–3000 Hz range, e.g. by making sure that the sound energy level is sufficient in this range. The level of presentation of auditory icons reviewed here, if given, was 57–85 dB(A). In an applied situation, the icon's level can influence its interpretation when presented among other sounds. In a general context, the signal-to-noise ratio for an auditory warning should be of 15–20 dB(A) (Begault et al., 2007). However, different environments and users may require a different signal-to-noise ratio. For example, for listeners with normal hearing auditory warnings should be presented at least 6 dB above any background sound. For listeners even with a slight hearing impairment, however, often at least 15 dB is necessary (Baldwin, 2002). In addition, the level of presentation of auditory warnings also affects the perception of urgency. For example, auditory warnings presented at 90 dB (A) were considered more urgent compared to others presented at 75 dB (A) (Momtahan, 1990). When presented against a normal background environment, a presentation level of 60–70 dB SPL should suffice for auditory icons. This presentation level range is similar to that commonly used in behavioral studies on complex sounds, such as speech. In any case we suggest that, given the general lack of design consensus and the lack of information about the physical characteristics of auditory icons, researchers either make their icon(s) available to the research and design community or show a spectrogram or the temporal envelope of the sound(s) used in an auditory icon. Such physical/ acoustic characteristics are indispensable for the replicability of (future) auditory icon studies. Finally, fruitful lines of research would be to investigate whether the perception of the physical characteristics of the same auditory icon changes according to factors such as the environment in which they are delivered, individual differences among users, and/or even the users' cultural background. For example, a preliminary study (Cabral et al., 2017) suggests that the perception of sound(s) in auditory icons used in automotive studies differs with the listeners’ cultural backgrounds. The potentially beneficial effect of auditory icons on visual information processing also deserves more attention. We hope this review can be an effective reference for all who seek a better understanding of how auditory icons have been applied and designed so far.

5.7. Consideration of design key points in the reviewed literature From the 31 articles written after Mynatt (1994), just three articles mentioned all four of Mynatt's design key points (Section 4.3). The other articles dealt with three key points or less. Authors discussed the key point about the identifiability of auditory icons in 28 articles. User preferences were discussed in 22 articles, (some) physical characteristics of auditory icons were mentioned in 20 articles, while the key point about conceptual mapping was considered in 11 articles. As mentioned in Section 4.3, following Mynatt's key points is important to avoid problems with the identification of an auditory icon, regardless of the listening context or the typology of the user. Regarding design methodology (Section 4.3.5), a striking problem in general is the poor description of the physical characteristics of the sound(s) used as icon. Even if some description was given, many of the reviewed articles lack precise description of the physical/acoustic characteristics of the sound(s) (Mynatt's Step I). Twenty-nine studies were covered in Tables 1–4, but they show a lot of empty entries for sound duration, level and frequency. Overall, 55.2% of the articles provided some information about the duration of the auditory icon. The percentage of studies that mentioned the sound level in dB (SPL, Aweighted, or Linear-weighted) was just 37.9%. Surprisingly, only 10.3% of the studies indicated frequencies or frequency ranges of the sound(s) used in an auditory icon. 6. General discussion and conclusion We reviewed the literature on a specific type of auditory warnings: the auditory icon. An auditory icon is a short sound message that informs the listener about an object, event, or situation, which is called the referent. The purpose of this review was to highlight how auditory icons have been designed and used in different areas over the last three decades, and to give a description of the sounds used to represent their referents. The review covers the explanation of the concepts, classification, advantages, and disadvantages behind auditory icon design. In order to understand how auditory icons have been designed, the most cited, relevant articles were selected by means of a search in an academic database and a journal search. The review shows that auditory icons were studied in informatics, the automotive industry, the aviation industry, mobile interaction, and fields regarding the operation of medical equipment and homecare systems. Tables 1–4 lists the auditory icons described in the reviewed literature in terms of the referents they represent and their physical characteristics. The main finding is that surprisingly few studies provided a precise description of the physical characteristics of the sound(s) used as auditory icons. Consideration of acoustic characteristics was Mynatt's (1994) first design guideline for auditory icons. With regard to the acoustic characteristics of auditory icons, we can say the following. The auditory icons reviewed here ranged from 270 to 7630 ms in duration. Psychophysical research has shown that auditory warnings, in general, should be at least 200 ms in duration, because shorter sounds are difficult to identify (Azrin, 1958; Reed and Yoshino, 2001). Since auditory icons consist of natural, every-day sound(s), however, it is recommendable that they have a minimum duration of at least 400 ms. Natural everyday sounds require a duration of at least 400 ms for correct interpretation (Fabiani et al., 1996). The desired maximum duration of an auditory icon likely depends on its purpose. Auditory icons with a relatively long duration could be suitable to inform referents that do not require an immediate action by the user, e.g., informing users that a new email has arrived (Table 1; Mynatt et al., 1998). However, since icons with a relatively long duration could postpone user reaction, in cases where a reaction is urgent a maximum duration of

Declaration of interest The authors declare that they have no competing interests. Acknowledgments The authors would like to thank the editor-in-chief/scientific editor and the anonymous reviewers for their time reading and making valuable comments and suggestions to improve the quality of this manuscript. References Alseid, M., Azzeh, M., Sheikh, Y.E., 2014. A comparative usability study on the use of

237

Applied Ergonomics 78 (2019) 224–239

J.P. Cabral and G.B. Remijn auditory icons to support virtual lecturers in e-learning interfaces. Int. J. Adv. Comput. Sci. Appl. 5 (1), 88–95. Azrin, N.H., 1958. Some effects of noise on human behavior. J. Exp. Anal. Behav. 1, 183–200 [PubMed: 16811219]. In Bellettiere, J., Hughes, S.C., Liles, S., BomanDavis, M., Klepeis, N., Blumberg, E., Mills, J., Berardi, V., Obayashi., S., Allen, T. T. & Hovell, M. F. (2014). Developing and Selecting Auditory Warnings for a Real-Time Behavioral Intervention. American Journal of Public Health Research, 1(6): 232-238. doi:10.12691/ajphr-2-6-3. Bakowski, D.L., Davis, S.T., Moroney, W.F., 2015. Reaction time and glance behavior of visually distracted drivers to an imminent forward collision as a function of training, auditory warning, and gender. Proc. Manufact. (3), 3238–3245. Baldwin, C.L., 2002. Designing in-vehicle technologies for older drivers: application of sensory-cognitive interaction theory. Theor. Issues Ergon. Sci. 3 (4), 307–329. Ballas, J.A., Sliwinski, M.J., Harding, J.P., 1986. Uncertainty and responses time in identifying non-speech sounds. J. Acoust. Soc. Am. 79, S47. Retrieved from. https:// doi.org/10.1121/1.2023244. Beattie, D., Baillie, L., Halvey, M., 2015. A comparison of artificial driving sounds for automated vehicles. In: Proceedings of the 2015 ACM International Joint Conference on Persuasive and Ubiquitous Computing, pp. 451–462. https://doi.org/10.1145/ 2750858.2807519. Begault, D.,R., Godfroy, M., Sandor, A., Holden, K., 2007. Auditory alarm design for NASA CEV applications. In: Proceedings of the 13th International Conference on Auditory Display, pp. 131–138. Retrieved from. https://smartech.gatech.edu/ handle/1853/49971. Belz, S.M., Robinson, G.S., Casali, J.G., 1999. A new class of auditory warning signals for complex systems: auditory icons. Hum. Factors: J. Hum. Factors Ergonom. Soc. 41 (4), 608–618. Retrieved from. https://doi.org/10.1518/001872099779656734. Brazil, E., Fernström, M., 2011. Auditory icons. 325-338. In: Hermann, T., Hunt, A., Neuhoff, J. (Eds.), The Sonification Handbook. Logos Verlag Berlin (2011). Buxton, W., Gaver, W., Bly, S., 1994. Auditory icons. Auditory interfaces: the use of nonspeech audio at the interface. Unfinished book manuscript. Retrieved from: http:// www.billbuxton.com/AudioUI06icons.pdf. Cabral, J.,P., Mendonça, R., M L, O., Cabral, J.A., 2015. Synesthetic metering for speed evaluation applied to young drivers' speeding. In: Proceedings of the Sixth International Conference on Applied Human Factors and Ergonomics, vol. 3. pp. 6198–6205. Retrieved from. https://doi.org/10.1016/j.promfg.2015.07.917. Cabral, J.P., Nakajima, Y., Ueda, K., Remijn, G.B., 2017. The perception of auditory icons by Japanese drivers. In: Ueda, K., Ito, H., Remijn, G.B., Nakajima, Y. (Eds.), Fechner Day 2017: Proceedings of the 33rd Annual Meeting of the International Society for Psychophysics. International Society for Psychophysics, Fukuoka, Japan, pp. 125. Chi, C.F., Dewi, R.S., Surbakti, Y.Y., Hsieh, D.Y., 2017. The perceived quality of in-vehicle auditory signals: a structural equation modelling approach. Ergonomics 60 (11), 1471–1484. Edworthy, J., Hellier, E.J., Hards, R., 1995. The semantic associations of acoustic parameters commonly used in the design of auditory information and warning signals. Ergonomics 38 (11), 2341–2361. In Palomäki, H. (2006). Meanings conveyed by simple auditory rhythms. Proceedings of the 12th International Conference on Auditory Display, 99-104. Retrieved from https://smartech.gatech.edu/handle/ 1853/50599. Edworthy, J., Page, R., Hibbard, A., Kyle, S., Ratnage, P., Claydon, S., 2014. Learning three sets of alarms for the same medical functions: a perspective on the difficulty of learning alarms specified in an international standard. Appl. Ergon. 45 (5), 1291–1296. Retrieved from. https://doi.org/10.1016/j.apergo.2013.10.003. Edworthy, J., Reid, S., McDougall, S., Edworthy, J., Hall, S., Bennett, D., Pye, E., 2017. The recognizability and localizability of auditory alarms: setting global medical device standards. Hum. Factors: J. Hum. Factors Ergonom. Soc. 59 (7), 1108–1127. Retrieved from. https://doi.org/10.1177/0018720817712004. Fabiani, M., Kazmerski, V.A., Cycowicz, Y.M., Friedman, D., 1996. Naming norms for brief environmental sounds: effects of age and dementia. Psychophysiology 33, 462–475 In Marcell, M., M., Borella, D., Greene, D., Kerr, E. & Rogers, S. (2000). Confrontation Naming of Environmental Sounds. Journal of Clinical and Experimental Neuropsychology, 22(6), 830-864. Fagerlönn, J., 2011. Making auditory warning signals informative: examining the acceptance of auditory icons as warning signals in trucks. In: Proceedings of the Sixth International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design, pp. 95–101. Retrieved from https://doi.org/10.17077/ drivingassessment.1383. Familant, M.E., Detweiler, M.C., 1993. Iconic reference: evolving perspectives and an organizing framework. Int. J. Man Mach. Stud. 39, 705–728. In Keller, P., & Stevens, C. (2004). Meaning From Environmental Sounds: Types of Signal-Referent Relations and Their Effect on Recognizing Auditory Icons. Journal of Experimental Psychology: Applied, 10(1), 3–12. Retrieved from https://doi.org/10.1037/1076-898X.10.1.3. Fricke, N., Thüring, M., 2009. Complementary audio-visual collision warnings. In: Human Factors and Ergonomics Society Annual Meeting Proceedings, vol. 53. pp. 1815–1819. Frimalm, R., Fagerlönn, J., Lindberg, S., Sirkka, A., 2014. How many auditory icons, in a control room environment, can you learn? In: The 20th International Conference on Auditory Displays, Retrieved from. https://smartech.gatech.edu/handle/1853/ 52084. Garcia-Ruiz, M.,A., Edwards, A., Martin, M.,V., Seoud, S.E., 2008. Auditory display as a tool for teaching network intrusion detection. Int. J. Emerg. Technol. Learn. 2 (3), 59–62. Retrieved from. http://online-journals.org/i-jet/article/view/202. Garzonis, S., Bevan, C., O'Neill, E., 2008. Mobile service audio notifications: intuitive semantics and noises. In: Proceedings of the 20th Australasian Conference on Computer-Human Interaction: Designing for Habitus and Habitat, pp. 156–163. Retrieved from. https://doi.org/10.1145/1517744.1517793.

Garzonis, S., Jones, S., Jay, T., O'Neill, E., 2009. Auditory icon and earcon mobile service notifications. In: Proceedings of the 27th International Conference on Human Factors in Computing Systems - CHI 09, pp. 1513. Retrieved from. https://doi.org/10.1145/ 1518701.1518932. Gaver, W.W., 1986. Human – Computer Interaction Auditory Icons : Using Sound in Computer Interfaces Auditory Icons : Using Sound in Computer Interfaces. pp. 37–41 December 2011. Gaver, W.W., 1989. The SonicFinder: an interface that uses auditory icons. Hum. Comput. Interact. 4 (1), 67–94. Retrieved from. https://doi.org/10.1207/ s15327051hci0401_3. Gaver, W.W., Smith, R.B., O'Shea, T., 1991. Effective sounds in complex systems: the Arkola simulation. Proceedings of CHI 1991, 85–90. Retrieved from. https://doi.org/ 10.1145/108844.108857. Graham, R., 1999. Use of auditory icons as emergency warnings: evaluation within a vehicle collision avoidance application. Ergonomics 42 (9), 1233–1248. Retrieved from. https://doi.org/10.1080/001401399185108. Gibson, J.J., 1979. The Ecological Approach to Visual Perception, Classic Edition. Psychology Press, New York. Haas, E.C., Schmidt, J., 1995. Auditory icons as warning and advisory signals in the U.S. Army battlefield combat identification system (BCIS). In: Proceedings of the Human Factors and Ergonomics Society 39th Annual Meeting, pp. 999–1003. Höferlin, B., Höferlin, M., Raschke, M., Heidemann, G., Weiskopf, D., 2011. Interactive auditory display to support situational awareness in video surveillance. In: The 17th International Conference on Auditory Display, pp. 156–163. Retrieved from. https:// smartech.gatech.edu/handle/1853/44426. International Organization for Standardization, 2003. Ergonomics – Danger signals for public and work areas – Auditory danger signals (ISO 7731:2003). In: Begault, D.,R., Godfroy, M., Sandor, A., Holden, K. (Eds.), Auditory Alarm Design for NASA CEV Applications. Proceedings of the 13th International Conference on Auditory Display, pp. 131–138. (2007) Retrieved from. https://smartech.gatech.edu/handle/1853/ 49971. Isherwood, J.S., Mckeown, D., 2016. Semantic congruency of auditory warnings. Ergonomics 60 (7), 1014–1023. Jacko, J.A., 1996. The identifiability of auditory icons for use in educational software for children. Interact. Comput. 8 (2), 121–133. Retrieved from. https://doi.org/10. 1016/0953-5438(96)01023-5. Keller, P., Stevens, C., 2004. Meaning from environmental sounds: types of signal-referent relations and their effect on recognizing auditory icons. J. Exp. Psychol. Appl. 10 (1), 3–12. Retrieved from. https://doi.org/10.1037/1076-898X.10.1.3. Larsson, P., Opperud, A., Fredriksson, K., Vaestfjaell, D., 2009. Emotional and behavioural response to auditory icons and earcons in driver-vehicle interfaces. In: Proceedings of the Twenty-First International Technical Conference on Enhanced Safety of Vehicles, 09-0104. Leung, Y.K., Smith, S., Parker, S., Martin, R., 1997. Learning and retention of auditory warnings. In: Proceedings of the 3rd International Conference on Auditory Display, pp. 129–133. Retrieved from. https://smartech.gatech.edu/handle/1853/50748. McKeown, D., Isherwood, S., 2007. Mapping candidate within-vehicle auditory displays to their referents. Hum. Factors: J. Hum. Factors Ergonom. Soc. 49 (3), 417–428. Retrieved from. https://doi.org/10.1518/001872007X200067. Mckeown, D., Isherwood, S., Conway, G., 2010. Auditory displays as occasion setters. Hum. Factors: J. Hum. Factors Ergonom. Soc. 52 (1), 54–62. Retrieved from. https:// doi.org/10.1177/0018720810366861. Momtahan, K.L., 1990. Mapping of Psychoacoustic Parameters to the Perceived Urgency of Auditory Warning Signals. Unpublished master's thesis. Carleton University, Ottawa, Ontario In Baldwin, C. L. (2012). Auditory Cognition and Human Performance. Research and Applications. CRC Press. USA. 244. Mynatt, E.D., 1994. Designing with auditory icons. In: Proceedings of the Conference on Auditory Display (ICAD 1994), pp. 109–119. Retrieved from http://hdl.handle.net/ 1853/50842. Mynatt, E., 1997. Transforming graphical interfaces into auditory interfaces for blind users. Hum. Comput. Interact. 12 (1), 7–45. Retrieved from. https://doi.org/10. 1207/s15327051hci1201&2_2. Mynatt, E.D., Back, M., Want, R., Baer, M., Ellis, J.B., 1998. Designing audio aura. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ’98. pp. 566–573. Retrieved from. https://doi.org/10.1145/274644.274720. Nees, M.A., Walker, B.,N., 2011. Auditory displays for in-vehicle technologies. Rev. Hum. Factors Ergonom. 1 (7), 58–99. Retrieved from. https://doi.org/10.1177/ 1557234X11410396. Nees, M.A., Helbein, B., Porter, A., 2016. Speech auditory alerts promote memory for alerted events in a video-simulated self-driving car ride. Hum. Factors: J. Hum. Factors Ergonom. Soc. 58 (3), 416–426. Retrieved from. https://doi.org/10.1177/ 0018720816629279. Nielsen, J., 1993. Usability Engineering. Morgan Kaufmann, USA. Norman, D.A., 1988. The Design of Everyday Things. Basic Books Inc., Publishers, New York. Ogden, C.K., Richards, I.A., 1923. The Meaning of Meaning. A Study of the Influence of Language upon Thought and of the Science of Symbolism. Harcourt, Brace & World, Inc., New York. Perry, N.C., Stevens, J.C., Wiggins, M.W., Howell, C.E., 2007. Cough once for danger: icons versus abstract warnings as informative alerts in civil aviation. Hum. Factors 49 (6), 1061–1071. Petrie, H., Morley, S., 1998. The use of non-speech sounds in non-visual interfaces to the MS-Windows GUI for blind computer users. In: Proceedings of the 5th International Conference on Auditory Displays, pp. 1–5. Retrieved from. https://smartech.gatech. edu/handle/1853/50734. Reed, P., Yoshino, T., 2001. The effect of response-dependent tones on the acquisition of

238

Applied Ergonomics 78 (2019) 224–239

J.P. Cabral and G.B. Remijn concurrent behavior in rats. Learn. Motiv. 32, 255–273. Retrieved from. https://doi. org/10.1006/lmot.2000.1074 In Bellettiere, J., Hughes, S.C., Liles, S., Boman-Davis, M., Klepeis, N., Blumberg, E., Mills, J., Berardi, V., Obayashi., S., Allen, T. T. & Hovell, M. F. (2014). Developing and Selecting Auditory Warnings for a Real-Time Behavioral Intervention. American Journal of Public Health Research, 2(6): 232–238. doi:10.12691/ajphr-2-6-3. Rigas, D., Algahtani, A., 2014. The role of expressive full-body avatars and earcons and auditory icons in e-assessment interfaces. In: Recent Advances in Electrical Engineering and Educational Technologies, pp. 63–69. Retrieved from http://inase. org/library/2014/athens/bypaper/ELECT/ELECT-10.pdf. Sanders, M.,S., McCormick, E.,J., 1993. Human Factors in Engineering and Design. McGraw-Hill, INC. Singapore. Sirkka, A., Fagerlönn, A., Delsing, K., Lindberg, S., 2014. The design of an auditory alarm concept for a paper mill control room. In: Advances in Ergonomics in Design, Usability & Special Populations: Part III, 188-129. Stanton, N., Edworthy, J., 1998. Auditory affordances in the intensive treatment unit. Appl. Ergon. 29 (5), 389–394. Ttephan, K.L., Smith, S.E., Martin, R.L., Parker, S.P.A., McAnally, K.I., 2006. Learning and retention of associations between auditory icons and denotative referents: implications for the design of auditory warnings. Hum. Factors: J. Hum. Factors Ergonom. Soc. 48 (2), 288–299. Retrieved from https://doi.org/10.1518/ 001872006777724426. Stevens, C., Brennan, D., Parker, S., 2004. Simultaneous manipulation of parameters of auditory icons to convey direction, size, and distance: effects on recognition and interpretation. In: Proceedings of the 10th International Conference on Auditory Display, Retrieved from. https://smartech.gatech.edu/handle/1853/50916. Stevens, C.J., Brennan, D., Petocz, A., Howell, C., 2009. Designing informative warning signals: effects of indicator type, modality, and task demand on recognition speed and accuracy. Adv. Cognit. Psychol. 5 (4), 84–90. https://doi.org/10.2478/v10053-0080064-6. Suzuki, Y., Takeshima, H., 2004. Equal-loudness-level contours for pure tones. J. Acoust. Soc. Am. 116, 918–933. Retrieved from https://asa.scitation.org/doi/10.1121/1. 1763601. Warnock, D., McGee-Lennon, M.R., Brewster, S., 2011. The impact of unwanted multimodal notifications. In: Proceedings of the 13th International Conference on Multimodal Interfaces - ICMI ’11, pp. 177. Retrieved from https://doi.org/10.1145/ 2070481.2070510. Wang, M.,J., Li, Y.,C., Chen, F., 2012. How can we design 3D auditory interfaces which enhance traffic safety for Chinese drivers? In: Proceedings of the 4th International Conference on Automotive User Interfaces and Interactive Vehicular Applications

(AutomotiveUI '12), pp. 77–83. https://doi.org/10.1145/2390256.2390268. Wei, C.L., Kenny, C.W.L., 2009. An evaluation of icon, tone and speech cues for detecting anomalies and auditory communications for onboard military vehicle displays. In: Human Factors and Ergonomics Society Annual Meeting Proceedings, pp. 1076–1080. Retrieved from. https://doi.org/10.1518/ 107118109X12524443344835 53. Winters, J.J., 1998. An Investigation of Auditory Icons and Brake Response Times in a Commercial Truck-Cab Environment. Virginia Polytechnic Institute and State University Unpublished Master of Science Thesis. Retrieved from. https:// vtechworks.lib.vt.edu/handle/10919/36714?show=full. Xu, J., Fang, Z.G., Dong, D.H., Zhou, F., 2010. An outdoor navigation aid system for the visually impaired. In: IEEM2010 - IEEE International Conference on Industrial Engineering and Engineering Management, pp. 2435–2439. Retrieved from https:// doi.org/10.1109/IEEM.2010.5674377.

Abbreviations ADAS: Advanced Driver-Assistance Systems BBC: British Broadcasting Corporation dB (A): decibel A-weighted dB (Lin): decibel Linear-weighted dB (SPL): decibel Sound Pressure Level Dur: Duration in milliseconds Freq: Frequency range in Hertz GPS: Global Positioning System Hz: Hertz ISO: International Organization for Standardization ITU: Intensive Treatment Unit ms: milliseconds PDA: Personal Digital Assistant rcp: rate control protocol rlogin: remote login rsh: remote shell SD: Standard Deviation SJR: SCImago Journal Ranking Indicator SMS: Short Message Service TV: Television

239