Audio in Modern Computing

With all the focus today on creating modern and more interactive interfaces with the computing platforms we use on a daily basis, it seems to me that we are focusing mostly on the senses of touch and sight, and neglecting the roll that sound could play in the way we use our computing devices.

I realize that most devices are targeted to work with a large variety of consumers, thus allowing someone to use a PC without needing to be able to experience audio-specific cues, but I have to wonder what sort of new ideas could come out of modern interface design if developers were allowed to use audio in a more functional manner.

As it is now, most software relies on audio as either a multimedia feature, or simply for system-level notifications.  With the general exception of video games, most software and operating systems are relegated to this realm of integration with regards to audio.

What Video Games Can Teach Us About Interface Design

Oh Shit!

Oh Shit!

When a player is engrossed in a new video game, they are treated to a variety of both visual and auditory cues that provide a deep immersion in the content.  In a game such as Left 4 Dead, the player knows what certain weapons sound like, and are provided with various ambient environmental clues which enhance the gameplay that cannot be simply understood by a player who has muted the audio.

When a Smoker is around, you hear coughing; a Boomer provides a sickening vomiting sound; and the presence of a Witch is hinted at with the creepy singing sounds, alerting the player to her presence.  When approaching the Witch, a player can tell whether she is oblivious to his presence, and likewise when she becomes agitated as the player is coming too close to her.

These cues are indispensable in a game environment where you are meant to feel like you are right there in the action, in a living and breathing world.  These cues can create feelings of dread, happiness, calm or any other emotion that a player could feel in certain situations.

While playing Left4Dead is nothing like creating spreadsheets or browsing the web, I think we can learn quite a bit from games in the arena of audio design, and how the overall environment not only relies on audio, but also excels in presenting immersion in even the slightest details.

Modern Touch Technology

Apple iPhone

Apple iPhone

Visuals in computer interface design are nothing new, and have been around since the blinking lights on the earliest room-sized calculating devices in the 40s.  Though traditionally a user can type and move a mouse to navigate an interface using their sense of touch, newer technology has given us a new paradigm of interaction through our fingers and even through the location and orientation of the device which we are using.

These modern visceral technologies, termed “multi-touch” are beginning to allow us to perform natural gestures using our hands to interact with virtual objects that were once relegated to the realm of the mouse and keyboard abstraction.

This level of interaction allows users to eschew the unnatural interface of the keyboard and mouse, and use the input devices that were provided to them from birth.  Allowing your users to manipulate virtual objects with the coordination that they have built with their hands since childhood could mean a quicker adoption of the abstract virtual world they are using.

As devices which rely on multi-touch interaction begin to proliferate, one has to question where the future lies after these interfaces are perfected and become more mainstream.

The Five Senses

When you think about the five sense and how they are used in modern computing, we have the visual and the touch down to a science.  Though multimedia applications such as listening to music or watching video use our sense of hearing, I have to think that we could be moving into a future reality where these three senses work together to provide a superior environment to get our work done and aid in many other educational and scientific pursuits.

Our current technological level excludes the use of smell and taste as a medium for interacting with computer interfaces.  For the purposes of this discussion, I am going to exclude those as possibilities in the future of computing.

Where a video game allows us to use our senses of sight, hearing and touch to interact with a virtual world, I feel like our normal operating environments are lacking in the are of our sense of hearing.

Sure, we get rich audio and video for entertainment purposes, but where is the level of interaction that we see in the virtual worlds of video games?

I can see two issues with using audio as a main interface mechanism in modern computing platforms:

  • What about people with hearing disabilities?
  • Is it really necessary?

Those With Hearing Disabilities
I understand the concern that backs the concept of disregarding audio in the realm of computing due to the reduced market for such products.  Even though this is a valid concern, I have to think of how current systems with interactive measures for those with visual disabilities work.

A person without the ability to see, although hindered in a more general sense, is able to use a computer through the aid of reading programs and other assistive technologies.  While they potentially cannot experience the virtual world before them with the same clarity that people without a disability can, the fact remains that the visual elements of an interface are not absolutely required to harness the information that can be found through the web and other interactive interfaces.

Modern multimedia makes up for hearing disabilities by providing subtitles to movies and other video sources.  While this may not be the full experience that the original artist wanted for the viewer, the technology behind these devices allows those who would otherwise be relegated to a muted experience to understand what is being said and heard by an audience without such disabilities.

I would imagine a computing interface that uses audio heavily to immerse a user in their virtual environment would provide a similar solution in order to allow those with hearing disabilities to use the system with minimal loss of experience.

Is It Really Necessary?
No, it is not absolutely necessary to integrate audio into every aspect of a user’s interaction with their virtual environment.  But it sure could make things a different experience for those willing to adopt the new technology.

Some Ideas

Talking about using audio as a stronger interface with modern computing is one thing, but actually providing an example of how using audio could create a new way of interacting with a virtual environment is another.

Thinking in a new dimension such as this can be pretty difficult, so some of my ideas may be a little rough around the edges.  It would take some time to sit and brainstorm more elegant uses for this sort of technology.  Trying to get away from the notion that audio is currently only being used as either multimedia or notifications, where can we go with this idea?

Let’s consider a situation where you have a multi-touch interface on a tablet PC (come on Apple, make it already!).  One use of audio in this case would be to provide feedback to the user in the form of buzzing or similar sound effects when a user encounters a boundary.

Take for instance if the user were to move a window to the edge of the screen, and instead of just having an onscreen snap, the user would also hear a snap or the sound of bumping, as if the window were actually bumping against the edge of the screen.  Similarly, when interacting with a photograph, and moving it around and arbitrarily rotating it, how about the sound of paper moving across the top of a wooden surface?

I enjoy the use of window snapping functionality, which comes with Gnome, and is available in Windows (using the AllSnap program).  I think it would add a bit of an extra flair if a sound effect were associated with the action of two windows bumping into each other, or a window being snapped to the side of the screen.

Of course, these sound effects could get old very quickly, and I’m sure there are plenty of ramifications that I’m not foreseeing here, but I think that as a base idea, audio needs to be used in a more fundamental way in modern interface design.

This was just a quick idea, and there weren’t any wireframe mockups or anything like that on my part.  I just wanted to put it out there to the universe, and see if someone can make something of it.

  • Russell

    Very nice idea. I liked the reference to Left4Dead and how you mentioned the audio cues. I’d have to say that the use of it in OS’s though would probably by annoying but I would see the future using audio more and more. Me personally, I would automatically turn off some of the annoying audio cues. Eventually I’d find it usual. Good points.

  • Benjamin M. Strozykowski

    I think I have a bit of a bitter taste in my mouth from the crap that goes on inside Windows regarding audio cues. The whole Explorer “clicking” thing just drives me bananas.

    Obviously, any kind of modification to the way we use sound in our operating systems would be editable preferences, which wouldn’t be absolutely necessary to use the system.