Multimodal - Annotated Bibliography

With entries by Yaro Brock and David Hruska

Jones, Matt (2006). Mobile Interaction Design. Hoboken, NJ: John Wiley & Sons. 3-37

Matt Jones assesses the current state of interaction design methods for electronic devices and discusses their strengths and weaknesses. He then moves forward to discuss possible interfaces for future designs, such as: haptic interfaces, gestural interfaces, tilt-based systems, and multimodality. He recognizes the need for innovation in the portable electronic space.

Lynne Baillie, et al. (July 22-27, 2005) Designing Mona: User Interactions with Multimodal Mobile Applications. HCI International 2005, 11th International Conference on Human-Computer Interaction, Las Vegas, Nevada, USA.

The important finding in this article is that users find a greater need for certain modalities depending on their context, and that multimodality does improve mobile device experience. This revelation was revealed while designing a mobile multimodal game. They focused on the possible use of speech interaction in multiple situations.

Chandler, C., Lo, G., & Sinha, A. (2002). Multimodal Theater: Extending Low Fidelity Paper Prototyping to Multimodal Applications. Conference on Human Factors in Computing Systems (CHI), Student Posters (pp. 874-875). ACM Press.

This is a report of how a team approached low fidelity prototyping for multimodal applications. It promotes the idea that multimodal interfaces can be simulated using basic non computer based supplies such as paper, overhead projectors and sticky notes. They were able to successfully prototype such objects as a PDA, mp3 player and a mobile assistant for use in an automobile. This is a demonstration of how the difficulties that come along with multimodal development can be overcome.

Coutaz, J., Nigay, L., Salber, D., Blandford, A., May, J., & Young, R. (1995). Four Easy Pieces for Assessing the Usability of Multimodal Interaction: The Care Properties. IFIP International Conference on Human-Computer Interaction (pp. 115-120). London: Chapman & Hall.

This article explains how the authors have assessed multimodal systems. The focus is on understanding the user’s perception of the multimodal interface as well as describing the hardware’s status. The paper provides new terminology that can be used to describe both the user and the hardware. Some examples of topics covered are: Does the user know which modality is available? Do users use all modalities or get stuck using one? Do the users have the cognitive resources to use specific modalities? Do they choose the most efficient modality?

Jameson, A. (2002). Usability Issues and Methods for Mobile Multimodal Systems. ISCA Tutorial and Research Workshop on Multi-Modal Dialogue in Mobile Environments. Kloster Irsee, Germany.

This article explains issues that come up when dealing with multimodal interfaces. For instance, some modes require less cognitive resources than others, and yet making the choice of which mode to use requires cognitive resources. It also highlights how the environment and the multi-modal device is always clamoring for the user’s attention. It offers the suggestion that multi-modal devices need to be aware of context in order to best know how to present information and yet also notices that they are not capable of effectively doing this.

Kaikkonen, A., Kallio, T., Kekalainen, A., Kankainen, A., & Cankar, M. (2005). Usability Testing of Mobile Applications: A Comparison between Laboratory and Field Testing. Journal of Usability Studies , 1 (1), 4-46.

This article, as its name connotes, compares the results of usability tests in lab and field settings. It concludes that lab studies can be equally effective as field studies with less effort and cost. The authors make the point that usability inspections are not to find every detail, but to find the largest issues that can be rectified.

Kim, S., Kim, M., Kim, S., & Kang, H. (2004). Usability Test Equipment for Mobile Devices. DIS2004. Cambridge, Massachusets.

This article analyzes and compares different equipment that might be used to test mobile devices. The equipment types were “Environment-mounted”, “Device-mounted”, and “Subject-mounted.” The results of the study showed that the “Subject-mounted” devices showed the greatest promise for getting the best information from testing, but that there were many drawbacks to each method.

Kondratova, I., Lumsden, J., & Langton, N. (2006). Multimodal Field Data Entry: Performance and Usability issues. The Proceedings of the Joint International Conference on Computing and Decision Making in Civil and Building Engineering. Montreal, Canada.

This is a study looking at a multimodal PDA type device being used in an outdoor work environment. The article states that multimodal devices are especially important in the field because environmental factors make different input/output methods favorable at different times. For instance, extremely bright sunlight makes it difficult to read screens, therefore audio input might be preferable. Also, environmental sensors could actually send back information about a work sight that a person would not even have to input if sensors were added to the device. It also includes an interesting argument for lab studies that represent outdoor environments without actually doing field studies.

Lee, K., & Grice, R. (2004). Developing a New Usability Testing Method for Mobile Devices. Professional Communication Conference (pp. 115-217). Minneapolis: IEEE.

In this article, the authors introduce the audience to usability theory in relation to mobile devices then go on to describe methods for evaluating usability. They go into depth about heuristic methods and then describe a unique method of usability testing that they have created. This new method involves the utilization of heuristics as well as scenario and questionnaire based approaches. It is notable in that it takes seriously the mobile context of PDA's, cell phones and other such devices. It also takes into account the social aspects of their use and considers how social feedback will affect use.

Oviatt, Sharon et al. When Do We Interact Multimodally? Cognitive Load and Multimodal Communication Patterns. 129-136.

Findings from this study reveal that multimodal interface users spontaneously respond to dynamic changes in their own cognitive load by shifting to multimodal communication as load increases with task difficulty and communicative complexity. Future multimodal systems will need to distinguish between instances when users are and are not communication multimodally so that accurate decisions can be made about when parallel input streams should be interpreted jointly vs. individually.

Oviatt, Sharon (1999). Ten Myths of Multimodal Interaction. Communications of the ACM. 42 Iss. 11, 74-81.

Sharon Oviatt addresses 10 issues regarding multimodal design and the myths surrounding them based on research on multimodal human-computer interaction. This article presents information “designed to replace popularized myths with a more accurate foundation for guiding the design of next-generation multimodal systems.”

Oviatt, Sharon. (Sept-Oct 2003) Advances in robust multimodal interface design. In IEEE Computer Graphics and Applications, 23, p62(7).

In this article, Sharon Oviatt explores the possibilities of combining multiple interface modalities into one multimodal mode. She states that users have a strong preference to interact multimodally, and that multimodal interaction greatly improves performance. Performance is improved by helping to avoid errors and recover from those errors when they happen.

Oviatt, Sharon (2000). Multimodal Interfaces that Process What Comes Naturally. Communications of the ACM. 43 Iss. 3, 45-53.

In this article, Sharon Oviatt discusses the benefits of multimodal interfaces when compared to unimodal speech input. She addresses the challenges of multimodal interfaces, ideas on how to alter the computing experience, and the multitude of varying disciplines that need to work together to make everything work together successfully.

Pieraccini, Roberto (2004). Multimodal Conversational Systems for Automobiles. Communications of the ACM. 47 Iss. 1, 47-49.

Discusses how current car speech recognition systems require that owners study a user manual and memorize 200 commands. SpeechWorks and Ford embarked on creating an easier-to-use system that incorporated a touch screen. They used the touch screen to help users as they used speech commands, and allowed the drivers to use the screen while the car was at a stoplight. The main goals were flexibility and complementary functions.

Raman, T. V. (2003). User Interface Principles For Multimodal Interaction. Retrieved February 7, 2007
Web site: http://www.almaden.ibm.com/cs/people/tvraman/chi-2003/mmi-position.html

This article is posted on the IBM Research web site from one of their Research Staff Members in the Human Language Technologies division. This particular article focus on multimodal interfaces, where he states that multiple modalities need to be synchronized, degrade gracefully, should share a common interaction state, and adapt to the user’s environment.

Ryu, Y., & Smith, T. (2006). Reliability and Validity of the Mobile Phone Usability Questionaire (MPUQ). Journal of Usability Studies , 39-53.

This is an article that discusses the authors’ attempt to create a valid psychometric questionnaire. I found the discussion of different types of mobile user’s particularly interesting. They are described as, “Display Mavens”, “Mobile Elitists”, “Minimalists” and “Voice/Text Fanatics”. This article makes the point that different users have vastly different methods of using the device. It is very important that users are identified and the effects of the interface be considered in relation to all of these user profiles.

Simon, Steven J. (2007). User Acceptance of Voice Recognition Technology: An Emperical Extention of the Technology Acceptance Model. Journal of Organizational and End User Computing. 19 Iss. 1, 24-50.

This article discusses the benefits of speech recognition, what fields it is already being used in, and the challenges it currently faces. The article then focuses on a study performed with the US Navy to determine the effectiveness of speech recognition devices while people performed their work. The main conclusion is that as speech recognition becomes more accurate and user friendly, peoples’ perception of usefulness and ease of use will change.

Zhang, D., & Adipat, B. (2005). Challenges, Methodologies, and Issues in the Usability Testing of Mobile Applications. International Journal of Human Computer Studies , 18, 293-308.

This article summarizes different techniques for doing usability tests and discusses how context, network reliability and bandwidth limitation affect usability. The article does a great job of laying out the challenges faced when dealing with mobile devices (context, connectivity, screen size, different resolutions, limited power and data entry modes). The authors emphasize the power that a tester has in a lab compared to in the field. This helped lead me to my conclusion that labs are still the superior place to do most usability studies.