User Research - Mobile Usability Testing - Full Report
By Yaro Brock
Mobile devices have many significant differences from personal computers. These differences are spurring interest in the efficacy of traditional methods of running usability tests when evaluating these devices. The following report is a summary of issues raised in several different articles relating to mobile usability.
What are some of the unique characteristics of mobile devices that need to be addressed by usability tests?
When examining mobile devices, one of the primary concerns is going to be with context. Mobile devices are by their very nature, mobile. As a result, they will be used differently than a static PC. They will have a much wider context in which they operate. For instance, a mobile phone can be used indoors and outdoors, in public or in private, in various weather conditions, and sundry locations. Mobile phone manufacturers have to concern themselves with things such as light, or lack thereof. (Kondratova et al., 2006) In some conditions, direct light may obscure the screen. In other conditions, a lack of light may cause issues with using the keypad. If a customer chooses to use a phone in a completely different country, that could also impact usability. If the phone is used in a snowy climate, can the buttons be used with gloves? If it is used in a hot location, will it burn the users hand if left in the sun? Can the user ascertain how much time he/she has out and about before returning home? When at home or at work on a PC, service providers can supply a relatively consistent experience in terms of access to things like Internet and power. When on the road, how do users react to the variability associated with connections to cell towers and limited power? All of these questions come into play because of the mobile nature of these devices yet are not nearly as important when testing the traditional computer.
Social context is another phenomenon that impacts mobile devices in a much more profound way than it has personal computers. This impact can be seen in the annoyed glares directed towards a teenagers talking on their cell phones while riding the bus, or the “friendly” reminders to turn off your cell phone before a movie. While personal computing is primarily individual, withdrawing people in some senses from society (and in another sense building new societies), mobile devices live among us in the “real world.” Social computing can be defined as the interplay between a person’s social behavior and technologies. (Lee & Grice., 2004) Some of the questions related to social computing might be as follows: Does the device distract or annoy non-users? Does the user mind being seen using the device? Does the use of the device distract the user in such a way that it impacts other more important functions? Does the device increase social interaction? Does the device create or exacerbate power differences between people? (Lee & Grice, 118) These questions certainly could be expanded upon, but they make the point that mobile devices are very much in the social sphere and thus have much more rich interactions than traditional personal computers and even laptops. As a result, these issues must be addressed in testing. Do users feel proud to use the product in public or embarrassed? Do they interact with the device differently when surrounded by people? Does the product make an alarming sound when being put into silent mode? These are just a taste of the questions that might be asked when running a usability test on a mobile device while considering the social implications of use.
“Mobile devices themselves, due to their heterogeneous characteristics and physical constraints, may play a much more influential role in usability testing of mobile applications than desktop computers do in usability testing of desktop applications.” (Zhang and Adipat, 302) This quote emphasizes the unique physical characteristics of mobile devices. Unlike the traditional PC with a mouse, keyboard, case, monitor and speakers, there is much variability in the form factor of a mobile device. They do not all support scrolling in the same way. They do not all power on and off the same way. You do not select items in the same way. They do not all have the same basic screen size or shape. They do not all have the same text and numeric input method. They are not always used with eyes on the screen (in the case of headset use). Because each phone is a unique artifact, it will have unique interactions with an already complex context.
Mobile phones also have a unique user group. One study (Ryu & Smith, 2006) described the users of mobile phones as, “Display Mavens”, “Mobile Elitists”, “Minimalists” and “Voice/Text Fanatics”. A “Display Maven” is someone who uses a PDA to do work normally associated with a laptop. A “Mobile Elite” would carry all sorts of mobile devices such as smart phones, mp3 players and cameras. A “Minimalist” might carry only a mobile phone. A “Voice/Text Fanatic” would be someone who uses the basic features of the phone, but uses them prodigiously. These are by no means the only way to classify users of mobile devices, but they illustrate the point that mobile users need to be grouped in order to understand how they interact with a product and ultimately how they perceive the product.
What are some of the practical ramifications of testing mobile devices?
One of the first decisions to be made when testing a device is whether it will be tested in a lab or in the field. As a result of a shifting social, physical, and situational scene, field studies might seem like the obvious choice for most usability tests as they can readily simulate common contextual situations. Surprisingly, several tests have shown that the benefits associated with field studies are not as pronounced as they might seem. (Kjeldskov & Graham, 2003) In fact, most current mobile studies are done in a lab setting and it does not appear that this trend is likely to change.
In a study done by several researchers (Kaiikonen et al, 2005), a comparison of a field test and a lab test was made. The author s had the expectation that field studies would yield more and different types of errors than an equivalent lab study that was seeking to measure the same interface. Although the errors were not exactly duplicated in each study, there was much overlap. The big takeaway was that there were exactly the same number of issues found in each test. This showed that the most time and cost effective method was the most beneficial. The report went on to state that lab studies are generally quicker and therefore more cost effective.
One way that lab studies can be effective at simulating real world contexts is through the use of scenarios. “Scenarios can help the developers imitate the mobility aspects of the task, which is the most difficult to simulate in a lab.” (Lee et al., 2004) By creatively engaging the minds of users, scenarios allow them to imagine using the device in a mobile context. Surprisingly, this method of finding issues that might occur in the outside world can be just as effective as being there, with the benefit of reduced cost. If necessary, test planners can create systems that support the simulation like wall displays of a busy worksite or the sounds of a rumbling engine. (Kondratava, 2006)
Field studies also carry baggage that lab studies do not. A field study can be difficult to administer, at least more so than a lab study. Think-aloud protocol and video recording are more difficult in the field than the lab. (Zang & Adipat, 2004) Field studies are more time-consuming and require more effort than lab studies. (Kaikkonen et al., 2005) Arrangements take longer to make for field studies and can easily become complicated by unanticipated interference (Kaikkonen et al., p 13). Field studies require the user to be connected to equipment that can easily inhibit the normal use of the device. Cameras can be attached to the device or the user. Although new technology allows for primitive forms of eye tracking, the ability to mount and power these video and audio devices without making the user conspicuous is still underdeveloped.
Questionnaires are also a valuable tool being used to determine usability in mobile devices. They are particularly helpful because they can go with a participant to places that it would not be convenient for a researcher to be. It is likened to an interview without the interviewer. (Lee et al., 2004) But making a questionnaire relevant to the mobile device is also very important. Many usability questionnaires are generic in nature and need to be adapted to the mobile genre. (Ryu & Smith, 2006) The effort to find valid and reliable methods of acquiring the user’s satisfaction via questionnaires is a task that is being approached, but still has work to be done until it is universally accepted.
We have examined some of the issues that impact the usability and testability of mobile products. The key concept for mobile devices is that of context. By clearly grasping the potential contexts, we will be better able to understand what the issues are that need to be addressed, as well as create methods of testing that adequately simulate those situations. Future work might seek for a way to limit the understanding of context so that it can be used without overwhelming the testers. Much effort needs to be put into finding technology that can track the user’s use of equipment without the conspicuous technology impacting the experience. Also, further work on creating questionnaires that are built into devices or accessed through devices would likely prove superior to paper questionnaires that are currently being used.