Summary

In user testing, website use on mobile devices got very low scores, especially when users accessed “full” sites that weren’t designed for mobile.

 Mobile use is one of the biggest challenges now facing many websites. It’s also important for some intranets, particularly in companies with many traveling employees.

Usability Research

To discover what makes websites easy or difficult to use on mobile devices, we combined three usability methods:

In the usability tests, we asked participants to perform typical tasks with their phones. In testing mobile.winespectator.com, for example, we showed people a bottle of wine and asked them to find information about it on the site. For m.lufthansa.com, one of the tasks was:


“Your friend is scheduled to arrive in London today around 12pm from Munich, Germany. Find out if her flight is on time.”


In all, we tested 36 websites and asked users to attempt particular tasks on each. These site-specific tasks let us systematically observe several users with different phones trying to do the same thing. We also tested 34 Web-wide tasks where participants could use any site they wanted. One such task was:


“You and your vegetarian friend want to find a good Indian restaurant nearby. Use the Web to locate one that you may want to go to and that serves vegetarian food.”


These tasks gave us usability insights into hundreds of additional sites as well as an understanding of how people decide which sites to visit on their mobiles.

The test tasks were inspired by the user activities recorded in the diary studies. The diaries also let us follow user behavior over a longer time period in more naturalistic settings than is feasible for lab studies.

The Mobile User Experience Is Miserable

The phrase “mobile usability“ is pretty much an oxymoron. It’s neither easy nor pleasant to use the Web on mobile devices. Observing user suffering during our sessions reminded us of the very first usability studies we did with traditional websites in 1994. It was that bad.
view Mobile Web 2009 = Desktop Web 1998

In our mobile studies, the average success rate was 59%, which is admittedly higher than success rates in the 1990s, but substantially lower than the roughly 80% success rate when testing websites on a regular PC today.
view Success Rate: The Simplest Usability Metric
view IA Task Failures Remain Costly

Before the study, we had expected to get better results in London because the UK has a stronger tradition for mobile services than the US. However, the actual sessions didn’t bear this out: the British sites were just as bad as the American sites, and users struggled about as much to get things done.

Main Mobile Problems

Mobile users face four main usability hurdles:

The first two problems seem fundamental. Yes, such problems impact newer phones less than older phones (as I discuss below), but still: mobile devices will never offer screens as big or input devices as good as a full-fledged PC.

Connectivity problems will hopefully diminish in the future, but it will take many years until mobile connections are as fast as even a modest cable modem—let alone as fast as the broadband connections promised by wireline improvements.

Mobile will never be the same as desktop. So, we’re left with the hope that websites will redesign for better mobile usability.

Mobile Sites Beat Full Sites

When our test participants used sites that were designed specifically for mobile devices, their success rate averaged 64%, which is substantially higher than the 53% recorded for using “full” sites—that is, the same sites that desktop users see.

Improving user performance by 1/5 is reason enough to create mobile-optimized sites. Such sites were also more pleasant to use and thus received higher subjective satisfaction ratings. This fact offers an additional rationale: When users are successful and satisfied, they’re likely to come back. So, if mobile use is important to your Internet strategy, it’s smart to build a dedicated mobile site.

Still, users often had trouble getting to mobile sites, even when companies offered them. The best approach is to auto-sense users’ devices and auto-forward mobile users to the mobile site (even if they’re using a high-end phone). You should also offer clear links from the desktop site to the mobile site, as well as a link back to the full site. As for link labels, we recommend “Mobile Site” and “Full Site,” respectively.

Linking to the full site supports users who want advanced features that the mobile site doesn’t support. Given this fallback solution, you should scale back the mobile site’s functionality and focus on features that people are actually likely to use in a mobile scenario. Users repeatedly told us that they don’t want to do tasks on their mobiles that involve heavy interaction or in-depth information perusal.

Better Phones Perform Better

There are 3 distinct classes of mobile user experience, and they’re mainly defined by screen size:

  1. Feature phones (regular cellphones) with a tiny screen and a numeric keypad. These devices account for the vast majority of the market (at least 85% in some statistics).
  2. Smartphones, in a range of form factors, typically with a mid-sized screen and a full A-Z keypad.
  3. Touch-screen phones (such as the iPhone) with a nearly device-sized screen and a true GUI driven by direct manipulation and touch gestures.

Unsurprisingly, the bigger the screen, the better the user experience when accessing websites. Average success rates were:

Feature phones 38%
Smartphones 55%
Touch phones 75%

With these numbers, the consumer advice is easy: buy a touch phone if using websites is important to you.

The advice for Internet managers is harder. Considering the horrible usability of feature phones, should you even support them? Alternatively, should you focus on smartphone and touch phone users who are more likely to use your site extensively? There’s no single answer.

For services highly suited for mobile use—such as news or social networking—you should probably create a dedicated feature-phone site, as well as a site optimized for higher-end phones. Most other websites might be better off concentrating their investment on a single mobile site optimized for smartphones and touch phones. Finally, if you focus on complex transactions or in-depth content, you’ll probably have too few mobile users to justify a separate site.

No Progress Since 2000?

In our London sessions, we repeated two tasks from our study of WAP usability in 2000. We expected to find reasonable improvements in task performance, but the results contradicted those expectations (which is obviously why we bother doing research). The mean task times from the two studies were:
view WAP Field Study Findings

Tasks WAP phones (2000) Current Phones
Find the local weather for tonight 164 sec. 247 sec.
Find what’s on BBC TV 1 tonight at 8 pm 159 sec. 199 sec.

Amazingly, users spent 38% more time on these two tasks now than they did in 2000. Are modern mobile devices really worse than the horrible WAP phones of times past? Has site usability declined that much? The answer is no on both counts; phones and sites are definitely better now.

What has changed is the usage environment. In 2000, users were restricted to the “walled garden” supplied by their mobile carrier. WAP phones came with a built-in “deck” that supplied direct access to a few selected services. While this approach limited users’ freedom and restricted them to only the simplest of tasks, they could get to the information with just a few key-presses.

Today’s mobile users are highly search-dominant. When we don’t specify which site they should use (and often even when we do), they turn first to their favorite search engine. Again, this means plenty of typing, which is slow, awkward, and error-prone on mobile devices.

Today, mobile users can do anything. The fact that doing most things takes so long further emphasizes the need for scaled-back mobile site designs.

In our current study, one user did really well—an iPhone user who had a weather application installed on the phone and used it to get the weather forecast in only 18 seconds (1/3 of the fastest speed from 2000). If any additional evidence were needed for mobile-dedicated design’s benefits, this example should surely suffice.

Mobile Usability Is Hard

All of our new research findings support a single conclusion: designing for mobile is hard. Technical accessibility is very far from providing an acceptable user experience. It’s not enough that your site will display on a phone. Even touch phones that offer “full-featured” browsers don’t offer PC-level usability in terms of users’ ability to actually get things done on a website.

When designing for mobile, there’s a tension between (a) making content and navigation salient so that people do not work too hard to get there, and (b) designing for a small screen and for slow downloading speeds. That’s why almost every design decision must be made in the context of the site being designed, and what works for a site may not work for another.

Unless websites are redesigned for the special circumstances of mobile use, the mobile Web will remain a mirage. Users won’t realize the benefits promised by mobile vendors, and site owners won’t reap the profits that would follow from gathering hordes of loyal mobile customers.

Learn More

132-page report on Mobile Usability with 85 design guidelines is available for download.
In-depth course on mobile usability, including many video clips of test participants using a wide range of sites on many different phones, at the Usability Week 2009 conference in Las Vegas and Berlin.
view Designing for Mobile Devices