UX Magazine

Defining and Informing the Complex Field of User Experience (UX)
Article No. 498 March 3, 2010

Where Do Heuristics Come From?

Last year, I had the honor and pleasure of working on a project for the National Institute of Standards and Technology (NIST) to develop style guidelines for voting system documentation. Yawner, right? Not at all, it turns out. It made me think about where guidelines and heuristics come from for all kinds of design. Yes, if you live in the United States, you paid for me to find this out. Thank you.

What I learned in the process of developing style guidelines for voting system documentation (which, astonishingly, took about a year) is that most heuristics—accepted principles—used in evaluating user interfaces come from three sources: lore or folk wisdom, specialist experience, and research.

Though style guidelines for content are important, I’m going to talk about each of these sources of heuristics with various design examples. I’m sure you’ll see something that you’ve encountered before.

Lore or Folk Wisdom

First comes guidance from “They,” as in “They say…,” for which no one knows the true source. For example: “Feed a cold, starve a fever.” “Never end a sentence with a preposition.” “Limit the number of items in the main navigation to seven plus or minus two.”

Where do these come from? Someone’s belief that this is a good practice. They may have heard something or done something that they think supports the practice, but there’s really no basis in fact for any of these.

A New York Times article by Anahad O’Connor says that recent research about whether to eat a lot when you have a cold and fast when you have a fever is inconclusive. No one seems to know how this one started. It may just feel like there’s some inherent logic to it.

Not ending sentences with prepositions was encoded by a British guy named Henry Fowler in 1926. He was a crotchety, proscriptionist pedant, but his book was a best seller. People wanted guidance about how to speak and write “properly,” especially in class-conscious England. So a rule to not use words like “to,” “in,” “for,” “with,” or “on” as the last word in a sentence became wildly popular as a marker of a well-bred, well-educated person. But it was really just Fowler’s personal preference, and today the practice seems like an affectation.

My favorite example in the Web design world is a guideline about limiting the number of items in a navigation menu or list to five to seven items. Most people don’t know where this came from; if they did, they’d know that this isn’t the best use of that “rule” and imposing it actually won’t make the design usable. This one does originate in research, specifically an article published in 1956 by George Miller in the Psychological Review called “The magical number seven, plus or minus two: Some limits on our capacity for processing information.” The findings from the research Miller describes are about working memory. The lore passed down from that article is that humans can only hold about seven things in their short-term memory at a time. But Miller very heavily qualifies this as “suggestive” and an “estimate.” More importantly, what in the world does this have to do with designing website navigation? Nothing. Navigation is persistent. We’re not asking people to remember from section to section where they can go. It’s right there for them to see and use. The number of items in navigation should be determined by the data from research about the users and their task goals.

If you find yourself saying “they say” or “I’ve heard” when you make an observation about a design issue, you may be caught without a lot of support for your point. Basing an inspection on your own experiences observing users can hold more authority.

Specialist Experience

Older adults who use the Web need high contrast and large targets. If they are not expert Web users, they can be easily distracted, so to ensure that they’re successful, we should design in smooth task paths and clear labeling that doesn’t use jargon.

One of my special interests since about 2003 has been Web design for older adults. I’ve internalized the design principles above (as well as many others) after watching dozens of people who are age 55 or older use a variety of Web sites. I am confident that implementing a design that takes these design principles into account will make the design easier for older people to use than designs that use subtle colors layered on one another, small buttons and links, and cluttered page layout with trendoid headings and labels. Though I have observed many types of people using lots of different kinds of websites, I have specialist experience from watching one audience try to do typical tasks on a variety of websites.

Specialist experience means expertise in a particular domain or product. You get it only after hours and hours and hours of seeing the same kinds of things happen to the same types of users. Basing an inspection on specialist experience is definitely a step up from working from lore, but if you haven’t distilled what you have found in the many hours of observing a type of users using a site or type of site, then you may be working from hunches and opinion that could make it difficult for you to justify the evaluation recommendations.

Evidence From Research

Some things that experienced designers have internalized do have data to support them: eliminate horizontal scrolling, design for working memory limitations, facilitate scanning.

These all rate a 5 (out of 5) for strength of evidence in the guidelines on usability.gov (I’ll get to the rating thing in a minute). Usability.gov started as a project at the National Cancer Institute (NCI), which is part of the US Department of Health and Human Services. NCI needed guidance in designing usable cancer information websites—a simple goal.

On the way, the NCI team realized that not all guidelines were equal. Some guidelines were supported by a lot of data from multiple studies (like the high-scoring heuristics above). Some guidelines might come from only one study. Still, evidence was evidence, and NCI wanted to use “quantified, peer-reviewed Web site design guidelines,” which they found simply didn’t exist. And as far as I know, there’s nothing like the resource NCI created at usability.gov.

To reach their goal, NCI put together panels of experts to review research. The panelists then rated each guideline for strength of evidence that, among other considerations, needed to be “cumulative and compelling” for a 5 out of 5. The idea was that teams could use the ratings to help them make design decisions. But the guidelines were not meant to be a substitute for usability testing. Why not? The main reason was that the guidelines at usability.gov were developed for information-rich websites (versus e-commerce or transaction-based sites) with content about major illnesses. That’s fairly specialized. But when you read through the 500 guidelines that NCI identified, it is obvious that almost all could apply to many types of websites or many types of pages within sites. Your mileage may vary.

The Basis of the Heuristic Matter

As the folks at NCI learned in developing usability.gov and I learned in the work for NIST[*], the provenance of a heuristic is important. This is true of all implicit and explicit heuristics applied in design decisions.

Learning about where heuristics come from—lore or folk wisdom, specialist experience, or research—helped me understand better where some of the teams I’ve worked with were coming from as they developed design principles. Sometimes they based the principles on lore, sometimes on expertise. Rarely did they go to the research.

Expertise is good, but research is better. Research-based heuristics simply have more heft: credibility, specificity, and applicability.

Still, there’s no substitute for primary research. Firsthand observation of your users in their context reveals subtleties of behavior that even research-based heuristics can’t match. And if your research of your users in their context contradicts the known research, what do you do? (You don’t get two guesses to answer this question.) If you go with what your users do, then even the most deeply researched heuristics are at best a poor substitute for doing the right thing.

[*] I couldn't have made the discoveries I did on that project without Susan Becker, my project partner, who did most of the heavy lifting.

Related Links

You can pore through the evidence-based guidelines for usability developed by the National Cancer Institute (NCI) at usability.gov.

You might also want to check out other sets of research-based heuristics and guidelines. I've worked on a couple of them:

AARP commissioned Janice ("Ginny") Redish and me to do an expert review of websites in 2005 in order to develop a set of heuristics for designing websites for older adults (Google doc).

The final report from our review of 50 websites is here (also a Google doc).

The National Institute on Aging (NIA) also published guidelines for designing senior-friendly websites.

The voting section of the National Institute of Standards and Technology (NIST) website now links to style guidelines for voting system documentation, which are based on research in technical communication, information design, usability, and instructional design. Check out the site’s Publications section to download the PDF of the guidelines and to view other design research related to voting systems.

Oh, and you might want to read Miller's article to judge for yourself.

ABOUT THE AUTHOR(S)

User Profile

Dana has helped hundreds of people make better design decisions by giving them the skills to gain knowledge about users. She's the co-author, with Jeff Rubin, of Handbook of Usability Testing Second Edition. (Wiley, 2008) Visit Dana's website at usabilityworks.net or follow her on Twitter @danachis.

Add new comment

Comments

21
20

Shameless plug, but I’ve just launched http://www.uxponder.com, a site providing user-submitted heuristics to ponder while you design.

17
20

Your paragraph on Fowler is the worst that I have ever read on this writer, and indeed on English usage. http://maryandleo.tumblr.com/post/476102903/from-a-dictionary-of-modern-english-usage-by-h-w

19
16

PJ,

  (This is cool. It's like writing to my sister, who also goes by PJ. You are clearly smarter than she, however.)

  Usability.gov is an awesome resource. I don't like the PDF thing, either. But the content is huge and the site is maintained on volunteer time. I don't know if there's even a CMS under the covers. So, I'm just thankful it's still there at all.   Use it in good health.

21
19

Hi Zeke,   I'm not critiquing Miller. I'm complaining about how designers use only the part that they remember. (Get it? That working memory thing puts a meme in our heads to remember something, a cue for recall, but we don't always remember all the relevant bits, so we have to look it up again, which is the recognition part.)   9 is not a magic number for navigation choices. There are sites that have more than 9 choices. If you're seeing issues with users being able to use the navigation when there are lots of choices, it is very likely that it isn't the memorability of the choices, but something else. For example, maybe the labeling in the information architecture isn't clear. Or the visual design is cluttered. Or the wrong things for supporting users goals are in the navigation or they're in the wrong place in the navigation. None of that has anything to do with what Miller was saying.   Your saying fewer than 9 choices as a design principle is like saying users should be able to get to anything within 3 clicks. Please tell me that you're not saying that, too.

20
15

Hi Charlotte,

Thanks much - so glad you liked it! It's not often people thank me for *making* them think. I blame Steve Krug for that.

17
24

Derek,

It's true. Most of the "heuristic evaluations" conducted are done badly. It's easy to get it wrong; the temptation of "checklist usability" is great. I've found that in the right circumstances, HE can be a great complement to usability testing. It often reveals things that might be embarrassing or wasteful for users to find. (See Jared Spool's piece at Johnny Holland on "Low Hanging Fruit and Penny Stocks.") But it's no substitute for observing real people use a design.

16
18

Interesting piece. Thanks. Led me to revisit the guidelines at usability.gov. Bummer that they're now all in PDFs instead of web content. Makes scanning the "Importance" and "Strength of Evidence" rankings, well, kind of impossible.

20
20

I have to question your George Miller 7 +/- 2 critique. While I could easily see the "rule" being often touted by those not fully understanding its root... at it's core is real research about working memory limitations (which you even acknowledge as a well supported design principal). After observing lots of user testing (mostly consumer web applications), I have consistently seen issues with more than 9 menu items starting to hinder usability in ways that were resolved by restructuring, reorganizing, or progressively disclosing to limit the number options the user is presented with at a time (within a given task space) to live within Miller's threshold.

So yes... user testing should always be the final confirmation, but the "less than 9 choices" goal during the design process has been consistently constructive for me... confirmed by user testing, and backed by research.

21
19

This is an excellent article, Dana! Thanks for writing it. It was fun to go back and re-read Miller after all these years. And in the fast paced UX world most of us live in (or as Fowlwer would say, in which most of us live - ha), who has time to research the history behind the various heuristics we've come to accept as everyday. Thanks for making me think!

20
22

I started out being very distrustful of heuristics because compared to "real" usability testing it felt like a cheat. As I've learned a bit more about that particular area of the discipline I'm realizing I made a snap judgement based on a non-representative sample of those working professionally with heuristics based reviews. Some people base heuristics based on "folk wisdom" which may or may not offer any real value to their clients. Those clients may then be left with a negative impression of the entire discipline and we all suffer for it (not just UX professionals, users and designers too). But that's a poor reason for me to discount heuristics as a whole. Thanks for explaining some of the root causes of my cognitive dissonance on heuristics. :)