Software development is built around quantitative measurements. Measurements such as the time it takes an application to load, the amount of memory used, or the load on the CPU. These measurements are all easy to calculate and are wonderfully quantitative. One of the reasons some organizations tend to discount usability (both in practice and in artifacts like the severity descriptions in bug tracking systems), is an inaccurate view that usability is an amorphous and subjective thing that simply can’t be scientifically quantified and measured. That assumption is incorrect.
The usability inspection technique of heuristic evaluation, which was introduced by Jakob Nielsen [2,3,4] has emerged as one of the most common ways for professional UX designers to evaluate the usability of a software application. Heuristic evaluations are extremely useful because they formally quantify the usability of a software application against a set of well defined and irrefutable principles. Usability violations can be quantified individually: either an interface supports undo, or it does not, either an interface is internally consistent, or it is not, etc. Usability violations can also be quantified in aggregate: the software application currently has 731 known usability issues.
Additionally, by establishing a tracking system on a set of agreed upon principles, much of the debate on the level of “there is no right or wrong with UI / every user is entitled to their personal opinion / all that matters is the ability to customize” that is currently found in some development communities can be significantly reduced. Usability heuristics will help ground these debates, just as currently in software development no one argues in favor of data loss, or in defense of crashing.
Injecting User Experience Principles Into a Bug Tracking Tool
Adapting a software development team’s bug tracker to capture usability issues defined by a set of specific heuristics can reshape the way developers think about usability. Just as developers currently have a shared vocabulary to describe good and bad with concepts such as performance, data loss, and crashing, usability heuristics can introduce additional concepts, like consistency, jargon, and feedback. All of these concepts, covering both the underlying implementation as well as the user interface, can now have an equal potential to impact the software application at any level of severity, from trivial to critical.
Modifying a bug tracking system to track a heuristic evaluation of software is reasonably straightforward. Each issue needs to be able to be associated with the specific usability heuristic being violated (for example: “using the term POSTDATA in a dialog is technical jargon“). Developers encountering these keywords likely won’t have any additional interface design training, so it is important that each heuristic is very clearly defined with specific examples and detailed explanations. Additionally, allowing developers to view all of the bugs in the software marked as the same type of issue, both current and resolved, serves as an effective way for them to further learn about the heuristic.
The Principles
These are the UX principles currently being used by developers working on Firefox and other projects in the Mozilla community. We are currently utilizing Bugzilla’s keyword functionality, similar to how current bugs can be flagged as violating implementation level heuristics, like data loss. These principles can be added to any bug tracking that allows bugs to be tagged.
- ux-feedback
- Interfaces should provide feedback about their current status. Users should never wonder what state the system is in. [Source: Nielsen]
- ux-implementation-level
- Interfaces should not be organized around the underlying implementation and technology in ways that are illogical, or require the user to have access to additional information that is not found in the interface itself. [Source: Nielsen, Cooper]
- ux-jargon
- Users should not be required to understand any form of implementation level terminology. (This principle is a special case of ux-implementation-level). [Source: Nielsen]
- ux-control
- Users should always feel like they are in control of their software. (This principle is often the nemesis of ux-interruption, especially in cases where developers assume users want more control than they actually want). [Source: Nielsen]
- ux-undo
- Actions should support undo so that users remain in control. (This principle is a special case of ux-control).
- ux-consistency
- In general software should be internally consistent with itself, and externally consistent with similar interfaces to leverage the user’s existing knowledge. [Source: Nielsen]
- ux-error-prevention
- Interfaces should proactively try to prevent errors from happening. [Source: Nielsen]
- ux-mode-error
- Users should not encounter errors because the interface is in a different state than they expected it to be. (This principle is a special case of ux-error-prevention).
- ux-error-recovery
- Interfaces should proactively help users recover from both user errors and technology errors. (A preferable case is to address through ux-error-prevention so that the error does not occur). [Source: Nielsen]
- ux-discovery
- Users should be able to discover functionality and information by visually exploring the interface, they should not be forced to recall information from memory. (This is often the nemesis of ux-minimalism since additional visible items diminish the relative visibility of other items being displayed). [Source: Nielsen]
- ux-efficiency
- Interfaces should be as efficient as possible, minimizing the complexity of actions and the overall time to complete a task. [Source: Nielsen]
- ux-minimalism
- Interfaces should be as simple as possible, both visually and interactively. Interfaces should avoid redundancy. (This principle is often the nemesis of ux-discovery since removing or hiding items deep into the interface forces the user to rely more on memory than recognition). [Source: Nielsen]
- ux-interruption
- Interfaces should not interrupt the user. Interfaces should never ask the user a question that they are not prepared to answer simply for a false sense of ux-control. In general software should only speak when spoken to.
- ux-tone
- Interfaces should not blame the user, or communicate in a way that is overly negative or dramatic.
- ux-natural-mapping
- Controls should be placed in the correct location relative to the effect that they will have. [Source: Norman]
- ux-affordance
- Controls should visually express how the user should interact with them. [Source: Norman]
- ux-visual-hierarchy
- Controls that are more important or more commonly used should leverage visual variables such as size and contrast so that they have more dominance and weight relative to other controls. (This principle is an adaption of ux-discovery).
Side Effects of Distributed Heuristic Evaluation
Today the process of heuristic evaluation is normally completed in corporations and academia by a small number of designers, who are extremely well practiced at identifying usability issues. However, it is worth noting two important aspects of the heuristic evaluation method from when it was first introduced:
Education
First, the method of heuristic evaluation has its roots not in the functional purpose of evaluating usability, but rather in the even more basic purpose of teaching usability. We see this in Nielsen’s 1989 SIGCHI bulletin: Teaching User Interface Design Based on Usability Engineering [2] that heuristic evaluation was introduced as part of the curriculum for a master’s degree in computer science. This is still true today: the road to becoming a good UX designer begins with mastering the identification of well defined heuristics.
Power in Numbers
The second important aspect of heuristic evaluations is that it was quickly found that the number of evaluators played a major role in how successful it was. Nielsen wrote in 1990 that “evaluators were mostly quite bad at doing such heuristic evaluations… they only found between 20 and 51% of the usability problems in the interfaces they evaluated. On the other hand, we could aggregate the evaluations several evaluators to a single evaluation and such aggregates do rather well” [3]. The more people an organization has identifying unique usability problems in a product, the better the product will become, even if your organization already has a very strong UX team.
Conclusion
This practice has the potential to drive UX improvements in organizations that haven’t yet fully embraced UX as part of their product development or IT processes. It helps influence and enlist engineers to create better usability outcomes by making usability testing integral to existing testing, quality control, and issue tracking systems. This approach can be a realistic, practical means of infusing better UX into companies because it embraces the way most IT organizations actually work (driven by quality control and testing) and uses tools already integral to their workflow.
Having your software development team adopt a shared vocabulary for UX principles and having them track these principles in the same bug tracking software they use for addressing all other issues can go a long way towards bridging any divide that exists between design and development teams. Adding these keywords to your bug tracking software obviously doesn’t remove the need for building a very strong UX team inside of your organization, but it will put them in a position to have more influence over the software you are developing.
- Schwartz, D. and Gunn, A. 2009. Integrating user experience into free/libre open source software: CHI 2009 special interest group. CHI EA ’09. ACM, New York, NY, 2739-2742.
- Nielsen, J. and Molich, R. 1989. Teaching user interface design based on usability engineering. SIGCHI Bull. 21, 1 (Aug. 1989), 45-48.
- Nielsen, J. and Molich, R. 1990. Heuristic evaluation of user interfaces. CHI ’90. ACM, New York, NY, 249-256.
- Nielsen, J. 1994. Enhancing the explanatory power of usability heuristics. CHI ’94. ACM, New York, NY, 152-158.
- Bugzilla Installation List, https://.bugzilla.org/installation-list
