- Email: [email protected]
Applying heuristics to accessibility inspections
Interacting with Computers 16 (2004) 507–521 www.elsevier.com/locate/intcom
Applying heuristics to accessibility inspections Claire Paddison*, Paul Englefield1 Ease of Use, IBM Warwick MP5, P.O. Box 31, Birmingham Road, Warwick CV34 5JL, UK Available online 7 May 2004
Abstract Accessibility heuristics have been developed to complement accessibility guidelines. The use of Web accessibility heuristics in heuristic evaluations considers a greater range of special needs, such as visual impairments to cognitive disabilities. Key advantages of heuristics are conciseness, memorability, meaningfulness and insight. The heuristics allow evaluators to understand effectively which areas of a site have accessibility issues and provide useful insight into how to create a solution. However, the heuristics will not tell evaluators whether a Web site conforms to legislation. Studies have confirmed the view that while heuristics do not substitute for expertise, they do act to cue the deeper body of knowledge defined by the guidelines. It is essential that evaluators receive accessibility education before completing a heuristic evaluation using the accessibility heuristics. q 2004 Elsevier B.V. All rights reserved. Keywords: Accessibility; Heuristic evaluation; Accessibility heuristics; IBM heuristic evaluation database
1. Introduction A heuristic ‘pertains to the process of gaining knowledge or some desired result by intelligent guesswork rather than by following some pre-established formula’ (Whatis?com, 2004). Heuristics are widely utilised in usability and considered useful in other fields, such as artificial intelligence and medicine. In this paper, heuristics have been extended to consider the notion of accessibility. By using specific accessibility heuristics to assess a Web site, evaluators consider a greater range of special accessibility needs. Examples include visual, motor, and cognitive impairments. By rigorously addressing the issues identified by such an evaluation, designers can ensure that the site will be more accessible to a larger and more diverse audience. * Corresponding author. Tel.: þ 44-1926-464978; fax: þ 44-1926-465323. E-mail addresses: [email protected] (C. Paddison), [email protected] (P. Englefield). 1 Tel.: þ44-1926-464329; fax: þ44-1926-465323. 0953-5438/$ - see front matter q 2004 Elsevier B.V. All rights reserved. doi:10.1016/j.intcom.2004.04.007
508
C. Paddison, P. Englefield / Interacting with Computers 16 (2004) 507–521
This paper aims to show that accessibility heuristics recognise diversity in heuristic evaluations and can provide an alternative representation of the guidelines that is suitable for use during an inspection process. ‘Barrier-free’ design is a concise way of describing an accessible interface for people with disabilities. In the same way, universal accessibility ‘ensures that sites are developed to serve the largest possible audience using the broadest range of hardware and software platforms, and that the needs of users with impairments are considered’ (McEwan et al., 2003). Being ‘barrier-free’ means that the information or service you provide must be available to everyone regardless of software, platform, environment, and user ability. Examples include: for software, access using desktop browser or voice browser; for platform use of a desktop, mobile phone or PDA (Personal Digital Assistant); for environment, working in a noisy or under-illuminated surroundings; and, for ability, visual impairment or dyslexia. IBM practitioners distinguish two aspects of accessibility: (a) technical accessibility and (b) usable accessibility. These aspects affect people with disabilities in different ways. Technical accessibility relates to the adoption of best practice in software engineering. Examples include labelling ‘alt’ tags appropriately, and ensuring that the interface can be accessed using the keyboard alone. Technical accessibility ensures that assistive technologies, used by people with disabilities, can understand and interact usefully with the different elements of the design. Automated tools, such as Bobby (2004), provide some level of support in testing for technical accessibility. By contrast, usable accessibility relates to the application of usability principles for the benefit of those with disabilities. Examples here include support for users’ tasks, consistent navigation, and management of cognitive load. Both aspects of accessibility are critical to users’ success. For example, a technically accessible Web site may still present challenges to users of assistive technologies. Such challenges may result directly from the need to use assistive technologies, or may arise from more fundamental usability issues that adversely affect all users. In summary, it is not enough to follow accessible guidelines and make the appropriate technical accessibility changes. People with special accessibility needs have a distinct user profile with unique requirements that need to be addressed within a user-centred design process. Designing for accessibility is increasingly seen as an effective response to a growing recognition of the needs of people with disabilities. Personal independence is a critical need that that can be supported by providing accessible online information and services. For example, simple adjustments, such as ensuring fonts are resizable, can help users with low vision to access the Web. Keyboard enabling supports users who cannot use a mouse, due to upper limb disorders. Graphical representations can help users with learning difficulties to better comprehend online information. With the introduction of legislation to develop Web accessibility, these issues are becoming increasingly important to organisations. In the US, Section 508 (2004) requires that all Web sites ‘developed, procured, maintained, or used by the Federal government’ must be accessible. In the UK, an amendment to the Disability Discrimination Act (Disability Discrimination Act 1995, 2004) states that organisations that provide goods, facilities and services via the Web are legally obliged to make their Web sites accessible to disabled people. Guidance is specifically offered to e-commerce Web sites. In addition to
C. Paddison, P. Englefield / Interacting with Computers 16 (2004) 507–521
509
the above US and UK legislation, there are number of related initiatives in progress at a European level. From a business perspective, making services and products accessible is likely to increase market share and audience reach. Tom Kelley of IDEO, one of the world’s leading design companies, believes that work for people with disabilities will increasingly result in innovations that benefit the general population. He says, “By studying people of all ages, shapes, cultures and sizes, we’ve learned that the best products embrace people’s differences” (Kelley, 2001). A good example of this is OXO Good Grips, a line of kitchen tools with cushioned, oversized handles that are comfortable and easy to use. In 1990, Sam Farber first questioned the effectiveness of kitchen gadgets in response to his wife’s difficulty in gripping kitchen tools due to a slight case of arthritis. Since then, OXO Good Grips have won praise for good design and grown to a line of 350 products with a 15% share of the US market (Industrial Designers Society of America Showcase, 2004). The European Union has been active in the area of accessibility since 1999. At this time, the eEurope objective was ‘to bring every citizen, school and business online and to exploit the potential of the new economy for growth, employment, and inclusion’ (e-Accessibility: Accessibility and Information Society, 2004). Six years on, the emphasis has shifted to achieving ‘an Information Society for All’ with the visionary goal of ensuring an ‘inclusive digital society that provides opportunities for all, thus minimising the risk of digital divide.’ The majority of UK government services are planned to be online by 2005. Insufficient attention to accessibility could result in the marginalisation of many groups and the consequent creation of a ‘digital divide’. This final point includes arguments from each of the three considerations discussed above. The digital divide is a major focus for Europe’s e-Accessibility initiative, helping to drive the arguments for accessibility across Europe. Clearly, accessibility is a critical consideration in design. Whether designers integrate accessibility guidelines into the design process or apply them retrospectively, heuristic evaluation provides a cost-effective way to inspect for compliance.
2. Heuristic evaluation Heuristic Evaluation is an established usability evaluation technique originally proposed by Jakob Nielsen as a cost-effective ‘discount’ usability technique (Nielsen, 1994). A panel of experts formally assesses an interface design with respect to a set of heuristics or rules of thumb for effective design in an inspection method. A typical study has five phases: (1) study design, (2) discovery, (3) editing to identify and resolve duplicate and related findings, (4) prioritisation to identify severities, and (5) analysis and report writing. During the design phase, a study manager selects or creates a set of heuristics appropriate for the study goals. In the discovery phase, evaluators use these heuristics as mnemonics to identify design features that conflict with some aspect of best practice. In commercial practice, heuristics classify findings at the time of discovery to support quantitative analysis during the report writing phase. Heuristics provide a convenient mnemonic framework for existing evaluator expertise. In the same way, the accessibility heuristics have been designed to act as a mnemonic for
510
C. Paddison, P. Englefield / Interacting with Computers 16 (2004) 507–521
evaluator’s own knowledge of the concepts of accessibility. An unskilled evaluator relying on heuristics alone expects to be less effective than a skilled practitioner is. Consequently, a good heuristic should be concise, memorable, expressive and easy to relate to underlying knowledge and principles. A set of heuristics should ideally be concise, exhaustive and mutually exclusive. Examples of general purpose heuristics include Nielsen’s original set (Nielsen, 1994) and Nielsen and Molich’s revised set (Ten Usability Heuristics, 2004). An example of a domain specific set is the draft public transport Web site heuristics (Internet PTI Best Practice, 2004) developed by Southampton University’s Transport Research Group. Heuristic sets are often aggregated. For example, to assess a metro site, either of the general purpose sets described above may combine successfully with the public transport Web site set. The literature reflects a vigorous debate about the relative effectiveness and costefficiency of inspection methods with respect to empirical testing (Gray and Salzman, 1998; Jeffries et al., 1991; Karat et al., 1992). However, within IBM (Englefield, 2003a), heuristic evaluation is used in a range of situations: (1) where project constraints limit access to representative users; (2) where rapid turnaround is required; (3) where evaluators cannot work together at the same time and location; (4) where breadth of focus is more important than sensitivity, and (5) where a strong client relationship reduces the need to support advice with evidence directly gathered from real users. Consequently, heuristic evaluation is widely applied to both commercial services offerings and the design of IBM products within a User Engineering (IBM, 2004) process (Nielsen, 2001). In some cases, heuristic evaluation is applied as a pre-screen before empirical testing or as a triangulated study in conjunction with a user test. Situations (1) and (4) may well apply to an inspection of a design intended to be inclusive for a diverse range of physical, cognitive and motivational factors. Furthermore, heuristic evaluation offers strong support for the established inspection-oriented approach to accessibility evaluation, adding structure and rigor to a mature knowledge-oriented discipline. The Heuristic Evaluation Database (HEDB) (Englefield, 2003b) (see Fig. 1) implements heuristic evaluation as a Lotus Notes database supporting a collaborative workflow process in which the roles of Evaluation Manager and Evaluator carry out tasks in each of the five phases described above. The database provides efficient support for the key tasks of entering findings, adding severities, and editing duplicates. The tool automatically generates a skeleton report (see Fig. 2) to provide quantitative cross-tabulated analysis of the distribution of errors by task, location and heuristic together with a structured list of finding details. Practitioners report that these analyses are helpful in forming client recommendations; “…absolutely…helps identify areas in the design that need particular focus”. A study was conducted to assess the effectiveness of the HEDB (Englefield, 2003b). Five evaluators and a study manager carried out a realistic inspection of a commercial Web site and commented on their experiences. Results indicate that the database is easy to learn and efficient to use while addressing many of the practical concerns of applying Heuristic Evaluation in industry. For example, practitioners rated the database 95% for learnability and 90% for efficiency where scores 0 and 100% represent the minimum and maximum scores on a Likert scale. Subsequent feedback from commercial use suggests
C. Paddison, P. Englefield / Interacting with Computers 16 (2004) 507–521
Fig. 1. Sample view of the user interface for the heuristic evaluation database.
Fig. 2. Extract from HEDB.
511
512
C. Paddison, P. Englefield / Interacting with Computers 16 (2004) 507–521
that using the database might save ‘two to three days’ otherwise required for low level data management and formatting of results.
3. Accessibility heuristics The HEDB incorporates various sets of reusable usability heuristics derived from IBM and external sources. These include a set of accessibility heuristics that have been developed by IBM as a complement to usability heuristics. Accessibility heuristics identify areas of a Web site that are not accessible for some reason. Two examples follow: 1. key JavaScript elements not activated because scripting is disabled; 2. non-standard proprietary controls used at key stages of the navigation prevent assistive technologies from interacting correctly. In addition to identifying design errors, heuristic evaluation can also help designers to plan which areas of a site to make accessible first. This can be particularly useful when retrospectively using guidelines to assess an implemented site. Accessibility guidelines also have legal implications. Using the accessibility heuristics to conduct a heuristic evaluation will not tell evaluators whether a Web site conforms to legislation. However, it will (a) highlight those tasks that users are likely to have difficulties in completing, and (b) identify those elements of the site, such as navigation or consistency, that may require attention before the site meets legislative requirements. Many of the accessibility guidelines follow usability best practices. By using a set of usability heuristics together with the accessibility heuristics, both aspects (usable accessibility and technical accessibility) are addressed. In both cases, the heuristics act as a mnemonic framework for accessing a broader knowledge of usability and accessibility knowledge. Usability heuristics are commonplace in usability practice; there is now a growing need for a workable and usable set of accessibility heuristics. It is therefore important to understand why accessibility heuristics are preferred to guidelines as a resource for accessibility assessment. Guidelines are typically bulky and contain large numbers of individual items. For example, the Web Content Accessibility Guidelines 1.0 (Web Content Accessibility Guidelines, 2004) contain 65 guidelines (2.1 Ensure that all information conveyed with color is also available without color, for example from context or markup), divided into 14 main areas (2. Don’t rely on color alone), and divided again into three priority levels (Priority 1.) Because it takes significant time and effort to search for guidelines, assessment can be slow and expensive. By contrast, the IBM heuristics, together with their brief descriptions, fit on one side of letter paper. A simple scan enables evaluators to rapidly locate and categorise accessibility problems; in some cases an assessment can be carried out in a single day. Guidelines can be hard to remember; without regular use, their size and number make them difficult to retain. On the other hand, heuristics are specifically designed as a concise,
C. Paddison, P. Englefield / Interacting with Computers 16 (2004) 507–521
513
memorable set of compact, pithy, and generative points. A typical example is, ‘Respect users’ browser settings’. Many of the guidelines are intuitive; some, however, require a technical background or further study to fully understand their true meaning. An example is, ‘Use markup and style sheets and do so properly.’ Heuristics are expressive. Evaluators should look at the heuristic and think, ‘Yes, I understand that.’ They also consider a broad range of factors such as navigation and consistency, rather than focusing on a very specialist area. Making accessibility heuristics more meaningful to a broad range of people develops understanding of accessibility issues. It also allows a multi-disciplinary team to be able to inspect an interface, involving stakeholders other than accessibility experts. During the heuristic evaluation process, evaluators prioritise findings. This gives the study manager a clear indication as to which aspects of the design are causing problems and what the solution may be. For example, if many of the problems are related to consistency, then a style guide may be indicated. Although the guidelines are grouped by category and priority, their relative size leads to a less distinct prioritisation scheme. The characteristics discussed above support the use of accessibility heuristics as the immediate representation of accessibility knowledge with the ability to perform an inspection. However, there are disadvantages associated with using heuristics in certain scenarios. The Web Content Accessibility Guidelines 1.0 (Web Content Accessibility Guidelines, 2004) are extremely detailed and have additional links to technical advice and other useful information resources. As a result, it is clear whether an aspect of a Web site conforms to the guidelines and therefore subsequent legislation. The accessibility heuristics, even with the descriptions, do not provide distinct information. A degree of interpretation is always required. Using the accessibility heuristics will not tell the evaluators whether a Web site conforms to legislation. The heuristics are only able to highlight which tasks users will have difficulties in performing. Guidelines are able to provide organisations with clarity that they have complied with current legislation. Heuristics are intended as a mnemonic framework to cue an evaluator’s existing expertise. Heuristic evaluators should relate the finding to a particular heuristic rather than use the heuristics alone to search for errors. Using evaluators with limited accessibility knowledge may lead to misapplication of the heuristics. Table 1 shows the nine IBM Web Accessibility Heuristics and their descriptions. IBM’s experience suggests two practical methods for developing a set of heuristics. The research-based method involves initially deriving topics from an analysis of the literature and subsequently authoring heuristics to address these topics. The evaluationbased method requires authors to derive heuristics from an analysis of findings acquired during prior evaluations. The heuristics shown in Table 1 were developed by analysing a broad range of existing accessibility guidelines. The team reviewed Web Content Accessibility Guidelines 1.0 (Web Content Accessibility Guidelines, 2004), Nielsen Accessibility Guidelines (Nielsen, 2001), IBM Web Accessibility Checklist 3.01 (IBM Web Accessibility Checklist 3.1, 2004), Guidelines for UK Government Web Sites (2004), and Section 508 Web Standards (2004). Qualitative analysis methods were then used to analyse the source guidelines and derive the heuristics. Firstly, the authors created a spreadsheet to aggregate
514
C. Paddison, P. Englefield / Interacting with Computers 16 (2004) 507–521
Table 1 IBM web accessibility heuristics Heuristic
Heuristic description
1. Provide meaningful and relevant alternatives to non-text elements
Images, graphs, video, sound, image maps, objects (applets, plug-ins, Portable Document Format (PDF) files) If important information is being conveyed, provide a suitable alternative Alternatives provide all users with the same relevant content Always provide a home page link
2. Support consistent and correctly tagged navigation
3. Allow complete and efficient keyboard usage 4. Respect users’ browser settings
5. Ensure appropriate use of standard and proprietary controls 6. Do not rely on colour alone to code and distinguish 7. Allow users control of potential distractions 8. Allow users to understand and control time restraints 9. Make certain the Web site is content compatible with assistive technologies
Always provide a skip to main content link Correctly tag headings Meaningfully title frames, correctly tag table headers Allows users to navigate by keyboard alone Permit large font sizes Ensure page layout adjusts itself to suit font size Allow preferred colour settings Text entry fields, drop-down menus, radio buttons, checkboxes, buttons Ensure controls do not stop the completion of the task Verify colour alone is not used for information Ensure contrast is sufficient in images Provide a means to stop blinking, flashing, flickering Allow users to control speed of scrolling for example ticker tape Notify user of time-out Allow users to request more time Make certain important and frequent tasks can be completed successfully using assistive technology tools for example screen reader, magnifier
the source guidelines and code entries by topic and user group. Secondly, they sorted and sifted these entries to identify useful relationships and categories. Thirdly, they authored a set of heuristics drawn from this scheme, referenced back to the original guidelines, and sequenced in order of decreasing power. Accessibility experts from a range of disciplines reviewed the heuristics and commented on correctness, coverage, and terminology. Reviewers included an expert involved in the development of standards as part of the World Wide Web Consortium (W3C) together with several usability professionals with broad experience of commercial goals and constraints. Following this review, the heuristics were tested for usability. 4. User evaluation In the user evaluation, participants applied the accessibility heuristics to undertake a heuristic evaluation of WorkTrain: The National Jobs and Learning Site (2004).
C. Paddison, P. Englefield / Interacting with Computers 16 (2004) 507–521
515
WorkTrain is a government Web site aimed at providing people with information about jobs and learning opportunities in the UK. The aim of the study was to understand whether participants found the heuristics easy to use, considering factors such as memorability, ease of use, and expressiveness. A sample of eight evaluators, with a range of experience in accessibility, was recruited. All were employed as consultants and had an interest in methods that would allow them to conduct their work more efficiently and effectively. These evaluators carried out a simple task, using the WorkTrain Web site to choose an appropriate training opportunity. Nine accessibility heuristics were used, together with an additional ‘catch-all’ accessibility heuristic. Evaluators were provided with a document titled ‘How to test for accessibility’ giving an explanation of how to check for specific accessibility issues. For example, ‘When testing images, image maps and graphs in Internet Explorer, Click Tools, Internet options, Advanced. Deselect Show pictures under the heading Multimedia. If conveying important information, images should be replaced with a suitable and meaningful alternative.’ The evaluation was conducted in two stages. After each stage, subjective data was recorded. In the first stage, participants logged their findings in the HEDB; in the second, they prioritised findings. Prioritising the findings involved evaluators reviewing all the findings and assigning a severity level to each finding. Severities were used to assess the degree of difficulty a design feature would be likely to cause a potential user (refer to Table 2). A questionnaire was administered before prioritising the findings. Evaluators were asked to agree or disagree with statements related to the heuristics, for example, ‘I found the heuristics sufficiently distinct from one another.’ A second questionnaire was administered after prioritising the findings. Evaluators were asked to agree or disagree with statements such as ‘After prioritising all the findings, I felt confident that I had logged my findings correctly.’ The pre-prioritisation questionnaire results (refer to Fig. 3 and Table 3) showed that 100% of participants found the heuristics easy to work with (A). Eighty-seven percent of participants found the heuristics distinct (B) and expressive (C), and the heuristic description sufficient (D). Seventy-five percent of participants found the scope of the heuristics sufficient (12% disagreed) (E); 75% were confident with their findings (12% disagreed) (F); 75% felt the heuristics enabled them to evaluate the Web site to the best of their ability (H); and 75% would use the heuristics in future evaluations
Table 2 Severities and severity descriptions Severity
Severity description
1 2 3 4 5
Some users will fail to complete this task Some users will experience severe difficulty in completing this task Some users will experience some difficulty in completing this task Some users will experience slight difficulties in completing this task This finding does not represent a real problem
516
C. Paddison, P. Englefield / Interacting with Computers 16 (2004) 507–521
Fig. 3. Graph showing pre-prioritisation questionnaire results.
(12% disagreed) (I). Sixty-two percent of participants felt that there was sufficient help documentation (G). Table 5 shows pre- and post-prioritisation subjective comments. The post-prioritisation questionnaire results (refer to Fig. 4 and Table 4) showed that 100% of participants felt that the severity ratings were relevant to the heuristics. Eighty-six percent of participants felt that the other evaluators had logged their findings correctly (M). Seventy-five percent of participants felt that they had logged their findings correctly (L) and were happy with the scope of the heuristics (N). Twenty-five percent of participants felt confident that all accessibility issues had been found (K). Thirty-eight percent were not confident that all accessibility issues had been found. Table 5 shows preprioritisation subjective comments. Although participants rated their satisfaction highly, their confidence in their findings was relatively low. Many participants compared the accessibility heuristic evaluation with a usability heuristic evaluation. They felt that the usability evaluation was much easier to carry out and reported that they had far more confidence in their findings. Many of Table 3 Table showing pre-prioritisation questionnaire results Pre-prioritisation questions
5
4
3
2
1
A. I found the heuristics easy to work with B. I found the heuristics sufficiently distinct from one another C. I felt the heuristic names were sufficiently expressive D. I felt the heuristic descriptions illustrated the heuristics sufficiently E. I found the scope of the heuristics was sufficient F. I felt confident that I had logged findings correctly G. I felt the help documentation provided was sufficient H. I felt the heuristics enabled me to evaluate the Web site to the best of my ability I. I feel happy about using the heuristics (as they are) in future evaluations
3 4 2 3 3 4 3 3 3
5 3 5 4 3 2 2 3 3
0 1 1 1 1 1 3 2 1
0 0 0 0 1 1 0 0 1
0 0 0 0 0 0 0 0 0
C. Paddison, P. Englefield / Interacting with Computers 16 (2004) 507–521
517
Fig. 4. Graph showing post-prioritisation questionnaire results.
the participants were not experienced in accessibility issues and were not familiar with the guidelines. These participants reported some difficulty in understanding how to apply the heuristics. Finally, participants commented on the usefulness of the help documentation. Many participants asked for further accessibility education and demonstrations to help increase their understanding of accessibility issues. Overall, these results indicate that the participating usability specialists found it somewhat difficult to carry out a heuristic evaluation for accessibility. This may be a result of limited prior knowledge of accessibility topics. With increased training and experience, confidence and effectiveness may improve. Table 4 Table shows post-prioritisation questionnaire results Post-prioritisation questions
5
4
3
2
1
J. How would you rate your accessibility knowledge? K. I found the severity ratings were relevant to the heuristics L. After prioritising all the findings, I felt confident that all accessibility issues had been found M. After prioritising all the findings, I felt confident that I had logged my findings correctly N. After prioritising all the findings, I felt confident that other evaluators had logged their findings correctly
0
0
3
4
1
4
4
0
0
0
1
1
3
2
1
0
6
2
0
0
1
5
1
0
0
518
C. Paddison, P. Englefield / Interacting with Computers 16 (2004) 507–521
Table 5 Pre- and post-prioritisation participant comments Participant comments from the pre-prioritisation questionnaire were
Participant comments from the post-prioritisation questionnaire
I thought that the explanations provided clarity as to what each heuristic meant. There was very clear information provided as to what an evaluator must do (in a practical sense) in order to check if a heuristic has been violated
Generally, the heuristics seemed very thorough. The extensive notes that were provided to support me were invaluable. I think that a run through demo with an accessibility expert would be useful to highlight some of the common problems that are found in sites The heuristics seem really good and all make sense—but I think (well it would help me) if we were given a bit of training on these heuristics and examples of ways in which Web sites can violate them
I was happy using the heuristics within the time constraints of this exercise—it wasn’t a complete review of the site but I did understand the heuristics
As a basic set, the heuristics are good. However, many evaluation tasks implied from the heuristics descriptions target-browsing difficulties that are caused by visual impairment. More work needed to target other impairments, such as motion impairment
5. Heuristic evaluation This test was run as an actual heuristic evaluation conducted within an on-going project to make the UsabilityNews (UN) (UsabilityNews, 2004) Web site accessible. The UN Web site is the sister site to the British HCI Group (2004). The site contains information on all the latest news from HCI and usability, events, products, jobs and paper calls. The overall aim of the UsabilityNews Accessibility Project was to review and improve the current Web site from an accessibility perspective. A key feature of the project was to understand ‘Is it relatively easy to make a Web site accessible after it is live, with minimal investment of time and money?’ The use of the accessibility heuristics was essential in order to keep the time and cost to a minimum. The objective of the study was to make UN more accessible, utilising accessibility heuristics to identify, categorise and prioritise problems with the current design. A task-based inspection method was used to carry out the evaluation, using a team of five evaluators from a range of disciplines. Task-based inspections allow important or frequent tasks to be tested, ensuring that key areas of the Web site undergo evaluation. Six typical tasks were followed, for example, ‘You are interested in jobs outside the UK. Use UsabilityNews to see if there are any suitable job vacancies.’ Nine accessibility heuristics were used plus one ‘catch-all’ usability heuristic. The findings showed accessibility issues relating to incorrect keyboard focus; some fixed font size issues; lack of navigation support; images which inhibit accessibility; visual and screen reader mismatches. The evaluation also highlighted a number of usability issues: inconsistent information structuring; invisible disclaimer; trust issues; inappropriately worded error messages; misunderstood instructions.
C. Paddison, P. Englefield / Interacting with Computers 16 (2004) 507–521
519
In this evaluation, no data was captured regarding evaluator’s use of the accessibility heuristics. However, the following paragraph lists feedback from the UN development team. ‘The heuristic evaluation proved extremely beneficial. The resulting report clearly broke down and prioritised the areas of the site, highlighting the more acute accessibility problems.’ This allowed the UN team to review the required changes and make decisions on which issues should be tackled now as part of this project, and which should wait until a complete redesign is carried out at a later date. The client was satisfied that the recommendations from the heuristic evaluation report could be implemented to make UN more accessible. The use of heuristics and the heuristic evaluation method ensure that findings are prioritised, providing the client with an easy to follow implementation plan.
6. Case study in practice The IBM Usability Competency Centre recently conducted a rapid, cost-effective accessibility audit of an IBM developed Web site. To satisfy this brief, the team recommended an ‘Accessibility Healthcheck’, conducted as a heuristic evaluation using three experienced evaluators. As the study would be conducted on a client site, the evaluators were briefed on the tasks and views to be assessed and to ensure access to the necessary assistive technologies. The team inspected the interface using the Web accessibility heuristics, described above, together with two sets of usability heuristics (Nielsen’s general interface heuristics (Nielsen, 1994) and an IBM internal set of Web heuristics). The team inspected for two hours, completing four key user tasks and logging 75 findings in the HEDB. On completing the inspection, the team returned to the office to edit the findings for good business English and to resolve any duplicates. On completion, 62 findings were returned to the evaluators for prioritisation. The prioritised data included one severity 1 finding, 20 severity 2 findings and 41 severity 3 findings. Accessibility findings included, JavaScript preventing progression in a wizard (Priority 1); heading markup not being utilised (Priority 2); and words being separated by only ‘/’ and therefore indistinguishable by a screen reader (Priority 3). A management summary provided a high level overview of issues and achievements with respect to data input, navigation support and browser support. The detailed findings and statistics were generated from the HEDB. Recommendations were also made on the need for style guides and training. By using the accessibility heuristics and HEDB, the team was able to design, execute, and communicate a comprehensive evaluation within two working days.
7. Conclusion Our experience suggests that, when used appropriately, the accessibility heuristics can usefully incorporate considerations of diversity within a heuristic evaluation.
520
C. Paddison, P. Englefield / Interacting with Computers 16 (2004) 507–521
The key benefits of the heuristics (conciseness, memorability, meaningfulness and insight) support their use as the primary resource to assess a Web site for accessibility, complementing the valuable representation of detailed knowledge in the guidelines. However, benefits depend on the evaluation objectives. It is practical to use the heuristics to understand to what degree the site is accessible and ascertain what specific areas of the site require changes. However, the heuristics should not be considered where legislation requires conformance with the guidelines. A key concern highlighted in this paper is the extent to which evaluators should have knowledge of accessibility issues before undertaking an evaluation using the heuristics. It is essential to educate evaluators to a level where they are sufficiently familiar with the guidelines and related resources to have confidence in their findings. The heuristics should cue rather than fully embody knowledge. Combining experience of conducting accessibility heuristic evaluations with other more formal training methods may help to give evaluators this confidence. The HEDB is a powerful resource that enables evaluators to inspect the accessibility of interfaces cheaply and effectively. This gives Web site owners incentives to assess the accessibility of their interface before planning manageable steps to create a ‘barrier-free’ site. The study critique suggests that providing evaluators with sufficient accessibility education before completing the evaluation would have resulted in a greater understanding of the accessibility findings. As a result, evaluators relied on the heuristics as a key source of information rather than using them as a mnemonic for accessing their own knowledge. It is much harder to evaluate a site for accessibility (rather than usability) if you are not proficient in that area. I found it easier to look at the heuristics and then go through the site looking for things that violated them (rather than trying out the site, finding a problem and recognising which heuristic it belonged to). Ideas for future work include: † Conduct a study to understand what problems evaluators encounter when working with accessibility guidelines and compare these results to the studies illustrated in this paper. † Develop accessibility training to bring evaluators knowledge up to a level where they understand each of the heuristics, to the extent that they are confident in their own findings. † Capture evaluator’s feedback in future accessibility heuristic evaluations, either as an informal feedback session or as a questionnaire, and use this feedback to inform future development of the accessibility heuristics.
References Bobby. http://bobby.watchfire.com/bobby/html/en/index.jsp. Last accessed on 05.01.2004. British HCI Group. http://www.bcs-hci.org.uk/. Last accessed on 05.01.2004. Disability Discrimination Act 1995. http://www.disability.gov.uk/dda. Last accessed on 05.01.2004.
C. Paddison, P. Englefield / Interacting with Computers 16 (2004) 507–521
521
e-Accessibility: Accessibility and Information Society. http://europa.eu.int/information_society/topics/citizens/ accessibility/index_en.htm. Last accessed on 05.01.2004. Englefield, P.J., 2003. A Pragmatic Framework for Selecting Empirical or Inspection Methods to Evaluate Usability, IBM Easy site http://www-306.ibm.com/ibm/easy/eou_ext.nsf/Publish/50?OpenDocument&./ Publish/1118/$File/paper1118.pdf. Last accessed on 05.01.2004. Englefield, P.J., 2003. HEDB: a software tool to support heuristic evaluation. In: Gray, P., et al. (Eds.), Proceedings of HCI 2003, Designing for Society 17th British HCI Group Annual Conference, Research Press International, Champaign, IL. Gray, W.D., Salzman, M.C., 1998. Damaged merchandise? A review of experiments that compare usability and evaluation methods. In: Human–Computer Interaction. p. 13. Guidelines for UK Government Web Sites. http://www.e-envoy.gov.uk/oee/oee.nsf/sections/ Webguidelines-handbook-top/$file/handbookindex.htm. Last accessed on 05.01.2004. IBM User Engineering. http://www.ibm.com/easy/userengineering. Last accessed on 05.01.2004. IBM Web Accessibility Checklist 3.1. http://www.ibm.com/able/guidelines/web/accessweb.html. Last accessed on 05.01.2004. Industrial Designers Society of America Showcase. http://new.idsa.org/galleries.htm. Last accessed on 05.01.2004. Internet PTI Best Practice—Preliminary Usability Heuristics. http://www.trg.soton.ac.uk/pti/principles.html. Last accessed on 05.01.2004. Jeffries, R. et al., 1991. User interface evaluation in the real world: a comparison of four techniques, Published in Human Factors in Computing Systems CHI’91 Conference Proceedings. Karat C-M: et al., 1992. Comparison of empirical testing and walkthrough methods in user interface evaluation. In: Human Factors in Computing Systems CHI’92 Conference Proceedings. Kelley, T., et al., 2001. The art of innovation, Currency Books. McEwan, T., et al., 2003. Learning about universal access, Published in Proceedings of HCI2003: Designing for Society, vol. 2. Nielsen J., 1994. Usability Engineering, Academic Press, Boston. Nielsen J., 2001. Beyond ALT Text: Making the Web Easy to Use for Users With Disabilities, Nielsen Norman Group Report. Section 508 Web Standards. http://www.section508.gov/index.cfm?FuseAction ¼ Content&ID ¼ 12 (part of the American’s with Disabilities Act (ADA) http://www.usdoj.gov/crt/ada/adahom1.htm). Last accessed on 05.01.2004. Ten Usability Heuristics. Nielsen, J., 1994 http://www.useit.com/papers/heuristic/heuristic . _list.html. Last accessed on 05.01.2004. UsabilityNews. http://www.usabilitynews.com. Last accessed on 05.01.2004. Web Content Accessibility Guidelines (WCAG). http://www.w3.org/TR/WCAG10/. Last accessed on 05.01.2004. Whatis?com, part of the TechTarget Network of IT Enterprise Web sites. http://whatis.techtarget.com/definition/ 0,sid9_gci212246,00.html. Last accessed on 05.01.2004. WorkTrain: The National Jobs and Learning Site. http://www.worktrain.gov.uk/. Last accessed on 05.01.2004.
Applying heuristics to accessibility inspections Claire Paddison*, Paul Englefield1 Ease of Use, IBM Warwick MP5, P.O. Box 31, Birmingham Road, Warwick CV34 5JL, UK Available online 7 May 2004
Abstract Accessibility heuristics have been developed to complement accessibility guidelines. The use of Web accessibility heuristics in heuristic evaluations considers a greater range of special needs, such as visual impairments to cognitive disabilities. Key advantages of heuristics are conciseness, memorability, meaningfulness and insight. The heuristics allow evaluators to understand effectively which areas of a site have accessibility issues and provide useful insight into how to create a solution. However, the heuristics will not tell evaluators whether a Web site conforms to legislation. Studies have confirmed the view that while heuristics do not substitute for expertise, they do act to cue the deeper body of knowledge defined by the guidelines. It is essential that evaluators receive accessibility education before completing a heuristic evaluation using the accessibility heuristics. q 2004 Elsevier B.V. All rights reserved. Keywords: Accessibility; Heuristic evaluation; Accessibility heuristics; IBM heuristic evaluation database
1. Introduction A heuristic ‘pertains to the process of gaining knowledge or some desired result by intelligent guesswork rather than by following some pre-established formula’ (Whatis?com, 2004). Heuristics are widely utilised in usability and considered useful in other fields, such as artificial intelligence and medicine. In this paper, heuristics have been extended to consider the notion of accessibility. By using specific accessibility heuristics to assess a Web site, evaluators consider a greater range of special accessibility needs. Examples include visual, motor, and cognitive impairments. By rigorously addressing the issues identified by such an evaluation, designers can ensure that the site will be more accessible to a larger and more diverse audience. * Corresponding author. Tel.: þ 44-1926-464978; fax: þ 44-1926-465323. E-mail addresses: [email protected] (C. Paddison), [email protected] (P. Englefield). 1 Tel.: þ44-1926-464329; fax: þ44-1926-465323. 0953-5438/$ - see front matter q 2004 Elsevier B.V. All rights reserved. doi:10.1016/j.intcom.2004.04.007
508
C. Paddison, P. Englefield / Interacting with Computers 16 (2004) 507–521
This paper aims to show that accessibility heuristics recognise diversity in heuristic evaluations and can provide an alternative representation of the guidelines that is suitable for use during an inspection process. ‘Barrier-free’ design is a concise way of describing an accessible interface for people with disabilities. In the same way, universal accessibility ‘ensures that sites are developed to serve the largest possible audience using the broadest range of hardware and software platforms, and that the needs of users with impairments are considered’ (McEwan et al., 2003). Being ‘barrier-free’ means that the information or service you provide must be available to everyone regardless of software, platform, environment, and user ability. Examples include: for software, access using desktop browser or voice browser; for platform use of a desktop, mobile phone or PDA (Personal Digital Assistant); for environment, working in a noisy or under-illuminated surroundings; and, for ability, visual impairment or dyslexia. IBM practitioners distinguish two aspects of accessibility: (a) technical accessibility and (b) usable accessibility. These aspects affect people with disabilities in different ways. Technical accessibility relates to the adoption of best practice in software engineering. Examples include labelling ‘alt’ tags appropriately, and ensuring that the interface can be accessed using the keyboard alone. Technical accessibility ensures that assistive technologies, used by people with disabilities, can understand and interact usefully with the different elements of the design. Automated tools, such as Bobby (2004), provide some level of support in testing for technical accessibility. By contrast, usable accessibility relates to the application of usability principles for the benefit of those with disabilities. Examples here include support for users’ tasks, consistent navigation, and management of cognitive load. Both aspects of accessibility are critical to users’ success. For example, a technically accessible Web site may still present challenges to users of assistive technologies. Such challenges may result directly from the need to use assistive technologies, or may arise from more fundamental usability issues that adversely affect all users. In summary, it is not enough to follow accessible guidelines and make the appropriate technical accessibility changes. People with special accessibility needs have a distinct user profile with unique requirements that need to be addressed within a user-centred design process. Designing for accessibility is increasingly seen as an effective response to a growing recognition of the needs of people with disabilities. Personal independence is a critical need that that can be supported by providing accessible online information and services. For example, simple adjustments, such as ensuring fonts are resizable, can help users with low vision to access the Web. Keyboard enabling supports users who cannot use a mouse, due to upper limb disorders. Graphical representations can help users with learning difficulties to better comprehend online information. With the introduction of legislation to develop Web accessibility, these issues are becoming increasingly important to organisations. In the US, Section 508 (2004) requires that all Web sites ‘developed, procured, maintained, or used by the Federal government’ must be accessible. In the UK, an amendment to the Disability Discrimination Act (Disability Discrimination Act 1995, 2004) states that organisations that provide goods, facilities and services via the Web are legally obliged to make their Web sites accessible to disabled people. Guidance is specifically offered to e-commerce Web sites. In addition to
C. Paddison, P. Englefield / Interacting with Computers 16 (2004) 507–521
509
the above US and UK legislation, there are number of related initiatives in progress at a European level. From a business perspective, making services and products accessible is likely to increase market share and audience reach. Tom Kelley of IDEO, one of the world’s leading design companies, believes that work for people with disabilities will increasingly result in innovations that benefit the general population. He says, “By studying people of all ages, shapes, cultures and sizes, we’ve learned that the best products embrace people’s differences” (Kelley, 2001). A good example of this is OXO Good Grips, a line of kitchen tools with cushioned, oversized handles that are comfortable and easy to use. In 1990, Sam Farber first questioned the effectiveness of kitchen gadgets in response to his wife’s difficulty in gripping kitchen tools due to a slight case of arthritis. Since then, OXO Good Grips have won praise for good design and grown to a line of 350 products with a 15% share of the US market (Industrial Designers Society of America Showcase, 2004). The European Union has been active in the area of accessibility since 1999. At this time, the eEurope objective was ‘to bring every citizen, school and business online and to exploit the potential of the new economy for growth, employment, and inclusion’ (e-Accessibility: Accessibility and Information Society, 2004). Six years on, the emphasis has shifted to achieving ‘an Information Society for All’ with the visionary goal of ensuring an ‘inclusive digital society that provides opportunities for all, thus minimising the risk of digital divide.’ The majority of UK government services are planned to be online by 2005. Insufficient attention to accessibility could result in the marginalisation of many groups and the consequent creation of a ‘digital divide’. This final point includes arguments from each of the three considerations discussed above. The digital divide is a major focus for Europe’s e-Accessibility initiative, helping to drive the arguments for accessibility across Europe. Clearly, accessibility is a critical consideration in design. Whether designers integrate accessibility guidelines into the design process or apply them retrospectively, heuristic evaluation provides a cost-effective way to inspect for compliance.
2. Heuristic evaluation Heuristic Evaluation is an established usability evaluation technique originally proposed by Jakob Nielsen as a cost-effective ‘discount’ usability technique (Nielsen, 1994). A panel of experts formally assesses an interface design with respect to a set of heuristics or rules of thumb for effective design in an inspection method. A typical study has five phases: (1) study design, (2) discovery, (3) editing to identify and resolve duplicate and related findings, (4) prioritisation to identify severities, and (5) analysis and report writing. During the design phase, a study manager selects or creates a set of heuristics appropriate for the study goals. In the discovery phase, evaluators use these heuristics as mnemonics to identify design features that conflict with some aspect of best practice. In commercial practice, heuristics classify findings at the time of discovery to support quantitative analysis during the report writing phase. Heuristics provide a convenient mnemonic framework for existing evaluator expertise. In the same way, the accessibility heuristics have been designed to act as a mnemonic for
510
C. Paddison, P. Englefield / Interacting with Computers 16 (2004) 507–521
evaluator’s own knowledge of the concepts of accessibility. An unskilled evaluator relying on heuristics alone expects to be less effective than a skilled practitioner is. Consequently, a good heuristic should be concise, memorable, expressive and easy to relate to underlying knowledge and principles. A set of heuristics should ideally be concise, exhaustive and mutually exclusive. Examples of general purpose heuristics include Nielsen’s original set (Nielsen, 1994) and Nielsen and Molich’s revised set (Ten Usability Heuristics, 2004). An example of a domain specific set is the draft public transport Web site heuristics (Internet PTI Best Practice, 2004) developed by Southampton University’s Transport Research Group. Heuristic sets are often aggregated. For example, to assess a metro site, either of the general purpose sets described above may combine successfully with the public transport Web site set. The literature reflects a vigorous debate about the relative effectiveness and costefficiency of inspection methods with respect to empirical testing (Gray and Salzman, 1998; Jeffries et al., 1991; Karat et al., 1992). However, within IBM (Englefield, 2003a), heuristic evaluation is used in a range of situations: (1) where project constraints limit access to representative users; (2) where rapid turnaround is required; (3) where evaluators cannot work together at the same time and location; (4) where breadth of focus is more important than sensitivity, and (5) where a strong client relationship reduces the need to support advice with evidence directly gathered from real users. Consequently, heuristic evaluation is widely applied to both commercial services offerings and the design of IBM products within a User Engineering (IBM, 2004) process (Nielsen, 2001). In some cases, heuristic evaluation is applied as a pre-screen before empirical testing or as a triangulated study in conjunction with a user test. Situations (1) and (4) may well apply to an inspection of a design intended to be inclusive for a diverse range of physical, cognitive and motivational factors. Furthermore, heuristic evaluation offers strong support for the established inspection-oriented approach to accessibility evaluation, adding structure and rigor to a mature knowledge-oriented discipline. The Heuristic Evaluation Database (HEDB) (Englefield, 2003b) (see Fig. 1) implements heuristic evaluation as a Lotus Notes database supporting a collaborative workflow process in which the roles of Evaluation Manager and Evaluator carry out tasks in each of the five phases described above. The database provides efficient support for the key tasks of entering findings, adding severities, and editing duplicates. The tool automatically generates a skeleton report (see Fig. 2) to provide quantitative cross-tabulated analysis of the distribution of errors by task, location and heuristic together with a structured list of finding details. Practitioners report that these analyses are helpful in forming client recommendations; “…absolutely…helps identify areas in the design that need particular focus”. A study was conducted to assess the effectiveness of the HEDB (Englefield, 2003b). Five evaluators and a study manager carried out a realistic inspection of a commercial Web site and commented on their experiences. Results indicate that the database is easy to learn and efficient to use while addressing many of the practical concerns of applying Heuristic Evaluation in industry. For example, practitioners rated the database 95% for learnability and 90% for efficiency where scores 0 and 100% represent the minimum and maximum scores on a Likert scale. Subsequent feedback from commercial use suggests
C. Paddison, P. Englefield / Interacting with Computers 16 (2004) 507–521
Fig. 1. Sample view of the user interface for the heuristic evaluation database.
Fig. 2. Extract from HEDB.
511
512
C. Paddison, P. Englefield / Interacting with Computers 16 (2004) 507–521
that using the database might save ‘two to three days’ otherwise required for low level data management and formatting of results.
3. Accessibility heuristics The HEDB incorporates various sets of reusable usability heuristics derived from IBM and external sources. These include a set of accessibility heuristics that have been developed by IBM as a complement to usability heuristics. Accessibility heuristics identify areas of a Web site that are not accessible for some reason. Two examples follow: 1. key JavaScript elements not activated because scripting is disabled; 2. non-standard proprietary controls used at key stages of the navigation prevent assistive technologies from interacting correctly. In addition to identifying design errors, heuristic evaluation can also help designers to plan which areas of a site to make accessible first. This can be particularly useful when retrospectively using guidelines to assess an implemented site. Accessibility guidelines also have legal implications. Using the accessibility heuristics to conduct a heuristic evaluation will not tell evaluators whether a Web site conforms to legislation. However, it will (a) highlight those tasks that users are likely to have difficulties in completing, and (b) identify those elements of the site, such as navigation or consistency, that may require attention before the site meets legislative requirements. Many of the accessibility guidelines follow usability best practices. By using a set of usability heuristics together with the accessibility heuristics, both aspects (usable accessibility and technical accessibility) are addressed. In both cases, the heuristics act as a mnemonic framework for accessing a broader knowledge of usability and accessibility knowledge. Usability heuristics are commonplace in usability practice; there is now a growing need for a workable and usable set of accessibility heuristics. It is therefore important to understand why accessibility heuristics are preferred to guidelines as a resource for accessibility assessment. Guidelines are typically bulky and contain large numbers of individual items. For example, the Web Content Accessibility Guidelines 1.0 (Web Content Accessibility Guidelines, 2004) contain 65 guidelines (2.1 Ensure that all information conveyed with color is also available without color, for example from context or markup), divided into 14 main areas (2. Don’t rely on color alone), and divided again into three priority levels (Priority 1.) Because it takes significant time and effort to search for guidelines, assessment can be slow and expensive. By contrast, the IBM heuristics, together with their brief descriptions, fit on one side of letter paper. A simple scan enables evaluators to rapidly locate and categorise accessibility problems; in some cases an assessment can be carried out in a single day. Guidelines can be hard to remember; without regular use, their size and number make them difficult to retain. On the other hand, heuristics are specifically designed as a concise,
C. Paddison, P. Englefield / Interacting with Computers 16 (2004) 507–521
513
memorable set of compact, pithy, and generative points. A typical example is, ‘Respect users’ browser settings’. Many of the guidelines are intuitive; some, however, require a technical background or further study to fully understand their true meaning. An example is, ‘Use markup and style sheets and do so properly.’ Heuristics are expressive. Evaluators should look at the heuristic and think, ‘Yes, I understand that.’ They also consider a broad range of factors such as navigation and consistency, rather than focusing on a very specialist area. Making accessibility heuristics more meaningful to a broad range of people develops understanding of accessibility issues. It also allows a multi-disciplinary team to be able to inspect an interface, involving stakeholders other than accessibility experts. During the heuristic evaluation process, evaluators prioritise findings. This gives the study manager a clear indication as to which aspects of the design are causing problems and what the solution may be. For example, if many of the problems are related to consistency, then a style guide may be indicated. Although the guidelines are grouped by category and priority, their relative size leads to a less distinct prioritisation scheme. The characteristics discussed above support the use of accessibility heuristics as the immediate representation of accessibility knowledge with the ability to perform an inspection. However, there are disadvantages associated with using heuristics in certain scenarios. The Web Content Accessibility Guidelines 1.0 (Web Content Accessibility Guidelines, 2004) are extremely detailed and have additional links to technical advice and other useful information resources. As a result, it is clear whether an aspect of a Web site conforms to the guidelines and therefore subsequent legislation. The accessibility heuristics, even with the descriptions, do not provide distinct information. A degree of interpretation is always required. Using the accessibility heuristics will not tell the evaluators whether a Web site conforms to legislation. The heuristics are only able to highlight which tasks users will have difficulties in performing. Guidelines are able to provide organisations with clarity that they have complied with current legislation. Heuristics are intended as a mnemonic framework to cue an evaluator’s existing expertise. Heuristic evaluators should relate the finding to a particular heuristic rather than use the heuristics alone to search for errors. Using evaluators with limited accessibility knowledge may lead to misapplication of the heuristics. Table 1 shows the nine IBM Web Accessibility Heuristics and their descriptions. IBM’s experience suggests two practical methods for developing a set of heuristics. The research-based method involves initially deriving topics from an analysis of the literature and subsequently authoring heuristics to address these topics. The evaluationbased method requires authors to derive heuristics from an analysis of findings acquired during prior evaluations. The heuristics shown in Table 1 were developed by analysing a broad range of existing accessibility guidelines. The team reviewed Web Content Accessibility Guidelines 1.0 (Web Content Accessibility Guidelines, 2004), Nielsen Accessibility Guidelines (Nielsen, 2001), IBM Web Accessibility Checklist 3.01 (IBM Web Accessibility Checklist 3.1, 2004), Guidelines for UK Government Web Sites (2004), and Section 508 Web Standards (2004). Qualitative analysis methods were then used to analyse the source guidelines and derive the heuristics. Firstly, the authors created a spreadsheet to aggregate
514
C. Paddison, P. Englefield / Interacting with Computers 16 (2004) 507–521
Table 1 IBM web accessibility heuristics Heuristic
Heuristic description
1. Provide meaningful and relevant alternatives to non-text elements
Images, graphs, video, sound, image maps, objects (applets, plug-ins, Portable Document Format (PDF) files) If important information is being conveyed, provide a suitable alternative Alternatives provide all users with the same relevant content Always provide a home page link
2. Support consistent and correctly tagged navigation
3. Allow complete and efficient keyboard usage 4. Respect users’ browser settings
5. Ensure appropriate use of standard and proprietary controls 6. Do not rely on colour alone to code and distinguish 7. Allow users control of potential distractions 8. Allow users to understand and control time restraints 9. Make certain the Web site is content compatible with assistive technologies
Always provide a skip to main content link Correctly tag headings Meaningfully title frames, correctly tag table headers Allows users to navigate by keyboard alone Permit large font sizes Ensure page layout adjusts itself to suit font size Allow preferred colour settings Text entry fields, drop-down menus, radio buttons, checkboxes, buttons Ensure controls do not stop the completion of the task Verify colour alone is not used for information Ensure contrast is sufficient in images Provide a means to stop blinking, flashing, flickering Allow users to control speed of scrolling for example ticker tape Notify user of time-out Allow users to request more time Make certain important and frequent tasks can be completed successfully using assistive technology tools for example screen reader, magnifier
the source guidelines and code entries by topic and user group. Secondly, they sorted and sifted these entries to identify useful relationships and categories. Thirdly, they authored a set of heuristics drawn from this scheme, referenced back to the original guidelines, and sequenced in order of decreasing power. Accessibility experts from a range of disciplines reviewed the heuristics and commented on correctness, coverage, and terminology. Reviewers included an expert involved in the development of standards as part of the World Wide Web Consortium (W3C) together with several usability professionals with broad experience of commercial goals and constraints. Following this review, the heuristics were tested for usability. 4. User evaluation In the user evaluation, participants applied the accessibility heuristics to undertake a heuristic evaluation of WorkTrain: The National Jobs and Learning Site (2004).
C. Paddison, P. Englefield / Interacting with Computers 16 (2004) 507–521
515
WorkTrain is a government Web site aimed at providing people with information about jobs and learning opportunities in the UK. The aim of the study was to understand whether participants found the heuristics easy to use, considering factors such as memorability, ease of use, and expressiveness. A sample of eight evaluators, with a range of experience in accessibility, was recruited. All were employed as consultants and had an interest in methods that would allow them to conduct their work more efficiently and effectively. These evaluators carried out a simple task, using the WorkTrain Web site to choose an appropriate training opportunity. Nine accessibility heuristics were used, together with an additional ‘catch-all’ accessibility heuristic. Evaluators were provided with a document titled ‘How to test for accessibility’ giving an explanation of how to check for specific accessibility issues. For example, ‘When testing images, image maps and graphs in Internet Explorer, Click Tools, Internet options, Advanced. Deselect Show pictures under the heading Multimedia. If conveying important information, images should be replaced with a suitable and meaningful alternative.’ The evaluation was conducted in two stages. After each stage, subjective data was recorded. In the first stage, participants logged their findings in the HEDB; in the second, they prioritised findings. Prioritising the findings involved evaluators reviewing all the findings and assigning a severity level to each finding. Severities were used to assess the degree of difficulty a design feature would be likely to cause a potential user (refer to Table 2). A questionnaire was administered before prioritising the findings. Evaluators were asked to agree or disagree with statements related to the heuristics, for example, ‘I found the heuristics sufficiently distinct from one another.’ A second questionnaire was administered after prioritising the findings. Evaluators were asked to agree or disagree with statements such as ‘After prioritising all the findings, I felt confident that I had logged my findings correctly.’ The pre-prioritisation questionnaire results (refer to Fig. 3 and Table 3) showed that 100% of participants found the heuristics easy to work with (A). Eighty-seven percent of participants found the heuristics distinct (B) and expressive (C), and the heuristic description sufficient (D). Seventy-five percent of participants found the scope of the heuristics sufficient (12% disagreed) (E); 75% were confident with their findings (12% disagreed) (F); 75% felt the heuristics enabled them to evaluate the Web site to the best of their ability (H); and 75% would use the heuristics in future evaluations
Table 2 Severities and severity descriptions Severity
Severity description
1 2 3 4 5
Some users will fail to complete this task Some users will experience severe difficulty in completing this task Some users will experience some difficulty in completing this task Some users will experience slight difficulties in completing this task This finding does not represent a real problem
516
C. Paddison, P. Englefield / Interacting with Computers 16 (2004) 507–521
Fig. 3. Graph showing pre-prioritisation questionnaire results.
(12% disagreed) (I). Sixty-two percent of participants felt that there was sufficient help documentation (G). Table 5 shows pre- and post-prioritisation subjective comments. The post-prioritisation questionnaire results (refer to Fig. 4 and Table 4) showed that 100% of participants felt that the severity ratings were relevant to the heuristics. Eighty-six percent of participants felt that the other evaluators had logged their findings correctly (M). Seventy-five percent of participants felt that they had logged their findings correctly (L) and were happy with the scope of the heuristics (N). Twenty-five percent of participants felt confident that all accessibility issues had been found (K). Thirty-eight percent were not confident that all accessibility issues had been found. Table 5 shows preprioritisation subjective comments. Although participants rated their satisfaction highly, their confidence in their findings was relatively low. Many participants compared the accessibility heuristic evaluation with a usability heuristic evaluation. They felt that the usability evaluation was much easier to carry out and reported that they had far more confidence in their findings. Many of Table 3 Table showing pre-prioritisation questionnaire results Pre-prioritisation questions
5
4
3
2
1
A. I found the heuristics easy to work with B. I found the heuristics sufficiently distinct from one another C. I felt the heuristic names were sufficiently expressive D. I felt the heuristic descriptions illustrated the heuristics sufficiently E. I found the scope of the heuristics was sufficient F. I felt confident that I had logged findings correctly G. I felt the help documentation provided was sufficient H. I felt the heuristics enabled me to evaluate the Web site to the best of my ability I. I feel happy about using the heuristics (as they are) in future evaluations
3 4 2 3 3 4 3 3 3
5 3 5 4 3 2 2 3 3
0 1 1 1 1 1 3 2 1
0 0 0 0 1 1 0 0 1
0 0 0 0 0 0 0 0 0
C. Paddison, P. Englefield / Interacting with Computers 16 (2004) 507–521
517
Fig. 4. Graph showing post-prioritisation questionnaire results.
the participants were not experienced in accessibility issues and were not familiar with the guidelines. These participants reported some difficulty in understanding how to apply the heuristics. Finally, participants commented on the usefulness of the help documentation. Many participants asked for further accessibility education and demonstrations to help increase their understanding of accessibility issues. Overall, these results indicate that the participating usability specialists found it somewhat difficult to carry out a heuristic evaluation for accessibility. This may be a result of limited prior knowledge of accessibility topics. With increased training and experience, confidence and effectiveness may improve. Table 4 Table shows post-prioritisation questionnaire results Post-prioritisation questions
5
4
3
2
1
J. How would you rate your accessibility knowledge? K. I found the severity ratings were relevant to the heuristics L. After prioritising all the findings, I felt confident that all accessibility issues had been found M. After prioritising all the findings, I felt confident that I had logged my findings correctly N. After prioritising all the findings, I felt confident that other evaluators had logged their findings correctly
0
0
3
4
1
4
4
0
0
0
1
1
3
2
1
0
6
2
0
0
1
5
1
0
0
518
C. Paddison, P. Englefield / Interacting with Computers 16 (2004) 507–521
Table 5 Pre- and post-prioritisation participant comments Participant comments from the pre-prioritisation questionnaire were
Participant comments from the post-prioritisation questionnaire
I thought that the explanations provided clarity as to what each heuristic meant. There was very clear information provided as to what an evaluator must do (in a practical sense) in order to check if a heuristic has been violated
Generally, the heuristics seemed very thorough. The extensive notes that were provided to support me were invaluable. I think that a run through demo with an accessibility expert would be useful to highlight some of the common problems that are found in sites The heuristics seem really good and all make sense—but I think (well it would help me) if we were given a bit of training on these heuristics and examples of ways in which Web sites can violate them
I was happy using the heuristics within the time constraints of this exercise—it wasn’t a complete review of the site but I did understand the heuristics
As a basic set, the heuristics are good. However, many evaluation tasks implied from the heuristics descriptions target-browsing difficulties that are caused by visual impairment. More work needed to target other impairments, such as motion impairment
5. Heuristic evaluation This test was run as an actual heuristic evaluation conducted within an on-going project to make the UsabilityNews (UN) (UsabilityNews, 2004) Web site accessible. The UN Web site is the sister site to the British HCI Group (2004). The site contains information on all the latest news from HCI and usability, events, products, jobs and paper calls. The overall aim of the UsabilityNews Accessibility Project was to review and improve the current Web site from an accessibility perspective. A key feature of the project was to understand ‘Is it relatively easy to make a Web site accessible after it is live, with minimal investment of time and money?’ The use of the accessibility heuristics was essential in order to keep the time and cost to a minimum. The objective of the study was to make UN more accessible, utilising accessibility heuristics to identify, categorise and prioritise problems with the current design. A task-based inspection method was used to carry out the evaluation, using a team of five evaluators from a range of disciplines. Task-based inspections allow important or frequent tasks to be tested, ensuring that key areas of the Web site undergo evaluation. Six typical tasks were followed, for example, ‘You are interested in jobs outside the UK. Use UsabilityNews to see if there are any suitable job vacancies.’ Nine accessibility heuristics were used plus one ‘catch-all’ usability heuristic. The findings showed accessibility issues relating to incorrect keyboard focus; some fixed font size issues; lack of navigation support; images which inhibit accessibility; visual and screen reader mismatches. The evaluation also highlighted a number of usability issues: inconsistent information structuring; invisible disclaimer; trust issues; inappropriately worded error messages; misunderstood instructions.
C. Paddison, P. Englefield / Interacting with Computers 16 (2004) 507–521
519
In this evaluation, no data was captured regarding evaluator’s use of the accessibility heuristics. However, the following paragraph lists feedback from the UN development team. ‘The heuristic evaluation proved extremely beneficial. The resulting report clearly broke down and prioritised the areas of the site, highlighting the more acute accessibility problems.’ This allowed the UN team to review the required changes and make decisions on which issues should be tackled now as part of this project, and which should wait until a complete redesign is carried out at a later date. The client was satisfied that the recommendations from the heuristic evaluation report could be implemented to make UN more accessible. The use of heuristics and the heuristic evaluation method ensure that findings are prioritised, providing the client with an easy to follow implementation plan.
6. Case study in practice The IBM Usability Competency Centre recently conducted a rapid, cost-effective accessibility audit of an IBM developed Web site. To satisfy this brief, the team recommended an ‘Accessibility Healthcheck’, conducted as a heuristic evaluation using three experienced evaluators. As the study would be conducted on a client site, the evaluators were briefed on the tasks and views to be assessed and to ensure access to the necessary assistive technologies. The team inspected the interface using the Web accessibility heuristics, described above, together with two sets of usability heuristics (Nielsen’s general interface heuristics (Nielsen, 1994) and an IBM internal set of Web heuristics). The team inspected for two hours, completing four key user tasks and logging 75 findings in the HEDB. On completing the inspection, the team returned to the office to edit the findings for good business English and to resolve any duplicates. On completion, 62 findings were returned to the evaluators for prioritisation. The prioritised data included one severity 1 finding, 20 severity 2 findings and 41 severity 3 findings. Accessibility findings included, JavaScript preventing progression in a wizard (Priority 1); heading markup not being utilised (Priority 2); and words being separated by only ‘/’ and therefore indistinguishable by a screen reader (Priority 3). A management summary provided a high level overview of issues and achievements with respect to data input, navigation support and browser support. The detailed findings and statistics were generated from the HEDB. Recommendations were also made on the need for style guides and training. By using the accessibility heuristics and HEDB, the team was able to design, execute, and communicate a comprehensive evaluation within two working days.
7. Conclusion Our experience suggests that, when used appropriately, the accessibility heuristics can usefully incorporate considerations of diversity within a heuristic evaluation.
520
C. Paddison, P. Englefield / Interacting with Computers 16 (2004) 507–521
The key benefits of the heuristics (conciseness, memorability, meaningfulness and insight) support their use as the primary resource to assess a Web site for accessibility, complementing the valuable representation of detailed knowledge in the guidelines. However, benefits depend on the evaluation objectives. It is practical to use the heuristics to understand to what degree the site is accessible and ascertain what specific areas of the site require changes. However, the heuristics should not be considered where legislation requires conformance with the guidelines. A key concern highlighted in this paper is the extent to which evaluators should have knowledge of accessibility issues before undertaking an evaluation using the heuristics. It is essential to educate evaluators to a level where they are sufficiently familiar with the guidelines and related resources to have confidence in their findings. The heuristics should cue rather than fully embody knowledge. Combining experience of conducting accessibility heuristic evaluations with other more formal training methods may help to give evaluators this confidence. The HEDB is a powerful resource that enables evaluators to inspect the accessibility of interfaces cheaply and effectively. This gives Web site owners incentives to assess the accessibility of their interface before planning manageable steps to create a ‘barrier-free’ site. The study critique suggests that providing evaluators with sufficient accessibility education before completing the evaluation would have resulted in a greater understanding of the accessibility findings. As a result, evaluators relied on the heuristics as a key source of information rather than using them as a mnemonic for accessing their own knowledge. It is much harder to evaluate a site for accessibility (rather than usability) if you are not proficient in that area. I found it easier to look at the heuristics and then go through the site looking for things that violated them (rather than trying out the site, finding a problem and recognising which heuristic it belonged to). Ideas for future work include: † Conduct a study to understand what problems evaluators encounter when working with accessibility guidelines and compare these results to the studies illustrated in this paper. † Develop accessibility training to bring evaluators knowledge up to a level where they understand each of the heuristics, to the extent that they are confident in their own findings. † Capture evaluator’s feedback in future accessibility heuristic evaluations, either as an informal feedback session or as a questionnaire, and use this feedback to inform future development of the accessibility heuristics.
References Bobby. http://bobby.watchfire.com/bobby/html/en/index.jsp. Last accessed on 05.01.2004. British HCI Group. http://www.bcs-hci.org.uk/. Last accessed on 05.01.2004. Disability Discrimination Act 1995. http://www.disability.gov.uk/dda. Last accessed on 05.01.2004.
C. Paddison, P. Englefield / Interacting with Computers 16 (2004) 507–521
521
e-Accessibility: Accessibility and Information Society. http://europa.eu.int/information_society/topics/citizens/ accessibility/index_en.htm. Last accessed on 05.01.2004. Englefield, P.J., 2003. A Pragmatic Framework for Selecting Empirical or Inspection Methods to Evaluate Usability, IBM Easy site http://www-306.ibm.com/ibm/easy/eou_ext.nsf/Publish/50?OpenDocument&./ Publish/1118/$File/paper1118.pdf. Last accessed on 05.01.2004. Englefield, P.J., 2003. HEDB: a software tool to support heuristic evaluation. In: Gray, P., et al. (Eds.), Proceedings of HCI 2003, Designing for Society 17th British HCI Group Annual Conference, Research Press International, Champaign, IL. Gray, W.D., Salzman, M.C., 1998. Damaged merchandise? A review of experiments that compare usability and evaluation methods. In: Human–Computer Interaction. p. 13. Guidelines for UK Government Web Sites. http://www.e-envoy.gov.uk/oee/oee.nsf/sections/ Webguidelines-handbook-top/$file/handbookindex.htm. Last accessed on 05.01.2004. IBM User Engineering. http://www.ibm.com/easy/userengineering. Last accessed on 05.01.2004. IBM Web Accessibility Checklist 3.1. http://www.ibm.com/able/guidelines/web/accessweb.html. Last accessed on 05.01.2004. Industrial Designers Society of America Showcase. http://new.idsa.org/galleries.htm. Last accessed on 05.01.2004. Internet PTI Best Practice—Preliminary Usability Heuristics. http://www.trg.soton.ac.uk/pti/principles.html. Last accessed on 05.01.2004. Jeffries, R. et al., 1991. User interface evaluation in the real world: a comparison of four techniques, Published in Human Factors in Computing Systems CHI’91 Conference Proceedings. Karat C-M: et al., 1992. Comparison of empirical testing and walkthrough methods in user interface evaluation. In: Human Factors in Computing Systems CHI’92 Conference Proceedings. Kelley, T., et al., 2001. The art of innovation, Currency Books. McEwan, T., et al., 2003. Learning about universal access, Published in Proceedings of HCI2003: Designing for Society, vol. 2. Nielsen J., 1994. Usability Engineering, Academic Press, Boston. Nielsen J., 2001. Beyond ALT Text: Making the Web Easy to Use for Users With Disabilities, Nielsen Norman Group Report. Section 508 Web Standards. http://www.section508.gov/index.cfm?FuseAction ¼ Content&ID ¼ 12 (part of the American’s with Disabilities Act (ADA) http://www.usdoj.gov/crt/ada/adahom1.htm). Last accessed on 05.01.2004. Ten Usability Heuristics. Nielsen, J., 1994 http://www.useit.com/papers/heuristic/heuristic . _list.html. Last accessed on 05.01.2004. UsabilityNews. http://www.usabilitynews.com. Last accessed on 05.01.2004. Web Content Accessibility Guidelines (WCAG). http://www.w3.org/TR/WCAG10/. Last accessed on 05.01.2004. Whatis?com, part of the TechTarget Network of IT Enterprise Web sites. http://whatis.techtarget.com/definition/ 0,sid9_gci212246,00.html. Last accessed on 05.01.2004. WorkTrain: The National Jobs and Learning Site. http://www.worktrain.gov.uk/. Last accessed on 05.01.2004.