- Email: [email protected]
Uphill and Downhill Radial Keratotomy Incisions
2004 ACM Symposium on Applied Computing
A Study into the Usability of E-encyclopaedias Ruth Wilson
Julie Shortreed
Monica Landoni
Department of Computer and Information Sciences University of Strathclyde Glasgow, UK +44(0)141 548 3705
Information Services University of Abertay Dundee UK +44(0)1382 308836
Department of Computer and Information Sciences University of Strathclyde Glasgow, UK +44(0)141 548 4949
[email protected]
[email protected]
[email protected]
ABSTRACT This paper describes a usability study performed on three different e-encyclopaedias, all of which were available on the web, contained similar content and targeted similar audiences, but differed in their styles of presentation. The purpose of the study was to discover the effectiveness of each design in terms of overall usability and subjective satisfaction, to extract general guidelines for the successful design of e-encyclopaedias, and to examine the requirements of different user groups.
Keywords electronic encyclopaedias, e-encyclopaedias, usability, evaluation
1. DESIGN OF EDUCATIONAL MATERIAL FOR THE WEB Educational websites and electronic textbooks are becoming progressively similar in terms of both content and presentation, as scholars mount their course material online in increasingly sophisticated ways and print resources are redefined for electronic delivery. Textbooks need no longer be supplied in their entirety to students; rather, selected chapters or sections can be provided electronically, alone or in a “package” with chunks of other textbooks or complementary material. The notion of the book as a static set of pages bound together by a cover is fading, as more and more of its traditional features are transformed. Educational websites, at the same time, are becoming more advanced, bringing together rich, scholarly material and employing many of the same design elements as electronic textbooks, such as internal and external hyperlinks and multimedia. Based on this observation, that the “look and feel” of educational websites and electronic textbooks are overlapping and the boundaries between them blurring, the EBONI (Electronic Book ON-screen Interface) Project [1] asked if design principles which have been proven to succeed in one area could be borrowed by the other.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The design of material for the web is a well-researched area, and principles are now well-established [2, 3, 4], whereas e-book design is still in its infancy and many of the usability studies which have taken place have focused on handheld devices [5, 6, 7]. EBONI investigated the area of on-screen e-book design, taking Morkes and Nielsen’s web usability guidelines as a starting point [8], and evaluated their applicability in a series of e-book evaluations involving over 100 students, lecturers and researchers from a variety of disciplines in UK Higher Education [9]. It was discovered that certain web design principles could indeed be borrowed by the field of e-book design, but that some traditional features should also be retained. Specifically, in the resulting set of Electronic Textbook Design Guidelines [10], two main ingredients for a successful electronic book are proposed: adherence to certain familiar aspects of the paper book metaphor, such as page length, tables of contents and indexes; and “scannability”, a notion inherited from web design principles which encourages the use of extra headings, coloured text and images to break up long streams of text [11]. During the summer of 2001, the project also conducted a study into the usability of electronic encyclopaedias (e-encyclopaedias) [12]. As resources which were previously published in print but which comprise short “bites” of non-linear content, these sit somewhere between electronic books and web sites and, as such, presented interesting design questions to the project. Whereas many books have arguably been comfortable within the “limitations” of print, encyclopaedias have been constrained by a linear, alphabetical structure since their beginnings in the 18th century, hindering access to the rich organisation and interconnections among articles [13]. It was just over a decade ago that a seamless “circle of learning” started to become a real possibility with hypertext, which can take the reader “out of a linear model and into a connected network of thought” [14]. The electronic medium, therefore, presents great opportunities for encyclopaedias, if designed well. EBONI’s experiment aimed to define what is meant by a well-designed e-encyclopaedia by examining the requirements of a range of users, and to determine the relationship between e-encyclopaedia, e-book and web usability. This paper describes the evaluation of three web-based encyclopaedias, asking to what extent design principles from the e-book and web usability arenas are applicable, and attempting to understand this dynamic, evolving genre that is highly adapted to the online environment.
SAC'2004, March 14-17, 2004, Nicosia, Cyprus Copyright 2004 ACM 1 -58113 -812-1/03/04…$5.00.
1688
2. E-ENCYCLOPAEDIA EXPERIMENT 2.1 Hypotheses The study involved three e-encyclopaedias, all of which were available via the web, contained similar material and targeted similar audiences but exhibited different styles of presentation. The purpose was to discover the effectiveness of each design in terms of overall usability and subjective satisfaction, and to extract general guidelines for the successful design of eencyclopaedias. The selected material ranged from an e-encyclopaedia which closely resembled its paper counterpart in terms of appearance to a highly interactive, complex hypermedia one which had a variety of added value content and bore little, if any, resemblance to a print encyclopaedia. As identified above, the web offers particular opportunities to eencyclopaedias in the form of interactivity and added value content. The first aim of the experiment was to measure the impact of these factors on usability; as such, the degree to which these elements were utilised by each e-encyclopaedia was measured: •
Interaction (the degree of interaction available to the user).
•
Added value (the degree of added value content available to the user).
The overall criteria for judging the effectiveness of these elements were: •
“Sense of directness”, which measures the user’s ability to learn and internalise the interface.
•
“Sense of engagement”, which measures the level of interest induced in the user and is related to the level of interaction available and the novelty of the system [15].
2.2 Selection of Material Three free e-encyclopaedias were selected for evaluation, all of which differed in terms of presentation, navigation and added value content. They were analysed in detail with reference to a range of current web usability guidelines [17, 18, 19, 20]. Britannica.com (Britannica) [21] was the least interactive of the three in terms of its functionality, and added value content was limited to links to external resources. It is a search-dominant resource, with a simple interface. The 6th edition (2001) Concise Columbia Encyclopaedia (Columbia) [22] bore most resemblance to its print counterpart in terms of content, layout and appearance, albeit enhanced by a search function and hyperlinked cross-references. Articles were indexed alphabetically and the page layout was consistent with the paper book metaphor. The third encyclopaedia, Microsoft’s Encarta (Encarta) [23] was the most interactive in terms of functionality and content and offered the most added value content including multimedia. It was also the largest and most complex encyclopaedia of the three.
2.3 Selection of Actors 36 participants took part in the experiment. An important aim of the study was to investigate how age and educational background would affect the way people interact with the selected material. To this end, the 36 participants belonged to two distinct groups: 1.
“School group”. This comprised a class of 5th and 6th year pupils from Perth Grammar School sitting their Higher Computing exams. All 15 participants were aged 16 or 17 and had obtained either a Standard Grade in Computing or Information Systems Intermediate 2 as entrance to the Higher course.
2.
“Online group”, composed of 21 postgraduate students and mailing list respondents, ranging from 19 to over 50 years of age. The postgraduate students were studying for a Postgraduate Diploma or Msc in the Department of Information Science, University of Strathclyde. The mailing lists were:
The following hypotheses were put forward: •
•
Hypothesis 1. A high degree of interaction and added value content will not have a positive effect on sense of directness (the added complexity will increase users’ confusion and disorientation and greater cognitive effort will be required to achieve a satisfactory level of performance). Hypothesis 2. A high degree of interaction and added value will have a positive effect on sense of engagement (users will experience more pleasure and satisfaction and their level of interest will be higher).
JISCmail lis -e-books, aimed at those in the academic community with an interest in e-books.
b.
The eBook Community mailing list, for the general discussion of e-books and the e-book industry.
It was assumed that online participants would all have a qualification or professional interest in information science in general and e-publishing in particular. Postgraduate students were motivated to complete the survey by a desire to help the evaluator (a fellow-student) and an interest in the subject; mailing list respondents’ motivation stemmed from their professional interest in the outcome of the study and the advancement of research into e-books.
A secondary aim of the study was to investigate how age and educational background affect the way people interact with the selected material. If e-encyclopaedia publishers are to successfully tailor content and design to suit the needs of their diverse audience, then an awareness of how age and background can influence use is required. The methodology was based on EBONI’s e-book evaluation framework, specially developed by the project to provide cohesiveness to all its individual experiments [16]. It comprises several stages: selection of material, selection of actors, selection of tasks, and selection of techniques.
a.
75% of online respondents used the Internet daily, and 80% of school participants used it at least two or three times a week; the Head of Computing confirmed that all pupils were familiar with the Internet and used it on a regular basis as part of the school curriculum. In both groups, the majority of respondents had not used an e-encyclopaedia before, and none used one on a regular basis.
1689
One evaluator coordinated all aspects of the study, and collaborated with the Head of Computing at Perth Academy with regard to task development and assessment.
2.4 Selection of Tasks and Techniques Tasks and techniques were inherited from EBONI’s earlier experiments. Scavenger Hunt [24] and memory tasks were used as a means of measuring accuracy, recognition and recall, and were designed with two aims in mind: •
•
Table 1. Techniques used in the e-encyclopaedia experiment, and their relationship to the evaluation criteria. Metacriteria Sense of engagement
To ensure that participants used all the features of the encyclopaedias, in terms of presentation, navigation and content. To ensure that neither group was disadvantaged due to differences in age and education. Participants were asked about astronomy, a topic of general interest, and tasks were designed to be challenging enough to require both groups to use the encyclopaedias in order to discover the answers, yet not to be beyond the capabilities of either group.
Subjective satisfaction questionnaires were completed by all participants after the tasks. Due to the limitations of time (school group) and resources (online group), interviews and think-aloud sessions were not conducted. However, the evaluator was able to observe the school pupils during the evaluation.
2.5 Procedure A between-subjects design was adopted, with each participant evaluating only one encyclopaedia. All participants were asked: •
To provide details about themselves regarding age, and their experience of the Internet and electronic encyclopaedias.
•
To visit a particular e-encyclopaedia and complete some tasks.
•
To complete a subjective satisfaction questionnaire.
Satisfaction
Quality Ease of use Likeability User affect Recognition Recall
Sense of directness
Usability
Task success
Data collection methods Subjective satisfaction questionnaires (qualitative and quantitative) Memory tasks (quantitative) Scavenger Hunts (quantitative)
By incorporating tasks which require a range of cognitive skills, and questionnaires designed to measure user satisfaction, the experiment aimed to measure usability comprehensively and at a variety of levels. Moreover, by adopting a methodology that was consistent with EBONI’s, it could be “mapped” and compared to past and future evaluations in the field.
2.7 Results Data was analysed individually for each measure, and compared across the three encyclopaedias and the two groups of participants. Figure 1 shows participants’ overall sense of directness and sense of engagement with the three encyclopaedias.
percentage
2.6 Measures The measures and criteria for evaluation were derived from EBONI’s e-book evaluation methodology, and gathered information about participants’ sense of engagement and sense of directness with the e-encyclopaedias:
100 90 80 70 60 50 40 30 20 10 0
Britannica Columbia Encarta
Sense of directness
Sense of engagement
Two criteria comprised sense of engagement: o
o •
Measures
Memorability
The tasks and subjective satisfaction questionnaires were identical for both groups; however the school group’s evaluation was conducted in a controlled environment with a fixed time limit, while the online group’s evaluation was conducted over the Internet, unsupervised and with no time limit.
•
Criteria
Satisfaction, including quality, ease of use, likeability and user affect, and measured via the subjective satisfaction questionnaire. Memorability, including recognition and recall, measured via the memory tasks.
In this study, the criterion for sense of directness was usability, and participants’ success at Scavenger Hunt tasks was used to measure the usability of each encyclopeadia.
The various tasks and techniques, their measurements and their relationship to the criteria for evaluation are outlined in Table 1 below.
Figure 1. Overall sense of directness and sense of engagement Hypothesis 1 was validated. Interaction and added value content did not have a positive effect on sense of directness (Encarta was the most interactive and had most added value content). When users were task driven and when the tasks involved specific factfinding, there was a clear indication that a design retaining elements of the book metaphor (Columbia), in terms of both functionality and presentation, made it easier for users to learn and internalise the system. Indeed, Columbia users scored significantly higher than Encarta users in the Scavenger Hunts (p<0.1).
1690
Hypothesis 2 was partially validated. Interaction and added value content had a positive effect overall on sense of engagement in terms of memorability and likeability, however there was no clear indication that ease of use, user affect or perceptions of quality of content were increased by the presence of added value and a high level of interaction. Therefore, when tasks involved browsing and memorising rather than specific fact-finding, there was some indication that interactive encyclopaedias with added value content improved task success and certain elements of user satisfaction. Distinct differences between the two groups were identified in terms of satisfaction, memorability and usability. Younger users were shown to be less successful at completing tasks involving cognitive skills and at extracting relevant information from highly interactive, information abundant web sites, and there was some indication that they had a lower tolerance for unfamiliar design. Older users, on the other hand, had a greater level of engagement and were able to learn and internalise an unfamiliar interface more quickly. Their recall scores were significantly higher (p<0.1) than the school pupils’. The following characteristics of the school group emerged: •
Pupils had a lower tolerance to learning how to use the eencyclopaedia and were more likely to lose interest and give up as tasks became more complex.
•
They also had difficulty in identifying relevant information when presented with a lot of choice or when required to exercise good judgement. Some could not distinguish between commercial and non-commercial; for example, one pupil was observed searching through the Barnes and Noble bookstore (a link from Encarta), while another ended up looking at fiction in Bartleby.com, Columbia’s parent site.
•
While they said that multimedia was important to them, not enough was available on the free, web-based encyclopaedias to significantly increase their levels of engagement and enjoyment.
•
While school pupils were more comfortable with reading text on screen, use of unfamiliar or specialist language and labelling increased their confusion.
•
They did not consider the experience of using eencyclopaedias to be an enjoyable one.
For online respondents, the level of engagement and general enjoyment of the experience was higher and quality of content was just as important as added value. There was a clear indication that this group was better at coping with more complex tasks and had greater recognition and recall. The online group was less happy with reading text on screen: one participant commented that the writing was very small but that the volume of information perhaps made this necessary; another commented, “I don’t enjoy reading from the screen for too long, so probably wouldn’t use an online encyclopaedia by choice… I find it easier to read larger print.” This group also requested more sophisticated search options. In the words of one participant, “a better search interface with more advanced methods of refining your query would have improved the whole experience for me” (Britannica).
3. DISCUSSION AND CONCLUSIONS These results have certain implications for the design of eencyclopaedias. First of all, the two design factors which formed the focus of this experiment (interactive elements and added value content) were liked by users and aided their memory of the information being conveyed. However, users found it more difficult to retrieve facts from encyclopaedias which were rich in such features. E-encyclopaedias with simple interfaces and most closely resembling a printed book were the easiest from which to retrieve facts. Therefore, it is important for designers to strike a balance, combining a clean presentation and familiar, simple functions inherited from the paper medium with interactive and added value elements which do not distract from the core content. These findings are in line with EBONI’s Electronic Textbook Design Guidelines, which promote a successful fusion of traditional ways of presenting information and those made possible via the electronic medium. In terms of adherence to the paper book metaphor, there was some indication that users’ confusion was increased by having to scroll for essential information. The school pupils, in particular, had a tendency not to scroll at all on home pages. EBONI’s guidelines recommend that the typical page length of a print textbook should be used as a guide for the length of pages in an electronic book, and that the amount of scrolling required should be minimised, especially for smaller screens. Similarly, the longer line lengths and paragraphs of Columbia were not conducive to a sense of engagement. Again, EBONI stipulates that pagination has to be designed carefully to enhance readability, with line lengths similar to that of a printed page (10 to 15 words) and plenty of “white space”. The following e-book design heuristics encouraging retention of features of the paper book were also shown to be applicable to eencyclopaedias: •
It should be made possible to treat the book as a closed environment, containing no links to external sources unless clearly labelled as such (for example in a reference section or bibliography). This is particularly relevant to eencyclopaedias, in which the maze of hyperlinks between references and articles has to be designed very carefully in order to help users to make best use of all available paths.
•
Readers gain a sense of their place in a printed book via the page numbers and by comparing the thickness and weight of the pages read against the thickness and weight of the pages still to be read. It is important for this “sense of place” also to be present in the electronic medium, via page numbers, section headings, or navigation bars highlighting the current position.
Moreover, with regard to effectively utilising the capabilities of the electronic medium, the benefits of positioning hyperlinked section headings at the top of each page were supported in both the e-book and the e-encyclopaedia experiments. Several other design guidelines emerging from EBONI’s e-book evaluations were also relevant: •
1691
Search engines can be worthwhile where they are intelligent enough to simulate and enhance the way readers search in paper books. However, they should not replace tables of contents and indexes; indeed, indexes in an encyclopaedia are essential.
• •
Fonts should be large enough to read comfortably for long periods of time. Within each page, breaking the text into short chunks improves its scannability. This can be achieved by, for example, interspersing text with images and diagrams and keeping paragraphs short, and by using meaningful subheadings, indented, bulleted lists, and colour to break the uniformity of the text.
However, perhaps because of their non-linear nature and, therefore, the additional opportunities offered by the electronic medium to break from a structure and style which had previously restricted usability, the e-encyclopaedias reviewed in this experiment contained more components of web page design than a typical e-book, such as frames, search boxes and drop-down menus. Consequently, design requirement s central to the field of web usability, but beyond the realm of e-book design, also emerged. For example, conciseness, a parameter of web design extolled by Nielsen [25], was found to assist users when searching for specific facts. Users of the more extensive Britannica, on the other hand, made more errors on search tasks and commented on the difficulty of finding facts: “There’s a lot of text in Britannica to read about a subject if you are only looking for a few detailed answers to specific questions”.
Further, the use of frames and animated advertising on the eencyclopaedias reduced users’ concentration. Users found it easiest to concentrate on Columbia which made minimal use of frames and the advertising disappeared when scrolling down a screen. Long drop -down menus and the inclusion of several search boxes on a page also contributed to users’ confusion. The area of web usability provides guidance and rules on each of these features. In short, like educational web sites and electronic textbooks, appearance has been shown to play an important role in the design of e-encyclopaedias. Moreover, principles for good design can be borrowed from the fields of e-book and web usability. However, the continuing evolution of e-encyclopaedias’ content, the complex relationships between their articles, and their broad range of uses and users make them a unique consideration, and they do not belong in either camp. Rather, a careful implementation of selected heuristics from both fields is required.
4. REFERENCES
[6] Dearnley, J. and McKnight, C. The revolution starts next week: the findings of two studies considering electronic books. Information Services and Use. 21 (2), 2001.
[7] Electronic Books in Libraries. http://www.lib.rochester.edu/main/e-books/index.htm
[8] Morkes, J. and Nielsen, J. Concise, SCANNABLE, and objective: how to write for the Web. 1997. http://www.useit.com/papers/webwriting/writing.html
[9] Wilson, R., Landoni, M. and Gibb, F. A user centred approach to e-book design. The Electronic Library. 20 (4), 2002.
[10] Wilson, R., Landoni, M. and Gibb, F. Guidelines for designing electronic textbooks. Sixth European Conference on Research and Advanced Technology for Digital Libraries (ECDL 2002). Rome, Italy, 16-18 September 2002.
[11] Wilson, R. The "look and feel" of an ebook: considerations in interface design. 17th ACM Symposium on Applied Computing (SAC 2002). Universidad Carlos III de Madrid, Spain, March 10-14, 2002.
[12] Shortreed, J. E-encyclopaedias : a usability study. Glasgow: Department of Information Science, University of Strathclyde (MSc Dissertation), 2001.
[13] Weyer, S. and Borning, A. A prototype electronic encyclopaedia. ACM Transactions on Office Information Systems. 3 (1), January 1985.
[14] Castelluccio, M. The future of books. Management Accounting. 79, 1998.
[15] Landoni, M. The visual book system: a study of the use of visual rhetoric in the design of electronic books. Glasgow: Department of Information Science, University of Strathclyde (PhD Thesis), 1997.
[16] Wilson, R. and Landoni, M. Evaluating electronic textbooks: a methodology. Fifth European Conference on Research and Advanced Technology for Digital Libraries (ECDL 2001). Darmstadt, Germany, 4-9 September 2001.
[17] Nielsen, J. op. cit. [18] Rosenfeld, L. and Morville, P. Information Architecture for the World Wide Web. O’Reilly, 2002.
[19] Dyson, M. and Kipping, G. The effect of line length and number of columns on reading performance. http://www.rdg.ac.uk/AcaDepts/It/main/resea/publ/titles/98.h tml
[1] EBONI (Electronic Books ON-screen Interface). http://ebooks.strath.ac.uk/eboni/
[2] Nielsen, J. Designing Web Usability. Indiana: New Riders, 2000.
[3] Spool, J., Scanlon, T., Schroeder, W., Snyder, C. and DeAngelo, T. Web Site Usability: A Designer’s Guide. California: Morgan Kauffman, 1999.
[4] NCI (National Cancer Institute). Research-based web design
[20] Spool, et. al. op. cit. [21] Britannica.com. http://www.britannica.com [22] Columbia Encyclopaedia (6th edition). New York: Columbia University Press, 2001. http://www.bartleby.com/65/
[23] Microsoft Encarta Online Encyclopaedia. http://www.encarta.msn.com/reference/
and usability guidelines. http://usability.gov/guidelines/
[5] Marshall, C. and Ruotolo, C. Reading-in-the-small: a study of reading on small form factor devices. Proceedings of the Second ACM/IEEE-CS Joint Conference on Digital Libraries. New York, 2002.
[24] Spool, et. al. op. cit. [25] Morkes, J. and Nielsen, J. op. cit.
1692
A Study into the Usability of E-encyclopaedias Ruth Wilson
Julie Shortreed
Monica Landoni
Department of Computer and Information Sciences University of Strathclyde Glasgow, UK +44(0)141 548 3705
Information Services University of Abertay Dundee UK +44(0)1382 308836
Department of Computer and Information Sciences University of Strathclyde Glasgow, UK +44(0)141 548 4949
[email protected]
[email protected]
[email protected]
ABSTRACT This paper describes a usability study performed on three different e-encyclopaedias, all of which were available on the web, contained similar content and targeted similar audiences, but differed in their styles of presentation. The purpose of the study was to discover the effectiveness of each design in terms of overall usability and subjective satisfaction, to extract general guidelines for the successful design of e-encyclopaedias, and to examine the requirements of different user groups.
Keywords electronic encyclopaedias, e-encyclopaedias, usability, evaluation
1. DESIGN OF EDUCATIONAL MATERIAL FOR THE WEB Educational websites and electronic textbooks are becoming progressively similar in terms of both content and presentation, as scholars mount their course material online in increasingly sophisticated ways and print resources are redefined for electronic delivery. Textbooks need no longer be supplied in their entirety to students; rather, selected chapters or sections can be provided electronically, alone or in a “package” with chunks of other textbooks or complementary material. The notion of the book as a static set of pages bound together by a cover is fading, as more and more of its traditional features are transformed. Educational websites, at the same time, are becoming more advanced, bringing together rich, scholarly material and employing many of the same design elements as electronic textbooks, such as internal and external hyperlinks and multimedia. Based on this observation, that the “look and feel” of educational websites and electronic textbooks are overlapping and the boundaries between them blurring, the EBONI (Electronic Book ON-screen Interface) Project [1] asked if design principles which have been proven to succeed in one area could be borrowed by the other.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The design of material for the web is a well-researched area, and principles are now well-established [2, 3, 4], whereas e-book design is still in its infancy and many of the usability studies which have taken place have focused on handheld devices [5, 6, 7]. EBONI investigated the area of on-screen e-book design, taking Morkes and Nielsen’s web usability guidelines as a starting point [8], and evaluated their applicability in a series of e-book evaluations involving over 100 students, lecturers and researchers from a variety of disciplines in UK Higher Education [9]. It was discovered that certain web design principles could indeed be borrowed by the field of e-book design, but that some traditional features should also be retained. Specifically, in the resulting set of Electronic Textbook Design Guidelines [10], two main ingredients for a successful electronic book are proposed: adherence to certain familiar aspects of the paper book metaphor, such as page length, tables of contents and indexes; and “scannability”, a notion inherited from web design principles which encourages the use of extra headings, coloured text and images to break up long streams of text [11]. During the summer of 2001, the project also conducted a study into the usability of electronic encyclopaedias (e-encyclopaedias) [12]. As resources which were previously published in print but which comprise short “bites” of non-linear content, these sit somewhere between electronic books and web sites and, as such, presented interesting design questions to the project. Whereas many books have arguably been comfortable within the “limitations” of print, encyclopaedias have been constrained by a linear, alphabetical structure since their beginnings in the 18th century, hindering access to the rich organisation and interconnections among articles [13]. It was just over a decade ago that a seamless “circle of learning” started to become a real possibility with hypertext, which can take the reader “out of a linear model and into a connected network of thought” [14]. The electronic medium, therefore, presents great opportunities for encyclopaedias, if designed well. EBONI’s experiment aimed to define what is meant by a well-designed e-encyclopaedia by examining the requirements of a range of users, and to determine the relationship between e-encyclopaedia, e-book and web usability. This paper describes the evaluation of three web-based encyclopaedias, asking to what extent design principles from the e-book and web usability arenas are applicable, and attempting to understand this dynamic, evolving genre that is highly adapted to the online environment.
SAC'2004, March 14-17, 2004, Nicosia, Cyprus Copyright 2004 ACM 1 -58113 -812-1/03/04…$5.00.
1688
2. E-ENCYCLOPAEDIA EXPERIMENT 2.1 Hypotheses The study involved three e-encyclopaedias, all of which were available via the web, contained similar material and targeted similar audiences but exhibited different styles of presentation. The purpose was to discover the effectiveness of each design in terms of overall usability and subjective satisfaction, and to extract general guidelines for the successful design of eencyclopaedias. The selected material ranged from an e-encyclopaedia which closely resembled its paper counterpart in terms of appearance to a highly interactive, complex hypermedia one which had a variety of added value content and bore little, if any, resemblance to a print encyclopaedia. As identified above, the web offers particular opportunities to eencyclopaedias in the form of interactivity and added value content. The first aim of the experiment was to measure the impact of these factors on usability; as such, the degree to which these elements were utilised by each e-encyclopaedia was measured: •
Interaction (the degree of interaction available to the user).
•
Added value (the degree of added value content available to the user).
The overall criteria for judging the effectiveness of these elements were: •
“Sense of directness”, which measures the user’s ability to learn and internalise the interface.
•
“Sense of engagement”, which measures the level of interest induced in the user and is related to the level of interaction available and the novelty of the system [15].
2.2 Selection of Material Three free e-encyclopaedias were selected for evaluation, all of which differed in terms of presentation, navigation and added value content. They were analysed in detail with reference to a range of current web usability guidelines [17, 18, 19, 20]. Britannica.com (Britannica) [21] was the least interactive of the three in terms of its functionality, and added value content was limited to links to external resources. It is a search-dominant resource, with a simple interface. The 6th edition (2001) Concise Columbia Encyclopaedia (Columbia) [22] bore most resemblance to its print counterpart in terms of content, layout and appearance, albeit enhanced by a search function and hyperlinked cross-references. Articles were indexed alphabetically and the page layout was consistent with the paper book metaphor. The third encyclopaedia, Microsoft’s Encarta (Encarta) [23] was the most interactive in terms of functionality and content and offered the most added value content including multimedia. It was also the largest and most complex encyclopaedia of the three.
2.3 Selection of Actors 36 participants took part in the experiment. An important aim of the study was to investigate how age and educational background would affect the way people interact with the selected material. To this end, the 36 participants belonged to two distinct groups: 1.
“School group”. This comprised a class of 5th and 6th year pupils from Perth Grammar School sitting their Higher Computing exams. All 15 participants were aged 16 or 17 and had obtained either a Standard Grade in Computing or Information Systems Intermediate 2 as entrance to the Higher course.
2.
“Online group”, composed of 21 postgraduate students and mailing list respondents, ranging from 19 to over 50 years of age. The postgraduate students were studying for a Postgraduate Diploma or Msc in the Department of Information Science, University of Strathclyde. The mailing lists were:
The following hypotheses were put forward: •
•
Hypothesis 1. A high degree of interaction and added value content will not have a positive effect on sense of directness (the added complexity will increase users’ confusion and disorientation and greater cognitive effort will be required to achieve a satisfactory level of performance). Hypothesis 2. A high degree of interaction and added value will have a positive effect on sense of engagement (users will experience more pleasure and satisfaction and their level of interest will be higher).
JISCmail lis -e-books, aimed at those in the academic community with an interest in e-books.
b.
The eBook Community mailing list, for the general discussion of e-books and the e-book industry.
It was assumed that online participants would all have a qualification or professional interest in information science in general and e-publishing in particular. Postgraduate students were motivated to complete the survey by a desire to help the evaluator (a fellow-student) and an interest in the subject; mailing list respondents’ motivation stemmed from their professional interest in the outcome of the study and the advancement of research into e-books.
A secondary aim of the study was to investigate how age and educational background affect the way people interact with the selected material. If e-encyclopaedia publishers are to successfully tailor content and design to suit the needs of their diverse audience, then an awareness of how age and background can influence use is required. The methodology was based on EBONI’s e-book evaluation framework, specially developed by the project to provide cohesiveness to all its individual experiments [16]. It comprises several stages: selection of material, selection of actors, selection of tasks, and selection of techniques.
a.
75% of online respondents used the Internet daily, and 80% of school participants used it at least two or three times a week; the Head of Computing confirmed that all pupils were familiar with the Internet and used it on a regular basis as part of the school curriculum. In both groups, the majority of respondents had not used an e-encyclopaedia before, and none used one on a regular basis.
1689
One evaluator coordinated all aspects of the study, and collaborated with the Head of Computing at Perth Academy with regard to task development and assessment.
2.4 Selection of Tasks and Techniques Tasks and techniques were inherited from EBONI’s earlier experiments. Scavenger Hunt [24] and memory tasks were used as a means of measuring accuracy, recognition and recall, and were designed with two aims in mind: •
•
Table 1. Techniques used in the e-encyclopaedia experiment, and their relationship to the evaluation criteria. Metacriteria Sense of engagement
To ensure that participants used all the features of the encyclopaedias, in terms of presentation, navigation and content. To ensure that neither group was disadvantaged due to differences in age and education. Participants were asked about astronomy, a topic of general interest, and tasks were designed to be challenging enough to require both groups to use the encyclopaedias in order to discover the answers, yet not to be beyond the capabilities of either group.
Subjective satisfaction questionnaires were completed by all participants after the tasks. Due to the limitations of time (school group) and resources (online group), interviews and think-aloud sessions were not conducted. However, the evaluator was able to observe the school pupils during the evaluation.
2.5 Procedure A between-subjects design was adopted, with each participant evaluating only one encyclopaedia. All participants were asked: •
To provide details about themselves regarding age, and their experience of the Internet and electronic encyclopaedias.
•
To visit a particular e-encyclopaedia and complete some tasks.
•
To complete a subjective satisfaction questionnaire.
Satisfaction
Quality Ease of use Likeability User affect Recognition Recall
Sense of directness
Usability
Task success
Data collection methods Subjective satisfaction questionnaires (qualitative and quantitative) Memory tasks (quantitative) Scavenger Hunts (quantitative)
By incorporating tasks which require a range of cognitive skills, and questionnaires designed to measure user satisfaction, the experiment aimed to measure usability comprehensively and at a variety of levels. Moreover, by adopting a methodology that was consistent with EBONI’s, it could be “mapped” and compared to past and future evaluations in the field.
2.7 Results Data was analysed individually for each measure, and compared across the three encyclopaedias and the two groups of participants. Figure 1 shows participants’ overall sense of directness and sense of engagement with the three encyclopaedias.
percentage
2.6 Measures The measures and criteria for evaluation were derived from EBONI’s e-book evaluation methodology, and gathered information about participants’ sense of engagement and sense of directness with the e-encyclopaedias:
100 90 80 70 60 50 40 30 20 10 0
Britannica Columbia Encarta
Sense of directness
Sense of engagement
Two criteria comprised sense of engagement: o
o •
Measures
Memorability
The tasks and subjective satisfaction questionnaires were identical for both groups; however the school group’s evaluation was conducted in a controlled environment with a fixed time limit, while the online group’s evaluation was conducted over the Internet, unsupervised and with no time limit.
•
Criteria
Satisfaction, including quality, ease of use, likeability and user affect, and measured via the subjective satisfaction questionnaire. Memorability, including recognition and recall, measured via the memory tasks.
In this study, the criterion for sense of directness was usability, and participants’ success at Scavenger Hunt tasks was used to measure the usability of each encyclopeadia.
The various tasks and techniques, their measurements and their relationship to the criteria for evaluation are outlined in Table 1 below.
Figure 1. Overall sense of directness and sense of engagement Hypothesis 1 was validated. Interaction and added value content did not have a positive effect on sense of directness (Encarta was the most interactive and had most added value content). When users were task driven and when the tasks involved specific factfinding, there was a clear indication that a design retaining elements of the book metaphor (Columbia), in terms of both functionality and presentation, made it easier for users to learn and internalise the system. Indeed, Columbia users scored significantly higher than Encarta users in the Scavenger Hunts (p<0.1).
1690
Hypothesis 2 was partially validated. Interaction and added value content had a positive effect overall on sense of engagement in terms of memorability and likeability, however there was no clear indication that ease of use, user affect or perceptions of quality of content were increased by the presence of added value and a high level of interaction. Therefore, when tasks involved browsing and memorising rather than specific fact-finding, there was some indication that interactive encyclopaedias with added value content improved task success and certain elements of user satisfaction. Distinct differences between the two groups were identified in terms of satisfaction, memorability and usability. Younger users were shown to be less successful at completing tasks involving cognitive skills and at extracting relevant information from highly interactive, information abundant web sites, and there was some indication that they had a lower tolerance for unfamiliar design. Older users, on the other hand, had a greater level of engagement and were able to learn and internalise an unfamiliar interface more quickly. Their recall scores were significantly higher (p<0.1) than the school pupils’. The following characteristics of the school group emerged: •
Pupils had a lower tolerance to learning how to use the eencyclopaedia and were more likely to lose interest and give up as tasks became more complex.
•
They also had difficulty in identifying relevant information when presented with a lot of choice or when required to exercise good judgement. Some could not distinguish between commercial and non-commercial; for example, one pupil was observed searching through the Barnes and Noble bookstore (a link from Encarta), while another ended up looking at fiction in Bartleby.com, Columbia’s parent site.
•
While they said that multimedia was important to them, not enough was available on the free, web-based encyclopaedias to significantly increase their levels of engagement and enjoyment.
•
While school pupils were more comfortable with reading text on screen, use of unfamiliar or specialist language and labelling increased their confusion.
•
They did not consider the experience of using eencyclopaedias to be an enjoyable one.
For online respondents, the level of engagement and general enjoyment of the experience was higher and quality of content was just as important as added value. There was a clear indication that this group was better at coping with more complex tasks and had greater recognition and recall. The online group was less happy with reading text on screen: one participant commented that the writing was very small but that the volume of information perhaps made this necessary; another commented, “I don’t enjoy reading from the screen for too long, so probably wouldn’t use an online encyclopaedia by choice… I find it easier to read larger print.” This group also requested more sophisticated search options. In the words of one participant, “a better search interface with more advanced methods of refining your query would have improved the whole experience for me” (Britannica).
3. DISCUSSION AND CONCLUSIONS These results have certain implications for the design of eencyclopaedias. First of all, the two design factors which formed the focus of this experiment (interactive elements and added value content) were liked by users and aided their memory of the information being conveyed. However, users found it more difficult to retrieve facts from encyclopaedias which were rich in such features. E-encyclopaedias with simple interfaces and most closely resembling a printed book were the easiest from which to retrieve facts. Therefore, it is important for designers to strike a balance, combining a clean presentation and familiar, simple functions inherited from the paper medium with interactive and added value elements which do not distract from the core content. These findings are in line with EBONI’s Electronic Textbook Design Guidelines, which promote a successful fusion of traditional ways of presenting information and those made possible via the electronic medium. In terms of adherence to the paper book metaphor, there was some indication that users’ confusion was increased by having to scroll for essential information. The school pupils, in particular, had a tendency not to scroll at all on home pages. EBONI’s guidelines recommend that the typical page length of a print textbook should be used as a guide for the length of pages in an electronic book, and that the amount of scrolling required should be minimised, especially for smaller screens. Similarly, the longer line lengths and paragraphs of Columbia were not conducive to a sense of engagement. Again, EBONI stipulates that pagination has to be designed carefully to enhance readability, with line lengths similar to that of a printed page (10 to 15 words) and plenty of “white space”. The following e-book design heuristics encouraging retention of features of the paper book were also shown to be applicable to eencyclopaedias: •
It should be made possible to treat the book as a closed environment, containing no links to external sources unless clearly labelled as such (for example in a reference section or bibliography). This is particularly relevant to eencyclopaedias, in which the maze of hyperlinks between references and articles has to be designed very carefully in order to help users to make best use of all available paths.
•
Readers gain a sense of their place in a printed book via the page numbers and by comparing the thickness and weight of the pages read against the thickness and weight of the pages still to be read. It is important for this “sense of place” also to be present in the electronic medium, via page numbers, section headings, or navigation bars highlighting the current position.
Moreover, with regard to effectively utilising the capabilities of the electronic medium, the benefits of positioning hyperlinked section headings at the top of each page were supported in both the e-book and the e-encyclopaedia experiments. Several other design guidelines emerging from EBONI’s e-book evaluations were also relevant: •
1691
Search engines can be worthwhile where they are intelligent enough to simulate and enhance the way readers search in paper books. However, they should not replace tables of contents and indexes; indeed, indexes in an encyclopaedia are essential.
• •
Fonts should be large enough to read comfortably for long periods of time. Within each page, breaking the text into short chunks improves its scannability. This can be achieved by, for example, interspersing text with images and diagrams and keeping paragraphs short, and by using meaningful subheadings, indented, bulleted lists, and colour to break the uniformity of the text.
However, perhaps because of their non-linear nature and, therefore, the additional opportunities offered by the electronic medium to break from a structure and style which had previously restricted usability, the e-encyclopaedias reviewed in this experiment contained more components of web page design than a typical e-book, such as frames, search boxes and drop-down menus. Consequently, design requirement s central to the field of web usability, but beyond the realm of e-book design, also emerged. For example, conciseness, a parameter of web design extolled by Nielsen [25], was found to assist users when searching for specific facts. Users of the more extensive Britannica, on the other hand, made more errors on search tasks and commented on the difficulty of finding facts: “There’s a lot of text in Britannica to read about a subject if you are only looking for a few detailed answers to specific questions”.
Further, the use of frames and animated advertising on the eencyclopaedias reduced users’ concentration. Users found it easiest to concentrate on Columbia which made minimal use of frames and the advertising disappeared when scrolling down a screen. Long drop -down menus and the inclusion of several search boxes on a page also contributed to users’ confusion. The area of web usability provides guidance and rules on each of these features. In short, like educational web sites and electronic textbooks, appearance has been shown to play an important role in the design of e-encyclopaedias. Moreover, principles for good design can be borrowed from the fields of e-book and web usability. However, the continuing evolution of e-encyclopaedias’ content, the complex relationships between their articles, and their broad range of uses and users make them a unique consideration, and they do not belong in either camp. Rather, a careful implementation of selected heuristics from both fields is required.
4. REFERENCES
[6] Dearnley, J. and McKnight, C. The revolution starts next week: the findings of two studies considering electronic books. Information Services and Use. 21 (2), 2001.
[7] Electronic Books in Libraries. http://www.lib.rochester.edu/main/e-books/index.htm
[8] Morkes, J. and Nielsen, J. Concise, SCANNABLE, and objective: how to write for the Web. 1997. http://www.useit.com/papers/webwriting/writing.html
[9] Wilson, R., Landoni, M. and Gibb, F. A user centred approach to e-book design. The Electronic Library. 20 (4), 2002.
[10] Wilson, R., Landoni, M. and Gibb, F. Guidelines for designing electronic textbooks. Sixth European Conference on Research and Advanced Technology for Digital Libraries (ECDL 2002). Rome, Italy, 16-18 September 2002.
[11] Wilson, R. The "look and feel" of an ebook: considerations in interface design. 17th ACM Symposium on Applied Computing (SAC 2002). Universidad Carlos III de Madrid, Spain, March 10-14, 2002.
[12] Shortreed, J. E-encyclopaedias : a usability study. Glasgow: Department of Information Science, University of Strathclyde (MSc Dissertation), 2001.
[13] Weyer, S. and Borning, A. A prototype electronic encyclopaedia. ACM Transactions on Office Information Systems. 3 (1), January 1985.
[14] Castelluccio, M. The future of books. Management Accounting. 79, 1998.
[15] Landoni, M. The visual book system: a study of the use of visual rhetoric in the design of electronic books. Glasgow: Department of Information Science, University of Strathclyde (PhD Thesis), 1997.
[16] Wilson, R. and Landoni, M. Evaluating electronic textbooks: a methodology. Fifth European Conference on Research and Advanced Technology for Digital Libraries (ECDL 2001). Darmstadt, Germany, 4-9 September 2001.
[17] Nielsen, J. op. cit. [18] Rosenfeld, L. and Morville, P. Information Architecture for the World Wide Web. O’Reilly, 2002.
[19] Dyson, M. and Kipping, G. The effect of line length and number of columns on reading performance. http://www.rdg.ac.uk/AcaDepts/It/main/resea/publ/titles/98.h tml
[1] EBONI (Electronic Books ON-screen Interface). http://ebooks.strath.ac.uk/eboni/
[2] Nielsen, J. Designing Web Usability. Indiana: New Riders, 2000.
[3] Spool, J., Scanlon, T., Schroeder, W., Snyder, C. and DeAngelo, T. Web Site Usability: A Designer’s Guide. California: Morgan Kauffman, 1999.
[4] NCI (National Cancer Institute). Research-based web design
[20] Spool, et. al. op. cit. [21] Britannica.com. http://www.britannica.com [22] Columbia Encyclopaedia (6th edition). New York: Columbia University Press, 2001. http://www.bartleby.com/65/
[23] Microsoft Encarta Online Encyclopaedia. http://www.encarta.msn.com/reference/
and usability guidelines. http://usability.gov/guidelines/
[5] Marshall, C. and Ruotolo, C. Reading-in-the-small: a study of reading on small form factor devices. Proceedings of the Second ACM/IEEE-CS Joint Conference on Digital Libraries. New York, 2002.
[24] Spool, et. al. op. cit. [25] Morkes, J. and Nielsen, J. op. cit.
1692