Practicing What We Preach? A Usability Evaluation of the HFES Proceedings CD-ROM
By Ryan Baker
When it comes to conferences, traditional paper proceedings are quickly being replaced by their electronic counterparts. The annual conference of the Human Factors and Ergonomics Society is no exception. Last year’s 2000 conference proceedings were available solely on the CD-ROM, "Ergonomics for the Millennium." After working with the proceedings ourselves and hearing other conference attendees comment on the difficulties accessing information from the proceedings, we decided to conduct a quick usability test on the software.
Participants
Participants in this study consisted of four graduate students or faculty from the Wichita State University human factors program. All participants were daily computer users and familiar with the Human Factors and Ergonomic Society.
Procedure
Participants were first asked to orient themselves with the Proceedings program by viewing the 'home' page and exploring the 'About HFES' screen. After this brief orientation, participants were presented the following tasks (one at a time) in random order:
- find a paper by title
- find a paper by author's affiliation
- find the author of a specific paper
- find a paper by subject I (i.e. chronic fatigue syndrome)
- find a paper by subject II (i.e. usability testing of web sites)
Participants were timed for each task. After they completed all 5 tasks, they were asked to complete a questionnaire reporting their satisfaction of the Proceedings program.
Results
Participants spent between 2.5 to 7.5 minutes on the tasks (average time on task: 3.5 minutes). Two of the participants correctly answered all 5 tasks, one participant gave up on Task 2 and one participant gave up on Task 1 and 4. Satisfaction scores are summarized in Table 1. In general, the program was rated user-friendly and easy to use only "some of the time." Moreover, it was rated worse than average in terms of overall satisfaction. However, because it is the only source available for these proceedings, most participants said they would use the CD-ROM again.
Table 1. Summary of Satisfaction scores.
| Question | Mean (SD) |
|---|---|
| Program was user-friendly (1=almost never 5=almost always) | 2.0 (1.41) |
| Program was easy to use (1=almost never 5=almost always) | 2.0 (1.41) |
| Got info quickly (1=almost never 5=almost always) | 2.5 (0.58) |
| Info was up-to-date (1=almost never 5=almost always) | 3.5 (1.29) |
| Would use again (1=definitely no 5=definitely yes) | 3.5 (1.73) |
| Overall satisfaction (1=one of best 5=one of worst) | 3.75 (0.96) |
Usability problems discovered
- Searching
Searching for a paper is the primary task that users perform with any conference proceeding. However, in searching for paper titles, subjects, and authors, the drop-down combination list box (combo list) used in this program was reported to be ineffective and surprising to our users. (Figure 1 shows what the users saw when they selected this kind of search).
The participants did not initially realize that the search field was indeed a combo list and began the search by typing the entire target keyword/phrase. Since the drop-down list is cued by one letter only -- the last letter typed -- users were surprised by what was displayed in the list. That is, as they typed each letter in their keyword/phrase, the list changed as well. Once users realized that the list was generated from the last letter typed, they attempted to find the most appropriate word or phrase from that list. However, they were forced to scroll through a lengthy list of items that began with that letter (authors, titles, subjects).
- Forced selection from a list
Users were particularly surprised that the Subject Search did not allow actual keyword searches, but instead only provided a long list of Subjects to choose from. This was especially problematic in cases when the keyword that the user was searching for did not have a direct match to the Subjects on the list. Participants were forced to either scroll through the Subjects presented or search for another keyword.
- Navigation using the 'Bookmarks' Menu
The 'Bookmarks' navigation menu along the left side of the screen appears when a user views a proceedings paper. However, it was unnoticed by some users and confusing to the users who did see it. Users were more inclined to return to the home page using the toolbar than the Bookmarks navigation menu. Several users who did use the Bookmarks menu, intending to return to the home page, selected the 'Exit' link and were unexpectedly exited from the program.
Figure 1. Search field. Participants typed the entire search term here not noticing that it was a combination list box.
Figure 2. Drop-down list for searching.
Figure 3. 'Bookmark' navigation menu on the article page.
CONCLUSION
Providing conference proceedings on CD-ROM instead of paper-based counterparts should be beneficial for a number of reasons:
- convenience for conference attendees (no bulky books to carry or to pack in suitcase)
- efficiency and flexibility when searching for papers (by title, author, subject matter, etc.)
- capability to incorporate additional mediums (i.e., video, animation)
Developers need to remember that these benefits only hold true when usability practices are incorporated into the design process. It seems evident that usability testing during the design of the HFES Proceedings program would have resulted in design recommendations to eliminate the usability problems encountered in our study.
Note: A presentation based on this work was presented at the Human Factors and Ergonomics Society's 45th (2001) Annual Meeting in Minneapolis/St. Paul, MN.
