Evaluation Model for Hypertext Document

(source: my thesis- improving learning ability through hypertext system)

Nielsen [1995] proposed five metrics to measure the usability of reading hypertext paper:

1. Task time was the number of seconds it took users to find answers for specific questions about the content element.

2. Errors was a percentage score based on the number of incorrect answers users gave for questions that had a known answer

3. Memory comprised two measures from an exam given to the users after they had finished using the hypertext paper. Recognition memory was a percentage score based on the number of correct answers minus the number of incorrect answers to 5 multiple-choice questions. Recall memory was a percentage score based on the number of item correctly recalled after the text minus the number incorrectly recalled.

4. Time to recall hypertext structure was the number of seconds it tool users to drawn a hypertext map. This is a measure of how well the users had understood the information organization.

5. Subjective satisfaction was determined from participant’s answers to questionnaire. Each question used a 10-point rating scale. Four satisfaction criteria were averaged to derive the subjective satisfaction score: perceived quality, perceived ease of use, likeability, and user affect.

Depending on the goal of a project, Nielsen in his research use different weight to each five usability metrics. For educational writing might give added weight to the memory measure and perhaps also some added weight to learning the hypertext structure.

Nielsen and Landauer [1993] also suggest to test only 5 participants in a group with the argument from their research that after the first iterative study with 5 users has found 85% of the important usability problems. The rest of problem can be fixed in maximum 2 subsequent groups of five. Following Figure 2.5 is the research result of their work in explaining why the evaluation of usability can be satisfied with 5 users in one group of observation to get 85% of the important usability problems. Then, to calculate final measurements, Nielsen [2001] as well suggest to use geometric mean calculation rather than arithmetic mean in order to not skewing the result by a single big number and for the cases in which some of the metrics are negative. The geometric mean formula of N numbers is N’th root of the product of the numbers.

Figure 2.5 Usability Evaluation Curve [Nielsen and Landauer, 1993, p.206-213]

References:

[Nielsen, 1995] Nielsen, J., 1995, Multimedia and Hypertext: The Internet and Beyond, Academic Press, Boston.

[Nielsen and Landauer, 1993] Nielsen, Jakob, and Landauer, Thomas K., A Mathematical model of the finding of usability problems, Proceedings of ACM INTERCHI’93 Conference, Amsterdam, the Netherlands, 24-29 April 1993, pp.206-213.

[Nielsen, 2001] Nielsen, J., 2001, Usability Metrics, Available on-line at http://www.useit.com/alertbox/20010121.html