Determining Cognitive Predictors of User Performance within
Complex User Interfaces

By Michael L. Bernard¹, Chris Hamblin, & Brett Scofield

It is quite apparent that the computer is now a ubiquitous tool for both home and work. It is also evident that computer interfaces have, in general, acquired more functions and present more information than at any other time in the past. For example, it is now common for users to have layouts with multiple and distinct information sources that are concurrently located on an interface. In light of this, the need to study the interaction between complex interfaces and the cognitive factors that affect user performance has become paramount. Unfortunately, most of the studies that have measured human performance with regard to interfaces were done prior to the advent of modern graphical user interfaces.

For instance, Vincente, Hayes, and Williges (1987) examined participants' performance while they searched for information within a hierarchical file system. The file system consisted of three levels, with a total of 15 files. Vincente et al. found that performance was affected by several cognitive variables that were independent of computer experience. These variables consisted of spatial ability, which generally predicted search performance, and verbal ability, which predicted their performance when reading was required. In fact, participants with low spatial ability took twice as long to find information than those with high spatial ability. However in the Vincente et al. study, the interface presented only a small number of functions and, thus, it is argued here that the interface used in the study does not reflect the type of complex user interface that is seen today.

The term 'complex' in this instance refers to the extent to which an interface adheres to a predictable visual scheme, such that the less predictable the interface layout, the more complex it should be (Tullis, 1983). Another factor that often adds to layout complexity is the overall density of the information displayed. That is to say, the more bits of information that are presented beyond a moderate amount, the more complex the interface is thought to be (Vitz, 1966). Since complex interfaces require users to exert higher amounts of cognitive effort, it is suggested that understanding which cognitive factors most affect users' actual and perceived performance should help designers create interfaces that conform more closely to the cognitive aptitude of individuals.

This study sought to assess the full extent of intellectual functioning across participants by administering the Wechsler Abbreviated Scale of Intelligence (WASI), along with administering a mirror-tracing task to assess their perceptual-motor skills. The participants' scores on the individual factors of intelligence and perceptual-motor skills were then examined in relation to their search performance on a complex website interface.

Method

Participants

Twenty-two undergraduate students between 18 and 34 (M = 22) years of age volunteered to participate in this study. All participants reported using the Web at least once per month (87 percent reported using the Web a few times per week or more). Two of the participants reported visiting a financial/brokerage website a few times per month and two reported visiting a financial/brokerage website less than once per month.

Experimental Task

Participants were presented with four interfaces that had approximately the same amount of layout complexity. This was accomplished by creating layouts with multiple information sources that were densely displayed (see Figure 1 for an example of one of the interfaces). Four interfaces were chosen rather than one in order to present as much information as possible without the need for participants to scroll in order to view the entire interface layout. All of the interfaces were as presented as an online brokerage portal.

Participants were given 20 search tasks and were required to find specific information pertaining to brokerage information that would most appropriately satisfy the context of the search task. For example, one question asked, "What is the pre-market percent change for Adobe Systems Inc.?" The tasks were designed to be moderately difficult. Yet, since the tasks only required participants to find but not interpret this information, they presented the same degree of difficultly for those with or without website brokerage experience. Also, all task information was presented without the need to search lower than one level of depth. The search tasks and the searching order within the four layouts were randomized by means of a Latin square design.

Figure 1. An example of the layout for one of the presented interfaces

Materials

A Pentium II based PC computer, using a 60 Hz, 96dpi 17-inch high-resolution RGB monitor with a resolution of 1024 x 768 pixels was used. The computer operating system used was Microsoft's Windows XP. The format of the text was presented as an HTML web page. Search time was recorded by means of the software tool, Ergobrowser^TM, which served as the Web browser.

Dependent Measures

The WASI Instrument

The instrument used to measure the participants' intellectual abilities was the Wechsler Abbreviated Scale of Intelligence (WASI) instrument (Wechsler, 1999). The WASI is similar in format and highly correlated with the Wechsler Adult Intelligence Scales (Goebel & Satz, 1975; Tulsky & Zhu, 2000). The WASI instrument was used because it is a standardized, normed, and validated short form of the Wechsler Adult Intelligence Scale. It also provided a reliable and valid estimate of verbal, performance, and general intellectual functioning (Kaufman & Kaufman, 2001).

The WASI instrument measures several facets of intelligence, such as verbal knowledge, visual information processing, spatial and nonverbal reasoning, and crystallized and fluid intelligence. The WASI instrument consists of four subtests - Vocabulary, Block Design, Similarities, and Matrix Reasoning (Wechsler, 1999):

Vocabulary measures expressive vocabulary, verbal knowledge, and fund of information. It is also a good measure of crystallized intelligence, as well as general intelligence. For this subtest, participants were required to name pictures and define words that are orally and visually presented.

Block Design measures spatial visualization, visual-motor coordination, abstract conceptualization, and perceptual organization by requiring participants to replicate modeled or printed two-dimensional geometric patterns within a specified time by using two-color cube patterns.

Similarities measures verbal concept formation, abstract reasoning ability, and general intellectual ability. For this measure, participants were presented four picture items and 22 verbal items. For the picture items, participants were shown a picture of three objects on the top row and four response options on the bottom row. Participants responded by indicating which response item was similar to the three target objects. For the verbal items, a pair of words was presented orally and participants explain the similarity between the object or concept that the two words represented.

Matrix Reasoning measures nonverbal fluid reasoning by requiring participants to complete a missing portion of an abstract, gridded pattern by indicating the correct completed pattern from five possible choices.
These four subtests combine to form the following general scores:

a) Performance IQ = block design + matrix reasoning

b) Verbal IQ = vocabulary + similarities

c) Full Scale IQ = block design + matrix reasoning + vocabulary + similarities

The Mirror-tracing Task

In addition to the WASI assessment, a mirror-tracing task was used to measure participants' perceptual-motor skills. For this measure participants were required to draw a line within and parallel to two printed circles that were 4mm apart. The diameter of the largest printed circle was approximately 180 mm. Participants could not directly view the circles or their hand when drawing the line. Instead, they viewed them through a mirror that reversed the image. Line completion times, which measured time taken to draw the circle, and drawing accuracy, which consisted of the number of time participants drew outside the printed circles, were measured.

Procedure

Participants first performed the mirror-tracing task. They then answered a computer/web experience and computer comfort questionnaire. After completing the questionnaire, participants were instructed to examine the contents of the four layout interfaces until they were familiar with its general layout. This lasted for approximately one minute per interface. Then participants were then given a practice question for each layout. When the practice session was completed, participants were given 20 search questions, five per layout. If a participant selected an item that was not designated as the target, he or she would be informed that the item was 'incorrect' and instructed to search again for an item that best satisfied the search task. For each task, participants searched until they found the correct task information, or until the allotted time (5 minutes) expired. After participants finished with the search tasks for all of the interfaces, they answered a perception of disorientation questionnaire. The question items were as follows: "It was difficult to find information on the computer screen," the amount of information presented was about right," and "The placement of information was disorientating." The questionnaire consisted of a 7-point Likert scale, with 1 = "Not at all" and 7 = "Completely" as anchors. Since the question items were significantly correlated with each other (p < .01), a mean score was used. Participants were then given the WASI intelligence instrument, which took approximately 45-50 minutes to administer, on a subsequent day.

Results

In order to compare search time and perception of disorientation scores to the WASI/Mirror-tracing scores, both the search time and perception of disorientation scores were equally divided into three separate categories, consisting of a fast, medium, and slow for reading speed (see Table 1) and low, medium, and high for the perception of disorientation (see Table 2).

Table 1. Search time categories

Table 2. Perception of disorientation categories

The three categories served as the independent variables, whereas the WASI and mirror-tracing scores served as the dependent variables. A 3 x 9 MANOVA was used to compare the three levels of search time and perceived disorientation with the dependent scores.

Analysis of the cognitive aptitudes for the three search time categories revealed significant differences between the three search time categories and Block Design, Performance IQ, Full Scale IQ, and Mirror-tracing Accuracy scores [F (7,14)= 3.50, p < .05; see Table 3 below]. Post hoc analysis using the Tukey HSD method indicated that participants with the slowest search time category had significantly lower cognitive aptitude scores than either the top or middle search time categories. The same was essentially true for the Matrix reasoning scores; however, the differences only approached significance. Correlations between search time across all three levels of aptitude and the dependent measures are also presented below. As shown in Table 3, the Block Design subtest correlated the highest with search time.

Search Time

Table 3. Search time and cognitive aptitude

Perception of Disorientation

Analysis of participants' perception of disorientation revealed significant differences between the three levels of perceived disorientation and the Vocabulary and Verbal IQ subtest scores [F (7,15)= 3.12, p < .05; see Table 4 below]. Post hoc analysis indicated that participants with the lowest level of perceived disorientation had significantly higher Vocabulary scores than those with high levels of perceived disorientation. Similar results were found for Full Scale IQ, but these results only approached significance. In addition, participants with the lowest level of perceived disorientation had significantly higher Verbal IQ scores than those with either the middle or highest levels of perceived disorientation. Interestingly, the perception of disorientation was not significantly correlated with search time (p = .61; r = 0.12). The correlations between perceived disorientation across all three levels and the dependent measures are also presented below. As shown in Table 4, the Verbal IQ subtest correlated the highest with perceived disorientation.

Table 4. Perceived disorientation and cognitive aptitude

Computer/web experience and computer comfort

Assessing participants' computer and web experience, as well as their comfort with using computers with regard to their search time and perceived perception, not surprisingly, did correlate significantly. Specifically, the participants' level of comfort with the Internet was significantly correlated with perceived disorientation (p < .05; r = -0.49). Moreover, participants' indications of frequent web visits significantly correlated with their search time in that frequent web users generally had faster search times (p < .05; r = -0.51).

Discussion

This study has shown that psychometric tests of cognitive abilities can generally predict search performance for complex interfaces, in that certain cognitive factors do significantly correspond to search time performance and perceived disorientation when searching within a complex interface. When considering search time, the subtest factors that most determined participant performance were Block Design, Performance IQ, and Mirror-tracing Accuracy. All of these subtests generally tap into three cognitive functions: spatial visualization, visual-motor perception and coordination, and fluid reasoning. Thus, it is proposed that these cognitive/motor functions play a substantial role in determining search performance within complex interfaces.

The Vocabulary subtest did not significantly contribute to search performance. It is certainly possible that the Vocabulary subtest, which generally measures verbal knowledge, was not as important to performance because the task involved mostly searching for information. The Similarities subtest, which generally measures intellectual ability, also did not significantly contribute to search performance, possibly for the same reason as above.

When assessing perceived disorientation, only the Vocabulary and Verbal IQ subtest factors significantly contributed to participants' perceived disorientation. Not surprisingly, both of these subtests have a common thread, in that they both measure intellectual ability. It is interesting that perceived disorientation was mostly affected by intellectual factors, rather than spatial factors. It is possible that complex interfaces burden the intellectual capacity of users, which is translated by means of higher correlations with the disorientation scores. Yet, apparently this burden is not great enough to affect the search time of the users.

Implications for Interface Design

It is very common these days to encounter user interfaces that contain multiple information sources that are densely displayed—such as with online travel and brokerage sites. When creating these types of interfaces, designers should take into consideration our cognitive and motor limitations. From these results certain design recommendations can be suggested. Specifically, layouts should be designed to reduce the cognitive burden associated with spatial visualization and visual-motor coordination. To help do this designers should focus their efforts on creating interfaces that appropriately group information by function (Dodson & Shields, 1978) and reduce overall information density to less than 50 percent of the screen area (see Horton, 1989 for a discussion of the empirical studies related to this).

References

Dodson, D. W., & Shields, N. L. (1987). Development of user guidelines for ECAS display design (Vol. 1) (Report No. NASA-CR-150877). Huntsville , AL : Essex Corp.

Goebel, R. A., & Satz, P. (1975). Profile analysis and the Abbreviated Wechsler Adult Intelligence Scale: A multivariate approach. Journal of Consulting and Clinical Psychology, 43, 780-785.

Horton, W. K. (1989). Designing and writing online documentation. New York : John Wiley & Sons.

Kaufman, J. C., & Kaufman, A. S. (2001). Time for the changing of the guard: A farewell to short forms of intelligence tests. Journal of Psychoeducational Assessment, 19, 245-267.

Tullis, T. S. (1983). The formatting of alphanumeric displays: A review and analysis. Human Factors, 25, 657-683.

Tulsky, D. S., & Zhu, J. (2000). Could test length or order affect scores on Letter Number Sequencing of the WAIS-III and WMS-III? Ruling out effects of fatigue. Clinical Neuropsychologist, 14, 474-478.

Vicente, K. J., Hayes, B. C., & Williges, R. C. (1987). Assaying and isolating individual differences in searching a hierarchical file system. Human Factors, 29, 349-359.

Vitz, P. C. (1966). Preference for different amounts of visual complexity. Behavioral Science, 11, 105-114.

Wechsler, D. (1999). Wechsler Abbreviated Scale of Intelligence^TM (WASI^TM). Wechsler Abbreviated^TM. San Antonio, TX: The Psychological Corporation.

¹Note: Michael Bernard is currently a post-doctorate fellow at Sandia National Laboratories