• In what ways are UI testing and general SW testing (say, of a function) alike?
  - test plans can be developed before the product is.
  - exhaustive testing generally not possible
  - testing can be conducted by someone other than developer
  - can be very expensive
• In what ways are UI testing and general SW testing (say, of a function) different?
  - UI testing can be conducted in different development phases (eg, requirements, design, testing, acceptance test)
  - in UI testing, there are no absolute correct or incorrect results.
• Importance of UI Evaluation depends on number of product-specific factors:
  - Number of users
  - Variety of users
  - Importance of safety and reliability
  - Consequences of user error
  (related to safety but not same: consequence could be loss of online customer)
  - Budget
• UI Evaluation (or Usability) Plan should be developed and followed.
• Many UI Evaluation Techniques exist (all require people and system and feedback)
  - Expert Reviews
  - Usability Lab Testing
  - Written User Surveys
  - Interviews with users
  - Focus Groups
  - Performance data logging
  - Online suggestion box / newsgroup / bb

What is usability? Paul Booth, in Introduction to HCI (1989) says

• Usefulness : degree to which users can achieve their goals
• Effectiveness : how well users can perform tasks
• Learnability : how long to reach defined level of competence
• Attitude : user perceptions and opinions

One approach is GQM : Goals - Questions - Metrics

All these things need to be determined before test:

• Can be staff member or outside consultant
• Can be expert in application domain or in usability (or both)
• If expert in app only, may not be able to detect and describe all shortcomings.
• If expert in usability only, will require training in app before testing
• Feedback can be:
• List of problems encountered
• Problems plus suggested solutions (carefully)
• Review can occur at several development points
  (perhaps after usability test has revealed that problems exist)
• Different experts will notice different things (this is good and bad)
• Difficult for expert to "simulate" typical user
• Some Review Methods:
• Heuristic evaluation: evaluate according to e.g. 8 Golden Rules
• Guidelines review : evaluate according to developer guidelines
• Cognitive walkthrough : simulate users working on tasks
• Formal Inspection : expert vs. developer over UI problems

• Gaining in popularity (reduces time and cost risk)
• Can run range of formality.
• Formal: designed experiment
• Informal: a few subjects running usability test and reporting problems.
• Requires: staff usability experts, participants, laboratory
• Staff usability experts:
  - Operate the lab
  - Assist developer in test planning
  - Conduct tests
  - Report results
• Test participants:
• Representative of system users
• The number of relevant attributes (cognitive/physical) depends on formality of test
• Voluntary (means not coerced, not necessarily free)
• Should be informed about test and consent in writing
• Laboratory:
• Possibly devoted to usability testing
• Equipped for monitoring/observation
• Methods of monitoring: audio/video recorder, software (input device usage)
• Methods of observation: seated beside participant, behind 2-way mirror.
• The test itself :
• participant works with system to complete a set of tasks.
• participant is encouraged to "think aloud"
• observer may sit with participant
  (only to clarify comments; should not coach)
• Test should be max 3 hours.
• At end, specific questions can be asked and comments given.
• Videotaping is valuable evidence to developer, and most participants quickly ignore the videotaping.
• Input device events can also be logged by software to provide trace and much valuable productivity information.
• Limitations to this approach
• Expensive; not all developers can afford it
• Participant doesn't have chance to become acquainted with system, so can't test aspects of system designed for expert user.

• Field test is one possibility (like beta) -- tested by actual users!
• formal experiment pitting system against previous or competitor.

Users performed a set of tasks.

Some findings:

WHAT QUESTIONS DO YOU HAVE ABOUT THE SURVEY?

Some answers (from Mandel book)

• Who funded the study? Microsoft
• PC memory config? Min for MacOS, twice min for Win95 (16 MB)
• According to Apple response:
• Over 20% of Mac users selected from Microsoft list
• Tests "unfairly highlighted" Win95 features
• Win95 terminology was used
• Did not include system-level functions for which Mac is superior
• One test had users change screen resolution but warned against changing # colors (forces reboot on Win95 but not Mac)

By the way, Apple does this sort of thing too.

• (1996 Apple vs Microsoft at Software Publishers Association convention, live on stage: setup system from box, install peripherals, connect to net, install and deinstall apps. Mac won.)
• (iMac demo you can view at SMSU bookstore.)

Last reviewed: 25 September 1998