Usability testing is a technique used in user-centered design to evaluate a product by testing it on real users. This helps you understand how your users experience the designed product/feature etc. Usability testing collects qualitative data through exploration of how our target audience interacts with your product/feature.
This article gives you insights into how to set up a usability test, what data to collect, and above all, how to quantify that data for your research results.
How to set up a usability test?
Define a goal
Before starting with usability testing, it is important to first define the purpose of the test, the goal. The most common goals for usability tests are:
- Effectiveness
- Efficiency
- Learnability
- Memorability
- Satisfaction
- Errors.
Define a story
Because our users are not UX experts, a usability test should be accompanied by a story or a set of scenarios that create a context for the user. These descriptions help our users interact with the product or service to complete the tasks given to them.
Define usability tasks
Important during a usability test is to set out user tasks (or sometimes referred to as usability tasks). These tasks are like mini assignments that you give to a participant during the test. These tasks can be actions or activities you want your participant to engage in (table 1).
| Goal | Scenario | Tasks |
|---|---|---|
| Find out the effectiveness of the redesign of the payment process when trying to upgrade the Spotify subscription from free to premium | You’ve had a Spotify account for about a month now and you have decided that you want to upgrade your subscription to premium. You no longer wish to hear commercials and want to enjoy the music without interruption | 1. On the Spotify website, navigate to your account. 2. Find and select the payment plan you are interested in. 3. Pay for your new subscription 4. Find your new subscription in your account |
Define the success
And finally, in the preparation of your usability test, define when a task is a success. Is that when the user performs a task in less than 3 clicks? Or under a minute? Or when they’ve taken a specific path to get to where you need them to go? By defining the success state of the task, you make it easier to measure the data collected through the test.
Measuring the usability data
Each usability tasks can be assessed for effectiveness, efficiency and satisfaction (Shukairy, 2022) which can be done with the following metrics:
- Task success rate
- Error rate
- Time-based efficiency
- Overall relative efficiency
- Post-task satisfaction
- Task level satisfaction
Usability testing metrics for effectiveness
1. Task-Success Rate
Task-success rate or completion rate refers to the percentage of users that were able to successfully complete the given task. Of course, every researcher wants a 100% success rate so the higher this score, the better. On average a success rate is about 78% and is very much dependent on the context in which the task is evaluated. This metric can be measured at the end of the test.
How to calculate the task-success rate?
The completion rate is calculated by assigning a binary value of ‘1’ if the test participant manages to complete a task and ‘0’ if they do not. Take the number of correctly completed tasks and divide them by the total number of attempts.
2. Error rate
Another method to measure effectiveness is by counting the number of errors the participant made when attempting to complete a user task. These can be unintended actions or mistakes the user makes. Make sure that you prepare beforehand what the errors are that might occur and give them a rating in terms of severity for the test.
An average number of errors per taks is 0.7, with 2 out of every 3 users making an error. On average, only 10% f users perform tasks without any errors, meaning that it is normal for users to make mistakes.
Usability testing metrics for efficiency
3. Time-Based Efficiency
Time-based Efficiency is calculated by first measuring time on task. This is the average time a user takes to successfully complete each task. This metric is useful when looking for problems with the usability and is calculated by subtracting the start time from the end time.
The formula for time-based efficiency is (Sergeev, n.d.):
Where:
N = The total number of tasks (goals)
R = The number of users
nij = The result of task i by user j; if the user successfully completes the task, then Nij = 1, if not = 0
tij = The time spent by user j to complete task i. If the task is not successfully completed, then time is measured till the moment the user quits the task
Template: Calculation of time-based efficiency
You can use the table below to help you calculate the time based efficiency of your usability task. For nij we fill in a 1 when the task was completed and a 0 when it was not. Time to complete is the seconds it took for the user to complete the task, or the seconds it took them to abandon the task. The total (last column) gets calculated by dividing nij by tij.
Task: [enter task] (N)
| User (R) | Completed (nij) | Time to complete (tij) | Total |
|---|---|---|---|
| User 1 | yes (1) / no (0) | [enter seconds] | = nij/tij |
| User 2 | yes (1) / no (0) | [enter seconds] | = nij/tij |
| User 3 | yes (1) / no (0) | [enter seconds] | = nij/tij |
| User 4 | yes (1) / no (0) | [enter seconds] | = nij/tij |
| Total | [leave blank] | [leave blank] | Sum of above |
Time-based efficiency (goals per second) = Sum of above / (task total (N) x user total (R))
4. Overall relative Efficiency
The overall relative efficiency uses the ratio of the time taken by the users who successfully completed the task in relation to the total time taken by all users. And the formula for that is:
Template: Calculation of overall relative efficiency.
With a similar table as we used before (and almost the same data) we can calculate the overall relative efficiency of the task(s). The big difference with the time-based efficiency is that we now multiply the completed number by the number of seconds it took to calculate the total. And we also add up the time to complete tasks.
Task: [enter task] (N)
| User (R) | Completed (nij) | Time to complete (tij) | Total |
|---|---|---|---|
| User 1 | yes (1) / no (0) | [enter seconds] | = nij * tij |
| User 2 | yes (1) / no (0) | [enter seconds] | = nij * tij |
| User 3 | yes (1) / no (0) | [enter seconds] | = nij * tij |
| User 4 | yes (1) / no (0) | [enter seconds] | = nij * tij |
| Total | [leave blank] | tij Sum of above | Sum of above |
Total overall relative efficiency (%) = (Sum of above / tij Sum of above) x 100
Usability Testing Metrics for Satisfaction
A simple way to measure user satisfaction is by using standardized satisfaction questionnaires, which can be administered after each task or at the end of the usability test.
5. Post-task satisfaction
After a user has attempted a task, they should be given a questionnaire to measure their task difficulty experience. These quick questionnaires never contain more than 5 questions and are then used after each task has been performed (or attempted to achieve). These post-task questionnaires often take the form of Likert scale ratings and their goal is to provide insight into task difficulty as seen from the participants’ perspective.
The most popular post-task questionnaires are:
- ASQ: After Scenario Questionnaire (3 questions)
- NASA-TLX: NASA’s task load index is a measure of mental effort (5 questions)
- SMEQ: Subjective Mental Effort Questionnaire (1 question)
- UME: Usability Magnitude Estimation (1 question)
- SEQ: Single Ease Question (1 question)
However, making a satisfaction questionnaire yourself via Google Forms is also a good option. The SEQ is simple and effective, leaving your users plenty of time to focus on the task without bothering them with questionnaires over and over.
6. Task level satisfaction
Task level satisfaction is done at the end of the usability test and not after each task. This serves to measure the users impression of the overall ease of use of the system (Interaction Design Foundation, n.d.).
The most popular task level satisfaction questionnaires are:
- SUS: System Usability Scale (10 questions)
- SUPR-Q: Standardized User Experience Percentile Rank Questionnaire (13 questions)
- CSUQ: Computer System Usability Questionnaire (19 questions)
- QUIS: Questionnaire For User Interaction Satisfaction (24 questions)
- SUMI: Software Usability Measurement Inventory (50 questions)
Where SUS is a relatively cheap and very accurate tool to use for testing the satisfaction of the task(s).
Conclusion
Usability metrics can be very useful to help you quantify the usability of your system, regardless of it being software, hardware, web-based or mobile application. These metrics are based on years of extensive research and are proven to work.
Moreover, these usability testing metrics cover the three core elements that constitute the definition of usability: effectiveness, efficiency, and satisfaction. This ensures an all-round quantification of the usability of the product being tested.
References
Interaction Design Foundation. (n.d.). What is Ease of Use? The Interaction Design Foundation. Retrieved July 25, 2022, from https://www.interaction-design.org/literature/topics/ease-of-use?ep=ug0
Sergeev, A. (n.d.). UI Designer – ISO-9241 Efficiency metrics – Theory of usability. UI Designer. Retrieved July 25, 2022, from http://ui-designer.net/usability/efficiency.htm
Shukairy, A. (2022, May 4). Usability Metrics: A Guide To Measuring The User Experience. Invesp. Retrieved July 25, 2022, from https://www.invespcro.com/blog/usability-metrics/
