Evaluating your product is an important part of the design process. Through evaluation, we learn whether or not the product has the intended user experience and where there is room for improvement. But there are many ways to evaluate your design. Your product goes through different stages of design and the method of evaluation depends on the stage of the product. This article helps you choose which method of evaluation can be useful based on the stage of design you are in.
Evaluating the stage of design
First, you need to take a look at the product you are intending to evaluate and find out in what stage of the design process you are. There are four phases of the design.
Phase 1: Design proposal
In the design proposal phase, there is an idea for a product. You may have done some preliminary research and put some design requirements together based on your research. Perhaps you’ve created a one-pager of what the product will be once it goes into development.
Phase 2: Global design
In this global design phase, there is a first elaboration of the product. For example, that one-pager you had made in phase 1, has now grown into a design plan, which details all the aspects of the design. Perhaps you have created a first, simple, paper prototype or lo-fi to see how your design would work.
Phase 3: Partly detailed product
In phase 3, you have a partly detailed product. In this phase, you have parts of the product in higher detail, perhaps a hi-fi prototype of a specific feature, or a hi-fi prototype of the overall flow of the app without the detailed features.
Phase 4: Finished product
And in phase 4, you have a fully finished product. Technically this product could hit the market soon or perhaps is already for sale. Whatever the case is, the product is finished.
Evaluating the quality aspect(s)
When you have evaluated the stage of design your product is in, that’s when you can look at what type of evaluation you are looking for. There are 6 types of quality aspects to evaluate. Not all of these can be evaluated for every stage of the design process. But we will get to that in a bit, first let’s have a look at the 6 quality aspects.
Aspect 1: Relevance
If you are interested in seeing whether or not there is a need for the product then you would be looking for the relevance of it. This is where you can check if your (scientific) research matches the design of your product.
Evaluating the relevance of a design is most useful during phases 1 and 2 when you have a design plan or a global design ready to be evaluated.
Aspect 2: Consistency
When you want to evaluate whether or not the product is logically designed, you would evaluate the consistency of the design. You do this during phases 1 and 2 when you have a design plan of global design
Aspect 3: Expected practicality
Evaluating how you think the product will work in the setting it was designed for is done by evaluating the expected practicality. You do this during phases 2 and 3 when you have a global design or a partly detailed product.
Aspect 4: Expected effectiveness
With expected effectiveness, the evaluation becomes a bit more difficult. With this aspect, you are going to evaluate whether the product is going to deliver the expected results. Evaluating for expected effectiveness is only useful in phases 3 and 4 when you have a partly detailed or finished product.
Aspect 5: Actual practicality
Evaluate the actual practicality in phases 3 and 4 when there is a partly detailed or finished product. With this aspect, you evaluate how the product is being used in the settings for which it has been designed.
Aspect 6: Actual effectiveness
And the final aspect, the actual effectiveness looks to evaluate if the desired outcomes actually result from the product. For example, if you have designed an app that helps people to stop smoking, your desired outcome would be for the user to quit smoking. When evaluating the actual effectiveness you look to see whether the user has reached that desired outcome. Because of this, the actual effectiveness can only be evaluated in phases 3 and 4 of the design process.
Choosing the right evaluation method for your design
When you know the phase of design you are in and you know which quality aspect to evaluate for you can choose the right evaluation method for the job. For this article, we will look at 5 well-known evaluation methods and see for which phases and quality aspects they would be right for.
Evaluation method 1: Screening
During a screening, members of the design research team check to see if the product matches the design requirements with the help of a checklist.
When to use a screening?
Use a screening when you have a design proposal or global design (phases 1 and 2) and you are looking to evaluate relevancy, consistency, or expected practicality
Evaluation method 2: Focus group
During a focus group, a group of users get together and react to a prototype of the product. There is an observer present, often behind a screen, that records the session and takes notes of what is happening. You can use prompts or questions to trigger a conversation about topics. Keep in mind that you give most of the control to the focus group to keep the results reliable.
When to use a focus group?
You can use a focus group as an evaluation method during any of the design phases mentioned earlier. Focus groups are best used when looking to evaluate relevancy, consistency, expected practicality, or expected effectiveness.
Evaluation method 3: Walkthrough
Choose a walkthrough method to simulate the use of the product together with the design research team or stakeholders of the product. For example, you have created a new payment system for your company’s SaaS system. Together with the stakeholders from finance and the design research team, you can walk through the new design to see how a user could use it.
When to use a walkthrough?
Use a walkthrough when you have a partially detailed product (phase 3) and when you are specifically looking for the expected practicality of your product.
Evaluation method 4: Micro-evaluation
During a micro-evaluation, you take a small group of target users to use parts of the product outside its normal user setting. For example when you have created a new feature for an app. You get users to come into your research room, or set up remote research and ask the user to use the product right then and there. This is a good method to learn about the usability of your product without having to launch.
When to use a micro-evaluation?
A micro-evaluation is best used in the later stages of design when there is a partly detailed or finished product ready for evaluation. You use a micro-evaluation to test for actual practicality or actual effectiveness.
Evaluation method 5: Try-out
Our final method is the try-out. During this evaluation method, you test how the target user uses the product in practice. In comparison to the micro-evaluation, you evaluate with a tryout after launch.
When to use a try-out?
A try-out is the best method when you have a finished product ready (phase 4). With the try-out method for evaluation, you can check for the actual practicality or actual effectiveness of the product.
Choosing the right activity for your evaluation
When you decide which evaluation method you need, you can then choose an activity to support the evaluation. For this article, I have chosen 5, well-known, activities from design research that you can use to evaluate your design based on your method.
Evaluation activity 1: Checklist
A simple activity is a checklist. This is a list of characteristics or requirements of the product that is used during the evaluation. A checklist is best used during the screening or walkthrough evaluation method.
Evaluation activity 2: Interviews
Interviewing users is a useful way of obtaining reliable information directly from the target user. Interviews can be done during any phase of the evaluation except for the screening method. The screening method is an internal evaluation process and excludes your target user.
Evaluation activity 3: Observation
Observing your users can lead to valuable insights into their behavior, actions, and reaction. Observations are a great activity during any phase of the evaluation, except (again) for the screening method. When observing your users during any of these phases, use the data you gather to design a journey map. This will help when presenting the results to your team and stakeholders as it is a visual approach to data presentation.
Evaluation activity 4: Surveys
Surveys are a quick way to gain answers from your users via a questionnaire. A survey focuses on quantitative data while interviews give you qualitative data. Surveys are therefore best used during a micro-evaluation or a try-out as an additional data gathering method but not as the sole one.
Evaluation activity 5: Diary study
You can ask your user to keep a diary during a certain period while using your product. A diary study is only useful when your product is finished and it is being tested in the actual environment of the user.
Combining evaluation methods & activities
My advice is to combine research activities if possible. For when you have a partly detailed product (phase 3) and you are doing a micro-evaluation to test for the expected practicality you can combine methods. Have your users work with the product and ask them questions about it during and after use (interviewing activity). And ask to record the session so you can observe their behavior and actions later (observation activity). Combining these activities will give you valuable data that will give you insights into your product in ways you haven’t thought of before. And most of all, choosing the right evaluation method will make your design better in the end.
References
SLO Evaluation Matchboard
