If you have ever heard someone say to you "uX dEsIgN iS sUbjEctive" and you have been at a loss of words after realising that there is complete lack of trust in design within the team, this research exercise is here for your rescue. While some might say, it's high effort and very complex to execute, IMO it's one of the most effective ways to build confidence in UX for a company and totally worth the effort.
To remove the 'subjectivity' from UX one must ask, how do product designers evaluate whether the product experiences they build are perceived to be ‘Good enough’ by their users?
It is worth noting that this UX measurement exercise can also helps benchmark product experiences against a compete, it can help you understand how the experience has improved with time and can build confidence in the team about what are other apps in the same market space providing UX wise.
When is the right time to do a UX measurement study?
If one of the biggest question your product team is trying to address are one of the following…
a) Does our product enable [type of users] to do [jobs that users hire your product for]?
Example: Does our product enable new users to save money through learning about financial management?
OR,
b) Does our product enable [type of users] to feel [core UX tenant] while using [feature or product area]?
Example: Does our product enable building managers to feel control and ease while using the building’s HVAC control panel?
…it is the right time to do a UX measurement study (Or a UX benchmarking study)
Few things you need for an effective measurement
1. Knowing what characteristics are sought by your users when engaging with your product and defining how to measure those characteristics. - Use foundational research.
For example, Paytm allows users to do UPI transactions and safety (from hackers, stuck payments, etc), fast service and reliability are it’s tenants and perceived safety, time on task and task completion rate can be used to measure it. Figure out what is the UX supposed to make the users feel, define how to identify in the product that the user is feeling that feeling. Is the UX supposed to make your users feel more control? Measure perceived control.
2. Learn about the core tasks user/s hire your product for, list them. For example, for Paytm this could look like, buy groceries, pay bills, shop online, etc.
3. Recruit users with varied levels of familiarity with your product, construct fictional scenarios where the user will have to perform your listed task to get the scenario done on your app.
4. Measure your defined metrics (Perceived time taken, task completion, perceived effort, perceived safety, etc - define your fit) compare the result with the compete apps tasks.
5. An outcome from such an exercise will look like this:
Creating a view for your team to understand in detail that, "new users feel, for paying bills, your compete is a better alternative because of higher reliability as the app ensures task completion." is incredibly useful outcome from this exercise. An insight like this develops a deep understanding about what's going wrong with the UX, why and how to fix it.
Understand the difference in ratings between the products and the reasons for the differences, and develop a product strategy on the outcomes of the success for both products. You shall have an objectively improved UX that will help achieve business goals with high confidence amongst the team.