Why quantitative and qualitative research needs to run together?

As a UX designer, I am in love with metrics this whole idea that we can just use data for anything and A/B tests is the best way to a great product. I think it’s just one of the worst aspects of the rise of this worship of data. Lots of teams think that quantitative data can solve all of their product problems. 

Only 24% UX professionals are reported using quantitative AND qualitative data to determine success

NN Group “Quant Qual Research in Practice survey with 429 participants”

Quantitative data doesn’t replace designers or design or replace listening to the users and it doesn’t tell us what we should be building. It gives us another channel for getting feedback from the users, but the information that data gives us isn’t the same as what we’re getting from connecting with actual human beings who are using our product. It only gives us information about what we have built.

  • Data doesn’t replace design or designers
  • Data doesn’t replace listening to the users
  • Data doesn’t tell us what we should be building

Quantitative data can help inform our design decisions, it doesn’t replace designer, but it gives us important feedback on the product decisions that we’ve made and understand whether our decisions helped or hurt user behaviour. Data can tell us whether we’re going in the right direction and it can also help us inform our research and tell us what we should be investigating.

Example:

We are going to use a checkout process funnel as our example, in this process users sign up for mobile plans. Everything you pour on top of the funnel comes out from the bottom that’s how funnels work. We are losing a certain percentage of the users who started right from the top and from each step users are dropping out. For every 100 users that are going through this funnel only 19 users are successfully getting through in other words you’re acquiring users through some means probably with google ads or getting customer referrals or any other expensive channels.

If you look at our checkout process, we are losing more users in some steps than others like specifically on payment information screen step looks poor, we are losing 38 users out of every 100, so that’s what we call a friction point. By looking at these numbers most of the teams immediately will start working on the ways of fixing the payment information screen. Some team members or stakeholders will suggest, “maybe we need to add new payment methods” or “may be it’s really confusing for user see the credit card type dropdown” or “maybe the user needs to feel secure and we should put a big lock on the page” or “may be we should try doing all of those things and then just see if a problem gets fixed”. Now at this point you should immediately stop thinking about solution. You are effectively trying to use quantitative information for something that it does not do you’re trying to use it to explain rather than to describe. If you really want to understand that friction point you need to do two things you need to describe it right and you need to say this is what is happening in the process that’s where the metrics and analytics are useful and using qualitative research you need to explain why it’s happening.

What we should be doing?

  • Step 1 Identify the biggest problem by using funnel metrics
  • Step 2 Understand WHY with observational usability testing
  • Step 3 Propose solution – Create solution hypotheses
  • Step 4 Learn and iterate

In our example, we can see payment information step in our check out process is the biggest problem. We understand that we are losing a lot of users but we still don’t know why it’s a problem. Observational usability testing technique can be used to understand why are we losing users on the payment information step. Observational usability testing is watching users use your product to understand all the pain they are facing and where they are getting stuck. Usually, this exercise is run with between 4 to 6 users. Now in our example by running usability testing, we came up with these sort of observations:

  • Some of the users didn’t have the payment information handy
  • All the users were confused with the box for promo code and sort of wandered off to look for a discount coupon

The first step in making a product change is to create several solution hypotheses because we need to test to see if the change was successful. In order to test to see if proposed changes are successful, we have to define what both success and failure look like. You should list all the possible negative consequences of the change as well as the positive ones. By listing these possible negative and positive consequences, you can evaluate later, which changes worked and which didn’t. We can use these steps to identify a new problem by looking at our quantitative data and try to fix it. We should keep iterating until we are consistently happy with the numbers that we’re seeing in the funnel.

Summary

There are lots of other ways to combine qualitative and quantitative data. The combinations are endless, you can start with qualitative research and identify user needs through user interviews or you can spot different patterns and very large sets of data or you can start with the usability test. It also depends on time, budget and resources available.

What is the difference between POC & MVP? and Which comes first?

In past couple of projects, several clients have asked me this question: What is the difference between POC & MVP? and Which comes first? Most businesses are familiar with these terms but don’t know the exact meaning that’s why they often experience mistakes while building a product. They jump into starting up a development team to build a product. Knowing what each term means is not good enough, you should know when to use them, is vitally important to the success of your next product launch.

First let’s understand their definitions

What is a Minimum Viable Product (MVP)?

A Minimum Viable Product (MVP) is a development technique in which a new product or website is developed with sufficient features to satisfy early adopters. The final, complete set of features is only designed and developed after considering feedback from the product’s initial users. [Source]

What is a Proof of Concept (POC)?

A proof of concept (POC) is a demonstration, the purpose of which is to verify that certain concepts or theories have the potential for real-world application. POC is therefore a prototype that is designed to determine feasibility, but does not represent deliverables. Proof of concept is also known as proof of principle. [Source]

Proof of Concepts (POC) are great way to validate the market on a small budget. Before you spend money on coding, you need to validate the assumption that people actually want your product. Discovering a problem that your idea will solve is easy, finding a solution people want is what you have to validate.

Proof of Concept (POC) Approach

There are 5 key activities associated with the Proof of Concept process as displayed below:

  1. Initiate the Proof of Concept – This activity is to initiate the proof of concept, which means getting everything ready for the POC to start. You will need to establish the team members that will be on the project. You will need to bring on board not just the IT and Finance departments but also include Operations, Sales, Marketing, etc.
  2. Confirm Overall Requirements – This activity  is to confirm the overall requirements and expectations for the proof of concept. You will identify the business processes and ask customer to provide details of all the key stakeholder and departments and how these departments and key stakeholder are going to impact the proof of concept.
  3. Business Process Workshops – This activity is to conduct the business process workshops, which will cover the current revenue scenarios. You will need to clearly document the proposed process changes, configuration changes, development changes, and custom rules that are needed.
  4. Business Process Deployment – This activity is for actual deployment of changes identified in the business process workshop. Once the specifications are documented the focus moves towards implementation, testing, analysis, iteration, and documentation of the changes. You will need to plan several cycles of testing to make sure that all the scenarios working correctly and the associated business processes and procedures are in place. The final output of this activity is to produce successful test results using the newly designed processes.
  5. Final Documentation – This activity is to complete the analysis and final documentation. This is where you combine all of the test results, specifications, process changes, etc. into one final deliverable.

A Minimum Viable Product (MVP) helps businesses start the process of learning as quickly as possible. It is not necessarily the smallest product imaginable, though; it is simply the fastest way to start learning how to build a sustainable business with the minimum amount of effort.

Most businesses try to please the customers and try to give everything but research shows that 64% of the features are rarely used or never used by users. I have described below 5 steps to resolve this issue (Jeff Patton book User Story Mapping)

mvp

Once you have your features prioritised, now, the first row on your map – this is the smallest possible representation of a working product. This is what business should build first.

 

In a nutshell, Proof of Concept validates two goals “People need that product” and “You have capability of building that product“. Minimum Viable Product validates that “people need the product and they will buy it”. Customers don’t see the POC’s, that kind of thing is done in-house. They do see the MVP’s and that is what they are willing to buy. MVP’s can be deployed into production but we cannot deploy POC’s into production.

proof-of-concept-minimum-viable-product

 

Follow Us!

Kloud Solutions Blog - Follow Us!