One of those dimensions seems to get a lot more attention than the rest of them: Qualitative vs. Quantitative. These two have seemingly split the entire industry in half, with the mythical “Mixed Methods” researcher being seen as a unicorn-like ideal.
Recent discussion about whether any one person is truly a “mixed methodologist” underscores a potentially ugly truth about the state of Research: most people come from (and are more comfortable with) a Qualitative background/approach.
This guide is meant to meet you where you are with quantitative familiarity to help you level up your Quant knowledge.
Before we get too deep into the weeds, let’s make sure we’re on the same page as to what exactly “quantitative” and “qualitative” mean. There isn’t a precise definition of this, and a quick search of Merriam-Webster and the Oxford University Press suggests that they’re basically defined as “not the other thing." 🙈
For our purposes, though, we’re going to use the following definitions:
Nielsen-Norman Group has a deeper dive on qual vs. quant if you want to read that, with a caveat that it’s more focused on usability testing than on research overall.
Surveying is easily the most popular quantitative research method. However, methods like A/B Testing, Card Sorting, Product Analytics, and Multivariate testing also fall under the quant umbrella. Common qualitative research methods include interviewing, usability studies, contextual inquiry and ethnographic observation, and diary studies. You can, of course, add qualitative elements to your quantitative study (e.g. asking follow-up questions during a moderated card sort), and vice versa (e.g. measuring time-on-task and success/error rates for usability studies).
Now that we have a working definition of these two terms, and a hint at how you might combine them, let’s discuss why mixing methods matters.
Ignoring our work as researchers for a moment, statistical literacy is a critical skill in today’s world. The sheer amount of data created, processed, and consumed has exploded in the past decade, and if we learned anything over the past few years, statistical literacy may be more important than ever. How many times have we all seen people mix up correlation and causation?
Jokes aside, having a solid understanding of the “other side” of research that you’re less familiar with is crucial. First, if your only tool is a hammer, it’s tempting to treat everything as if it were a nail. Too many researchers, when faced with a tough problem, will fall back on the methodologies they’re comfortable with instead of using what’s actually best for the questions they’re being asked to answer. If you’re uncomfortable working with large data sets (or don’t even know how you’d collect that much data), it can be far too tempting to conduct a handful of interviews and then start reporting percentages based on your findings.
“If your only tool is a hammer, it’s tempting to treat everything as if it were a nail.” – Silvan Thompson? Abraham Maslow? E.S. Dallas?
I know I just said 30 is the magic number, but remember: that’s per segment. If you have hundreds of thousands of users, it’s virtually impossible to capture all of that with 30 interviews, and turning 16/20 into “80% of users” can create a crucial failure in your findings.
I’ve seen research plans that call for 10 interviews, thinking they’re going above and beyond the typical 5 to 8 users — except they’re covering 8 subsegments. That puts their minimum number of interviews at 40. An understanding of quantitative reasoning and analysis is required to recognize that flaw and to understand the dangers of sampling bias.
Improving your understanding of quantitative research also makes you a better teammate, especially when it comes to cross-functional collaboration.
Whether you’re in a critique session with fellow Researchers or you’re working alongside colleagues in Marketing (especially Growth Marketing), Data Science, Engineering, or Product, having a solid grasp of the core concepts in quant allows you to make meaningful contributions.
This could be as simple as calling out potential sources of bias, or finding ways to weave some quantitative metrics into a study design (bonus points for leveraging existing data, whether proprietary to your company or publicly available for your industry). If you’re trying to find ways to work together across disciplines, understanding quantitative research will make it easier to see opportunities for collaboration.
For many of us, the building blocks of qualitative research are things we learned through school or socialization. Active listening, follow-up questions, note-taking, and affinity diagramming are concepts that are familiar to most, though mastering them to the level required for Research is its own challenge.
Depending on how far you are into your career, you might need to think back to the earlier days to understand what it’s like to learn a new skill. Take that same beginner mindset, and go through the same motions you did back then: focus on learning the basics, then start to build proficiency with execution, analysis, and synthesis. After that, you can build deeper proficiencies and start to become more T-shaped.
ℹ️ If you’re feeling uncertain about research design in general, check out this excellent introductory textbook.
Now that I’ve convinced you it’s important, you may be thinking “Great, I’m convinced — but I’m equally convinced I don’t know a mean from a mode, and I have no idea how or where to get started”. Intimidation and fear are powerful forces and often prevent people from even trying. Before you even worry about quantitative research methodology and analysis, it's helpful to build a basic understanding of statistics. Most educational systems prioritize calculus over statistics, so it’s entirely possible you went your entire academic career without taking much of any statistics coursework.
Start with the basics:
These are the building blocks of any quantitative research study. Think of them as a new language you hope to speak with colleagues. When discussing any data set (say, the results of a survey), being fluent in what the standard deviation means in relation to the mean or median is the building block of a fruitful critique.
Once you have a solid grasp of the core concepts, you can decide where to focus your attention and dive deeper. While it might be fairly easy to pick up the basic concepts, it can take years to build proficiency in any of the following topic areas, and doing so will open up new career paths for you:
Of course, you can also stay in the Research world and use these skills to drive better outcomes by delivering better insights. 💁 Deepening your understanding of what Research can do and the value it can deliver helps you advocate for yourself and your team, raising the profile of your work within your organization.
If you’re already comfortable with one or more of the above topics, it’s time to get your hands dirty. By improving your technical expertise, you move beyond being able to recognize the application or design the study, opening up more flexibility in your analysis. It’s a very different experience to reuse an existing analysis provided by a vendor or a teammate, versus being able to get into the data tool yourself and follow whatever piques your interest. That interest might again lead you into an entirely different career arc, which proficiency in these skills would certainly open up if you wanted to pursue.
And certainly, when I come across Researchers doing things like writing Python, SQL, or R alongside their diary studies or interviews, I do not hesitate to call them Mixed Methods Researchers.
No matter where you are in your career development, what company you work for, or what your role is, having a deep understanding of and comfort with quantitative research and analysis will improve the quality of your work and the interactions you have with your colleagues. Finding novel ways to triangulate your findings with qualitative and quantitative data leads to higher levels of confidence and stronger recommendations. The path to quantitative excellence may be a long and scary one, but it's worth it.
Brad (they/them) is a UX Leader, User Researcher, Coach, and Dancer who's been helping companies from early-stage startup to Fortune 500 develop engaging, fulfilling experiences and build top-tier Research & Design practices since 2009. They have helped launch dozens of products, touched hundreds of millions of users, managed budgets ranging from $0 to $10M+, and coached hundreds of Researchers. Born in Buffalo and currently based in Brooklyn, NY, Brad dances with the Sokolow Theatre Dance Ensemble and Kanopy Dance Company, co-organizes the NYC User Research meetup, and served on the Board of ResearchOps from 2018-2021.
