Research in an Evolving Landscape: Frameworks for Impact

Katya Hott applied at SeatGeek out of sheer curiosity. It was the summer of 2020, and she was job-hunting amid a global pandemic.

“How is this company hiring in the throes of the pandemic for live events, where there are almost none?” Hott wondered.

SeatGeek is a live events company, ticketing everything from sports and theater to monster truck rallies and rodeos. After pressing pause in the early days of the COVID-19 outbreak, SeatGeek had begun to piece together how to make it through this period without live events and come out stronger on the other side. Part of this plan included hiring a user researcher.

“They opened a role for someone to be the only user researcher, with a mandate of starting up the function. This was part of their strategy for making it through the pandemic, and coming out with more understanding and empathy toward this new world of music, theater, and sports fans that would eventually emerge after everything was shut down to zero.”

Hott got the job, making it the third time she’s been hired as a company’s first researcher. Given her past experience, she knew she would have to gain trust before getting opportunities to make a real impact with her work.

In this session from People Who Do Research 2023, Hott shares the three frameworks she has implemented at SeatGeek to instill trust in her team and make an impact with research.

Listen to Hott’s talk here or read our recap below.

‍

Trust & impact: A positive reinforcement loop

“The more impact you make with your research, the more trust you build in an organization. And the more trust you gain, the more opportunities you have to make an impact, and this loop goes around and around and around.”

But it’s hard to build trust if you don’t have an opportunity to demonstrate impact. As a research team of one, there was no clear starting point. She just knew she had to get this positive reinforcement loop of trust and impact spinning.

Finding a place for research at SeatGeek

At the time, SeatGeek was a company of a few hundred employees and “a whole bunch of product space that could leverage user research.” But most didn’t understand where user research could fit in.

Hott decided to launch her research journey with SeatGeek’s most user-facing team – Shopping.

“This was a team that was currently working on whole-hog improvements to our event shopping screen. Since they were the most user-facing team and the most interested in learning from fans, I thought ‘these are my people. I’ll start here.’”

Balancing exploratory & tactical research

Hott embedded herself in SeatGeek’s Shopping team and got to work. She quickly discovered the biggest barrier to making an impact was a lack of clarity around research outcomes. Her coworkers were asking two types of questions:

1. Exploratory questions. These questions lead to risky bets on big research opportunities that may produce ambiguous outcomes (which are often met with skepticism). Ex: How do shoppers think about ticket value?
2. Tactical questions. These questions prompt tactical work which directly informs design decisions. However, the lightweight outcomes may lead stakeholders to question the need for research in the first place. Ex: Are our icons easy to understand?

“What was missing was how we used the answers to these questions.”

Hott compares her early days on the SeatGeek Shopping to a pendulum, swinging between exploratory and tactical work, seeing what stuck and made people excited. Soon, other teams became interested in research, so she began swinging from team to team, sometimes for a 2-week sprint, other times just for an afternoon.

The turning point for research operations at SeatGeek

One year into Hott’s time at SeatGeek, the company was growing fast and hiring a lot. She made the case to hire two more researchers, while SeatGeek also hired a new round of product managers and engineers with backgrounds in UX. But the biggest change at this time? 

“Live events came back in a big way that was maybe not even seen before the pandemic.”

In year one, Hott focused on leading by example, with little emphasis on standardizing processes. Now, she had a team to support. Tons of research requests were coming in, but they weren't sure these were even the right questions to be asking. They needed to develop ResearchOps frameworks.

CHROME: A framework for asking the right question

In order to figure out how to ask the right research question, Hott and her team pulled questions from their research plans that described the goal and mapped them into groups, creating CHROME.

• Context: The background on why we're doing this study.
• Hypothesis: How we expect a proposed design change to impact the user experience. Ex: By learning about theater ticket shopper needs, we can tailor experience to theater audience.
• Research question: What we hope to learn from this study. Ex: To learn more about baseball game experience.
• Outcome: What the findings from this study will inform (actionable next steps). Ex: To inform designs for ticket displays.
• Metrics of success: Changes in product/business that will let us know this study had an impact. Ex: To increase funnel conversions.

Today, Hott’s team uses CHROME as an alignment exercise with stakeholders to stop unrealistic projects before they begin, ensure outcomes can be measured, uncover stakeholder assumptions driving the request, and ask questions that lead to concrete decisions.

Risk vs. Clarity: A framework for finding the right effort

The CHROME framework helped create excitement around research at SeatGeek, especially among the new UX-savvy product managers and engineers. But it didn’t provide a way to determine how hard to work on any given project.

“We weren’t thinking about the level of effort required to answer these research questions. We were just following the excitement of our newfound team and collaborators.”

So, they borrowed several frameworks for finding the level of effort to exert from the industry’s best: Leisa Reichelt, Head of Research and Insights at Atlassian, and Jeanette Fuccella, Director of Research and Insights at Pendo.io.

‍

Hott’s team has internalized these frameworks by adding ‘risk vs. clarity’ to every research roadmaps. This has helped set clear expectations and instill trust in their team.

“It’s made stakeholders confident we won’t spend too much time on something that’s not risky, and vice versa.”

This trust unlocked the positive reinforcement loop, increasing appetite for research across the company and giving researchers a seat at the table.

Desirability, Viability, Feasibility: A framework to find the right opportunities

With the trust-impact loop spinning like a fine-tuned machine, Hott saw the need for another framework – this time, to help them target the right research opportunities. It’s a simple Venn diagram that helps them complex decisions by evaluating:

• Desirability – The UX team figures out what is desirable. What opportunities will meet user needs?
• Viablity – The product team figures out what is viable for the business.
• Feasibility – The engineer team figures our what is feasible to build.

They use sticky notes to map opportunities and find what lands in the middle so they can prioritize those opportunities first as a team.

‍

Key takeaways

Hott credits these three frameworks for providing concrete ways to walk metrics of success back to their research projects and for instilling confidence in her team that people see the impact of their work. 

“This is what has unlocked that trust-and-impact positive reinforcement loop.”

• CHROME tells them they're asking questions worth investigating
• Risk vs. Clarity tells them they’re putting the right level of effort into investigating those questions
• The Desirability, Viability, Feasibility Venn diagram tells them they’re helping their team select the right opportunities to prioritize

A lot has changed since Hott arrived at SeatGeek. Live events are back like never before. SeatGeek has doubled in size, while her team has quadrupled. But maybe the biggest change? 

“Now I don’t have to wonder where to start in making an impact with research. Using these three frameworks has allowed my team to make sure our research affects outcomes.”