Our journey building the ultimate scheduler for UX research interviews

Since Great Question’s early days, our scheduler for UX research interviews has always been one of our platform’s most popular features.

And it made sense — the old way of scheduling interviews with users was a nightmare. We offered a quicker, easier way to schedule them in all their flavors.

Still, our scheduler hasn’t always been best-in-class.

Building a best-in-class product is a long journey with twists and turns. It doesn’t happen overnight. Having made significant strides over the past year in our scheduler’s reliability and performance, we want to share more about:

• Where our journey began and how far we’ve come
• Why it’s so damn hard to build a best-in-class scheduler for UX research interviews
• The other improvements on our product roadmap that will continue to raise the bar

Where our journey began

In 2021, Great Question launched with a simple premise: help companies talk to their customers. 

Our workflow to help researchers conduct moderated interviews was one of the earliest features on the platform — and one of the most popular, with nearly 100,000 interviews scheduled in the last 4 years. 

We originally integrated with Calendly, which worked for simple cases. But the limitations were quickly apparent: 

• No way to accommodate observers, multiple stakeholders, and other elements unique to researchers. 
• No way to easily integrate with the Great Question Repository. 

So, we decided to build our own scheduler instead.

To launch an interview study, researchers configured their availability, set the booking details, and sent invitations. Candidates then booked interview slots to participate in the study.

Great Question would send out a calendar invite to both the researcher and the candidate, ensuring that the event would appear in their calendars.

At the scheduled time, you would simply conduct the interview with the participant, and follow up with a thank-you. Easy, right?

When the scheduling gets tough

From a product standpoint, scheduling is an incredibly challenging problem. It’s complex for many reasons:

• We want researchers to set their availability dynamically, using their calendar bookings. This means integrating with researchers’ Google and Microsoft calendars.
• Studies can have multiple moderators — both where multiple moderators are present in calls, or in a round robin set up where bookings are allocated to different moderators based on an algorithm.
• Studies can have observers who watch a live-streamed version of the call. This required a separate, parallel calendar event, sent to only the moderator and the observers, as well as a livestream bot that would join the call at the correct time.
• Research often happens across numerous timezones.
• Participants and researchers must have the ability to reschedule, cancel, propose new times, and join waitlists.
• Zoom and Webex meeting rooms need to be created for each booking and kept up to date when bookings were modified or rescheduled
• Researchers would often need to make changes to event details after booking — like to change the call location, switch the video call provider, hand it over to another moderator, or add additional participants to the call.
• Calendar events, once generated, can also be modified off-platform. They can be forwarded to new recipients, declined, or updated with new details. This means that the two records that represent the booking (Great Question and the calendar event) can easily slip out of sync with each other.

All of this was built on a simple foundation that ultimately proved inadequate for representing complex bookings and maintaining close synchronization with a host of external services.

Over time, some cracks appeared.

Only selected booking details — such as the event date and time — were saved against the booking. Other details, such as the attendees list, were only defined at the study level, on the assumption that these would not need to change. So if study settings were modified after booking, the changes were propagated to existing booked events when the events were re-synced (e.g. upon reschedule), leading to more confusing side-effects and unpredictable behavior.

When our booking record slipped out of sync with external calendar events, a reschedule could trigger an update that would unintentionally trample previous changes. For example, if invites had been forwarded to other attendees outside our system, they would fall off the attendees list (and receive a cancellation) when we issued an updated invitation for the rescheduled time.

Given the complexity of the system, these issues were initially difficult to detect. Compounding the issue was that these capabilities (and changes made to them) were very difficult to test. Manual testing required multiple accounts across several services, while the number of live integrations created significant hurdles for effective automated testing.

As a result, we went through a challenging phase in which many of our customers experienced numerous issues with booking accuracy and calendar event reliability. This impact was felt beyond users, also impacting colleagues and research participants.

How we’re fixing this & what comes next

In mid-2024, we began rebuilding our scheduling system from the ground up to serve as a more capable, extensible foundation for the other capabilities we had built. 

Our new system has three key elements:

1. A more declarative approach. Initially, our system used a step-by-step update method: when one element changed, we would manually trigger an update for the related element to keep things in sync. As the complexity of the system grew, this rigid approach proved unsustainable. Our new declarative method simply defines the desired outcome — for example, ensuring that any change in one area automatically keeps all related elements in sync — making the system easier to manage and extend.
2. Full two-way synchronization. In order to capture all changes made to calendar events, we needed to maintain a full replica of the event on our side. This meant that our bookings would reference study-level settings upon creation, but then maintain its own copy of every relevant event detail. This meant no more trampling of changes or unexpected changes during syncs.
3. A better way to edit bookings and understand the impact of changes. With all relevant information stored against each booking, the only missing part was to provide users the ability to change these values. Our new Edit Booking feature allows users to not only view and edit all booking details, but understand the impact of every change; it clearly explains who will receive event invites, updates, and cancellations.

We’re excited to launch these features this March. We believe this update will end a challenging period built on an outdated and overextended foundation. As a startup, choosing to pay down technical debt isn’t easy, but we’re confident that the investment will result in a rock-solid scheduling system that will serve our customers well into the future.