Although incentives may seem like a harmless way to encourage participation, it’s not always that simple. What if incentives are not only encouraging research participation but also subtly nudging respondents toward specific responses? The implications of bias are profound and can ripple across the fabric of user experiences, shaping the digital world in ways that are neither intended nor desirable.
In this article, we analyze academic research to dissect how incentives can introduce or remove bias in UX research. We explore their intricate effects and unravel strategies for their ethical use.
Using incentives in UX research can positively influence participant recruitment and response rates. The type of incentive offered — be it monetary, non-monetary, or account credits — appeals to different participant demographics, which may result in various biases.
Academic studies have explored this interaction and found that incentives can boost participation. But they may also skew the results due to response or selection bias. That’s why researchers should carefully consider the nature of the incentives they use. Leveraging platforms like Great Question can aid in the strategic use of research incentives, helping you manage and track your budget effectively while maintaining a diverse participant pool.
As a researcher, your choice of incentives can directly impact the quality and quantity of participation in your studies. Let's explore the different types of incentives you might consider when conducting customer research.
Cash is king, and research incentives are no exception. The appeal of cash incentives is their simplicity — you can easily disburse them to attract a wide range of participants, including customers and non-customers. Participants can use the cash reward as they wish, motivating them to participate in research now and perhaps again in the future.
However, cash rewards have a few potential pitfalls. Certain demographic groups, such as government employees, may have restrictions against accepting cash, since it could be misconstrued as a bribe.
Others, such as high-income earners, loyal customers, or business-to-business (B2B) prospects, might not be as intrigued by cash. These groups may instead find more value in incentives like account credits, merchandise, or other rewards.
It’s best to use cash incentives when participants are not necessarily engaged with your brand or product, since product discounts or branded swag will be less enticing. Cash incentives tend to motivate participation faster, so they’re also ideal when you need quick high response rates.
Gift cards are a popular alternative to cash, allowing participants to purchase from selected stores or online platforms. While not as flexible as cash, they still provide a tangible reward that can encourage user participation. With their widespread acceptance, gift cards can effectively encourage participation from diverse demographic groups, including those who might resist a direct cash incentive.
However, gift cards come with their own challenges. They can come with usage restrictions, such as expiration dates or limitations on the retailers or platforms where they can be used. When dealing with a global pool of participants, you should recognize that gift cards' relevance can vary greatly across borders.
For instance, a participant in Tokyo may find little use for a £40 gift card to the British department store Marks & Spencer, just as individuals in Canada might not be thrilled by an offer of a 3,000 Indian Rupee voucher at a popular Indian restaurant chain like Barbeque Nation. Always consider the geographical relevance and currency of gift cards to ensure they hold meaningful value for your participants.
If you choose to use gift cards, provide options for participants to choose from depending on their preferred retailers. Also keep in mind that not everyone will accept them due to the same ethical rules that apply to cash incentives.
Offering account credits or product discounts as incentives is an effective strategy when conducting customer research for a specific product or service. This type of incentive not only motivates participation but also increases product usage and customer engagement.
Credits and discounts eliminate upfront costs and reduce administrative work, as there's no need to manage the distribution of physical cash or gift cards. Better yet, these incentives can bolster sustained customer interaction with your research.
That said, credits and discounts may be less effective for non-users or infrequent users of your product or service. These incentives may also not appeal to your users if they’re not directly involved in the purchase decision. They may be less motivated by account credits since their organization would realize the rewards, not them personally.
Offering branded products as incentives, such as T-shirts, mugs, or notebooks, can create a sense of community among participants. High-quality branded merchandise can make participants feel valued and serve as a lasting reminder of their engagement with your brand.
However, swag may not resonate with all potential participants, particularly those who aren't already users or fans of your brand. Plus, creating and shipping physical items can entail additional costs and logistics.
For socially conscious participants, offering a charitable donation in their name can be a strong incentive. This type of reward motivates participants and adds a philanthropic dimension to your research, potentially enhancing your brand's reputation.
However, the impact of charitable donations as incentives may vary significantly based on the participant's personal values and their alignment with the chosen charity. Charitable donations can be a good incentive choice when your brand values align with social responsibility, and you want this to be reflected in your research methods.
As a UX researcher, understanding and addressing bias is key to ensuring the validity of your findings. Here are some common types of bias you might encounter in UX research and strategies for mitigating them:
Confirmation bias is the tendency to seek information that confirms pre-existing beliefs while ignoring or discounting information that contradicts them. This can distort the findings of UX research, leading to skewed interpretations that reflect the researcher's personal beliefs more than the actual data. To mitigate confirmation bias, ask open-ended questions when conducting user interviews, survey research, and focus groups, and be willing to challenge your assumptions and beliefs.
Selection bias occurs when the participants chosen for a study do not accurately represent the overall population. This can lead to findings that only apply to a specific group and not the wider user base.
For instance, selection bias might occur if you only test a new app feature with long-term, highly-engaged users, and disregard new or infrequent users. The feedback you receive may be positive, but it won't represent the experience of your entire user base.
To overcome selection bias, ensure that your sample is representative of your user population. This might involve stratified sampling methods, where you intentionally select participants from different demographic groups to balance representation.
Observer bias, often called the Hawthorne Effect, is the change of participant behavior due to their awareness of being observed. In a usability test, participants might use a product in an "ideal" way, rather than their usual way, because they know they're being observed.
This can lead to inaccurate data as participants may alter their natural behavior to fit what they perceive as expected. Minimize this bias by employing methods like unmoderated usability testing or ensuring the testing environment is as similar as possible to the users' natural setting.
Response bias is the tendency of participants to answer questions in a way they think is socially acceptable or pleasing to the researcher rather than expressing their true thoughts and feelings. For example, a participant in a user interview might say they love a new product feature — not because they genuinely do, but because they think that's what you want to hear and saying otherwise might be uncomfortable.
This could lead you to overestimate the success of that feature. To avoid this, communicate to participants that all types of feedback are equally valuable and reassure them of the anonymity and confidentiality of their responses.
If you're thinking about introducing incentives into your customer research, you might be concerned about the potential impact on your results. Will incentives introduce bias, or could they help mitigate existing biases?
The relationship between incentives and bias is complex and varies across research disciplines, from clinical and medical studies to user experience (UX) research. Various academic studies have examined whether incentives introduce or remove bias in research. Here’s what researchers have found.
Offering cash rewards to attract study participants is a common strategy employed across various research fields, including UX research. Research by Bentley and Thacker found that monetary incentives can significantly boost response rates in medical research, regardless of the risk level involved.
However, these incentives could introduce a form of response bias. The study found that cash incentives had some influence on the participants’ behavior regarding concealing information about restricted activities.
For instance, monetary payments influenced the respondents to neglect to disclose their involvement in activities that would disqualify them from participating in the research, such as alcohol abuse. This means the results would be distorted if the respondents actually engaged in these activities. The incentive potentially skewed their responses to align with what they perceived as socially desirable.
A study by Abdelazeem et al. further establishes that monetary incentives can increase the number of participants in randomized control trials (RCTs). This can significantly alleviate challenges related to recruitment failure and premature termination of studies.
While this approach can effectively boost participation rates, it's also important to consider the potential bias that such incentives might introduce. Participants mainly motivated by monetary rewards might not fully represent the target population, leading to selection bias. These individuals might also exhibit response bias, providing answers they believe the researchers want to hear to secure their reward.
Research by Knoll et al. reveals that non-monetary incentives, such as coupons, may introduce a socioeconomic bias by being more appealing to individuals from lower socioeconomic backgrounds. This could potentially skew the study results since these individuals might be more motivated to participate in the research for the incentive than for genuine interest in the study topic. For example, in UX research, this bias could lead to feedback that doesn't accurately represent the user base if the product or service being tested is not typically used by individuals from lower socioeconomic backgrounds.
Hsieh and Kocielnik examined whether incentives might lead to participation bias, as different rewards might appeal to different types of people. They share the results of two studies about crowdworking, with both studies showing that the type of reward given can influence who decides to join in.
For instance, when the reward is a lottery, it attracts participants who are more open to change. Meanwhile, when the reward was a donation to charity, participants who valued helping others more were attracted.
The study also found that how participants self-selected their participation resulted in different task results. The lottery reward brought in more independent people, and as a result, they produced more ideas in a brainstorming task. This study suggests that researchers can use different rewards to attract specific types of participants or to encourage a diverse range of people to participate.
A Central University Research Ethics Committee report argues that incentives may inappropriately commercialize the relationship between researchers and research participants. This relationship could potentially distort the genuine engagement required for unbiased outcomes.
Incentives could lead participants to view the research process as a transaction, where their responses are given in exchange for a reward rather than for contributing to the advancement of knowledge. This viewpoint might compromise the quality of the data gathered, as participants may prioritize earning the incentive over providing honest, thoughtful input.
Given the insights from the academic research discussed above, there are several practical implications for UX research. These include:
Because different types of incentives can attract different segments of the population, it's important for UX researchers to carefully strategize how they reward participants. This means understanding the demographic you wish to attract and their preferences, motivations, and values.
Monetary incentives can quickly boost participation rates, which is particularly useful when gathering data within a short timeframe. However, to ensure that the feedback is representative and not just driven by the reward, consider mixing monetary incentives with non-monetary incentives.
Non-monetary incentives like coupons or product discounts can introduce a socioeconomic bias. This can lead to skewed data in UX research if a different demographic typically uses the product or service. To mitigate this, provide a range of incentives that appeal to a broader demographic, ensuring that the participant pool is representative of the entire user base.
Contextual factors are important. For example, if a UX study aims to understand the behaviors and motivations of a specific user segment, you should use incentives appealing to that segment. On the other hand, a wider range of incentives might be more appropriate if the goal is to get a broad range of ideas or feedback.
During customer research, aim for balance. Incentives should encourage participation, but shouldn’t reduce research to a mere transaction. Communicate the importance of the participants’ feedback in improving the product or service. Ensuring that participants understand the value of their input beyond the incentives can help maintain genuine engagement.
Given the findings and practical implications, you should follow established industry best practices when offering incentives for participation in your research. Some best practices to consider include:
Customize the incentives according to the target audience’s preferences. For example, tech-savvy users of a B2B software company may be more inclined to participate if you offer them access to premium product features or an Apple gift card.
Recognize the complexity and time involved in your research. An hour-long user interview should offer a higher incentive than a short online survey.
Offering a range of incentives (e.g., cash, gift cards, product discounts) can attract a more diverse participant pool. This reduces the biases associated with any single type of incentive.
The incentive type, amount, and conditions for your study should be clear from the start. Transparent communication ensures that participants enter the study with the right expectations and motivations, and don’t leave unsatisfied.
Reiterate the anonymity and confidentiality of your participants’ responses. This step can encourage participants to be more honest and open in their answers, countering any bias towards what’s deemed socially desirable.
Engage with other professionals in the industry through forums, workshops, and conferences to understand the most effective incentives to use. Understanding how others are successfully employing incentives can provide valuable insights on how to level up your own research practice.
After each study, evaluate the effectiveness of your incentives. Did they work as intended, or did they introduce unexpected biases? Use this insight to improve your future incentive strategies.
The use of incentives in UX research is a double-edged sword, with the potential to either enhance or skew results. A thoughtful, strategic approach to research incentives is vital for attracting a diverse, representative participant pool and maintaining the integrity of the data collected.
Leveraging a platform like Great Question is a smart way to seamlessly integrate incentives into your research process. Great Question allows you to set incentives during study creation and automate their distribution upon study completion, helping to boost response rates. You can easily manage and track your incentive budget for each study, ensuring cost-effective research practices.
Get started with Great Question today to unlock the power of automated, globally-accepted research incentives.
Jack is the Content Marketing Lead at Great Question, the end-to-end UX research platform for customer-centric teams. Previously, he led content marketing and strategy as the first hire at two insurtech startups, Breeze and LeverageRx. He lives in Omaha, Nebraska.
