As pioneers in the industry, we consistently work on elevating the standards of research excellence. We recognize that in the world of insights, participant and data quality are paramount. In this blogpost, we’re excited to introduce you to Sasha Basson, our Network Innovation Manager, who embodies our commitment to excellence. Join us as we delve into the art of quality control and its pivotal role in crafting superior insights.

Q: Let’s start with the basics first: what is good- and bad-quality data?

“We define ‘good quality’ as real data coming from real people and serving a specific purpose. This data possesses four characteristics: accuracy, completeness, consistency, and validity.

Historically, we have considered participants, and by extension their data, as either good or bad. However, bad-quality data varies in severity. For example, all participants can make mistakes in their responses, such as a typo here or the wrong click there, and this may reduce the quality of the data. We are not concerned about these minor data-quality issues. What keeps us awake at night is severely bad quality, which can arise for one of two reasons: fraud, and lack of engagement.”

Q: What is the difference between fraud and lack of engagement amongst survey respondents?

“Fraudulent responses refer to the intentional falsification of data. One example is a bot completing a survey multiple times. These types of responses are highly undesirable since the data is false and often meaningless. On the other hand, our data can also be severely compromised when participants are disengaged – they complete the survey as quickly as possible with the least amount of effort. The resulting data tends to contain a minimal level of detail, very little variability, and although not deceptive, it does not contribute much to the overall insights.

Both motivations result in poor-quality data, but for completely different reasons. In the first instance, a purposefully deceptive participant is providing data through a bot or is untruthful about their experiences. In the second instance, the participant is providing poor-quality data because they are not motivated or engaged in the survey. We cannot influence the behaviour of participants who fall into the first category – but we have measures in place to avoid recruiting these types of participants. However, we can influence the behaviour of participants in the second category and reduce the likelihood of them becoming disengaged.”

Q: Can you elaborate a bit more on the processes Human8 installed to ensure superior data quality, and then in particular around participant engagement?

“Sure. More generally, we have put into place extensive quality-check processes that follow every step of the research process, including project set-up, data collection, and post-data collection to safeguard the best possible data quality. In the first step, we focus on carefully crafting the research design, tailored to our clients’ needs. Through thoughtful source selection and innovative thinking, we ensure that questionnaires help us safeguard data quality. A well-designed, well-phrased, and engaging questionnaire that is tailored to the project’s objectives ensures excellent quality data in at least two ways. First, a questionnaire that is properly designed and contains appropriate questions is more likely to yield data that actually provide an answer to the research question. Second, an engaging questionnaire is more likely to keep participants focused, ensuring they complete all questions, which subsequently results in better-quality data.

As an example, assume that our client was interested in a sensitive topic, like contraceptive use and preferences, or personal saving habits. A poorly designed questionnaire is one that begins with these personal questions; participants might be put off by being asked such personal questions right away, and therefore might decide to terminate the survey. It is important to remember that our participants are human; they have insights, experiences, and opinions that we are interested in, and that is why we want their input. However, participants can also get bored, have bad days, make mistakes, or feel rushed. Knowing this, we must consider how the research experience can affect our participants’ behaviour. If surveys are too long, too boring, or too demanding, then participants will no longer provide good-quality data. Therefore, it becomes increasingly important to optimize the participants’ experience and show that we value their time and effort.

There are many things that we can do as researchers to ensure the quality of the data that the participant provides. Everything from deciding on the right question type (for example, rating scales where you can to capture quantitative variability, and open questions for richer insights), preventing boredom by varying the types of questions asked; wording the question clearly so that the participant understands what is asked, avoiding jargon, formal language or internet speak; ordering and numbering the answer options properly; and even the overall flow of the questions in the questionnaire can influence participants.

We must keep in mind that we are increasingly competing for respondents’ time and attention: they are rethinking how and what they spend their time on. A third of respondents have indicated that they are not willing to spend more than 10 minutes completing surveys, and participants are increasingly demanding improved online experiences. Bad survey experiences can affect participant behaviour to such an extent that they can become less engaged (i.e., provide poor-quality information) and ultimately stop participating in surveys altogether. This in turn can also negatively affect the overall data quality, and impact feasibility and recruitment planning for future projects.

So, with all of this in mind, we try to provide a more enjoyable survey experience, by using a mix of different question and answer types, engagement techniques, and an easy-to-follow flow. One example of a tool that makes surveys more fun is the swiping tool. Similar to Tinder, participants are shown an idea or concept; if they like it, they swipe right, if they don’t, they swipe left. This gamified experience keeps participants engaged.”

“Besides improving the research experience, we have conducted extensive research on innovative solutions to use in our surveys. For example, we use digital fingerprinting to distinguish fraudulent participants from valid participants, as well as to ensure unique participants. Other methods like quality and attention measures also help us detect poor-quality responding. Our in-house software also allows us to track data quality in real time and identify participants whose answers can negatively affect clients’ data. When data collection is complete, the dataset is carefully inspected and evaluated to ensure there are no records causing any anomalies. This is the final step in ensuring data quality.

The key to good-quality data thus is connecting with the right people through the right design, to accurately answer the research objectives.”