Two Truths and a Lie

You’ve probably played “Two Truths And A Lie” – the game where you tell people three things about yourself, and everyone has to guess which one is the lie. It’s a fun game – if you’re not betting your business on getting the answer right.

But that’s the problem. Most businesses doing market research are playing Two Truths And A Lie – and they don’t even know it. 

Research is only valuable when respondents tell us two truths. The first, and arguably most important truth is what they think. We ask them questions about our product and service, and we need honest answers. The second truth is who they are. We need a proper segmentation of respondents based on demographics, location, job, income, preferences, and anything else marketers look to target. While these truths are foundational, the industry has built a model of research which incentivises participants to be dishonest and by not telling the end-client this is happening, the lie begins.

HOW THE INDUSTRY HAS ALLOWED THIS TO HAPPEN

In Two Truths And A Lie, players know you have to tell a convincing lie to win. In market research, participants know exactly the same thing – to win, to get those ‘breakage model’ incentives or to get past an online survey to get what you actually want, you might have to lie. Likewise, many panel providers are engaging in a lie when they do not provide radical transparency about the system they’ve created and continue to support, and most likely, rely on for profitability.  

What creates dishonesty from participants? Bad incentives, which make respondents feel being dishonest pays. And bad experiences, which make them feel being honest doesn’t pay. Let’s look deeper at the two truths in market research – and the lie being peddled to those who are paying for the research. 

THE FIRST TRUTH

The first truth is that respondents may not be telling the truth. In fact, the companies who pay for market research have no right to expect honest answers if the respondents have not been fairly compensated. This reality that survey takers may not be providing honest feedback might be news to some, but why should they be honest when they’re being misled and often cheated? With impossible hoops to jump through to get the promised payout, many give up as, in effect, they’re being asked to work for free. Few do a great job when they know their pay is going to be withheld. 

Not only does the lack of fair compensation affect the outcome, but in this race to get the right number of completes on a survey, little or no money is invested in the experience of the consumer who participates either. Both of these realities drive consumers to simply not care enough to answer surveys honestly. Take polling. In the 2016 election, the polls missed badly because they underestimated how many people would vote for Donald Trump. Lots of Trump voters simply didn’t answer truthfully when called during dinner or as they walked out of the polls. The polling firms held post-mortems, worked out the error, and made adjustments for it. And in 2020… exactly the same thing happened! Trump voters weren’t truthful, and in the battleground states the election was far closer than polls predicted.

Your consumer survey probably doesn’t feel much like a swing state poll, but the consequences of dishonesty are the same. When your data comes from people who are not telling the truth, the veracity of the conclusions are at risk.

Outside of polling, a person who signed up to take surveys is doing them to get rewards or simply to get past the pop up on their screen – that’s the deal we make with participants. On most panels that means they have to rack up enough points or completed surveys to reach a certain reward level in order to reach payout. This ’breakage model’  incentive to “level up” creates an urgency to rush through their answers as quickly as they can, not really giving thought to them. THAT is a dishonest or at least, inaccurate answer.

Also, too often, that bad experience leaves truthful respondents feeling they haven’t been fairly compensated. Maybe surveys that claim to be 5 minutes last 20. Maybe they spend 10 minutes on a screener and it gives them a “thanks but no thanks”. Maybe it takes them too many surveys to get any pay or, what is more common, they never receive anything because they never reach the payout threshold put in place to maintain the breakage model. There are plenty of ways surveys create negative experiences and ultimately devalue the most valuable commodity in market research – the time and voice of the consumer. When treated unfairly, most quit outright. But plenty decide that two can play that game, and begin to tailor their answers to get through surveys faster. They begin to leave no comments on open questions and the thoughtfulness of the answers relax. When respondents lose interest in providing the truth, noisier, less honest and inaccurate data is the result. 

These panel companies are not being transparent with the end-client who is relying on the veracity and authenticity of these responses. Those paying for the research are often unaware that many of the completes they receive represent survey takers who were never compensated for their time or opinion because they could not jump through the many hoops for final payout. With this breakage model in place, the client may not even know that the panel providers they are using are pocketing the difference between what they pay for completes and what the panel company never pays out to respondents based on the payout schemes in place. 

The initial problem here is that clients, whether a market research firm or any end-client using a panel provider to help their in-house research team, are paying for these completes, but respondents are not being paid. This known margin is the survey world’s dirty little secret. To make matters worse, the long-term problem is that when people don’t feel fairly compensated and well-treated, the contract with them breaks down. They stop trusting us. They stop feeling the obligation to be honest or to provide thoughtful responses, so we get bad data now and poison the well of future research efforts.

THE SECOND TRUTH

The second truth is that not all respondents are who they claim to be or who you think they are. How do you prove that someone who claims to be an 18-year old gamer dude is actually a 45-year old housewife? 

Many surveys are structured conveniently for respondents to game the system. Placing lists of surveys for respondents to choose from online is a set up inviting professional survey takers and bots. What dishonest participants have learned is if they can get away with lying about who they are, they have a better chance of getting through screening and getting rewards or other compensation. What off-shore coders have learned is, if they can find these lists of options, they can create a bot to first test the screener and when it is figured out, they can “take” the survey many times over and far exceed the payment threshold and get the payout. 

Both of these have been known problems and many sample suppliers have tried to implement verified data on its panelists. Working with the large public data companies to certify data on income or occupation is offered as an industry solution. While this is touted as a viable option for verification, the truth about how inaccurate these databases can be are not shared with the end-client. 

WHERE THESE TWO TRUTHS COLLIDE

An interesting blend of the first and second truth emerges when these professional survey takers and bots arrive on the scene. Any system that posts lists of surveys available will always have this problem. These systems are created not to uncover the truth through thoughtful completes, nor are they set up to correctly represent the demographic audience. These are built solely to maximize completes as quickly as possible. This data is folded into the data set and can seriously skew outcomes. 

When looking at these two truths, you can see that a truthful answer from a known person (even if they are not exactly in the right demographically targeted audience) is still more helpful than a straight out untruthful answer. Dishonesty around the first truth – what people say – is much worse than dishonesty around the second truth – who they are. You can’t prove it happens, and your clients might only suspect it when the decisions they base on the research turn out to be wrong. Dishonest respondents reduce the value of everyone’s answers. This can only be solved by treating consumers fairly and compensating them transparently and immediately. This means instant payouts – no games, unkept promises or stupid sweepstakes entries. 

When these two truths collide, it literally allows business decisions to be made based on false data and could result in tens of millions of dollars of loss.

THE LIE

Panel suppliers are quick to emphasize efforts to improve quality. They downplay the realities of the two truths. Even more insidiously, they don’t share how they are married to a system that compensates respondents poorly or not at all. This is a lie to end-clients about how they obtain their data and remain profitable. Perhaps, even, they cannot survive financially, without this dishonesty. That’s why companies like 1Q, with a radically different business model and leading-edge technology can offer such a compelling alternative, and at a lower price!

JOIN THE REVOLUTION

At 1Q, we own our proprietary panel of over one million screened respondents. We pay fairly and immediately, and by doing so have brokered a relationship of trust within our community. Our panel members are incentivized to be honest. They never reach survey fatigue because we don’t ask one respondent too many questions. Because of our unique payout system we know who these respondents are. And because of this connection, we offer immediate or later recontact with a singular person or groups of respondents. This is an efficient way clients can iterate on research or probe deeper to get real answers for pressing business decisions while at the same time verifying that respondents are who they say they are. We continue to build trust with this industry’s most valued resource – the consumer. They respond in kind. 

And because of our unique business model, which doesn’t list surveys available to be taken, we are certified 100% bot-free and have no professional survey takers!

Two Truths And A Lie is more than a game for research and the businesses that use it. The two truths – that respondents are often incentivized to be dishonest and may in fact not even be who they say they are strikes at the core of the integrity of any survey. The lie – that market research companies are benefiting from this broken system, is still the biggest challenge our industry faces.

Respondents have joined 1Q because they want to be treated fairly. Join the ranks of brands looking for the same. When you need quality data to make informed decisions for real business impact, look to the power of 1Q.