Researchers from the School of Psychological Science published a new research methods paper in Ethics & Behaviors offering strategies to prevent fraudulent responses to studies and interventions
Jessica Dietch, Kathleen Bogart, and Jillian Coleman
By Colin Bowyer, CLA Communications Manager - October 23, 2024
Psychologists Jessica Dietch and Kathleen Bogart were hoping for a few dozen participants for their 2022 study on the sleep habits of adults and children with Moebius syndrome, a rare congenital condition. After just three days, they held thousands of responses to an online survey.
“First, we publicized the survey to an exclusive audience,” said Dietch, assistant professor of psychology. “Second, Moebius syndrome is a very rare condition so seeing thousands of people respond to our survey was jarring, to say the least.”
Dietch and Bogart, professor of psychology, halted the survey and started looking at the individual response data. Most answers to the qualitative, open-ended questions were plausible, though oddly worded, but what confirmed Dietch and Bogart’s suspicions were hundreds of the exact same answers. It became clear that there was fraudulent data, including submissions from automated bots.
Evidence on the rapid growth in fraudulent activity in internet research is increasing. Fraudulent participation in online research introduces a litany of concerns and risks to the research process, including the compromise of data quality, the misallocation of resources (e.g., incentives, researcher time), and the potential distortion of findings in studies addressing individual mental health and well-being. These concerns can contribute to health inequity by jeopardizing interventions, leading to the development of ineffective policies that fail to address the unique challenges faced by marginalized groups, which, in turn, can exacerbate health disparities by diverting attention and resources away from those who need them most.
“As a researcher focused on disability and rare disorders, I have long relied on online communities and social media to reach participants,” said Bogart. “But in the past few years, there has been an alarming rise in fake responses and bots infiltrating our research. This is a huge problem, not just for data integrity, but for equity. These fraudsters are muddying the data and preventing us from truly understanding the most marginalized communities. Many researchers probably have fake respondents and don't even know it. Others may feel ashamed to admit it. It’s important to raise awareness about this issue.”
Hoping to address these challenges, Bogart and Dietch, in partnership with Jillian Coleman, research coordinator at the School of Psychological Science, and others, collaborated to write a new research methods paper.
Published in Ethics & Behavior in October 2024, the authors present four case studies drawn from body image and appearance research, highlighting the obstacles confronted in dealing with fraudulent responses, and offering potential strategies. A few of the paper’s 12 recommendations include using verified research panels instead of broad internet samples, instituting more stringent identity verification methods, utilizing anti-bot tools, e.g. capturing IP addresses, and closely monitoring responses as they’re being collected. The authors' outcomes were not necessarily to present findings, but raise awareness and inform researchers of the possibility of fraudulent data in responses to internet surveys.
“We wanted to write a paper to warn people that this is becoming increasingly common,” said Dietch. “It’s a huge undertaking to halt a study and weed out fraudulent responses, particularly in mental health studies, where fraudulent responses jeopardize interventions and care efforts. We’re hoping this research can help all researchers prevent fraudulent responders from invading their data collection.”