New tools help detect fraudulent images in scientific papers

Research sleuths are turning to new tools to tackle the growing problem of artificial intelligence-generated or forged images in scientific publications, which has emerged as a major form of academic misconduct in recent years.

But experts have warned that while detectors can help, they are struggling to keep up with the scale and complexity of the problem.

Elisabeth Bik, a microbiologist and science integrity consultant, told Times Higher Education that she is “very worried” about what generative AI could mean for the future of scientific research because of its capacity to produce datasets and photos “that look absolutely convincing”.

Bik previously identified image duplication by eye, but now she uses AI-enabled software tools, ImageTwin and Proofig, for observing pattern recognition.

Both tools scan PDFs or images from scientific papers and compare them against a database of millions of other published photos to detect if images have been reused or if there are duplicating elements.

“They are not perfect, we image sleuths still scan papers also by eye,” said Bik, “but they are very helpful in the large-throughput scanning of papers”.

Bik stressed that image fraud is particularly problematic in scientific research because a photo in a scientific paper is “not just an illustration”.

“The photos in a scientific paper are data. They are proof that the experiment happened.” 

Both ImageTwin and Proofig help to determine whether an image has been Photoshopped or reused, but Bik said that they “have trouble” detecting AI images. 

“To prove that an image is AI generated is really hard,” she said. This is especially true by the time images are published because they have been heavily compressed, resulting in a loss of the “pixel-level detail that the camera once had”.

“They might flag low-contrast images from older papers that are likely false-positives, and if you feed it an AI-generated image, they don’t always detect it, or with low confidence. These tools might do better in the future but are not currently very reliable.”

Given that real photographs can include AI-generated elements makes the task of identifying misconduct even more challenging, said Jinjin Gu, a tenure-track faculty member at the Institute for Computer Science, Artificial Intelligence and Technology (INSAIT).

“A photograph might be genuinely captured by a camera but then denoised, upscaled, relit, or partially inpainted with AI. Perhaps only one small object or section of the background was generated. In such cases, asking whether the entire image is AI-generated becomes an ill-defined binary question,” he explained.

The questions that Gu felt academics should ask instead are around “positive authentication and provenance”, with an emphasis on verifying the origins of an image.

“In academic publishing, the most reliable investigation would usually go beyond uploading the final JPEG to an online detector,” said Gu. “A forensic detector may help identify cases that deserve closer examination, but it should not be the sole basis for accusing someone of misconduct.”

Haihong E, professor at the School of Computer Science at Beijing University of Posts and Telecommunications (BUPT), said that many practices related to image manipulation are “perceived by researchers as minor modifications”, which is contributing to “their high occurrence” and making “effective supervision more challenging”.

Haihong is leading a team of researchers in China who are developing AI tools to detect manipulated images in research publications and she told Times Higher Education that these tools “have already [been] adopted by a growing number of universities, academic publishers, and research institutions in China”.

The detection model developed by Haihong and her team analyses academic image manipulation cases in international and domestic datasets to strengthen its capability of identifying different types of academic image manipulation. The team “continuously collect emerging AI-generated fraud cases” to update the models regularly, Haihong explained. As part of the evaluation, Haihong’s team compares machine-generated detection results with human reviews. 

While detection tools have undoubtedly helped many members of the scientific community to identify instances of image fraud, drawing attention to details invisible to the human eye, INSAIT’s Gu cautioned that they are not “a universal truth machine”.

He also raised the distinction that “detecting AI involvement is not the same as detecting fraud”.

“A disclosed AI-generated illustration may be entirely legitimate, while a conventionally edited photograph or scientific figure may be seriously misleading. A detector cannot establish authorship, intention, scientific validity, or whether a particular use of AI violated a policy.”

rosalind.skillen@timeshighereducation.com