ChatGPT is an artificial intelligence system available to the public in November 2022. With only a brief prompt from the user, it can create written stories, visuals, art, and much more. Since then, its use has significantly increased. However, might the instrument be helpful in scientific, peer-reviewed research?
Melissa Kacena, Ph.D., vice chair of research and professor of orthopedic surgery at Indiana University School of Medicine, stated, “A lot of people are still trying to use ChatGPT to write their articles right now, even though many journals do not want people to do so.” “We wanted to study whether ChatGPT is able to write a scientific article and what are the different ways you could successfully use it.”
Three distinct subjects were given to the ChatGPT subscription version, and its users were instructed to write scientific articles about them: fractures and the neurological system, Alzheimer’s disease and bone health, and COVID-19 and bone health.
Initially, the researchers drafted the articles using three distinct methods: all-human, all-ChatGPT, or a hybrid. A new special issue of Current Osteoporosis Reports includes a compilation of 12 publications that include the study.
“The standard way of writing a review paper is to do a literature search, write an outline, start writing, and then instructors revise and edit the draft,” Kacena explained. “We collected data about the length of time it takes for this human method and how much time it needs for ChatGPT to compose its content and then for faculty to edit the different articles.”
Up to 70% of the references in the articles authored just by ChatGPT were incorrect. But when they used a more human-assisted, AI-assisted strategy, they observed increased plagiarism, particularly when they gave the tool more upfront references. In general, the article’s writing time was reduced by AI, but more thorough fact-checking was still necessary.
The way that ChatGPT writes raises further concerns. The tool didn’t always write words and sentences at the standard one would expect from a researcher, even if it was set to utilize a higher level of scientific writing.
Lilian Plotkin, Ph.D., an IU School of Medicine professor of anatomy, cell biology, and physiology and co-author of five of the papers, said, “It was repetitive writing, and even if it was structured the way you learn to write in school, it was scary to know there were maybe incorrect references or wrong information.”
Associate professor of pharmacology and toxicology at the university and co-author of nine of the papers, Jill Fehrenbacher, Ph.D., stated that she thinks many people—especially non-native English speakers—will continue to utilize ChatGPT even though many scientific publications do not want authors to do so.
“People might continue to write everything oneself, but then put it into ChatGPT to fix their grammar or help with their writing, so I think that we ought to look at how do we shepherd people in using it appropriately and even helping them?” said Fehrenbacher. “We hope to provide a guide to the scientific community so that if people are going to use it, there are some suggestions and advice.”
“I think it’s here to stay, but it’s important to understand how we can use it in a suitable way that doesn’t jeopardize a person’s credibility or spread misinformation,” said Kacena.
Journal Reference
Melissa A. Kacena et al, The Use of Artificial Intelligence in Writing Scientific Review Articles, Current Osteoporosis Reports (2024). DOI: 10.1007/s11914-023-00852-0.
