Indiana University School of Medicine's Melissa Kacena wanted to see who is better at writing review articles: people or ChatGPT.

For her study in Current Osteoporosis Reports, she split nine students and the AI program into three groups and asked each group to write a review article on a different topic.

For one group, she told the students to write review articles on the topics; for another, she instructed ChatGPT to write articles on the same topics; and for the last group, she gave each of the students their own ChatGPT account and told them to team up with the AI program to write articles.

That let her compare articles written by people, by AI, and by a mix of people and AI. She asked her colleagues and the students to mark the facts in each article and compare the three types of articles on accuracy, readability, and language.

The results were shocking. The articles written by ChatGPT were easy to read and were even better written than the students'. But up to 70 per cent of the references were wrong: they were either mashed up from different studies or made up. The AI versions were also more likely to be plagiarised.

"ChatGPT was pretty convincing with some of the phoney statements it made, to be honest. It used the proper syntax and integrated them with proper statements in a paragraph, so sometimes there were no warning bells. It was only because the faculty members understood the data well, or because the students fact-checked everything, that they were detected."

ChatGPT used better grammar than the students. But it couldn't always read the room: AI tended to employ more flowery language that wasn't always appropriate for scientific journals (unless the students had told ChatGPT to write it from the perspective of a graduate-level science student.)

So the technology is still garbage in, garbage out and is only as good as the information it receives.

While ChatGPT isn't quite ready to author scientific journal articles, she said that with the proper programming and training, it could improve and become a useful tool for researchers.

"Right now, it's not great by itself, but it can be made to work," says Kacena.

For example, if queried, the algorithm was suitable at recommending ways to summarise data in figures and graphical depictions.

"The advice it gave on those was spot on, and exactly what I would have done," she says.

The more feedback the students provided on ChatGPT's work, the better it learned—and that represents its most tremendous promise. In the study, some students found that when they worked with ChatGPT to write the article, the software continued to improve and provide better results if they told it what things it was doing right and what was less helpful.

That means that addressing problems like questionable references and plagiarism could potentially be fixed.

ChatGPT could be programmed, for example, not to merge references to treat each scientific journal article as its separate reference and to limit copying consecutive words to avoid plagiarism.

With more input and some fixes, Kacena believes that AI could help researchers smooth the writing process and gain scientific insights.

"I think ChatGPT is here to stay, and figuring out how to make it better, and how to use it in an ethical and conscientious and scientifically sound manner, is going to be important," Kacena said.