Study of AI as a creative writing helper finds that it works, but there’s a catch

A finger points to the ChatGPT app on a phone screen.

Researchers are exploring the existential implications of generative AI, including whether or not the advancing technology will actually make humans more creatively capable — or narrow our views.

The new study, published in Science Advances by two University College London and University of Exeter researchers, tested hundreds of short stories created solely by humans against those created with the creative help of ChatGPT’s generative AI. One group of writers had access solely to their own ideas, a second group could ask ChatGPT for one story idea, and a third could work with a set of five ChatGPT made prompts. The stories were then rated on “novelty, usefulness (i.e. likelihood of publishing), and emotional enjoyment,” reported TechCrunch.

“These results point to an increase in individual creativity at the risk of losing collective novelty,” the study reads. “This dynamic resembles a social dilemma: With generative AI, writers are individually better off, but collectively a narrower scope of novel content is produced.”

Participants were “measured” for creativity prior to the writing session with a commonly used word-production task that builds a standard of creativity among respondents. Those who tested lower on these creativity proxy tests received better scores on their personal writing when given access to AI-generated ideas. But for those with already high creativity scores, AI ideas had little to no benefit on their story ratings.

Additionally, the pool of stories aided by AI-generated prompts were deemed to be less diverse and displayed less unique writing characteristics, suggesting the limits of ChatGPT’s all-around ingenuity. The new study’s literary findings add to concerns about AI’s self-consuming training loops, or the problem of AI models trained only on AI outputs degrading AI models themselves, Mashable’s Cecily Mauran reported.

Study author Oliver Hauser said in a comment to TechCrunch: “Our study represents an early view on a very big question on how large language models and generative AI more generally will affect human activities, including creativity… It will be important that AI is actually being evaluated rigorously — rather than just implemented widely, under the assumption that it will have positive outcomes.”