Study proves ChatGPT knows us better than we know ourselves

If you've ever felt like ChatGPT knows exactly what you mean before you've finished typing, you're not alone.

Now, new research suggests there may be a scientific reason why.

A study published this week in iScience found that ChatGPT could generate entirely new personality tests and, more remarkably, predict how groups of people would respond before a single participant completed the survey.

Not your answers specifically.

But ours.

Collectively.

It's a finding that hints at something bigger than another clever AI trick. 

It raises an uncomfortable question about large language models: after consuming trillions of words written by humans across the internet, have they become startlingly good at modelling human behaviour itself?

Spoiler alert: Alarmingly yes.

More than a chatbot

Most of us think of ChatGPT as a machine that predicts the next word, perhaps using it as a quickfire way to reply to an email or build a presentation deck, but increasingly, researchers are wondering whether it's also predicting the next thought.

The team from The Hebrew University of Jerusalem asked GPT-4 to create personality questionnaires from two wildly different sources: the Diagnostic and Statistical Manual of Mental Disorders (DSM-5), the clinical handbook used by psychologists, and, in a deliberately odd twist, an astrology textbook.

The goal wasn't to prove astrology works; actually, it set out to prove quite the opposite.

The researchers wanted to see whether a large language model could extract meaningful personality information from almost any text.

"We wanted to choose texts that describe human personality in very rich detail but also sit on opposite ends of a spectrum in terms of scientific grounding," says lead author Rotem Monsa. 

"The DSM-5 was refined through decades of clinical research and is the standard diagnostic manual in clinical psychiatry, known worldwide. The astrology text is very culturally based but not scientifically validated."

It wasn't just writing the questions

Now, here's where it gets very Black Mirror-esque.

After generating the questionnaires, researchers asked 600 people to complete them alongside the widely accepted Big Five Inventory.

The DSM-5 version performed much as psychologists would expect.

The astrology version was less scientifically coherent, with personality traits failing to group together in meaningful ways.

"Our data suggest that the astrological elements don't reflect coherent psychological dimensions. Personality traits that were together in, for example, the fire elements don't actually go together in the real population," Monsa says.

Yet even the astrology-based questionnaire could predict outcomes such as depression, anxiety and wellbeing at levels comparable to established personality measures.

Then came the surprise.

Before any participant responses were analysed, ChatGPT had already estimated what the average answers and relationships between questions would look like.

The result? It was remarkably, spookily close.

Has AI learned human psychology?

The researchers believe the answer may lie in language itself.

Large language models aren't explicitly taught psychology. Rather, they're trained on an enormous archive of human conversation, books, websites and social media posts.

If personality leaves fingerprints in language, perhaps AI has been learning psychology without anyone explicitly teaching it.

"Given that personality traits are reflected in language, LLMs may have learned the structure of human personality as a natural byproduct of their training," says Monsa. 

"So, while they were not taught specifically psychology or personality theories, these are already embedded in the language that LLMs learn from."

It's an idea that shifts how we think about AI.

Rather than simply being a sophisticated autocomplete, large language models may be statistical maps of humanity, recognising patterns in how billions of people think, write and behave.