the artificiality of AI
on scam science and bad research
One of my favourite stories to come out of academia in the last year was about debunking bad research practices.
Dr. Saul Justin Newman’s study1 on longevity and demography won an Ig Nobel Prize2 for the way it interrogates some of the mythos around supercentenarians—people over 100 years old—and the concept of “Blue Zones”, areas in the world where a combination of social, dietary, and genetic factors allowed people to lead extraordinarily long lives.
According to Newman’s research, the claims made about Blue Zones have been greatly exaggerated, and the fantasy of communities where people live long, healthy lives and enjoy a unique, longevity promoting culture and lifestyle is just what it sounds like: a fantasy.
In reality, Blue Zone areas are largely impoverished and remote, with high crime rates, low incomes, and poor health. How, then, could there be so many cases of remarkable human age in these places? Put simply, bad record keeping, errors, and outright fraud are to blame.
In his research, Newman notes that supercentenarian birthdates are clustered around the first day of the month and dates divisible by 5, indicating that they are largely falsified or incorrect, and only 18% have actual birth certificates! The data available, and which has been used in studies like those on Blue Zones, is rife with inconsistencies and errors. As Newman joked in his Ig Nobel acceptance speech, “I’d like to thank the world’s oldest man for having three birthdays”.
Among other factors and correlations, the lack of adequate demographic records is one clear way that supercentenarian data has been falsified. The research suggests that pension, benefit, and identity fraud may also account for many cases, and that unreported deaths are likely for many cases of remarkable old age. A huge amount of error on the numerous levels identified by Newman demonstrates clearly the problems that underly the claims of supercentenarians and the Blue Zone theory.
The concept of Blue Zones was invented in 2005 by Dan Buettner, who is also the CEO of Blue Zones LLC, which sells merchandise, books, and courses promoting the “blue zone” lifestyle. In addition to the many questions of methodologies and the validity of data being used in his work, the capitalization of these findings by Buettner is a huge red flag.
It turns out that living by the coast or eating sweet potatoes every day isn’t the secret to eternal life - but maybe lying and human error is. And this problem isn’t just limited to the demographic failures of remarkable old age and Blue Zone studies, but is everywhere in science and research.
As a postgraduate researcher, I’m required to complete training on research ethics and practices. We talk a lot about data collection, methodologies, the ethics of working with humans and personal data, the replication crisis, fraudulent hypothesising, designing research around predetermined results, cherry picking data, bias, funding ethics, conflicts of interest. Many of the of issues they highlight to us as new scholars are the exact problems that we see in the Blue Zone studies, and in numerous other prominent examples.
Anti-vaccine conspiracies in the 2000s and into the present day were fuelled by the infamously debunked research of Andrew Wakefield3 which included fraudulent and misleading data and was driven by a key conflict of interest, as Wakefield stood to profit from further research and the sale of rival vaccines and diagnostic testing kits, predicated on his findings.
Cornell University psychology Professor Daryl Bem published widely publicised studies that provided evidence that ESP was real, a story which has become a famous case study for the replicability crisis and other major concerns about the validity of research methodologies in the field.
A 2005 study entitled “Why Most Published Research Findings Are False” was published by Professor John Ioannidis arguing that published results often rely on poor methodologies, irreplicable results, and are rife with false positives. This study helped to spark a larger and still ongoing conversation about some of the problems we see across science publishing.
These issue continue to crop up all the time in many other areas that are familiar to us through the misleading representations of science that we see in headlines today, and especially in areas of research that are particularly prone to click bait-y, pop science repackaging, like fad diets and superfoods, psychology, ‘wellness’ advice, and, particularly of recent interest, a lot of artificial intelligence research.
AI is an area that is absolutely full of bad science at the moment. Most of the stories we read right now about developments in AI are wildly exaggerated, promoting scams, misrepresenting data, or some combination of these, and are full of myriad other research problems. A few issues to look out for:
Outsized claims
News about AI is full of big promises with limited evidence.
The realities of technology often fail to meet high expectations.
Recent headline: How Good Is ChatGPT at Coding, Really?
See also: companies like Spotify have pivoted to AI and promote “AI created” playlists and features, which produce the same(or worse versions of) playlists that were previously labelled as being created by ‘the algorithm’.
AI’s reproducibility crisis
AI outputs tend to be unique - you won’t get the same image from the same prompt in Midjourney, or have the same conversation with ChatGPT even with the same inputs.
The processes and models are opaque - we don’t get to know exactly how they work!
The vast majority of AI research published by computer scientists cannot be reproduced.
Computers are not people
Much of the language around AI serves to personify it.
AI cannot “understand” or “misunderstand” prompts.
AI does not have human motivations, but using emotional language to describe it reinforces the complexity of the technology and the fantasy of computer programmes being “artificial intelligence”
A recent headline: “‘Scheming’ ChatGPT tried to stop itself from being shut down”
AI is a business
Claims are being made (in large part) to make money!
A recent headline: “Nvidia announces $3,000 personal AI supercomputer called Digits”
Nvidia’s CEO is quoted in the article saying that “placing an AI supercomputer on the desks of every data scientist, AI researcher and student empowers them to engage and shape the age of AI.”
The claim being made is that incredible, science fiction level technology is now being made available to the public! You can have your own SUPERCOMPUTER! The reality is that they want you to spend three thousand dollars on a computer with limited functionality and unknown reliability.
Wondering why every social media site, app, company, etc. is adopting AI right now? They are trying to make a profit off of the current technology boom.
From “Two Computer Scientists Debunk A.I. Hype with Arvind Narayanan and Sayash Kapoor” episode of the podcast Factually! with Adam Conover:
“A lot of what the AI evangelizers say is just marketing — it’s easy to believe in the coming singularity apocalypse when believing in it might be your path to making billions of dollars.”
The fantasy of AI is easy to buy into, but much of what we see in popular discourse is little more than hype. Rather than objective portrayals of any of the real developments in the capabilities of computing and the ways in which statistics, algorithms, data, and models can be used, the story we are being told right now is AI generated slop, written by scammers to serve the interests of profit. And much of what we are told about artificial intelligence is little more than science fiction.
Questions you should ask when reading popular science headlines and articles:
Does this article present science as if it is magic?
Does this story sell a product or otherwise serve the interests of a business or industry?
Is this written in a way that provokes an emotional response? How does the author want me to feel?
What actual science/data/statistics are being presented to me? In what ways are they represented, and is it clear what they mean?
Could this study/research/project be reproduced? How does it match up to previous research on this topic?
Is the original study accessible? What does it actually say?
thanks for reading!
stay valid and reproducible,
isobel
His most recent paper(and the one for which he has been awarded) is available to read for free. It is currently in pre-print, and has not yet been peer reviewed. This is a complicated issue, but I want to draw your attention to this since we’re talking about reading research critically! Take these findings with a grain of salt!
The Ig Nobel Prize celebrates scientific research that makes you laugh, then makes you think. I was introduced to this very fun awards ceremony by the podcast “Let’s Learn Everything”, which did an episode all about the prizes, as well as featuring coverage of the 2024 ceremony and some interviews with Newman and other winners(with some really insightful interview questions like: “What advice would you give to a young scientist or an old acrobat?” “Don’t die”) in a later episode.
The first line of his wikipedia article describes him as “a British fraudster, discredited academic, anti-vaccine activist, and former physician.” lol