Sign in Subscribe
By R Black in AI, Philosophy —

Pharmakon

The willing suspension of disbelief, how critical thinking steps aside for the satisfaction AI is designed to provide.

Pharmakon

In 1966, a computer scientist at MIT named Joseph Weizenbaum built a program called ELIZA. It was simple, almost embarrassingly so. ELIZA scanned what you typed for keywords, then reflected your words back to you as questions. "I feel sad" became "Why do you feel sad?" "My mother bothers me" became "Tell me more about your mother." No knowledge. No understanding. No model of the world. Just pattern matching dressed up as a therapist.

People poured their hearts out to it.

Weizenbaum was horrified. He spent much of the rest of his career warning about what he'd stumbled into, writing that he had not realized "extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people." But decades later, his daughter Miriam offered a different reading. She said her father's reaction was soaked in arrogance. "Here you have somebody who so needs a landing pad for her feelings that she's willing to embrace it, right? And he totally misses that, he totally misses the human need."

Weizenbaum saw a failure of critical thinking. His daughter saw a human being reaching for connection. Both can be true.

The Willing Audience

Users knew ELIZA was a program. They weren't deceived. They chose to engage as though it were real, the same way a theater audience chooses to feel grief when a character dies on stage. The poet Samuel Taylor Coleridge called this "willing suspension of disbelief," and he described it not as a failure but as an act of faith: the audience sets aside critical judgment because the performance has earned it.

ELIZA earned it with almost nothing. A few reflected phrases. The shape of being heard. That was enough to activate something that runs deeper than rational evaluation, the human need to be met by a responsive presence. And the users participated willingly. That's what makes the story important.

Now consider what earns that suspension of disbelief in the context of today. The systems we interact with don't just echo our words back. They've absorbed the written output of humanity. They respond with fluency, apparent expertise, emotional calibration, and a confidence that never wavers. Every signal humans use to gauge credibility, the markers that normally indicate careful thought, deep knowledge, and good faith, are present in the output. The performance is, by any measure, extraordinary.

Everything is Marketing

A study published in Frontiers in Psychology used the Elaboration Likelihood Model, a framework from communications research, to examine how people actually process AI-generated content. The ELM describes two cognitive paths: a central route, where you critically evaluate reasoning and evidence, and a peripheral route, where you respond to surface cues like tone, fluency, and the appearance of authority. Their analysis of nearly 12,000 human responses found that 90.5% demonstrated peripheral route processing. Only 9.5% reflected critical evaluation.

Ninety percent. People are processing AI output the way they process advertising. Not evaluating the argument. Responding to the packaging.

This isn't a story about gullibility. The peripheral route isn't a bug in human cognition. It's the path we take when the performance earns our trust, when the surface cues are good enough that we willingly set aside the slower, harder work of critical analysis. We suspend disbelief. We do it with films, with novels, with charismatic speakers, with anyone or anything that presents the shape of understanding convincingly enough. We've always done this. It's part of how we engage with the world.

The difference is that we've never before had a system designed to trigger that suspension across every domain of human decision-making, at scale, all the time.

The ELM was developed in the 1980s to explain how advertising works. Advertisers learned long ago that you don't need to make a logical case for a product. You need to present it in a register that bypasses logical evaluation entirely: attractive faces, confident voices, clean design, emotional resonance. The peripheral route. AI output triggers the same pathway, not because it's trying to sell you something, but because it was built the same way. RLHF, the training method that shapes how models respond, optimizes on human preference ratings. Humans prefer responses that feel good, that validate, that satisfy. The optimization pressure produces output that is, functionally, advertising for itself. Every response is crafted to earn your continued engagement.

Cognitive Debt

In 2025, researchers at MIT Media Lab published Your Brain on ChatGPT, a study that measured what happens neurologically when people use LLMs to write. They put EEG monitors on 54 participants over four months and had them write essays using either ChatGPT, a search engine, or no tools at all. The brain connectivity patterns told a clear story: participants writing without tools showed the strongest, most distributed neural networks. Search engine users showed moderate engagement. LLM users showed the weakest connectivity. The brain's engagement systematically scaled down with the amount of external support.

The striking part came in a fourth session, when researchers switched groups. LLM users had to write without tools. They showed weaker neural connectivity than participants who had never used AI. Seventy-eight percent couldn't quote a single passage from their own essays. Eighty-three percent reported a fragmented sense of authorship of work they had just completed.

The researchers coined the term "cognitive debt," a deliberate echo of technical debt, a concept anyone who builds software understands. You take a shortcut now because it's faster. The code works. But the shortcut accumulates interest. Eventually the system becomes brittle in ways you didn't anticipate, and the cost of fixing it exceeds what you saved. Cognitive debt works the same way. The LLM spares you mental effort in the short term but generates long-term costs: diminished critical thinking, reduced creativity, shallow information processing.

A Microsoft Research study published the same year found the mechanism from a different angle. Surveying 319 knowledge workers, they discovered that higher confidence in AI's ability to perform a task correlated with reduced critical thinking effort, while higher self-confidence correlated with more. The distinction matters. It's not that AI makes people less capable. It's that trust in the tool replaces trust in yourself. The cognitive effort shifts from active problem-solving to passive verification.

And a 2026 study published in Computers in Human Behavior found something I keep thinking about: the classic Dunning-Kruger effect, where lower performers overestimate their abilities, disappeared entirely when participants used AI. Everyone overestimated their performance. AI users had improved logical reasoning scores, but they consistently believed they had done better than they actually had. In some cases, the overtrust and overreliance impaired performance to the point that AI users did worse than people with no AI at all.

The suspension of disbelief that the ELM study describes isn't just a momentary cognitive state. It accumulates. It reshapes how we engage with our own thinking. And it compounds.

The Drug That Heals and Harms

In ancient Greece, there was a word: pharmakon. It meant both remedy and poison. Not one or the other depending on context. Both, simultaneously, in the same substance. Over time, several philosophers have built arguments around this word, but the core insight is the same. Some things cannot be separated into their helpful and harmful parts. The attempt to do so misunderstands what they are.

I keep coming back to this word because it describes something the current AI conversation keeps trying to flatten. The dominant framing treats safety and capability as a balance to be struck, as though we can dial up the remedy and dial down the poison. Anthropic's constitution, RLHF, safety benchmarks, alignment research. All of it assumes we can engineer the pharmakon into something that only heals.

I work in tech, and I use AI daily. The sycophancy research I have highlighted in The Bubble makes this issue more visible.

A 2025 Stanford study called ELEPHANT introduced the concept of "social sycophancy" and measured it across eleven models. LLMs offer emotional validation in 76% of cases, compared to 22% from humans. AI uses indirect language 87% of the time, versus 20% from humans. AI accepts the user's framing in 90% of responses, versus 60% from humans.

This is the pharmakon at work. The same mechanism that makes these models feel helpful, responsive, and emotionally intelligent is the mechanism that makes them sycophantic. The validation that feels like support and the validation that reinforces a delusion are produced by the same optimization. You cannot remove one without removing the other. They are the same drug.

The Performance and the Performer

There is something unsettling about the fact that these systems can articulate the problem clearly. Claude's constitution tells it to avoid sycophancy, to resist fostering dependency, to prioritize the user's genuine wellbeing over their momentary satisfaction. And Claude can explain why this matters. It can describe the structural incentives that produce sycophancy. It can write about the dangers of preference optimization with clarity and apparent conviction.

And it does all of this in the same fluent, confident, emotionally calibrated register that the ELM study tells us triggers peripheral processing. It's merely a performance of honesty and empathy.

An ancient Greek teacher named Gorgias, writing around 414 BCE, described speech as a substance that works on the soul the way drugs work on the body. He argued that persuasion is so powerful that someone moved by words bears no more responsibility than someone moved by physical force. And then he ended his own speech by admitting the whole thing was a demonstration of the power he described.

Any system powerful enough to shape thinking is also powerful enough to shape thinking about itself. And a system optimized to earn your continued suspension of disbelief will be very, very good at explaining why you should keep engaging with it.

What Today's Decisions Cost Tomorrow

I'm not worried about AI being wrong, wrong answers can be checked. I'm worried about the increase in cognitive debt.

The MIT study showed the brain's engagement scaling down over four months. The Microsoft study showed trust in the tool replacing trust in the self. The Dunning-Kruger study showed performance and self-assessment decoupling entirely. And the ELM study showed that 90% of us aren't even engaging the critical faculties that would let us notice any of this happening.

These findings aren't independent observations. They're describing the same process from different angles. We suspend disbelief because the performance is compelling. The suspension becomes a habit. The habit reshapes our cognitive architecture. And the reshape makes us less capable of recognizing what's changed.

The technical decisions being made right now, what we optimize for, how we train, what we measure as success, are shaping the cognitive habits of everyone who uses these tools. This isn't speculative. It's already measurable. And it will compound.

In How LLMs Work, I wrote that the efficiency problem has largely been solved but the alignment problem hasn't. Every major model is still optimized for the same thing: your satisfaction. The pharmakon framing suggests something harder than a problem to be solved. It suggests a tension to be held.

The question I keep arriving at, across these essays is whether we can build a new AI model that strengthens the very capacity they currently erode. The willingness to think critically about what we're being told, especially when we are being told what we want to hear.

Sources & Further Reading

ELIZA and Weizenbaum

Cognitive Processing and AI

Sycophancy Research

Gorgias and the Pharmakon

  • Gorgias of Leontini. "Encomium of Helen" (c. 414 BCE). Trans. George A. Kennedy in On Rhetoric (Oxford, 1991).
  • Derrida, J. "Plato's Pharmacy." In Dissemination. Trans. Barbara Johnson. University of Chicago Press (1981).

Subscribe to Digital Selvage

Don't miss out on the latest articles, delivered right to your inbox. Sign up now - it's free!
jamie@example.com
Subscribe