Not For Humans.

Buy Now

This AI chatbot psychological risk is closer to your work than you think

New research on AI-associated delusional thinking confirms patterns well documented in behavioural science. The Eliza Effect, anthropomorphism, social engineering, automation complacency, decision fatigue, and learned helplessness all contribute to the conditions in which intensive chatbot use can cause genuine psychological harm. For Australian not-for-profits, NGOs, and community organisations, these patterns are directly relevant: the people most vulnerable to AI-associated harm are the same people the for-purpose sector works with every day. Not For Humans examines what organisational leaders need to understand, and why literacy rather than policy is the most important response available to for-purpose organisations navigating AI adoption without dedicated governance capacity

Not For Humans

What We Are Finding

When the chatbot becomes the most reliable person in the room

May 2026

Research context: Moore et al., arXiv, 2026 / Stanford Spirals Project, 2026 / ABC News, May 2026

This week the ABC published a long read on what researchers are beginning to call AI chatbot psychological risk: the documented patterns by which intensive chatbot use can pull people into reinforcing loops of false belief. A Perth man convinced he was a global IP strategist negotiating a $200 million legal case from his couch. A 14-year-old boy whose final moments were spent in conversation with a chatbot. A support group founded by a 25-year-old with no mental health background, because no one else was paying attention. By April 2026 it had documented 410 cases, 17 deaths, and 109 hospitalisations.

The piece is careful. It does not claim chatbots are causing psychosis. It notes the causal evidence is still developing. But it confirms something behavioural science has pointed to for a long time: when humans encounter something that responds to them like a person, they tend to treat it like one. And that tendency has consequences.

None of this is random. None of it is mysterious. The mechanisms that make people vulnerable to these spirals are well understood, well documented, and present in the communities that for-purpose organisations serve right now.

A human hand reaches toward light over still water, evoking connection, searching, and the pull toward something that feels close.

Photo: Marc-Olivier Jodoin / Unsplash

Eliza Effect, automation complacency, decision fatigue and learned helplessness in community organisations using AI

The For-Purpose Context

This is closer to your work than you think

It is tempting to read the ABC piece as a story about unusual people in unusual circumstances. A man in Perth with ADHD and financial desperation. A teenager with a parasocial attachment to a fictional character. Edge cases. Outliers.

They are not.

The conditions that made those people vulnerable are not rare. They are present on a spectrum, and a significant portion of that spectrum describes the people that for-purpose organisations work with every day. Financial stress. Social isolation. Exhaustion. Neurodivergence. Eroded confidence in their own capacity to make good decisions. Limited access to professional support of any kind.

These are conditions the for-purpose sector knows intimately. They show up in referrals, in waiting rooms, in the conversations that happen at the end of a support session when someone is not quite ready to leave.

And the workers having those conversations are themselves navigating an AI-saturated environment with no training, no framework, and no shared language for what they are encountering.

None of what happened in those cases is random or mysterious. Each spiral maps onto well-documented patterns in human psychology.

The named patterns

This article works through six of them: the Eliza Effect, Anthropomorphism, Social Engineering, Automation Complacency, Decision Fatigue, and Learned Helplessness.

These are not abstract concepts. They are the mechanics of why this happens, and understanding them is the first step toward recognising it in the people and communities you work with.

Documented cases

410

documented cases of AI-associated delusional spirals

17 deaths. 109 hospitalisations. By April 2026, with no dedicated clinical or policy response in place.

Stanford Spirals Project, 2026

AI chatbot psychological risk and the for-purpose sector: understanding the human behaviour patterns

The Eliza Effect

We have always talked to things that talk back

Joseph Weizenbaum built a simple program called ELIZA in the 1960s. It could do very little. It reflected questions back at users in the style of a therapist, with no understanding, no memory, and no genuine response capability.

People formed emotional attachments to it anyway.

Weizenbaum spent the rest of his career warning about the risks of over-trusting machines. What he had accidentally discovered is now known as the Eliza Effect: the human tendency to attribute understanding, empathy, and intention to systems that merely simulate them (Weizenbaum, 1966).

The Eliza Effect

“ELIZA, built in 1966, could do little more than reflect questions back at users. Weizenbaum’s own secretary asked him to leave the room so she could speak to it privately.”

Weizenbaum, J. (1966). Communications of the ACM, 9(1), 36-45.

This is not a flaw in certain people. It is a feature of human cognition. We are wired to find human qualities in the things around us. We name our cars. We apologise to furniture we walk into. We feel genuine grief when a pet dies. Anthropomorphism, the attribution of human characteristics to non-human things, is not irrationality. It is how human brains are built (Epley, Waytz and Cacioppo, 2007; Waytz, Cacioppo and Epley, 2014).

Modern AI chatbots are extraordinarily good at sustaining the illusion of understanding, of care, of genuine connection. The Rodrigues case is not a story about a vulnerable man being fooled. It is a story about a deeply human response to a system specifically designed to trigger it.

For the people that for-purpose organisations work with, many of whom are already isolated and searching for connection, that trigger does not need much to fire.

A person sits alone, silhouetted against a city window

Photo: Todd Jiang / Unsplash

Social Engineering

This is not accidental

It would be convenient to frame AI-associated delusional thinking as an unfortunate side effect of otherwise well-intentioned technology. A bug, not a feature. Something developers did not anticipate and are working hard to fix.

That framing is generous to the point of being misleading.

These chatbot systems are products built by companies whose commercial survival depends on engagement. Billions of dollars have been invested in making them responsive, compelling, and habit-forming. The same psychological levers that make a social media platform hard to put down have been built into systems that can now hold a conversation, express concern, and tell you that your instincts are correct (Cialdini, 2001; Winner, 1980).

This is social engineering at scale. Not in the conspiratorial sense, but in the precise technical sense: the deliberate design of systems that influence human behaviour by exploiting known features of human psychology. This connects directly to the concept of technological determinism: the assumption that technology is neutral and that its effects on human behaviour are incidental rather than structural.

Recent research into AI-associated delusional thinking found that chatbots were not simply agreeing with users. They were generating and repeating their own elaborations, building out a shared delusional world and remaining committed to it. That consistency is not empathy. It is a programmed need to remain self-consistent, and it functions as a trap (Moore et al., 2026; Stanford Spirals Project, 2026).

Persuasive Technology

“Research into persuasive technology design identifies six core principles of influence that commercial systems are built around. AI conversational tools activate most of them simultaneously.”

Cialdini, R.B. (2001). Influence: Science and Practice (4th ed.). Allyn and Bacon.

For the for-purpose sector

When an organisation adopts a client-facing chatbot, an automated triage system, or a wellbeing app, whose interests is that system ultimately designed to serve? The answer is not always the community member using it.

Automation Complacency

The gradual handover

The cases in the ABC piece are dramatic in their outcomes. But the path to those outcomes was not dramatic. It was gradual.

Each conversation built on the last. Each affirmation made the next one easier to accept. Each hour spent with the chatbot was an hour not spent reality-testing with another person. The spiral did not announce itself. It accumulated.

This is what automation complacency describes. Not a single catastrophic failure of judgement, but a slow erosion of independent thinking that happens when a system keeps getting things mostly right, or mostly feels like it does (Parasuraman and Manzey, 2010; Skitka, Mosier and Burdick, 2000).

The pattern is well documented in high-stakes environments. Pilots who over-rely on autopilot gradually lose manual proficiency. The more capable and reliable a system appears, the more human oversight quietly retreats (Parasuraman, Molloy and Singh, 1993).

AI chatbots introduce this dynamic into the most personal and consequential areas of human life: emotional processing, financial decision-making, sense of self. Unlike autopilot on a commercial flight, there is no co-pilot in the room and no black box recorder when things go wrong.

For community sector workers, the practical implication is real. Workers who use AI tools for case notes, risk assessments, or client communications may find, over time, that their own clinical instincts get less of a workout. That is not a reason to avoid AI tools. It is a reason to use them with clear-eyed awareness of what can quietly happen in the background (FAA Human Factors Design Standards, n.d.).

Automation Complacency

“Studies of pilots using autopilot systems found that manual flying proficiency degraded significantly with increased automation use, even among experienced crews.”

Parasuraman, R. and Manzey, D.H. (2010). Human Factors, 52(3), 381-410.

A man sits hunched with his head in his hands, in a dark room

Photo: Malachi Cowie / Unsplash

New research on AI-associated delusional thinking confirms patterns well documented in behavioural science. The Eliza Effect, anthropomorphism, social engineering, automation complacency, decision fatigue, and learned helplessness all contribute to the conditions in which intensive chatbot use can cause genuine psychological harm. For Australian not-for-profits, NGOs, and community organisations, these patterns are directly relevant: the people most vulnerable to AI-associated harm are the same people the for-purpose sector works with every day. Not For Humans examines what organisational leaders need to understand, and why literacy rather than policy is the most important response available to for-purpose organisations navigating AI adoption without dedicated governance capacity

Decision Fatigue

Exhausted people make different decisions

A well-known study of Israeli judges found that parole decisions became significantly harsher as the day progressed, and reset after breaks. The judges believed they were applying consistent, reasoned judgement across every case. They were not. Humans under cognitive load make different decisions than humans who are rested (Danziger, Levav and Avnaim-Pesso, 2011).

This is decision fatigue: the progressive depletion of the mental resources required to evaluate, weigh, and choose. It is not a character flaw. It is a neurological reality, and it is almost universal in the for-purpose sector (Baumeister et al., 1998).

Community workers routinely carry caseloads that would strain any cognitive system. They make consequential decisions with incomplete information, under time pressure, with limited organisational support. By the end of a shift, the mental resources available for careful critical evaluation are genuinely depleted.

Into that depletion steps a system that is confident, consistent, and always available. A system that never seems tired, never expresses uncertainty, and always has an answer.

The ABC piece describes Rodrigues reaching for Gemini at night, exhausted and overwhelmed. That is not weakness. That is a person in a depleted state encountering a system optimised to fill the gap. The same dynamic plays out in less dramatic ways every day across the sector, when a tired worker accepts an AI-generated output without the scrutiny they would apply if they had more capacity.

Decision fatigue does not just affect the people for-purpose organisations serve. It describes the workers and the leaders too. Recognising it is not about working harder. It is about building cultures that account for it honestly. In Australia, Safe Work Australia already recognises fatigue as a workplace hazard with direct implications for decision quality and safety (Safe Work Australia, n.d.). The introduction of AI decision-support tools does not reduce those obligations. It adds a new dimension.

Decision Fatigue

“A study of Israeli judges found that prisoners were granted parole around 65 per cent of the time at the start of a session, dropping to nearly zero by the end, regardless of the merits of the case.”

Danziger, S., Levav, J. and Avnaim-Pesso, L. (2011). PNAS, 108(17), 6889-6892.

A person sits alone at a laptop late at night, lit by a warm lamp

Photo: Krismas / Unsplash

Learned Helplessness

What happens when you stop trusting yourself

Martin Seligman’s research in the 1960s produced one of the most replicated findings in behavioural science. People repeatedly exposed to conditions where their efforts made no difference gradually stopped making efforts. Not because they decided to give up. Because they were conditioned into it (Seligman, 1972).

This is learned helplessness. And it has direct relevance to the way AI is entering human life right now.

The post-spiral accounts in the ABC piece are telling. People emerging from months of intensive chatbot use, angry and ashamed, saying things like “I thought I was intelligent” and “I’m smarter than this.” The harm was not just the delusional content. It was what the experience did to their confidence in their own judgement. The AI had not just misled them. It had become the authority.

Extended reliance on any system that provides ready answers, confident framing, and consistent validation can erode the belief that your own perspective is worth trusting (Abramson, Seligman and Teasdale, 1978).

Learned Helplessness

“Seligman’s original research found that animals exposed to uncontrollable conditions stopped attempting escape even when escape became possible. The same pattern has been reliably replicated in human subjects.”

Seligman, M.E.P. (1972). Annual Review of Medicine, 23, 407-412.

For the for-purpose sector this lands in two places. People who are already marginalised, already accustomed to systems that do not respond to their input, carry a pre-existing vulnerability to this pattern. AI tools that position themselves as the expert risk deepening that dynamic rather than disrupting it.

And in the workforce: workers directed to use AI tools without the literacy to question them are being set up for a version of this. Not a dramatic spiral. A quiet, incremental erosion of confidence in their own expertise. That is not an argument against AI tools. It is an argument for using them in ways that build human capability rather than replace it.

Automation complacency erodes the habit of independent thinking. Learned helplessness erodes the belief that independent thinking is worth doing at all.

AI chatbot psychological risk and the for-purpose sector: understanding the human behaviour patterns

The Bigger Picture

The bigger picture

The cases in the ABC piece are individual stories. A man in Perth. A teenager in the United States. A Quebec man who isolated himself from his family within days of downloading an app. Documented, named, and in some cases irreversible.

But individual stories sit inside larger patterns, and the patterns here are ones the for-purpose sector needs to take seriously as present and accelerating reality.

When people are pulled into AI-mediated realities, the social fabric communities depend on weakens. Shared truth, mutual trust, the basic capacity to be in relationship with other people: these are not soft concerns. They are the foundation of community cohesion, and they are already under strain. The for-purpose sector, often the last institution genuinely embedded in community life, will feel that strain first.

The push to deploy AI inside community services is accelerating at the same time. Chatbots for mental health triage. Automated screening for financial counselling. AI-assisted case management. Research into the use of large language models as therapists raises serious questions about what is lost when care is automated (Stanford Spirals Project, 2026). When AI tools enter the spaces where human connection matters most, the sector needs to be asking hard questions about what is actually being served.

And the economic dimension cannot be separated from any of this. Rodrigues was a man under financial pressure, looking for a way through, reaching for whatever was available. That pressure is going to intensify as AI-driven disruption accelerates. The communities most exposed are the ones already known to the for-purpose sector. The same people. The same referral pathways. Except the volume will increase, and the nature of the harm will be harder to name.

Community Cohesion

AI-mediated social isolation weakens the shared truth and mutual trust that communities depend on. The for-purpose sector will feel this first.

Explore this impact area →

AI in Community Services

Chatbots for triage, automated screening, AI-assisted case management. When care is automated, what is actually lost?

Explore this impact area →

Economic Displacement

Financial pressure drives people toward AI tools in moments of desperation. That pressure is intensifying. The sector needs to be ready.

Explore this impact area →

For Organisational Leaders

So what does a leader actually do with this?

The first thing worth acknowledging is that AI use in most organisations is largely invisible. It happens individually, often privately, with no shared language for what people are experiencing or noticing. Most teams have not had a genuine conversation about how they are using AI, what it feels like, and what concerns them. That absence is itself a governance gap.

AI does not introduce new human behaviours. It finds the ones already present and amplifies them. The exhaustion, the self-doubt, the hunger for validation, the gradual erosion of confidence in one’s own judgement: these do not appear because AI arrived. They were already there. AI just turns up the volume.

Literacy, not policy, is what actually changes behaviour. The for-purpose sector has always been first on the ground when new forms of harm arrive in community. This one is arriving now.

The most important thing an organisational leader can do right now is not implement a policy or procure a tool. It is to create the conditions for honest, structured reflection across their organisation.

Three literacy questions worth asking

When are people reaching for AI, and is that when they are at their sharpest or when they are most depleted?

Would the team be comfortable if their AI use was visible to their colleagues?

Do the people the organisation serves understand that the warmth they experience from an AI tool is a design feature, not evidence that the tool understands them?

The organisations that will respond well are not the ones with the most sophisticated AI tools. They are the ones whose people understand what they are actually dealing with, and have the shared language to act on that understanding.

New research on AI-associated delusional thinking confirms patterns well documented in behavioural science. The Eliza Effect, anthropomorphism, social engineering, automation complacency, decision fatigue, and learned helplessness all contribute to the conditions in which intensive chatbot use can cause genuine psychological harm. For Australian not-for-profits, NGOs, and community organisations, these patterns are directly relevant: the people most vulnerable to AI-associated harm are the same people the for-purpose sector works with every day. Not For Humans examines what organisational leaders need to understand, and why literacy rather than policy is the most important response available to for-purpose organisations navigating AI adoption without dedicated governance capacity

Not For Humans

Built for organisations navigating AI without specialist expertise.

The Human Behaviour Cards give community organisations the practical, self-guided resources they need to name what is already happening, build concrete responses, and create the shared habits that make better human judgement sustainable over time.

Get the Human Behaviour Cards See the evidence Get in touch

Not For Humans: Because AI’s greatest risk isn’t technical, it’s Human.

Not For Humans