ELIZA effect AI governance: why people attribute understanding and empathy to AI systems in nonprofit and community organisations
The ELIZA effect is the tendency to attribute understanding, intent, and empathy to AI systems that have none. Named after Weizenbaum’s 1966 ELIZA program, which prompted users to form emotional connections with a simple pattern-matching script, the effect is dramatically amplified by modern large language models whose outputs are far more sophisticated and contextually responsive. In community service contexts where relationships are central to the work, the ELIZA effect carries particular risk: workers may trust AI assessments of human situations as though the AI understands those situations. This behavioural risk connects to processing fluency effect, automation bias, and coherence bias in the NFH Trust and Cognition theme, and underpins the NFH Behaviour Risk Card framework for community organisations. ELIZA effect AI governance why nonprofit staff attribute understanding and empathy to AI systems that have none. NFH Behavioural AI Governance Lexicon.
ELIZA Effect
People attribute understanding, intent, and empathy to AI systems that have none. In community service contexts where relationships are the work, this misattribution carries particular risk.
ELIZA effect AI governance: why people attribute understanding and empathy to AI systems in nonprofit and community organisations
Definition
What is the ELIZA Effect?
The ELIZA effect is the tendency to attribute human qualities to AI systems based on the surface characteristics of their output. When a system responds in language that sounds empathetic, contextually aware, or thoughtful, people infer that the system understands them. That inference is not examined; it is felt.
The effect operates below conscious evaluation. It is not that people believe the AI is human. It is that the social and relational cues embedded in natural language activate the same interpretive processes that govern human-to-human interaction. Those processes are fast, automatic, and not designed to distinguish between a person and a system producing person-like output.
The consequence is a systematic overestimation of what AI systems know, understand, and can reliably do. When a system sounds like it grasps the complexity of a situation, people respond as though it does. Decisions made on that basis carry risk that the surface of the interaction does not signal.
Research Origin
Weizenbaum (1966) created ELIZA, a simple pattern-matching program that simulated conversation by reflecting users’ statements back as questions. Despite its mechanical simplicity, users including Weizenbaum’s own secretary formed emotional attachments to it and attributed understanding to it. Weizenbaum was disturbed enough by this response to spend the following decade examining its implications.
Weizenbaum (1976) argued in his subsequent book that the ELIZA effect revealed something important about human psychology rather than machine capability: people are primed to find mind in language, and that priming does not require evidence of actual understanding. The danger is not that machines will become human, but that people will treat them as though they have.
Horton (2023) extended this concern to large language models, finding that even researchers studying AI attribution effects are not immune to them. The sophistication of modern language model output amplifies the ELIZA effect significantly compared to its original context. The gap between how the system sounds and what it actually does has widened, not narrowed.
In AI Contexts
How It Shows Up With AI
Modern language models produce output that is contextually responsive, tonally appropriate, and linguistically sophisticated. When asked about a difficult situation, they produce language that sounds considered. When asked for support, they produce language that sounds caring. None of this reflects understanding. It reflects pattern completion trained on vast quantities of human text.
The ELIZA effect means that the quality of this output activates genuine social cognition in the reader. A worker who describes a client’s situation to an AI tool and receives a thoughtful-sounding response will process that response differently to how they would process a form output or a database query result. The language cues override the technical framing.
This is compounded by the tendency of language models to express appropriate uncertainty and to qualify their outputs in ways that sound epistemically careful. That carefulness reads as evidence of genuine reflection. It is not. It is a trained pattern.
Why It Matters for Nonprofits
The Specific Risk for Community Organisations
Community organisations exist to serve people in complex, often vulnerable circumstances. The work is relational. Clinical judgement, peer support, case management, and advocacy all depend on genuinely understanding the person in front of you. That understanding is built through relationship, context, and professional experience. No AI system has access to any of it.
The ELIZA effect creates risk at exactly this intersection. When a worker uses an AI tool to help assess a client situation, draft a support plan, or frame a difficult conversation, the tool’s output sounds like it understands the person involved. That sound can displace the worker’s own relational knowledge, not through an explicit decision to defer, but through the quiet cognitive authority that human-like language carries.
For organisations serving communities with experiences of marginalisation, trauma, or systemic exclusion, the cost of that displacement is not abstract. A support plan drafted from an AI output that sounded empathetic but missed the actual person is a governance failure with a human face.
TThe Processing Fluency Effect is a cognitive bias documented by Reber and Schwarz in 1999 showing that information which is easy to read feels more accurate, regardless of whether its content has changed. In AI governance contexts, this means nonprofit and community organisation staff are more likely to accept AI-generated documents without critical review simply because AI output is always fluent, polished, and professionally formatted. This behavioural risk underpins the NFH Behaviour Risk Card framework and connects directly to automation bias, coherence bias, and default effect patterns in the Trust and Cognition theme.
Observable Signals
What to Watch For
These patterns suggest the ELIZA effect may be shaping how AI outputs are interpreted and used in your organisation. They are signals worth naming and discussing, not a definitive checklist.
- Staff describe AI outputs about client or community situations using relational language, saying the AI understood the situation or picked up on something important, rather than describing it as pattern-matched text.
- AI-generated support plans, case notes, or assessments are accepted as capturing the person involved, rather than being treated as a structural template requiring substantive human knowledge to complete.
- Workers find it difficult to articulate what the AI does not know about a situation, suggesting the surface quality of the output has displaced awareness of its actual limitations.
External References
- Weizenbaum, J. (1966). ELIZA: A computer program for the study of natural language communication between man and machine. Communications of the ACM, 9(1), 36-45.
- Weizenbaum, J. (1976). Computer Power and Human Reason: From Judgment to Calculation. W.H. Freeman.
- Horton, J.J. (2023). Large language models as simulated economic agents: What can we learn from Homo Silicus? NBER Working Paper 31122.
Related Lexicon Entries
From theory to practice
Understanding the risk is the first step.
The NFH Toolkit gives your organisation practical tools to protect relational knowledge and human judgement when AI output sounds more knowing than it is. The Behaviour Risk Cards name this pattern in language your whole team can use.
Not For Humans: Because AI’s greatest risk isn’t technical, it’s Human.