Preserving Human Judgement When AI Gets It Right and Wrong
New research from the Wharton School has found that as AI becomes more reliable, the conditions that sustain genuine human oversight quietly deteriorate a key factor in considering human judgement AI governance for-purpose organisations. Not because people stop caring. Because vigilance is expensive, and reviewing outputs that are almost always correct is hard to sustain without deliberate structure.
Observations
Preserving Human Judgement When AI Gets It Right and Wrong
New research from the Wharton School has found that as AI becomes more reliable, the conditions that sustain genuine human oversight quietly deteriorate. Not because people stop caring. Because vigilance is expensive, and reviewing outputs that are almost always correct is hard to sustain without deliberate structure.
A second study, published by the American Psychological Association, found something equally uncomfortable: passive acceptance of AI output is associated with reduced confidence in independent reasoning. The problem is not when AI fails. It is what happens to human judgement when AI keeps succeeding.
In this article
The research behind the paradox
What it means for for-purpose organisations specifically
Why policy commitments to oversight are not enough
What structured judgement looks like in practice, and a free resource to help
Photo: Museums Victoria / Unsplash
The paradox, stated plainly.
The Paradox
The paradox, stated plainly.
Researchers at the Wharton School set out to understand what happens to human oversight as AI systems become more capable. The answer was counterintuitive.
When AI rarely fails, humans face a specific and underappreciated problem. They must expend consistent cognitive effort reviewing outputs that will almost always be correct. Over time, that effort becomes difficult to sustain. Not through negligence, but through rational adaptation. The cost of genuine vigilance rises as AI reliability improves, because evidence that vigilance is necessary keeps failing to arrive.
The researchers call this the human-AI contracting paradox. Organisations end up with oversight that looks intact from the outside while quietly becoming nominal on the inside. The policy says humans are reviewing AI outputs. The practice tells a different story.
This paradox, the paper argues, explains why many AI governance arrangements stall after early success. The structure was designed for a tool that fails visibly. It was not designed for one that almost never does.
83%
Share of workers who say AI needs human review to be reliable. The oversight is assumed. The structure to make it real is rarely there.
What the research shows about judgement and confidence.
The Research
What the research shows about judgement and confidence.
Photo: Zachary Keimig / Unsplash
A peer-reviewed study published in April 2026 in the American Psychological Association’s journal Technology, Mind, and Behavior examined how 1,923 adults engaged with AI during complex reasoning tasks. The findings are worth reading carefully, because they are more nuanced than the summaries suggest.
AI use did not harm people’s thinking. What mattered was how people engaged with it.
Participants who actively modified, questioned, or pushed back on AI output reported significantly greater confidence in their own reasoning. Those who accepted outputs with minimal modification reported lower confidence, reduced sense of ownership over their ideas, and what one participant described plainly as getting an answer faster but not thinking as deeply as they normally would.
The study found that active engagement preserved judgement. Passive acceptance eroded confidence in it. The AI was identical in both cases. The interaction style was not.
58%
Share of participants who agreed AI did most of the thinking across tasks. Higher passive reliance was directly associated with lower confidence in independent reasoning.
Source: Baldeo, 2026. APA
Participants who actively modified or contradicted AI output reported significantly greater confidence in their own reasoning. Those who accepted outputs with minimal modification reported less. The difference was not about AI capability. It was about how people engaged with it.
Source: Baldeo, 2026. APA
For-Purpose Organisations
Why this lands differently in for-purpose organisations.
The Wharton and Baldeo research draws primarily on professional and commercial contexts. The for-purpose sector has its own version of this problem, and the stakes look different.
The work of community organisations depends on contextual human judgement in ways that are difficult to overstate. A funding submission is not just a document. It is an argument grounded in community evidence, organisational knowledge, and values-based reasoning that no AI system has access to. A case note is not just a record. It carries professional judgement about a person’s situation that requires the worker to bring what they know about that person, that community, and that service to what they write.
A funding submission that is faster but shallower is not a better funding submission. A case note that accepts AI framing without scrutiny is not a safer one.
The Baldeo study found the highest rates of passive AI acceptance occurred in tasks involving planning under uncertainty and interpretation of ambiguous information. Those are not edge cases in for-purpose work. They describe the conditions the sector operates in most of the time. This is where autonomy and judgement matter most, and where the risk of quietly losing both is greatest.
0.8
Average number of times per task that participants actively modified or contradicted AI output. Less than once per task. In high-stakes community sector work, that number matters.
Source: Baldeo, S. (2026). Technology, Mind, and Behavior, APA
The Governance Gap
Why the standard governance response is not enough.
The most common organisational response to AI risk comes down to a version of the same phrase. Keep humans in the loop. Ensure oversight. Maintain review of AI outputs before they are used.
These commitments appear in AI policies, funding agreements, and governance frameworks across the sector. They are reasonable starting points. The Wharton research suggests they are structurally insufficient on their own.
Oversight named in a document is not the same as oversight that actually happens. When the structures and incentives around review do not account for the real cognitive cost of vigilance, one of two things tends to occur. Organisations invest heavily in genuine scrutiny, which is rarely resourced in the for-purpose sector. Or oversight quietly becomes performative, a checkbox that is ticked rather than a judgement that is exercised, invisible from the outside but absent in the moment that matters.
Neither outcome looks like a failure from the outside. That is precisely what makes it a governance risk worth taking seriously. It is automation bias made structural, and a textbook instance of the accountability gap at work.
Mandating oversight without designing for it creates the appearance of governance without the reality. The policy says judgement is being exercised. The research shows what that actually means in practice when the structure to support it is absent.
What to Do
What structured judgement looks like.
Photo: Mike Tinnion / Unsplash
The Baldeo research points to something specific and useful. Active engagement with AI output, modifying it, questioning it, pushing back on it, preserves confidence in independent reasoning. The implication is not that people need to try harder. It is that workflows need to be designed so that active engagement is the default rather than an optional extra that gets dropped under pressure.
Judgement does not preserve itself. It requires conditions. A moment where the human is asked, explicitly and with consequence, to account for their role in what the AI helped produce.
That structure does not have to be complex. It has to be real. A deliberate pause, a named responsibility, a declaration of what the human actually contributed to the output before it is signed off or sent.
Not For Humans has developed the AI Authorship Statement, a free resource for for-purpose organisations. It is designed for the outputs where accuracy and integrity matter most: grant applications, funder reports, board documents, policy positions, and public-facing content.
It asks the human to declare their role before sign-off. It gradates scrutiny by the stakes of the work. It creates a reviewer loop so judgement is documented rather than assumed.
It is new, and we would genuinely like to know how it lands in your organisation.
Email to request your copy →For Leaders
What this asks of organisational leaders.
The research does not suggest AI should be used less. It suggests the conditions for genuine human engagement with AI output need to be deliberately built rather than assumed into existence.
For organisational leaders, that starts with a few honest questions.
When does your team reach for AI, and is that when they are at their sharpest or when they are most stretched? The Baldeo study found the highest passive acceptance occurred under planning and sequencing pressure, which describes the conditions community sector workers operate in most of the time.
What does active engagement with AI output actually look like in your organisation? Not in principle but in practice. When a worker is uncertain about an AI-generated claim, is there a shared language for naming that uncertainty, or does it go unspoken?
And if your governance framework says humans are reviewing AI outputs, how would you know whether that review is genuine rather than performative? These are not compliance questions. They are leadership questions. The answers will not be found in the policy document. They sit at the intersection of governance and accountability and what Not For Humans calls practice over paper.
When does your team reach for AI? What does active engagement look like in your organisation, not in principle, in practice? And how would you know if oversight is genuine rather than assumed? For for-purpose organisations, these questions sit at the heart of human judgement and AI governance.
Photo: Kaleidico / Unsplash
In Closing
The governance problem that needs a governance response.
Judgement is not automatic. It does not persist because a policy requires it. It requires conditions, structure, and a culture where exercising it is expected and visible rather than assumed and invisible.
The research offers something hopeful alongside the concern. Active engagement with AI, questioning it, modifying it, bringing your own knowledge to bear on what it produces, preserves confidence in independent reasoning. The human capacity for good judgement does not disappear in the presence of AI. It needs to be exercised to remain.
That is a governance problem. And governance problems have governance responses. Not policies that name the outcome, but structures that make it possible.
Explore the human behaviour lexicon for the concepts behind this article, and the community impact section for the broader context of what AI change means for the sector.