Encoded bias AI governance: how AI systems inherit historical inequity and produce discriminatory outcomes affecting vulnerable communities in not-for-profit, NGO, and community organisations in Australia
Encoded bias AI governance describes the risk that AI systems adopted by not-for-profit organisations, NGOs, and community organisations reproduce historical patterns of discrimination and inequity because those patterns are embedded in the training data those systems learn from. Drawing on Obermeyer and colleagues’ 2019 empirical demonstration that a widely used healthcare algorithm systematically disadvantaged Black patients due to biased proxy variables, Buolamwini and Gebru’s 2018 Gender Shades research documenting intersectional accuracy disparities in commercial AI systems, and Barocas and Selbst’s 2016 legal analysis establishing how algorithmic systems produce disparate impact under existing anti-discrimination frameworks, encoded bias AI governance risk is directly relevant to NFPs and community organisations whose clients are disproportionately drawn from communities that historical data systems have under-served, misrepresented, or actively harmed. When AI tools are trained on historical data, they do not learn a neutral account of the world. They learn a record of who has mattered, whose experiences have been captured, and whose outcomes have been shaped by systems that were not designed with them in mind. Encoded bias AI governance connects to mechanical objectivity, epistemic justice, social license to operate, and automation bias in the NFH Lexicon, and underpins the NFH Behaviour Risk Card framework for not-for-profit and community organisations across Australia.
Encoded Bias
AI systems do not learn from neutral data. They learn from historical records, and history is not neutral. The communities most likely to be harmed by encoded bias are those whose data has been shaped by systems that were never designed to serve them. For community organisations, that is not a distant concern. It is the population they exist to support.
Encoded bias AI governance: how AI systems inherit historical inequity and produce discriminatory outcomes affecting vulnerable communities in not-for-profit, NGO, and community organisations in Australia
Definition
What is Encoded Bias in AI Governance?
Encoded bias describes the process by which historical patterns of discrimination and inequity become embedded in AI systems through their training data. Because AI systems learn from data generated by past human decisions, they inherit the values, assumptions, and structural inequities that shaped those decisions. The bias is not introduced by the AI system itself. It arrives with the data and is amplified through the system’s application to new decisions.
Encoded bias operates through several distinct mechanisms. Proxy discrimination occurs when a variable that correlates with a protected characteristic substitutes for explicit discrimination, producing biased outcomes without any direct reference to race, gender, or disability. Underrepresentation bias occurs when training data contains fewer examples from particular communities, producing less accurate outputs for those communities. Feedback loops occur when biased outputs become inputs for future training, progressively amplifying the original bias over time.
The Invisibility of Encoded Bias
Encoded bias is structurally difficult to detect because it operates within systems that present themselves as objective. The bias is not visible in the algorithm itself. It is distributed across the training data, the variable selection, the proxy choices, and the threshold settings that collectively produce the output. Identifying encoded bias requires access to the system’s design and training data that most organisations adopting AI tools never receive, and most vendors never provide.
Research Origin
Obermeyer and colleagues (2019) provided one of the most significant empirical demonstrations of encoded bias in a high-stakes real-world system. Their analysis of a widely used US healthcare algorithm found that it systematically assigned lower risk scores to Black patients than to white patients with equivalent health conditions. The mechanism was proxy discrimination: the algorithm used healthcare cost as a proxy for health need. Because historical inequity meant Black patients had less spent on their care, the algorithm learned they were healthier. The bias was structural, invisible within the system, and produced by a variable that appeared entirely reasonable in isolation. The study established that encoded bias can operate in systems designed with no discriminatory intent, through entirely legitimate-seeming design decisions.
Buolamwini and Gebru (2018) documented intersectional accuracy disparities in commercial facial recognition systems from major technology companies, finding error rates of up to 34% for darker-skinned women compared to under 1% for lighter-skinned men. The systems had been trained predominantly on lighter-skinned faces. Their Gender Shades research established two critical governance principles: first, that accuracy varies systematically across demographic groups in ways that are invisible in aggregate performance metrics; and second, that the communities most misidentified are those least represented in training data. A system that performs well on average can simultaneously produce severe discrimination for specific communities, and aggregate accuracy metrics will not reveal that disparity.
Barocas and Selbst (2016) examined the legal implications of algorithmic bias under existing anti-discrimination frameworks, establishing that AI systems can produce unlawful disparate impact without any discriminatory intent on the part of their designers or adopters. Their analysis showed that the standard legal defences available to organisations using biased systems, such as business necessity and job-relatedness, transfer imperfectly to algorithmic contexts. Organisations that adopt AI tools producing disparate impact on protected groups may carry legal liability for that impact regardless of whether they designed the system, understood its mechanisms, or intended any discriminatory outcome.
In AI Contexts
How Encoded Bias Shows Up With AI
Encoded bias enters community organisations through the AI tools they adopt, the platforms they use, and the data systems they connect to. A risk assessment tool trained on population data that underrepresents particular communities will produce less accurate risk scores for those communities. A client communication tool trained on mainstream language patterns will perform less well for clients from culturally and linguistically diverse backgrounds. A hiring tool trained on historical recruitment data will reproduce the demographic patterns of past hiring decisions.
The Aggregate Accuracy Problem
Aggregate accuracy metrics do not reveal encoded bias. A tool that is correct 92% of the time may still systematically disadvantage a specific community if that community represents a small proportion of the training data. Vendors who report aggregate accuracy figures without demographic breakdowns are providing information that cannot identify this risk. Organisations that accept aggregate accuracy as evidence of fairness are accepting a metric that is structurally unable to detect the harms most relevant to their client population.
In the Field: In October 2024, the Australian Information Commissioner found that Bunnings Group Limited had violated the privacy of individuals by deploying facial recognition technology across 63 stores in Victoria and New South Wales. The case illustrated how biometric AI systems — trained predominantly on non-representative data — can produce discriminatory identification errors while being presented as routine security tools. The OAIC’s subsequent statement following Administrative Review Tribunal proceedings confirmed the significance of the case as a precedent for AI-driven data collection affecting community members without meaningful consent or transparency. OAIC statement on the Bunnings decision ↗
Feedback Loops in Community Data
Community organisations that use AI tools and feed client data back into those systems risk contributing to feedback loops that amplify existing bias. If an AI tool produces less accurate outputs for particular client groups and those outputs inform subsequent decisions, the data generated by those decisions will reinforce the original bias in future training cycles. Organisations that do not monitor outcomes by demographic group cannot detect this pattern, and the communities most affected are least likely to have the standing to surface it.
Why It Matters for Not-For-Profits
The Specific Risk for Community Organisations
Not-for-profit and community organisations disproportionately serve the communities most likely to be harmed by encoded bias. People experiencing poverty, homelessness, mental health challenges, disability, family violence, or migration are systematically underrepresented in the mainstream datasets that train most AI systems. They are also the people about whom historical data has most often been collected by systems designed for surveillance, risk management, or resource rationing rather than support.
Mission Contradiction
An organisation that adopts AI tools producing encoded bias against its own client population is not simply taking an operational risk. It is acting in direct contradiction of its mission. The harm does not require intent. Barocas and Selbst established that disparate impact can arise through entirely routine adoption decisions. Organisations that do not assess the demographic accuracy of AI tools before adoption, and monitor outcomes after adoption, have no basis on which to claim their AI use is consistent with their equity commitments.
Legal and Compliance Dimensions
Australian anti-discrimination law applies to outcomes, not intentions. Organisations using AI tools that produce disparate impact on the basis of race, gender, disability, or other protected characteristics may carry liability under the Racial Discrimination Act 1975, Sex Discrimination Act 1984, Disability Discrimination Act 1992, or Age Discrimination Act 2004 regardless of whether they designed the system or understood its bias. The Australian Human Rights Commission has documented this risk explicitly, establishing that AI adoption is not a shield from discrimination obligations.
Australian regulatory context: The Australian Human Rights Commission has published multiple resources establishing that AI systems producing discriminatory outcomes may breach Australian anti-discrimination law. Key resources include the AHRC’s thematic report on AI and racial discrimination, its submission on centring human rights in global AI governance, its technology and human rights guidance, and a dedicated technical paper on addressing algorithmic bias. AHRC: AI and Racial Discrimination ↗ AHRC: Centring Human Rights in AI Governance ↗ AHRC: Technology and Human Rights ↗ AHRC: Technical Paper on Algorithmic Bias ↗
Encoded bias AI governance describes the risk that AI systems adopted by not-for-profit organisations, NGOs, and community organisations reproduce historical patterns of discrimination and inequity because those patterns are embedded in the training data those systems learn from. Drawing on Obermeyer and colleagues’ 2019 empirical demonstration that a widely used healthcare algorithm systematically disadvantaged Black patients due to biased proxy variables, Buolamwini and Gebru’s 2018 Gender Shades research documenting intersectional accuracy disparities in commercial AI systems, and Barocas and Selbst’s 2016 legal analysis establishing how algorithmic systems produce disparate impact under existing anti-discrimination frameworks, encoded bias AI governance risk is directly relevant to NFPs and community organisations whose clients are disproportionately drawn from communities that historical data systems have under-served, misrepresented, or actively harmed. When AI tools are trained on historical data, they do not learn a neutral account of the world. They learn a record of who has mattered, whose experiences have been captured, and whose outcomes have been shaped by systems that were not designed with them in mind. Encoded bias AI governance connects to mechanical objectivity, epistemic justice, social license to operate, and automation bias in the NFH Lexicon, and underpins the NFH Behaviour Risk Card framework for not-for-profit and community organisations across Australia.
Observable Signals
What to Watch For
These patterns suggest encoded bias may be producing inequitable outcomes through AI adoption in your organisation. They are signals worth naming and discussing, not a definitive checklist.
- The organisation has adopted AI tools on the basis of aggregate accuracy metrics provided by the vendor without requesting or receiving demographic breakdowns of accuracy across the client groups the tool will affect, meaning the organisation has no basis to assess whether the tool performs equitably for its specific client population.
- Staff have observed that AI outputs seem less accurate, less relevant, or less useful for clients from particular communities, including First Nations clients, clients from culturally and linguistically diverse backgrounds, or clients with complex histories of contact with government systems, but the organisation has no formal process for capturing, escalating, or acting on those observations.
- The organisation has not assessed whether its use of AI tools is consistent with its obligations under Australian anti-discrimination law, and has not sought assurance from vendors that their systems have been tested for disparate impact on the demographic groups the organisation serves.
Encoded bias AI governance: how AI systems inherit historical inequity and produce discriminatory outcomes affecting vulnerable communities in not-for-profit, NGO, and community organisations in Australia
Encoded bias AI governance describes the risk that AI systems adopted by not-for-profit organisations, NGOs, and community organisations reproduce historical patterns of discrimination and inequity because those patterns are embedded in the training data those systems learn from. Drawing on Obermeyer and colleagues’ 2019 empirical demonstration that a widely used healthcare algorithm systematically disadvantaged Black patients due to biased proxy variables, Buolamwini and Gebru’s 2018 Gender Shades research documenting intersectional accuracy disparities in commercial AI systems, and Barocas and Selbst’s 2016 legal analysis establishing how algorithmic systems produce disparate impact under existing anti-discrimination frameworks, encoded bias AI governance risk is directly relevant to NFPs and community organisations whose clients are disproportionately drawn from communities that historical data systems have under-served, misrepresented, or actively harmed. When AI tools are trained on historical data, they do not learn a neutral account of the world. They learn a record of who has mattered, whose experiences have been captured, and whose outcomes have been shaped by systems that were not designed with them in mind. Encoded bias AI governance connects to mechanical objectivity, epistemic justice, social license to operate, and automation bias in the NFH Lexicon, and underpins the NFH Behaviour Risk Card framework for not-for-profit and community organisations across Australia.
External References
- Obermeyer, Z., Powers, B., Vogeli, C. and Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447-453.
- Buolamwini, J. and Gebru, T. (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. Proceedings of Machine Learning Research, 81, 1-15.
- Barocas, S. and Selbst, A.D. (2016). Big data’s disparate impact. California Law Review, 104(3), 671-732.
Related Lexicon Entries
From theory to practice
Understanding the risk is the first step.
The NFH Toolkit gives your organisation practical tools to assess AI tools for encoded bias before adoption, build monitoring processes that detect inequitable outcomes across your client population, and establish accountability frameworks that keep your AI use consistent with your equity commitments and your legal obligations. The Behaviour Risk Cards name this pattern in language your whole team can use.
Concerned about how AI is affecting your community or the organisations that serve them?
Not For Humans: Because AI’s greatest risk isn’t technical, it’s Human.