What are data broker dynamics in AI governance for community organisations?
Data broker dynamics AI governance describes how commercial data brokers supply the personal information that trains and operates AI systems, creating hidden risks for community organisations and not-for-profits. Understanding data broker dynamics helps organisations assess what they do not know about where their AI tools get their data.
Data Broker Dynamics
Data broker dynamics AI governance addresses a risk that most community organisations have never considered: the AI tools they use are almost certainly trained on data sourced, in part, from commercial data brokers. Those brokers collect personal information about individuals from hundreds of sources, combine and sell it to technology companies, and have no direct relationship with the people whose information they hold. Consequently, when a community organisation adopts an AI tool, it may unknowingly be placing client-affecting decisions in the hands of a system whose underlying data supply chain runs through some of the most opaque commercial practices in the digital economy.
How do data broker dynamics affect not-for-profit AI governance and client privacy?
Definition
What are Data Broker Dynamics in AI Governance?
Data brokers are companies that collect personal information about individuals from a wide range of sources — including public records, loyalty programs, social media platforms, mobile applications, purchase histories, and other data brokers — and then sell, license, or share that information with other businesses. Unlike the companies that collect data directly from consumers, data brokers typically have no direct relationship with the individuals whose information they hold. Those individuals are generally unaware that their information exists in broker databases, cannot access it, and have no practical means of correcting or removing it.
In AI governance contexts, data broker dynamics matter because many commercial AI systems are trained on data that was sourced, in part, from broker networks. Furthermore, once a community organisation adopts an AI tool, data about the clients who interact with that tool may flow back into commercial data pipelines through vendor data sharing arrangements. In consequence, the relationship between a community organisation and its AI vendor may create data flows that extend well beyond the vendor relationship itself, connecting client information to a commercial data ecosystem that the organisation neither controls nor fully understands.
The Transparency Problem
Data broker dynamics are characterised by structural opacity. Brokers do not disclose what information they hold, how it was obtained, how it has been combined with other data, or who it has been sold to. Moreover, because brokers compile information from many sources and combine it in ways that create inferences beyond the original data, the information they hold about individuals is often more detailed and more sensitive than any single source would suggest. For AI systems trained on broker-sourced data, this opacity means that neither the vendor nor the adopting organisation can fully account for the information embedded in the model’s training data.
Australian regulatory context: The ACCC’s March 2024 interim report on the Digital Platform Services Inquiry examined data brokers operating in Australia, finding that they collect, combine, and sell detailed consumer information with limited regulatory oversight. The report identified significant gaps in Australian consumer and privacy protections with respect to data broker practices. This is directly relevant to community organisations whose AI vendors may source training or operational data from broker networks active in Australia.
Key References
Zuboff (2019) provided the foundational analysis of data broker dynamics within the broader framework of surveillance capitalism. Her analysis established that the commercial collection, combination, and sale of personal data is not incidental to the digital economy but constitutive of it: the prediction and modification of human behaviour is the core product of the surveillance capitalism model, and data brokers are essential infrastructure for the data supply chains that make those predictions possible. For community organisations, Zuboff’s framework establishes that AI tools developed within the surveillance capitalism model are not neutral instruments. Rather, they are products of an economic system that treats personal information as raw material for commercial prediction, a framing that has direct implications for how organisations should assess AI vendor data practices.
Crain (2018) provided a critical analysis of data broker industry transparency mechanisms, demonstrating that the transparency frameworks developed to address data broker opacity are structurally inadequate. His analysis showed that requiring brokers to disclose their practices does not give individuals meaningful ability to understand or contest what information is held about them, because the volume and complexity of broker data operations makes disclosure essentially unintelligible to affected individuals. Furthermore, Crain’s work established that transparency without meaningful access and correction rights fails as a governance mechanism. For community organisations, this finding suggests that vendor assurances about data source transparency are not adequate governance information if individuals have no practical ability to know what information is held about them in broker networks.
Reviglio (2022) provided a transnational and interdisciplinary overview of data broker dynamics within surveillance capitalism, confirming that data broker practices are internationally pervasive, underregulated, and poorly understood even by policymakers. Their analysis is relevant to Australian community organisations because it establishes that the regulatory gap identified in the Australian context is not an anomaly but reflects a global pattern in which data broker practices have expanded faster than governance frameworks can address.
Further reading: Reviglio’s Internet Policy Review article provides an accessible interdisciplinary overview of data broker dynamics and their governance implications across jurisdictions. Internet Policy Review: Data Brokers and Surveillance Capitalism ↗
In AI Contexts
How Data Broker Dynamics AI Governance Risks Operate
In AI governance contexts, data broker dynamics create risk through three pathways. First, AI systems trained on broker-sourced data embed the quality, biases, and gaps of that data into their outputs. Second, vendor data-sharing arrangements may mean that operational data generated by a community organisation’s AI use flows into commercial broker networks, potentially without the organisation’s knowledge. Third, the opacity of broker data supply chains means that organisations cannot conduct meaningful due diligence on the data provenance of AI tools they adopt, even if they want to.
The Data Provenance Problem
When a community organisation asks an AI vendor where their training data came from, the answer is typically a general description that does not account for broker-sourced components. Moreover, because AI vendors often purchase data from other technology companies rather than directly from brokers, the chain of data provenance may run through multiple intermediaries before reaching the original collection point. Consequently, even a vendor acting in good faith cannot always provide a complete account of their training data provenance, because that provenance may be invisible even to them.
Vendor Data-Sharing as a Downstream Risk
Beyond training data, data broker dynamics affect AI governance through vendor data-sharing practices. Many AI vendors reserve the right, in their terms of service, to use data generated through client use of their products for model improvement, analytics, or commercial purposes. In some cases, that data may be shared with third parties including data brokers. For community organisations, this means that client interaction data generated through AI tool use may enter commercial data supply chains that the organisation explicitly sought to keep separate from its AI adoption. Reviewing vendor terms of service for data-sharing provisions is accordingly a basic governance requirement that many community organisations have not yet built into their procurement processes.
A practical procurement question: Before adopting any AI tool, ask the vendor directly: does your product share data generated through our use with any third parties, including for model training, analytics, or commercial purposes? Ask for this answer in writing. If the vendor cannot provide a clear negative answer, review the terms of service carefully and treat any data-sharing provisions as a governance risk that requires specific assessment before proceeding.
Why It Matters for Not-For-Profits
The Specific Risk for Community Organisations
Community organisations hold sensitive client information under strong ethical, professional, and legal obligations. Their clients are often people whose personal information is particularly sensitive — information about health status, legal history, housing situation, family circumstances, and service use — and whose consent to data sharing is often circumscribed by their vulnerability and their dependence on the services the organisation provides. In consequence, the entry of client information into commercial data broker networks would represent a significant breach of organisational obligations even if it were technically permitted by a provision buried in vendor terms of service.
Privacy Act Obligations and Data Broker Risks
Under the Privacy Act 1988, community organisations have obligations to ensure that personal information is used only for the purposes for which it was collected and that it is not disclosed to third parties without consent or legal authority. If vendor data-sharing arrangements result in client information entering commercial data pipelines without organisational knowledge or client consent, those obligations may be breached. Furthermore, the OAIC’s guidance on data analytics and the Australian Privacy Principles establishes specific expectations for organisations using data analytics tools that process personal information, expectations that apply directly to AI tool adoption.
The ACCC’s Australian Findings
The ACCC’s March 2024 interim report on data brokers is particularly significant for Australian community organisations because it provides regulatory documentation of the practices and risks that data broker dynamics create in the Australian market. Moreover, the report establishes that Australian regulators are aware of and actively investigating these practices, meaning that the governance gap organisations face is not invisible to regulatory authorities even if it remains poorly understood at the organisational level.
OAIC guidance: The OAIC’s Guide to Data Analytics and the Australian Privacy Principles provides specific guidance for organisations using data analytics tools that process personal information, with direct relevance to AI tool adoption and vendor data-sharing assessment. OAIC: Guide to Data Analytics and the Australian Privacy Principles ↗
Data broker dynamics AI governance describes how commercial data brokers supply the personal information that trains and operates AI systems, creating hidden risks for community organisations and not-for-profits. Understanding data broker dynamics helps organisations assess what they do not know about where their AI tools get their data.
Observable Signals
What to Watch For
These patterns suggest data broker dynamics may be creating governance risk in your organisation’s AI vendor relationships. They are signals worth naming and discussing, not a definitive checklist.
- The organisation has not reviewed AI vendor terms of service specifically for data-sharing provisions, meaning it does not know whether client data generated through AI tool use may be shared with third parties including commercial data brokers, and therefore cannot assess whether vendor data practices are compatible with its privacy obligations.
- When asked about training data provenance, the AI vendor provided general assurances rather than specific information about data sources, meaning the organisation cannot determine whether the system was trained on broker-sourced data and therefore cannot assess the governance implications of that data’s collection and combination history.
- The organisation’s client privacy notices and consent frameworks were developed before AI tool adoption and do not address the possibility that client data may flow into commercial data pipelines through vendor data-sharing arrangements, meaning clients have not been informed of and have not consented to data uses that may have become relevant since AI adoption.
- Leadership is unaware of the ACCC’s findings on data broker practices in Australia or the OAIC’s guidance on data analytics and the Australian Privacy Principles, suggesting the organisation has not assessed its AI governance obligations in light of the regulatory attention that data broker dynamics are receiving from Australian authorities.
Data broker dynamics AI governance and hidden data supply chains in community sector AI tools
Data broker dynamics AI governance describes how commercial data brokers supply the personal information that trains and operates AI systems, creating hidden risks for community organisations and not-for-profits. Understanding data broker dynamics helps organisations assess what they do not know about where their AI tools get their data.
External References
- Zuboff, S. (2019). The age of surveillance capitalism: The fight for a human future at the new frontier of power. PublicAffairs.
- Crain, M. (2018). The limits of transparency: Data brokers and commodification. New Media and Society, 20(1), 88-104.
- Reviglio, U. (2022). The untamed and discreet role of data brokers in surveillance capitalism: A transnational and interdisciplinary overview. Internet Policy Review, 11(3).
- Australian Competition and Consumer Commission. Digital Platform Services Inquiry: March 2024 interim report on data brokers.
- Office of the Australian Information Commissioner. Guide to data analytics and the Australian Privacy Principles.
- Office of the Australian Information Commissioner. Digital Platform Services Inquiry: March 2024 report on data brokers — issues paper submission.
- Consumer Policy Research Centre. Submission to ACCC Digital Platforms Inquiry: Data brokers.
Related Lexicon Entries
From theory to practice
Understanding the risk is the first step.
The Not For Humans Toolkit gives your organisation practical tools to review vendor terms of service for data-sharing risk, build procurement questions that surface data provenance and broker involvement, and design governance frameworks that protect client information from entering commercial data pipelines through AI vendor relationships. The Behaviour Risk Cards name data broker dynamics in language your whole team can use.
Not For Humans: Because AI’s greatest risk isn’t technical, it’s Human.