Testing algorithms for discrimination: lessons for public bodies
Short answer
Algorithms rarely discriminate on purpose. It happens through data that correlates with origin, age or neighbourhood, and through choices nobody recorded. So test in advance: which purpose, which data, which groups may be affected, and how you check the outcomes. The Dutch IAMA method helps here and is aligned with the fundamental rights assessment the AI Act requires for high-risk AI.
What went wrong
- Rotterdam (2021). The Rotterdam audit office examined the risk model that decided which welfare recipients were re-examined. Its conclusion: responsibility was poorly assigned, ethical choices could not be traced, and there was a real risk of biased outcomes.
- SyRI (2020). The district court in The Hague ruled that the legislation behind the SyRI risk system breached Article 8 ECHR, partly because it was insufficiently clear how it worked. See AI in social services.
The common pattern: nobody had recorded in advance which data could weigh in and how outcomes per group would be checked.
How discrimination creeps in
| Mechanism | Example |
|---|---|
| Proxies | Postcode, language or education correlate with origin |
| Skewed history | The model learns from earlier checks, which were already selective |
| Feedback loops | Those checked more often are found more often, and so checked even more |
| Language of language models | A summary or assessment that rewards writing style disadvantages the less fluent |
A practical test
- Purpose and necessity. What must the algorithm achieve, and is there a less intrusive alternative?
- Data. Which characteristics weigh in, and which could act as proxies for protected characteristics?
- Groups. Who may be affected, and how will you see if a group more often gets an adverse outcome?
- Human assessment. Who assesses the outcome, with what authority to deviate?
- Checks afterwards. How often do you measure outcomes per group, and what happens if they skew?
- Openness. Publish high-impact algorithms in a register. See Algorithm registers.
The Dutch Impact Assessment for Human Rights and Algorithms (IAMA) works these questions out and is now aligned with the fundamental rights assessment in the AI Act. See The EU AI Act in 2026.
What werqly contributes
werqly does not make an algorithm fair. It does supply the facts you need for the assessment and for checks afterwards: which model, which sources, which roles, and per use what was processed. Because everything runs on a dedicated node, you can analyse outcomes without sharing residents’ data with an outside party. See AI for local government.
Frequently asked questions
Questions on this topic
How does a public body test an algorithm for discrimination?
By recording purpose, data and possibly affected groups in advance, assessing which characteristics could act as proxies, arranging human assessment and measuring outcomes per group periodically. The IAMA provides a worked-out questionnaire for this.
What is the IAMA?
The Impact Assessment for Human Rights and Algorithms, developed in 2021 by Utrecht University for the Dutch Ministry of the Interior. It has since been updated and aligned with Article 27 of the AI Act, the fundamental rights assessment for high-risk AI.
What did the Rotterdam welfare algorithm teach?
The Rotterdam audit office concluded in 2021 that responsibility for the risk model was poorly assigned, that ethical choices could not be traced and that the model could lead to biased outcomes.
Can language models discriminate too?
Yes, for example by rewarding writing style over content, or by repeating biases from their training data. So do not let language models judge people, and spot-check summaries for skewed representation.
Sources
- Association of Audit Offices — Gekleurde technologie (Rotterdam audit office, 2021, Dutch)
- Government of the Netherlands — Impact Assessment for Human Rights and Algorithms (Dutch)
This article is general information, not legal advice. Rules and guidance change; check the sources listed or ask your lawyer or DPO if in doubt.