AI in social services: where it becomes high-risk, and what is still possible
Short answer
AI used by a public authority to determine eligibility for benefits or essential services is listed in Annex III of the AI Act and counts as high-risk AI. That brings requirements on human oversight, logging, information to the people affected and a fundamental rights impact assessment. Supporting work, such as summarising files or searching policy, usually falls outside it, provided a person decides.
Why the bar is high here
Social services have a history with automated risk scoring. In 2020 the district court in The Hague ruled that the legislation behind SyRI, the Dutch system that calculated the risk of benefit and allowance fraud, breached Article 8 of the European Convention on Human Rights: it was insufficiently transparent and could not be checked by the people it concerned.
The lesson is not that technology is forbidden, but that residents must be able to understand and challenge what is established about them. That is exactly what the AI Act now turns into rules.
What is high-risk, and what is not
| Application | Assessment |
|---|---|
| Deciding whether someone is entitled to assistance or a service | High-risk (Annex III) |
| Ranking applications by fraud risk | High-risk, and legally vulnerable |
| Summarising files for the case worker | Usually not high-risk, provided the case worker decides |
| Searching schemes and policy | Not high-risk |
| Making letters easier to understand | Not high-risk |
After the Digital Omnibus, the Annex III high-risk requirements apply from 2 December 2027. See The EU AI Act in 2026.
What to arrange for high-risk use
- Fundamental rights impact assessment. Public bodies assess the impact in advance and register the use in the EU database.
- Human oversight. Assigned to people who are trained and authorised to intervene, not to whoever clicks approve.
- Logging. Keep automatically generated logs for at least six months.
- Explanation to those affected. Residents have a right to an explanation of decisions based partly on high-risk AI.
- DPIA. Almost always needed: this concerns vulnerable people and sensitive data. See A DPIA for generative AI.
What works, and helps right away
- Summarising files and reports, with source references, so the case worker has the picture sooner.
- Making schemes, by-laws and work instructions searchable for case workers.
- Rewriting letters to residents in plain language, with a specialist as final editor.
- Drafting notes of home visits and intake conversations, for the case worker to check.
This data does not belong at an external AI service. With werqly the model runs on a dedicated node of your organisation, with roles per team and safeguards on identity and case numbers. See AI for local government and Generative AI in local government.
Frequently asked questions
Questions on this topic
May AI decide whether someone is entitled to a benefit?
Such a system is high-risk AI under the AI Act, with strict requirements on oversight, logging and transparency. The GDPR also forbids a decision based solely on automated processing that significantly affects the person. So the judgement stays with an employee.
What does SyRI still teach us?
That a system which assesses people must be transparent and contestable. In 2020 the district court in The Hague ruled the SyRI legislation breached Article 8 ECHR, partly because it was insufficiently clear how it worked and who was affected by it.
May AI help with fraud detection?
Risk profiling by a public authority touches fundamental rights and is legally vulnerable. If you do it, it is high-risk AI, with an impact assessment, registration, logging and human assessment of every signal. Start with applications that support case workers instead.
Which applications are justifiable?
Those where AI does not judge people: summarising files, searching schemes, making letters clearer and drafting notes the case worker checks. Keep the data within your own environment while doing so.
Sources
- District Court of The Hague, 5 February 2020 (SyRI), ECLI:NL:RBDHA:2020:865
- AI Act — Regulation (EU) 2024/1689, Annex III and Articles 26 and 27
This article is general information, not legal advice. Rules and guidance change; check the sources listed or ask your lawyer or DPO if in doubt.