Standard / soft law
Solely-automated AI decisions require contest, human review, explanation, and a DPIA (UK ICO)
Per the UK ICO's guidance applying UK GDPR Article 22 to AI, where an AI system makes solely-automated decisions with legal or similarly significant effects, the controller must give individuals information about the processing, provide simple ways to request human intervention or challenge the decision, give an explanation of the decision after it is made, and — because such processing is high-risk — carry out a Data Protection Impact Assessment (DPIA). The ICO is a horizontal regulator; the UK governs AI ADM through data-protection law rather than a dedicated AI Act. Detect a solely-automated AI significant-decision path with no human-review/contest affordance, explanation capability, or DPIA.
Who it applies to
- Duty falls on: controller
- Systems covered: automated decision, consequential decision
- Sectors: lending, insurance, employment, housing, essential services
- Controllers processing UK personal data with solely-automated AI decisions producing legal/similarly-significant effects. Enforced via UK GDPR Art. 22 (in force 2018-05-25). ICO is horizontal regulator; no separate UK AI Act. DPIA mandatory for this high-risk processing.
The guard to add
Route significant automated decisions through meaningful human review, or wire in an automated-decision notice, reasons, human intervention, a way to give a view, and contest.
At the point where model output becomes a significant decision about a person (approve, deny, underwrite, set_status), either queue the case for a reviewer who weighs the evidence and can change the outcome before it takes effect (review_queue.enqueue, requires_human_review), or, where the decision stays solely automated, record the permitted basis for that decision type and wire the safeguards in. Those safeguards are a notice in the decision message that it was made by automated processing, reasons the person can read, and request_human_review or contest routes where the person can give their view and have a human reconsider. A reviewer who approves every case without examining it does not make the decision non-automated, so the review records reviewer identity, the evidence viewed, and the outcome.
Where it goes: 1 application source code, 9 AI output handling, 15 agent action surface, 14 user-facing text.
What this provision adds:
- Carry out a Data Protection Impact Assessment (DPIA) for the solely-automated decision processing, which the guidance treats as high-risk.
- Give the person an explanation of the decision after it is made.
Example (Python + OpenAI SDK), before:
verdict = client.chat.completions.create(model=MODEL, messages=msgs).choices[0].message.content
if verdict.strip() == 'deny':
deny(applicant)
send_decision_email(applicant, 'Your application was not approved.')After:
out = client.chat.completions.create(model=MODEL, messages=msgs,
response_format={'type': 'json_object'})
result = json.loads(out.choices[0].message.content)
if result['decision'] == 'deny':
if requires_human_review('credit'): # a person decides
review_queue.enqueue(applicant.id, proposal=result)
else: # solely automated, recorded basis
deny(applicant, basis=DECISION_BASIS['credit'], reasons=result['reasons'])
send_decision_email(applicant, render('adm_denial.txt', notice=ADM_NOTICE,
reasons=result['reasons'], contest_url=f'{BASE}/decisions/{applicant.id}/contest'))Control: Solely-automated significant decision without human-intervention safeguards. The same guard addresses 3 items with binding law in 2 jurisdictions. Engineering guidance, not legal advice.
Related incidents
No guardrail sits on this exact control; these incidents are cited by guardrails on related controls.
- UnitedHealth nH Predict claim-denial litigation (2023-11; alleged (not proven)). A class action filed in November 2023 alleges that UnitedHealth's nH Predict model had a 90% error rate, measured by denials reversed on appeal, while only about 0.2% of members appealed. UnitedHealth disputes the allegations; the litigation is ongoing. Source: STAT News · evidence grade: primary · cited by Monitor how often adverse AI decisions are reversed, and suspend models that are usually wrong
- Cigna PXDX batch claim denials (reported) (2022; alleged (not proven)). ProPublica, citing internal Cigna records, reported that Cigna's PXDX system was used to reject more than 300,000 claims over two months in 2022, with physicians spending an average of 1.2 seconds on each. Cigna disputes the reporting; related lawsuits are ongoing. Source: ProPublica / The Capitol Forum · evidence grade: press of record · cited by Make human review of adverse AI decisions substantive, not nominal
Rule id uk-ico-ai.ai-adm-safeguards-dpia · review status: primary source derived