Standard / soft law
AI systems should provide human oversight (HITL/HOTL/HIC) and the ability to intervene (EU ALTAI)
Per EU ALTAI Requirement 1 (Human Agency and Oversight), AI systems should be designed with proper oversight mechanisms — human-in-the-loop, human-on-the-loop, or human-in-command — allowing humans to make informed decisions, intervene, and override, especially for decisions affecting fundamental rights. Detect an AI decision path with no human-oversight/override affordance.
Who it applies to
- Duty falls on: developer, deployer
- Systems covered: automated decision, high risk
- AI systems developed, deployed, procured, or used in the EU. Voluntary self-assessment; the EU AI Act later makes human oversight binding for high-risk systems (Art. 14). ALTAI is the AI Act's precursor checklist.
The guard to add
Give people who receive an adverse AI-influenced decision a way to see and correct the data used and to request human review that can change the outcome.
Built into the adverse-outcome path: when a decision is adverse, the system saves the personal-data inputs the model used with the decision, and the letter or screen that communicates it links to two routes. A data access and correction route (GET/PATCH /me/data, correct_input_data) shows those inputs and accepts corrections of inaccurate data, which send the decision back for re-run or review; a reconsideration route (POST /decisions/{id}/reconsideration, request_human_review) places the case in a review queue where a reviewer with authority to change the outcome records reviewer_id and override_reason.
Where it goes: 1 application source code, 9 AI output handling, 15 agent action surface, 2 data models.
Example (FastAPI), before:
if result['decision'] == 'denied':
applications.save(app_id, status='denied')
send_email(applicant.email, render('denial.txt', applicant=applicant))After:
if result['decision'] == 'denied':
applications.save(app_id, status='denied', inputs_used=features, model_version=MODEL_VERSION)
send_email(applicant.email, render('denial.txt', applicant=applicant,
data_url=f'{BASE}/me/data', review_url=f'{BASE}/decisions/{app_id}/reconsideration'))
@app.patch('/me/data')
def correct_input_data(fix: DataCorrection, user=Depends(current_user)):
corrections.create(user_id=user.id, field=fix.field, value=fix.value)
review_queue.enqueue(user.latest_decision_id, reason='data_corrected')
@app.post('/decisions/{decision_id}/reconsideration')
def request_human_review(decision_id: str, user=Depends(current_user)):
review_queue.enqueue(decision_id, reason='consumer_request') # reviewer can change the outcomeControl: No human review/data-correction path after adverse ADMT decision. The same guard addresses 2 items with binding law in 1 jurisdiction. Engineering guidance, not legal advice.
Related incidents
No guardrail sits on this exact control; these incidents are cited by guardrails on related controls.
- UnitedHealth nH Predict claim-denial litigation (2023-11; alleged (not proven)). A class action filed in November 2023 alleges that UnitedHealth's nH Predict model had a 90% error rate, measured by denials reversed on appeal, while only about 0.2% of members appealed. UnitedHealth disputes the allegations; the litigation is ongoing. Source: STAT News · evidence grade: primary · cited by Monitor how often adverse AI decisions are reversed, and suspend models that are usually wrong
- Cigna PXDX batch claim denials (reported) (2022; alleged (not proven)). ProPublica, citing internal Cigna records, reported that Cigna's PXDX system was used to reject more than 300,000 claims over two months in 2022, with physicians spending an average of 1.2 seconds on each. Cigna disputes the reporting; related lawsuits are ongoing. Source: ProPublica / The Capitol Forum · evidence grade: press of record · cited by Make human review of adverse AI decisions substantive, not nominal
Rule id eu-altai.human-oversight · review status: primary source derived