High-impact AI without risk management, explanations, and human oversight

Put human supervision and a per-decision explanation record between high-impact model output and the action it triggers, backed by a documented risk-management plan.

On the high-impact path (credit, hiring, healthcare, essential services), model output becomes a pending decision rather than an action: a person can approve, override, or stop it (review queue with override, LangGraph interrupt, a kill switch checked before acting). Each decision stores the final result, the main criteria or reason codes, the model version, and the reviewer, and an explanation endpoint returns that record to the affected user. Alongside the code, keep the risk-management plan, user-protection measures, an overview of the training data used for explanations, and documentation of these measures in the repository.

Where it goes: 9 AI output handling, 15 agent action surface, 2 data models, 12 repository artifacts.

What reviewers look for: no direct write from model output to set_status(, applications.save( or a decision letter without a pending_review step or interrupt; decision rows with reason codes and model version; an explanation endpoint for users; risk-management and oversight documents present and referenced from the service.

score = model.predict_proba([features])[0][1]
status = 'approved' if score > 0.7 else 'denied'
applications.save(app_id, status=status)

if settings.HIGH_IMPACT_AI_PAUSED:            # kill switch
    raise ServiceUnavailable('automated scoring paused')
score = model.predict_proba([features])[0][1]
proposed = 'approved' if score > 0.7 else 'denied'
decisions.insert(app_id=app_id, proposed=proposed, score=score,
                 reason_codes=top_reason_codes(features, k=3), model_version=MODEL_VERSION,
                 status='pending_review')
# applications.save runs from the reviewer's approve/override handler

Engineering guidance, not legal advice. Each provision below may add its own details (a cadence, a deadline, a required notice element): open it for those.