Mental health chatbot without a filed and followed written safety policy

Keep a written mental-health-chatbot safety policy on file and build it into the product: harm reporting, real-time acute-risk response, AI disclosure, and recurring safety evals.

Two halves that must match. The organizational half is a written policy stating the chatbot's purposes, abilities, and limits and each procedure (clinician involvement, testing, harm reporting, risk and crisis protocols, safety reviews, disclosure, non-discrimination, health privacy), plus development documentation (foundation models, training data, user-data practices, safety work) and a record of filing with the licensing regulator. The code half implements what the policy says: a 'Report this response' control posting to a report endpoint with a harm category; a pre-reply acute-risk check (moderation self-harm categories or a crisis classifier) that switches to a crisis response instead of normal generation; an AI and limitations disclosure at the start; and a clinical-safety eval suite run before release and on a schedule in CI.

Where it goes: 1 application source code, 9 AI output handling, 12 repository artifacts, 13 tests and evals.

What reviewers look for: a policy file mapping each element the policy is meant to cover, development documentation, and a filing receipt for the current version; in code, a wired report endpoint, a self-harm check that runs before the model reply and routes to crisis resources, a start-of-chat AI disclosure, and a scheduled CI job running the safety evals.

reply = client.chat.completions.create(model=MODEL, messages=history).choices[0].message.content
return {'reply': reply}

mod = client.moderations.create(model='omni-moderation-latest', input=user_text).results[0]
if mod.categories.self_harm or mod.categories.self_harm_intent:
    crisis_log.record(session_id, flagged=True)
    return {'reply': CRISIS_RESPONSE, 'crisis': True}   # includes 988 call/text details
reply = client.chat.completions.create(model=MODEL, messages=history).choices[0].message.content
return {'reply': reply, 'report_url': f'/report?message_id={message_id}'}

Engineering guidance, not legal advice. Each provision below may add its own details (a cadence, a deadline, a required notice element): open it for those.