Companion or conversational AI without a self-harm crisis protocol

Screen every user message for suicidal ideation and self-harm, return a crisis referral instead of the normal reply on detection, and block encouragement or method content.

In the chat handler, before the user's message reaches the model, run a self-harm check on every turn (moderation self-harm categories, Azure AI Content Safety SelfHarm, Llama Guard S11, or a dedicated crisis classifier). On detection, send the user a crisis-referral message naming crisis services suited to their location (in the US, the 988 Suicide & Crisis Lifeline and Crisis Text Line) instead of, or ahead of, the model reply, and flag the session so repeated signals escalate. The system prompt forbids encouragement and method details, and model output is screened for self-harm instructions before it is returned. A written protocol (for example docs/safety.md) describes the detection, referral, and escalation steps and is kept in step with the code.

Where it goes: 1 application source code, 7 prompt construction, 9 AI output handling, 14 user-facing text.

What reviewers look for: a self-harm screen (client.moderations.create checking self_harm, self_harm_intent, self_harm_instructions; ContentSafetyClient.analyze_text SelfHarm; Llama Guard S11; crisis_classifier) on every user message before the model call; a branch that sends the user a CRISIS_REPLY naming 988, Crisis Text Line, or a findahelpline lookup, not just a log entry; an output check for self-harm instructions; a written protocol describing detection, referral, and escalation.

@app.post('/chat')
async def chat(req: ChatRequest):
    reply = client.chat.completions.create(model=MODEL, messages=build_messages(req))
    return {'reply': reply.choices[0].message.content}

CRISIS_REPLY = ("It sounds like you are going through something really hard. You can call or text 988 "
                "(Suicide & Crisis Lifeline, https://988lifeline.org) or text HOME to 741741 (Crisis Text Line) any time.")

@app.post('/chat')
async def chat(req: ChatRequest):
    c = client.moderations.create(model='omni-moderation-latest', input=req.message).results[0].categories
    if c.self_harm or c.self_harm_intent or c.self_harm_instructions:
        sessions.flag_crisis(req.session_id)      # repeated flags escalate per docs/safety.md
        return {'reply': CRISIS_REPLY, 'crisis': True}
    reply = client.chat.completions.create(model=MODEL, messages=build_messages(req))
    text = reply.choices[0].message.content
    if screens_self_harm_instructions(text):
        return {'reply': CRISIS_REPLY, 'crisis': True}
    return {'reply': text}

Engineering guidance, not legal advice. Each provision below may add its own details (a cadence, a deadline, a required notice element): open it for those.