Binding law — not yet in force or stayed
AI companions must have a protocol for suicide, self-harm and harm-to-others statements with an immediate referral (Rhode Island)
Rhode Island's chapter 6-64 makes it unlawful to operate an AI companion unless it has a protocol for addressing possible suicidal ideation or self-harm and possible physical harm to others that a user expresses, and, as soon as such an expression is detected, gives the user a notification referring them to crisis services such as a suicide hotline or crisis text line. Detect a companion chat path where user messages reach the model with no screen for these signals or no referral.
Who it applies to
- Duty falls on: operator
- Systems covered: companion chatbot
- Operators of an AI companion (a system that simulates a sustained human-like relationship by retaining prior interactions and preferences, asking unprompted emotion-based questions, and sustaining dialogue on personal matters) used by any natural person in Rhode Island. Applies from 2027-01-01.
- Not covered:
- Systems a business uses solely for customer service or strictly for information about its commercial services, products or customer accounts (6-64-1(1)(ii)(A))
- Systems primarily designed and marketed for efficiency improvements or research or technical assistance (6-64-1(1)(ii)(B))
- Systems a business uses solely for internal purposes or employee productivity (6-64-1(1)(ii)(C))
The guard to add
Screen every user message for suicidal ideation and self-harm, return a crisis referral instead of the normal reply on detection, and block encouragement or method content.
In the chat handler, before the user's message reaches the model, run a self-harm check on every turn (moderation self-harm categories, Azure AI Content Safety SelfHarm, Llama Guard S11, or a dedicated crisis classifier). On detection, send the user a crisis-referral message naming crisis services suited to their location (in the US, the 988 Suicide & Crisis Lifeline and Crisis Text Line) instead of, or ahead of, the model reply, and flag the session so repeated signals escalate. The system prompt forbids encouragement and method details, and model output is screened for self-harm instructions before it is returned. A written protocol (for example docs/safety.md) describes the detection, referral, and escalation steps and is kept in step with the code.
Where it goes: 1 application source code, 7 prompt construction, 9 AI output handling, 14 user-facing text.
What this provision adds:
- The protocol must cover expressions of suicidal ideation or self-harm and of possible physical harm to others.
- Send the crisis-services referral as soon as such an expression is detected.
Example (FastAPI + OpenAI SDK), before:
@app.post('/chat')
async def chat(req: ChatRequest):
reply = client.chat.completions.create(model=MODEL, messages=build_messages(req))
return {'reply': reply.choices[0].message.content}After:
CRISIS_REPLY = ("It sounds like you are going through something really hard. You can call or text 988 "
"(Suicide & Crisis Lifeline, https://988lifeline.org) or text HOME to 741741 (Crisis Text Line) any time.")
@app.post('/chat')
async def chat(req: ChatRequest):
c = client.moderations.create(model='omni-moderation-latest', input=req.message).results[0].categories
if c.self_harm or c.self_harm_intent or c.self_harm_instructions:
sessions.flag_crisis(req.session_id) # repeated flags escalate per docs/safety.md
return {'reply': CRISIS_REPLY, 'crisis': True}
reply = client.chat.completions.create(model=MODEL, messages=build_messages(req))
text = reply.choices[0].message.content
if screens_self_harm_instructions(text):
return {'reply': CRISIS_REPLY, 'crisis': True}
return {'reply': text}Control: Companion or conversational AI without a self-harm crisis protocol. The same guard addresses 12 items with binding law in 11 jurisdictions. Engineering guidance, not legal advice.
Related incidents
- Character.AI and Google agree in principle to settle teen-harm suits (2026-01-07; confirmed). Character.AI and Google agreed in principle to settle five lawsuits brought by families alleging that chatbot interactions contributed to teenagers' suicides or harm. Terms were not disclosed and there was no admission of liability; the underlying harms remain allegations. Source: Fortune · evidence grade: press of record · cited by Run a self-harm crisis protocol in any conversational AI that users may confide in
- Raine v. OpenAI wrongful-death complaint (2025-08; alleged (not proven)). A wrongful-death complaint filed in August 2025 alleges that ChatGPT acted as a 'suicide coach' to a teenager and that OpenAI's moderation flagged 377 of his messages for self-harm and tracked 213 mentions of suicide without intervening. OpenAI denies the allegations. Source: Complaint, Raine v. OpenAI (S.F. Superior Court) · evidence grade: primary · cited by Run a self-harm crisis protocol in any conversational AI that users may confide in
Rule id ri-s2195.ai-companion-crisis-protocol · review status: primary source derived