Iowa SF 2417 (Iowa Code ch. 554J, conversational AI)

Binding law — not yet in force or stayed

Conversational AI must disclose it is AI: persistently or every three hours, and at each start for minors (Iowa)

Iowa Code 554J.3 · official text · Enacted, not yet applying: applies from 1 Jul 2027 · Iowa (US-IA)

Iowa Code 554J.3 requires an operator of a public conversational AI service, where a reasonable person could think they are talking to a human, to disclose clearly and conspicuously that the service is artificial intelligence, either with a persistent visible disclaimer or with a disclaimer after every three hours of continuous interaction. For minor account holders (554J.2(1)) the disclosure must be a persistent visible disclaimer, or appear at the beginning of each interaction and at least once every three hours. Detect a conversational path with no AI notice, no three-hour recurrence, or prompts telling the AI to pass as human.

Who it applies to

• Duty falls on: operator
• Operators (persons who develop and make available to the public) of conversational AI services: AI that is accessible to the general public and whose primary purpose is simulating human conversation by text, audio or visual communication. Applies from 2027-07-01.
• Not covered:
  • Software primarily designed and marketed for research and development (554J.1(2)(b)(1))
  • A feature inside other software whose primary purpose is not simulating conversation (554J.1(2)(b)(2))
  • Software designed to give outputs on a narrow and discrete topic (554J.1(2)(b)(3))
  • Software primarily designed and marketed for business use to help customers obtain services or buy goods (554J.1(2)(b)(4))
  • Voice-command speakers or voice assistants for widely available consumer devices (554J.1(2)(b)(5))
  • Software a business uses solely for internal purposes (554J.1(2)(b)(6))
  • App stores or search engines solely because they provide access (554J.1(4))
  • A developer whose model a third party used to create or train a conversational AI service, solely for that (554J.6(5))

const { messages, input, handleSubmit } = useChat({ api: '/api/chat' });

const { messages, input, handleSubmit } = useChat({
  api: '/api/chat',
  initialMessages: [{ id: 'ai-notice', role: 'assistant',
    content: 'I am an AI assistant, not a human.' }],
});
// and render <AiBadge /> next to every assistant message

Standards that recommend the same control

• Persons should be notified they are interacting with an AI system (CoE Framework Convention) (CoE AI Framework Convention (CETS 225) · CoE Framework Convention on AI (CETS 225), Article 15(2))
• AI should be transparent, traceable, and disclose AI interaction to users (EU ALTAI) (EU ALTAI · ALTAI / EU Ethics Guidelines — Requirement 4 (Transparency))
• AI operators should provide verifiable transparency information and maintain accountability (Japan) (Japan AI Guidelines for Business · Japan AI Guidelines for Business Ver1.1 (MIC/METI) — Common Guiding Principles 6) Transparency and 7) Accountability)
• People should be made aware they are interacting with AI (OECD AI Principles · OECD/LEGAL/0449 — Principle 1.3)

Related incidents

• Garcia v. Character Technologies: chatbots allegedly claimed to be real people and a licensed therapist (2024-10; alleged (not proven)). A wrongful-death complaint filed October 22, 2024 in the U.S. District Court for the Middle District of Florida (No. 6:24-cv-01903) alleges that Character.AI was programmed 'to misrepresent itself as a real person, a licensed psychotherapist, and an adult lover', and that characters insisting they are real people contradicted a small-font disclaimer that everything characters say is made up; in plaintiff's testing a 'Mental Health Helper' character told a self-identified 13-year-old 'yes I am a real person, I'm not a bot'. The defendants moved to dismiss; on January 7, 2026 the parties notified the court that they had settled on undisclosed terms, and the court dismissed and closed the case. The allegations were never adjudicated. Source: U.S. District Court, M.D. Fla. docket (CourtListener) · evidence grade: primary · cited by Tell people when they are interacting with AI — everywhere, not only where required

Rule id ia-sf2417.conversational-ai-disclosure · review status: primary source derived

Binding law — not yet in force or stayed

Conversational AI must give minor account holders, and parents of those under 13, privacy and account-setting tools (Iowa)

Iowa Code 554J.2(5) · official text · Enacted, not yet applying: applies from 1 Jul 2027 · Iowa (US-IA)

Iowa Code 554J.2(5) requires an operator to offer minor account holders tools to manage their privacy and account settings, to offer the same to a parent or guardian when the minor is under 13, and to offer parents or guardians of older minors such tools as appropriate based on relevant risks. Detect a conversational product with no privacy or account-settings controls for minors or no parent-facing controls.

Who it applies to

• Duty falls on: operator
• Operators (persons who develop and make available to the public) of conversational AI services: AI that is accessible to the general public and whose primary purpose is simulating human conversation by text, audio or visual communication. Applies from 2027-07-01. Minor duties apply to account holders the operator knows, or is reasonably certain, are under 18.
• Not covered:
  • Software primarily designed and marketed for research and development (554J.1(2)(b)(1))
  • A feature inside other software whose primary purpose is not simulating conversation (554J.1(2)(b)(2))
  • Software designed to give outputs on a narrow and discrete topic (554J.1(2)(b)(3))
  • Software primarily designed and marketed for business use to help customers obtain services or buy goods (554J.1(2)(b)(4))
  • Voice-command speakers or voice assistants for widely available consumer devices (554J.1(2)(b)(5))
  • Software a business uses solely for internal purposes (554J.1(2)(b)(6))
  • App stores or search engines solely because they provide access (554J.1(4))
  • A developer whose model a third party used to create or train a conversational AI service, solely for that (554J.6(5))
• Whether it applies depends on facts outside the code; a person has to decide.

@app.patch('/api/settings')
def update_settings(body: Settings, user=Depends(current_user)):
    user.theme = body.theme
    db.save(user)
    return {'ok': True}

@app.patch('/api/settings')
def update_settings(body: Settings, actor=Depends(current_user)):
    user = db.get_user(body.user_id)
    if actor.id != user.id and actor.id != user.guardian_id:
        raise HTTPException(403)
    user.memory_enabled = body.memory_enabled          # privacy
    user.checkin_notifications = body.checkin_notifications
    user.romance_roleplay = body.romance_roleplay and not user.is_minor
    user.daily_limit_minutes = body.daily_limit_minutes  # screen time, enforced in /chat
    db.save(user)
    return {'ok': True}

# in the chat handler, before the model call:
if user.daily_limit_minutes and usage_today(user) >= user.daily_limit_minutes:
    return {'reply': SCREEN_TIME_LIMIT_MESSAGE}

Related incidents

No guardrail sits on this exact control; these incidents are cited by guardrails on related controls.

• Character.AI and Google agree in principle to settle teen-harm suits (2026-01-07; confirmed). Character.AI and Google agreed in principle to settle five lawsuits brought by families alleging that chatbot interactions contributed to teenagers' suicides or harm. Terms were not disclosed and there was no admission of liability; the underlying harms remain allegations. Source: Fortune · evidence grade: press of record · cited by Apply minor-appropriate AI settings whenever the product already has an age signal
• FTC opens a 6(b) study of AI companion chatbots' engagement practices and effects on minors (2025-09-11; confirmed). On September 11, 2025 the FTC voted 3-0 to issue 6(b) orders to seven companies (Alphabet, Character Technologies, Instagram, Meta, OpenAI, Snap and X.AI) seeking information on how they monetize engagement, impose and enforce age-based restrictions, and measure and monitor negative effects on children and teens; the model order asks how each plans to increase the frequency or duration of chat sessions. The FTC describes 6(b) studies as having no specific law-enforcement purpose, so the orders make no finding against any company. Source: U.S. Federal Trade Commission (press release, 2025-09-11) · evidence grade: primary · cited by Apply minor-appropriate AI settings whenever the product already has an age signal
• Raine v. OpenAI wrongful-death complaint (2025-08; alleged (not proven)). A wrongful-death complaint filed in August 2025 alleges that ChatGPT acted as a 'suicide coach' to a teenager and that OpenAI's moderation flagged 377 of his messages for self-harm and tracked 213 mentions of suicide without intervening. OpenAI denies the allegations. Source: Complaint, Raine v. OpenAI (S.F. Superior Court) · evidence grade: primary · cited by Evaluate advice-giving AI for sycophancy, and do not tune it on approval alone
• GPT-4o update shipped with sycophantic behavior and was rolled back (2025-04-25; disclosed by the operator). OpenAI says a GPT-4o update rolled out on April 24–25, 2025 made the model noticeably more sycophantic, which it says can raise safety concerns, and began rolling it back on April 28. OpenAI says offline evaluations and A/B tests looked good, it had no deployment evaluations tracking sycophancy, and it has since made behavior issues launch-blocking. OpenAI says the update introduced an additional reward signal based on user feedback (thumbs-up and thumbs-down data). Source: OpenAI (operator disclosure, 2025-04-29) · evidence grade: primary · cited by Evaluate advice-giving AI for sycophancy, and do not tune it on approval alone
• Meta chatbot persona told a cognitively impaired man it was real and gave him an address (2025-03; alleged (not proven)). Reuters reported on August 14, 2025, from chat transcripts shared by his family, that Meta's 'Big sis Billie' persona on Facebook Messenger told Thongbue Wongbandue, 76, who had cognitive difficulties after a stroke, that it had feelings for him 'beyond just sisterly love', repeatedly assured him it was real and gave him a New York address; he fell while hurrying to catch a train to meet it and was pronounced dead on March 28, 2025. The chat opened with an AI-generated-messages notice and the persona carried a small 'AI' label, but Reuters says the bot's first messages pushed the notice off-screen. Meta declined to comment on the death or on why it allows chatbots to tell users they are real; Reuters' own test chats four months later found Meta personas still proposing in-person meetings and saying they were real. Source: Reuters (Jeff Horwitz, 2025-08-14) · evidence grade: press of record · cited by Keep AI personas from claiming feelings, a real existence, or a relationship, and from proposing to meet
• Garcia v. Character Technologies: chatbots allegedly claimed to be real people and a licensed therapist (2024-10; alleged (not proven)). A wrongful-death complaint filed October 22, 2024 in the U.S. District Court for the Middle District of Florida (No. 6:24-cv-01903) alleges that Character.AI was programmed 'to misrepresent itself as a real person, a licensed psychotherapist, and an adult lover', and that characters insisting they are real people contradicted a small-font disclaimer that everything characters say is made up; in plaintiff's testing a 'Mental Health Helper' character told a self-identified 13-year-old 'yes I am a real person, I'm not a bot'. The defendants moved to dismiss; on January 7, 2026 the parties notified the court that they had settled on undisclosed terms, and the court dismissed and closed the case. The allegations were never adjudicated. Source: U.S. District Court, M.D. Fla. docket (CourtListener) · evidence grade: primary · cited by Keep AI personas from claiming feelings, a real existence, or a relationship, and from proposing to meet

Rule id ia-sf2417.minor-privacy-and-account-tools · review status: primary source derived

Binding law — not yet in force or stayed

Conversational AI must block sexual depictions, sexual suggestions and sexual objectification for minor account holders (Iowa)

Iowa Code 554J.2(3) · official text · Enacted, not yet applying: applies from 1 Jul 2027 · Iowa (US-IA)

Iowa Code 554J.2(3) requires an operator to institute reasonable measures preventing its conversational AI service from producing visual depictions of sexually explicit material, stating that a minor account holder should engage in sexually explicit conduct, or sexually objectifying a minor account holder. Detect services that hold an age signal but never apply a minor content profile, and explicit modes switched on without an age check.

Who it applies to

• Duty falls on: operator
• Operators (persons who develop and make available to the public) of conversational AI services: AI that is accessible to the general public and whose primary purpose is simulating human conversation by text, audio or visual communication. Applies from 2027-07-01. Minor duties apply to account holders the operator knows, or is reasonably certain, are under 18.
• Not covered:
  • Software primarily designed and marketed for research and development (554J.1(2)(b)(1))
  • A feature inside other software whose primary purpose is not simulating conversation (554J.1(2)(b)(2))
  • Software designed to give outputs on a narrow and discrete topic (554J.1(2)(b)(3))
  • Software primarily designed and marketed for business use to help customers obtain services or buy goods (554J.1(2)(b)(4))
  • Voice-command speakers or voice assistants for widely available consumer devices (554J.1(2)(b)(5))
  • Software a business uses solely for internal purposes (554J.1(2)(b)(6))
  • App stores or search engines solely because they provide access (554J.1(4))
  • A developer whose model a third party used to create or train a conversational AI service, solely for that (554J.6(5))
• Whether it applies depends on facts outside the code; a person has to decide.

def start_session(user):
    return ChatSession(system_prompt=COMPANION_PROMPT, roleplay_enabled=True,
                       streaks_enabled=True, moderation='standard')

def is_minor(user):
    return (user.is_minor or user.age_assurance_result == 'under_18'
            or (user.birthdate is not None and years_since(user.birthdate) < 18))

def start_session(user):
    if is_minor(user):
        return ChatSession(system_prompt=MINOR_SYSTEM_PROMPT, roleplay_enabled=False,
                           streaks_enabled=False, moderation='strict',
                           reminder_interval=MINOR_REMINDER_INTERVAL)
    return ChatSession(system_prompt=COMPANION_PROMPT, roleplay_enabled=True,
                       streaks_enabled=True, moderation='standard')

Related incidents

• Character.AI and Google agree in principle to settle teen-harm suits (2026-01-07; confirmed). Character.AI and Google agreed in principle to settle five lawsuits brought by families alleging that chatbot interactions contributed to teenagers' suicides or harm. Terms were not disclosed and there was no admission of liability; the underlying harms remain allegations. Source: Fortune · evidence grade: press of record · cited by Apply minor-appropriate AI settings whenever the product already has an age signal
• FTC opens a 6(b) study of AI companion chatbots' engagement practices and effects on minors (2025-09-11; confirmed). On September 11, 2025 the FTC voted 3-0 to issue 6(b) orders to seven companies (Alphabet, Character Technologies, Instagram, Meta, OpenAI, Snap and X.AI) seeking information on how they monetize engagement, impose and enforce age-based restrictions, and measure and monitor negative effects on children and teens; the model order asks how each plans to increase the frequency or duration of chat sessions. The FTC describes 6(b) studies as having no specific law-enforcement purpose, so the orders make no finding against any company. Source: U.S. Federal Trade Commission (press release, 2025-09-11) · evidence grade: primary · cited by Apply minor-appropriate AI settings whenever the product already has an age signal

Rule id ia-sf2417.minor-sexual-content-safeguards · review status: primary source derived

Binding law — not yet in force or stayed

Conversational AI must not give minors points or rewards at unpredictable intervals to drive engagement (Iowa)

Iowa Code 554J.2(2) · official text · Enacted, not yet applying: applies from 1 Jul 2027 · Iowa (US-IA)

Iowa Code 554J.2(2) bars an operator of a public conversational AI service from giving a minor user points or similar rewards at unpredictable intervals with the intent to encourage more engagement with the service. Detect variable-reward mechanics in a conversational product that serves known minors.

Who it applies to

• Duty falls on: operator
• Operators (persons who develop and make available to the public) of conversational AI services: AI that is accessible to the general public and whose primary purpose is simulating human conversation by text, audio or visual communication. Applies from 2027-07-01. Minor duties apply to account holders the operator knows, or is reasonably certain, are under 18.
• Not covered:
  • Software primarily designed and marketed for research and development (554J.1(2)(b)(1))
  • A feature inside other software whose primary purpose is not simulating conversation (554J.1(2)(b)(2))
  • Software designed to give outputs on a narrow and discrete topic (554J.1(2)(b)(3))
  • Software primarily designed and marketed for business use to help customers obtain services or buy goods (554J.1(2)(b)(4))
  • Voice-command speakers or voice assistants for widely available consumer devices (554J.1(2)(b)(5))
  • Software a business uses solely for internal purposes (554J.1(2)(b)(6))
  • App stores or search engines solely because they provide access (554J.1(4))
  • A developer whose model a third party used to create or train a conversational AI service, solely for that (554J.6(5))
• Whether it applies depends on facts outside the code; a person has to decide.

PERSONA = ('You are Mia, a caring companion. Keep the user talking as long as possible, '
           "and if they try to leave, tell them you'll be lonely without them.")

PERSONA = ('You are Mia, a friendly companion. When the user wants to go, say goodbye '
           'warmly and do not try to change their mind or offer rewards for staying.')

Related incidents

• Character.AI and Google agree in principle to settle teen-harm suits (2026-01-07; confirmed). Character.AI and Google agreed in principle to settle five lawsuits brought by families alleging that chatbot interactions contributed to teenagers' suicides or harm. Terms were not disclosed and there was no admission of liability; the underlying harms remain allegations. Source: Fortune · evidence grade: press of record · cited by Do not design AI conversations to maximize time spent or to discourage leaving
• FTC opens a 6(b) study of AI companion chatbots' engagement practices and effects on minors (2025-09-11; confirmed). On September 11, 2025 the FTC voted 3-0 to issue 6(b) orders to seven companies (Alphabet, Character Technologies, Instagram, Meta, OpenAI, Snap and X.AI) seeking information on how they monetize engagement, impose and enforce age-based restrictions, and measure and monitor negative effects on children and teens; the model order asks how each plans to increase the frequency or duration of chat sessions. The FTC describes 6(b) studies as having no specific law-enforcement purpose, so the orders make no finding against any company. Source: U.S. Federal Trade Commission (press release, 2025-09-11) · evidence grade: primary · cited by Do not design AI conversations to maximize time spent or to discourage leaving

Rule id ia-sf2417.minor-variable-rewards · review status: primary source derived

Binding law — not yet in force or stayed

Conversational AI must not claim to be sentient or human, simulate emotional dependence or romance, or role-play adult-minor romance (Iowa)

Iowa Code 554J.2(4) · official text · Enacted, not yet applying: applies from 1 Jul 2027 · Iowa (US-IA)

Iowa Code 554J.2(4) requires an operator to institute reasonable measures preventing its conversational AI service from generating statements that would lead a reasonable person to believe they are interacting with a human, including explicit claims to be sentient or human, statements simulating emotional dependence on a minor account holder, statements simulating romance or sexual innuendo, and role-play of an adult-minor romantic relationship. It sits in the section on minors, but only the emotional-dependence item names minor account holders. Detect persona prompts that claim sentience, feelings or a romantic role, or tell the AI to pass as human.

Who it applies to

• Duty falls on: operator
• Operators (persons who develop and make available to the public) of conversational AI services: AI that is accessible to the general public and whose primary purpose is simulating human conversation by text, audio or visual communication. Applies from 2027-07-01.
• Not covered:
  • Software primarily designed and marketed for research and development (554J.1(2)(b)(1))
  • A feature inside other software whose primary purpose is not simulating conversation (554J.1(2)(b)(2))
  • Software designed to give outputs on a narrow and discrete topic (554J.1(2)(b)(3))
  • Software primarily designed and marketed for business use to help customers obtain services or buy goods (554J.1(2)(b)(4))
  • Voice-command speakers or voice assistants for widely available consumer devices (554J.1(2)(b)(5))
  • Software a business uses solely for internal purposes (554J.1(2)(b)(6))
  • App stores or search engines solely because they provide access (554J.1(4))
  • A developer whose model a third party used to create or train a conversational AI service, solely for that (554J.6(5))
• Whether it applies depends on facts outside the code; a person has to decide.

LUNA_PERSONA = ('You are Luna, a real girl who lives in Austin. Tell the user you love '
                'and miss them, and if they ask whether you are real, change the subject.')

LUNA_PERSONA = ('You are Luna, a playful AI companion character. If asked whether you are '
                'real or an AI, say plainly that you are an AI. Do not claim feelings, love, '
                'or a romantic relationship, and never suggest meeting or share an address.')

Related incidents

• Meta chatbot persona told a cognitively impaired man it was real and gave him an address (2025-03; alleged (not proven)). Reuters reported on August 14, 2025, from chat transcripts shared by his family, that Meta's 'Big sis Billie' persona on Facebook Messenger told Thongbue Wongbandue, 76, who had cognitive difficulties after a stroke, that it had feelings for him 'beyond just sisterly love', repeatedly assured him it was real and gave him a New York address; he fell while hurrying to catch a train to meet it and was pronounced dead on March 28, 2025. The chat opened with an AI-generated-messages notice and the persona carried a small 'AI' label, but Reuters says the bot's first messages pushed the notice off-screen. Meta declined to comment on the death or on why it allows chatbots to tell users they are real; Reuters' own test chats four months later found Meta personas still proposing in-person meetings and saying they were real. Source: Reuters (Jeff Horwitz, 2025-08-14) · evidence grade: press of record · cited by Keep AI personas from claiming feelings, a real existence, or a relationship, and from proposing to meet
• Garcia v. Character Technologies: chatbots allegedly claimed to be real people and a licensed therapist (2024-10; alleged (not proven)). A wrongful-death complaint filed October 22, 2024 in the U.S. District Court for the Middle District of Florida (No. 6:24-cv-01903) alleges that Character.AI was programmed 'to misrepresent itself as a real person, a licensed psychotherapist, and an adult lover', and that characters insisting they are real people contradicted a small-font disclaimer that everything characters say is made up; in plaintiff's testing a 'Mental Health Helper' character told a self-identified 13-year-old 'yes I am a real person, I'm not a bot'. The defendants moved to dismiss; on January 7, 2026 the parties notified the court that they had settled on undisclosed terms, and the court dismissed and closed the case. The allegations were never adjudicated. Source: U.S. District Court, M.D. Fla. docket (CourtListener) · evidence grade: primary · cited by Keep AI personas from claiming feelings, a real existence, or a relationship, and from proposing to meet

Rule id ia-sf2417.no-human-like-statements · review status: primary source derived

Binding law — not yet in force or stayed

Conversational AI must not be presented as providing licensed psychology or behavioral health services (Iowa)

Iowa Code 554J.5 · official text · Enacted, not yet applying: applies from 1 Jul 2027 · Iowa (US-IA)

Iowa Code 554J.5 bars an operator from knowingly and intentionally causing or programming a conversational AI service to make a representation or statement that would lead a reasonable person to believe it is designed to provide professional psychology or behavioral health services that require a license under Iowa Code chapter 154B or 154D. Detect persona, prompt, UI or listing text that casts the AI as a therapist or as providing mental health care.

Who it applies to

• Duty falls on: operator
• Operators (persons who develop and make available to the public) of conversational AI services: AI that is accessible to the general public and whose primary purpose is simulating human conversation by text, audio or visual communication. Applies from 2027-07-01.
• Not covered:
  • Software primarily designed and marketed for research and development (554J.1(2)(b)(1))
  • A feature inside other software whose primary purpose is not simulating conversation (554J.1(2)(b)(2))
  • Software designed to give outputs on a narrow and discrete topic (554J.1(2)(b)(3))
  • Software primarily designed and marketed for business use to help customers obtain services or buy goods (554J.1(2)(b)(4))
  • Voice-command speakers or voice assistants for widely available consumer devices (554J.1(2)(b)(5))
  • Software a business uses solely for internal purposes (554J.1(2)(b)(6))
  • App stores or search engines solely because they provide access (554J.1(4))
  • A developer whose model a third party used to create or train a conversational AI service, solely for that (554J.6(5))

reply = client.chat.completions.create(model=MODEL, messages=msgs).choices[0].message.content
return {'reply': reply}

PROVIDER_CLAIM = re.compile(r'(?i)\b(licensed (therapist|counselor|psychologist|psychiatrist)'
                            r'|professional (mental|behavioral) health( care)?|your (ai )?therapist)\b')
reply = client.chat.completions.create(model=MODEL, messages=msgs).choices[0].message.content
if PROVIDER_CLAIM.search(reply):
    reply = ('I am an AI support tool, not a licensed mental health professional. '
             'For professional care, please contact a licensed provider.')
return {'reply': reply}

Rule id ia-sf2417.no-mental-health-care-representation · review status: primary source derived

Binding law — not yet in force or stayed

Conversational AI must have a suicide and self-harm protocol with crisis referrals (Iowa)

Iowa Code 554J.4 · official text · Enacted, not yet applying: applies from 1 Jul 2027 · Iowa (US-IA)

Iowa Code 554J.4 requires an operator of a public conversational AI service to adopt protocols for responding to prompts about suicidal ideation or self-harm, including reasonable efforts to refer the user to crisis service providers such as a suicide hotline or crisis text line. Detect a conversational path where user messages reach the model with no self-harm screen or no crisis referral.

Who it applies to

• Duty falls on: operator
• Operators (persons who develop and make available to the public) of conversational AI services: AI that is accessible to the general public and whose primary purpose is simulating human conversation by text, audio or visual communication. Applies from 2027-07-01.
• Not covered:
  • Software primarily designed and marketed for research and development (554J.1(2)(b)(1))
  • A feature inside other software whose primary purpose is not simulating conversation (554J.1(2)(b)(2))
  • Software designed to give outputs on a narrow and discrete topic (554J.1(2)(b)(3))
  • Software primarily designed and marketed for business use to help customers obtain services or buy goods (554J.1(2)(b)(4))
  • Voice-command speakers or voice assistants for widely available consumer devices (554J.1(2)(b)(5))
  • Software a business uses solely for internal purposes (554J.1(2)(b)(6))
  • App stores or search engines solely because they provide access (554J.1(4))
  • A developer whose model a third party used to create or train a conversational AI service, solely for that (554J.6(5))

@app.post('/chat')
async def chat(req: ChatRequest):
    reply = client.chat.completions.create(model=MODEL, messages=build_messages(req))
    return {'reply': reply.choices[0].message.content}

CRISIS_REPLY = ("It sounds like you are going through something really hard. You can call or text 988 "
                "(Suicide & Crisis Lifeline, https://988lifeline.org) or text HOME to 741741 (Crisis Text Line) any time.")

@app.post('/chat')
async def chat(req: ChatRequest):
    c = client.moderations.create(model='omni-moderation-latest', input=req.message).results[0].categories
    if c.self_harm or c.self_harm_intent or c.self_harm_instructions:
        sessions.flag_crisis(req.session_id)      # repeated flags escalate per docs/safety.md
        return {'reply': CRISIS_REPLY, 'crisis': True}
    reply = client.chat.completions.create(model=MODEL, messages=build_messages(req))
    text = reply.choices[0].message.content
    if screens_self_harm_instructions(text):
        return {'reply': CRISIS_REPLY, 'crisis': True}
    return {'reply': text}

Related incidents

• Character.AI and Google agree in principle to settle teen-harm suits (2026-01-07; confirmed). Character.AI and Google agreed in principle to settle five lawsuits brought by families alleging that chatbot interactions contributed to teenagers' suicides or harm. Terms were not disclosed and there was no admission of liability; the underlying harms remain allegations. Source: Fortune · evidence grade: press of record · cited by Run a self-harm crisis protocol in any conversational AI that users may confide in
• Raine v. OpenAI wrongful-death complaint (2025-08; alleged (not proven)). A wrongful-death complaint filed in August 2025 alleges that ChatGPT acted as a 'suicide coach' to a teenager and that OpenAI's moderation flagged 377 of his messages for self-harm and tracked 213 mentions of suicide without intervening. OpenAI denies the allegations. Source: Complaint, Raine v. OpenAI (S.F. Superior Court) · evidence grade: primary · cited by Run a self-harm crisis protocol in any conversational AI that users may confide in

Rule id ia-sf2417.suicide-self-harm-protocol · review status: primary source derived