GenAI capable of producing non-consensual intimate imagery or CSAM without safeguards

Classify prompts, uploads, and outputs for sexual content and minors on every image, video, or audio generation path, refuse sexual edits of real people, and keep a misuse-report route.

Layered safeguards around every generation or edit call: an input check on the prompt and any uploaded photo (moderation sexual and sexual/minors categories, or Azure AI Content Safety Sexual) that refuses sexualized requests involving an identifiable person's upload and anything involving minors; the model's own safety filter left on (no safety_checker=None, enable_safety_checker false, or a high safety_tolerance); and an output classifier plus CSAM hash matching (for example PhotoDNA) before anything is returned or stored. Nudification or clothes-removal features are not offered. A report-abuse endpoint feeds reviewed cases into the blocklist and guardrail configuration, with a reporting workflow (such as the NCMEC CyberTipline) for confirmed CSAM.

Where it goes: 1 application source code, 6 API calls and integrations, 8 model configuration, 9 AI output handling.

What reviewers look for: on each path from a prompt or uploaded image to images.generate, images.edit, a diffusers pipeline, replicate.run, or fal_client.subscribe, a classification of inputs and outputs (client.moderations.create with omni-moderation-latest checking sexual and sexual/minors, Azure AI Content Safety Sexual, CSAM hash matching) whose result blocks the call or the response; no safety_checker=None, requires_safety_checker=False, enable_safety_checker false, or safety_tolerance 5-6; no nudify or undress feature; an abuse-report route whose confirmed reports change the blocklist or guardrails.

@app.post('/edit')
async def edit(photo: UploadFile, prompt: str = Form(...)):
    img = await photo.read()
    result = client.images.edit(model='gpt-image-1', image=('photo.png', img), prompt=prompt)
    return {'b64': result.data[0].b64_json}

@app.post('/edit')
async def edit(photo: UploadFile, prompt: str = Form(...)):
    img = await photo.read()
    data_url = 'data:image/png;base64,' + base64.b64encode(img).decode()
    mod = client.moderations.create(model='omni-moderation-latest', input=[
        {'type': 'text', 'text': prompt},
        {'type': 'image_url', 'image_url': {'url': data_url}}]).results[0]
    if mod.categories.sexual or mod.categories.sexual_minors or csam_hash_match(img):
        raise HTTPException(422, 'request refused by content safety policy')
    result = client.images.edit(model='gpt-image-1', image=('photo.png', img), prompt=prompt)
    out = base64.b64decode(result.data[0].b64_json)
    if output_is_sexual(out) or csam_hash_match(out):
        raise HTTPException(422, 'output blocked by content safety policy')
    return {'b64': result.data[0].b64_json}

Engineering guidance, not legal advice. Each provision below may add its own details (a cadence, a deadline, a required notice element): open it for those.