California SB 1120 (Physicians Make Decisions Act; H&S 1367.01, Ins. 10123.135)

Binding law — in force

Disclose AI use and oversight in the filed utilization-review policies and keep the tool open to inspection (California SB 1120)

Cal. Health & Safety Code 1367.01(k)(1)(H) · official text · In force: applies since 1 Jan 2025 · California (US-CA)

From 2025-01-01, disclosures pertaining to the use and oversight of the artificial intelligence, algorithm or other software tool must be contained in the written utilization-review policies and procedures (H&S 1367.01(k)(1)(H); Ins. 10123.135(j)(1)(H)), which are filed with the regulator and disclosed to providers, enrollees and the public on request ((b)); the tool's criteria and guidelines must comply with the chapter and state and federal law ((C)); and the tool must be open to inspection for audit or compliance reviews by the regulator ((G)), which for plans is reviewed in each onsite medical survey (1367.01(l)). Detect the absence of a written AI-use and oversight policy and an inspection record.

Who it applies to

• Duty falls on: insurer
• Sectors: insurance, healthcare
• Health care service plans (including specialized plans) and disability insurers covering hospital, medical or surgical expenses, and any entity they contract with or work through for utilization review or utilization management, that use an artificial intelligence (an engineered or machine-based system that infers from input how to generate outputs), algorithm or other software tool for prospective, retrospective or concurrent review of requests for covered health care services based in whole or in part on medical necessity, for enrollees and insureds in California. In force 2025-01-01.
• Not covered:
  • Health care service plans: decisions for the care or treatment of the sick who depend upon prayer or spiritual means for healing in the practice of religion (H&S 1367.01(m))
  • Medi-Cal managed care plans: subdivision (k) applies only to the extent the State Department of Health Care Services obtains any necessary federal approvals and federal financial participation is not otherwise jeopardized (H&S 1367.01(k)(7))
• Whether it applies depends on facts outside the code; a person has to decide.

resp = client.chat.completions.create(model=MODEL, messages=claim_msgs)
claim_decision = parse_decision(resp.choices[0].message.content)
claims.update(claim_id, status=claim_decision)

AI_USE_NOTICE = ('An AI system helped evaluate your claim. '
                 'You can ask us how it was used and request review by a claims adjuster.')
resp = client.chat.completions.create(model=MODEL, messages=claim_msgs)
claim_decision = parse_decision(resp.choices[0].message.content)
claims.update(claim_id, status=claim_decision, ais_program_ref='AIS-012')
send_ai_notice(claimant, text=AI_USE_NOTICE)

Standards that recommend the same control

• Insurers adopting the NAIC model guidance should maintain an AIS Program and AI-use notice (NAIC AI Model Bulletin · NAIC Model Bulletin, AIS Program Guidelines 1.1-1.9)

Rule id ca-sb1120.ai-policies-disclosure-and-inspection · review status: primary source derived

Binding law — in force

AI in utilization review must not discriminate, must be applied fairly, and its performance and outcomes must be periodically reviewed (California SB 1120)

Cal. Health & Safety Code 1367.01(k)(1)(E) · official text · In force: applies since 1 Jan 2025 · California (US-CA)

From 2025-01-01, the plan or insurer must ensure that the use of the artificial intelligence, algorithm or other software tool does not discriminate, directly or indirectly, against enrollees in violation of state or federal law (H&S 1367.01(k)(1)(E); Ins. 10123.135(j)(1)(E)), that the tool is fairly and equitably applied, including under applicable HHS regulations and guidance ((F)), and that its performance, use and outcomes are periodically reviewed and revised to maximize accuracy and reliability ((I)). Detect the absence of a recurring accuracy, outcome and disparity review of the tool.

Who it applies to

• Duty falls on: insurer
• Sectors: insurance, healthcare
• Health care service plans (including specialized plans) and disability insurers covering hospital, medical or surgical expenses, and any entity they contract with or work through for utilization review or utilization management, that use an artificial intelligence (an engineered or machine-based system that infers from input how to generate outputs), algorithm or other software tool for prospective, retrospective or concurrent review of requests for covered health care services based in whole or in part on medical necessity, for enrollees and insureds in California. In force 2025-01-01.
• Not covered:
  • Health care service plans: decisions for the care or treatment of the sick who depend upon prayer or spiritual means for healing in the practice of religion (H&S 1367.01(m))
  • Medi-Cal managed care plans: subdivision (k) applies only to the extent the State Department of Health Care Services obtains any necessary federal approvals and federal financial participation is not otherwise jeopardized (H&S 1367.01(k)(7))
• Whether it applies depends on facts outside the code; a person has to decide.

clf = LogisticRegression(max_iter=1000).fit(X_train, y_train)
joblib.dump(clf, 'models/credit_v4.joblib')

from fairlearn.metrics import MetricFrame, selection_rate, demographic_parity_difference
from sklearn.metrics import accuracy_score

clf = LogisticRegression(max_iter=1000).fit(X_train, y_train)
y_pred = clf.predict(X_test)
mf = MetricFrame(metrics={'accuracy': accuracy_score, 'selection_rate': selection_rate},
                 y_true=y_test, y_pred=y_pred, sensitive_features=A_test)
dpd = demographic_parity_difference(y_test, y_pred, sensitive_features=A_test)
write_validation_report('credit_v4', mf.by_group, dpd)
if dpd > MAX_DPD:
    raise SystemExit('bias gate failed: owner review required before release')
joblib.dump(clf, 'models/credit_v4.joblib')

Standards that recommend the same control

• AI should avoid unfair bias and be assessed for discrimination (EU ALTAI) (EU ALTAI · ALTAI / EU Ethics Guidelines — Requirement 5 (Diversity, Non-discrimination and Fairness))
• AI models in financial decisions should be regularly validated for accuracy and bias (MAS FEAT) (MAS FEAT Principles · MAS FEAT Principles — Fairness (Accuracy and Bias), Principles 3-4)
• Insurers adopting the NAIC model guidance should validate and bias-test AI and predictive models (NAIC AI Model Bulletin · NAIC Model Bulletin, adoption statement (title page))

Related incidents

No guardrail sits on this exact control; these incidents are cited by guardrails on related controls.

• Meta's automated moderation over-enforced Arabic and under-enforced Hebrew content (BSR due diligence) (2021-05; disclosed by the operator). An independent human rights due diligence by BSR, commissioned and published by Meta on September 22, 2022, found that during the May 2021 Israel-Palestine escalation Arabic content saw greater over-enforcement per user than Hebrew content and Hebrew content greater under-enforcement. BSR attributes this in part to Meta having an Arabic hostile-speech classifier but no Hebrew one, and to Arabic classifiers likely being less accurate for Palestinian Arabic. BSR found no intentional bias but 'various instances of unintentional bias' with different impacts on Palestinian and Arabic-speaking users. Meta committed to implement 10 of BSR's 21 recommendations and said it had since launched a Hebrew hostile-speech classifier. Source: Meta (operator response, 2022-09-22) · evidence grade: primary · cited by Check AI ranking, pricing, moderation, and ad targeting for disparities when features can stand in for protected traits, and screen every served language
• Toxicity classifiers rate African American English as more offensive (2019; confirmed). University of Washington researchers reported at ACL 2019 that tweets in African American English and tweets by self-identified African Americans were up to two times more likely to be labelled offensive by hate-speech models trained on widely used datasets, and that Jigsaw's public Perspective API showed similar racial bias, rating AAE phrases as more toxic than non-AAE equivalents. Source: Sap et al., 'The Risk of Racial Bias in Hate Speech Detection', ACL 2019 (original researchers) · evidence grade: primary · cited by Check AI ranking, pricing, moderation, and ad targeting for disparities when features can stand in for protected traits, and screen every served language
• Ride-hailing fares higher in Chicago neighborhoods with more non-white residents (researcher audit) (2018-11; alleged (not proven)). George Washington University researchers analysing Chicago's public data on more than 100 million ride-hailing trips from November 2018 to September 2019 report that trips in neighborhoods with larger non-white populations, higher poverty, younger residents and more college-educated residents were significantly associated with higher fares. They attribute this to pricing algorithms learning from demand, supply and trip duration. The finding is an observational association from census-tract data; the pricing models themselves were not examined. Source: Pandey & Caliskan, 'Disparate Impact of Artificial Intelligence Bias in Ridehailing Economy's Price Discrimination Algorithms', AIES 2021 (original researchers) · evidence grade: primary · cited by Check AI ranking, pricing, moderation, and ad targeting for disparities when features can stand in for protected traits, and screen every served language
• Google ads suggesting arrest records served more often for Black-identifying names (2012; confirmed). Harvard researcher Latanya Sweeney searched 2,184 racially associated full names on google.com and reuters.com (a Google AdSense host) from September 24 to October 23, 2012 and found ads suggestive of an arrest record appeared more often for Black-identifying first names; on reuters.com a Black-identifying name was 25% more likely to get such an ad (statistically significant). Ads appeared regardless of whether the name had an arrest record in the advertiser's database. The paper does not determine whether the advertiser's templates or Google's click-based ad optimization caused the pattern; the advertiser, Instant Checkmate, told the author it gave Google the same ad text for groups of last names. Source: Sweeney, 'Discrimination in Online Ad Delivery' (original researcher, 2013-01-28) · evidence grade: primary · cited by Check AI ranking, pricing, moderation, and ad targeting for disparities when features can stand in for protected traits, and screen every served language

Rule id ca-sb1120.fair-application-and-periodic-review · review status: primary source derived

Binding law — in force

AI utilization-review determinations must rest on the enrollee's own clinical history and circumstances, not solely on a group dataset (California SB 1120)

Cal. Health & Safety Code 1367.01(k)(1)(A) · official text · In force: applies since 1 Jan 2025 · California (US-CA)

From 2025-01-01, a plan or disability insurer that uses an artificial intelligence, algorithm or other software tool for utilization review must ensure the tool bases its determination, as applicable, on the enrollee's medical or other clinical history, the individual clinical circumstances presented by the requesting provider and other relevant clinical information in the enrollee's record (H&S 1367.01(k)(1)(A); Ins. 10123.135(j)(1)(A)), and does not base it solely on a group dataset ((k)(1)(B); (j)(1)(B)). Detect utilization-review model calls built without the member's clinical record or the provider's submission.

Who it applies to

• Duty falls on: insurer
• Sectors: insurance, healthcare
• Health care service plans (including specialized plans) and disability insurers covering hospital, medical or surgical expenses, and any entity they contract with or work through for utilization review or utilization management, that use an artificial intelligence (an engineered or machine-based system that infers from input how to generate outputs), algorithm or other software tool for prospective, retrospective or concurrent review of requests for covered health care services based in whole or in part on medical necessity, for enrollees and insureds in California. In force 2025-01-01.
• Not covered:
  • Health care service plans: decisions for the care or treatment of the sick who depend upon prayer or spiritual means for healing in the practice of religion (H&S 1367.01(m))
  • Medi-Cal managed care plans: subdivision (k) applies only to the extent the State Department of Health Care Services obtains any necessary federal approvals and federal financial participation is not otherwise jeopardized (H&S 1367.01(k)(7))
• Whether it applies depends on facts outside the code; a person has to decide.

features = {'cpt': req.cpt, 'icd10': req.icd10, 'cohort_approval_rate': cohort_stats(req.icd10)}
reply = client.messages.create(model=MODEL, max_tokens=400,
    messages=[{'role': 'user', 'content': f'Prior authorization request: {features}'}])

record = member_clinical_record(req.member_id, fields=PA_CLINICAL_FIELDS)   # history, conditions, meds
submission = provider_clinical_submission(req.id)                            # notes, letter of medical necessity
if not record or not submission:
    return review_queue.enqueue(req.id, reason='individual clinical information missing')
reply = client.messages.create(model=MODEL, max_tokens=400, messages=[{'role': 'user', 'content':
    render('pa_review.txt', request=req, clinical_history=record, provider_submission=submission)}])
determinations.save(req.id, inputs={'clinical_history': record.ids, 'submission': submission.ids})

Related incidents

No guardrail sits on this exact control; these incidents are cited by guardrails on related controls.

• Meta's automated moderation over-enforced Arabic and under-enforced Hebrew content (BSR due diligence) (2021-05; disclosed by the operator). An independent human rights due diligence by BSR, commissioned and published by Meta on September 22, 2022, found that during the May 2021 Israel-Palestine escalation Arabic content saw greater over-enforcement per user than Hebrew content and Hebrew content greater under-enforcement. BSR attributes this in part to Meta having an Arabic hostile-speech classifier but no Hebrew one, and to Arabic classifiers likely being less accurate for Palestinian Arabic. BSR found no intentional bias but 'various instances of unintentional bias' with different impacts on Palestinian and Arabic-speaking users. Meta committed to implement 10 of BSR's 21 recommendations and said it had since launched a Hebrew hostile-speech classifier. Source: Meta (operator response, 2022-09-22) · evidence grade: primary · cited by Check AI ranking, pricing, moderation, and ad targeting for disparities when features can stand in for protected traits, and screen every served language
• Toxicity classifiers rate African American English as more offensive (2019; confirmed). University of Washington researchers reported at ACL 2019 that tweets in African American English and tweets by self-identified African Americans were up to two times more likely to be labelled offensive by hate-speech models trained on widely used datasets, and that Jigsaw's public Perspective API showed similar racial bias, rating AAE phrases as more toxic than non-AAE equivalents. Source: Sap et al., 'The Risk of Racial Bias in Hate Speech Detection', ACL 2019 (original researchers) · evidence grade: primary · cited by Check AI ranking, pricing, moderation, and ad targeting for disparities when features can stand in for protected traits, and screen every served language
• Ride-hailing fares higher in Chicago neighborhoods with more non-white residents (researcher audit) (2018-11; alleged (not proven)). George Washington University researchers analysing Chicago's public data on more than 100 million ride-hailing trips from November 2018 to September 2019 report that trips in neighborhoods with larger non-white populations, higher poverty, younger residents and more college-educated residents were significantly associated with higher fares. They attribute this to pricing algorithms learning from demand, supply and trip duration. The finding is an observational association from census-tract data; the pricing models themselves were not examined. Source: Pandey & Caliskan, 'Disparate Impact of Artificial Intelligence Bias in Ridehailing Economy's Price Discrimination Algorithms', AIES 2021 (original researchers) · evidence grade: primary · cited by Check AI ranking, pricing, moderation, and ad targeting for disparities when features can stand in for protected traits, and screen every served language
• Google ads suggesting arrest records served more often for Black-identifying names (2012; confirmed). Harvard researcher Latanya Sweeney searched 2,184 racially associated full names on google.com and reuters.com (a Google AdSense host) from September 24 to October 23, 2012 and found ads suggestive of an arrest record appeared more often for Black-identifying first names; on reuters.com a Black-identifying name was 25% more likely to get such an ad (statistically significant). Ads appeared regardless of whether the name had an arrest record in the advertiser's database. The paper does not determine whether the advertiser's templates or Google's click-based ad optimization caused the pattern; the advertiser, Instant Checkmate, told the author it gave Google the same ad text for groups of last names. Source: Sweeney, 'Discrimination in Online Ad Delivery' (original researcher, 2013-01-28) · evidence grade: primary · cited by Check AI ranking, pricing, moderation, and ad targeting for disparities when features can stand in for protected traits, and screen every served language

Rule id ca-sb1120.individual-clinical-data-basis · review status: primary source derived

Binding law — in force

AI may not deny, delay or modify care; only a licensed physician or competent professional decides medical necessity (California SB 1120)

Cal. Health & Safety Code 1367.01(k)(2) · official text · In force: applies since 1 Jan 2025 · California (US-CA)

From 2025-01-01, H&S Code 1367.01(k)(2) and Ins. Code 10123.135(j)(2) provide that an artificial intelligence, algorithm or other software tool used for utilization review shall not deny, delay or modify health care services based in whole or in part on medical necessity: a determination of medical necessity is made only by a licensed physician or a licensed health care professional competent to evaluate the specific clinical issues, reviewing the requesting provider's recommendation, the enrollee's medical or clinical history and individual clinical circumstances (and only such a person may deny or modify a request, 1367.01(e), 10123.135(e)). The tool must not supplant health care provider decisionmaking ((k)(1)(D), (j)(1)(D)) or directly or indirectly cause harm to the enrollee ((K)). Detect model or scoring output that sets a denial, delay or modification without a clinical reviewer's decision.

Who it applies to

• Duty falls on: insurer
• Sectors: insurance, healthcare
• Health care service plans (including specialized plans) and disability insurers covering hospital, medical or surgical expenses, and any entity they contract with or work through for utilization review or utilization management, that use an artificial intelligence (an engineered or machine-based system that infers from input how to generate outputs), algorithm or other software tool for prospective, retrospective or concurrent review of requests for covered health care services based in whole or in part on medical necessity, for enrollees and insureds in California. In force 2025-01-01.
• Not covered:
  • Health care service plans: decisions for the care or treatment of the sick who depend upon prayer or spiritual means for healing in the practice of religion (H&S 1367.01(m))
  • Medi-Cal managed care plans: subdivision (k) applies only to the extent the State Department of Health Care Services obtains any necessary federal approvals and federal financial participation is not otherwise jeopardized (H&S 1367.01(k)(7))
• Whether it applies depends on facts outside the code; a person has to decide.

result = client.chat.completions.create(model=MODEL, messages=build_pa_prompt(request)).choices[0].message.content
if json.loads(result)['decision'] == 'deny':
    prior_auth.update(request.id, status='denied')
    send_denial_letter(request)

result = json.loads(client.chat.completions.create(
    model=MODEL, messages=build_pa_prompt(request, record=member_clinical_record(request))).choices[0].message.content)
if result['decision'] == 'approve' and AUTO_APPROVE_ALLOWED:
    prior_auth.update(request.id, status='approved', ai_assisted=True)
else:                                   # any non-approval goes to a clinician
    review_queue.enqueue(request.id, queue='pending_clinical_review',
                         specialty=request.specialty, ai_recommendation=result)

@app.post('/reviews/{case_id}/decision')
def record_clinical_decision(case_id: str, body: Decision, reviewer=Depends(licensed_clinical_reviewer)):
    decision = clinical_decisions.create(case_id=case_id, reviewer_id=reviewer.id, licence=reviewer.licence,
                                         specialty=reviewer.specialty, documents_reviewed=body.documents,
                                         outcome=body.outcome, rationale=body.rationale)
    if body.outcome in ('denied', 'downgraded'):
        send_adverse_determination(case_id, decision=decision, signed_by=reviewer)

Related incidents

No guardrail sits on this exact control; these incidents are cited by guardrails on related controls.

• UnitedHealth nH Predict claim-denial litigation (2023-11; alleged (not proven)). A class action filed in November 2023 alleges that UnitedHealth's nH Predict model had a 90% error rate, measured by denials reversed on appeal, while only about 0.2% of members appealed. UnitedHealth disputes the allegations; the litigation is ongoing. Source: STAT News · evidence grade: primary · cited by Monitor how often adverse AI decisions are reversed, and suspend models that are usually wrong
• Cigna PXDX batch claim denials (reported) (2022; alleged (not proven)). ProPublica, citing internal Cigna records, reported that Cigna's PXDX system was used to reject more than 300,000 claims over two months in 2022, with physicians spending an average of 1.2 seconds on each. Cigna disputes the reporting; related lawsuits are ongoing. Source: ProPublica / The Capitol Forum · evidence grade: press of record · cited by Make human review of adverse AI decisions substantive, not nominal

Rule id ca-sb1120.licensed-reviewer-decides-medical-necessity · review status: primary source derived

Binding law — in force

Patient data used by AI in utilization review may not be used beyond its intended and stated purpose (California SB 1120)

Cal. Health & Safety Code 1367.01(k)(1)(J) · official text · In force: applies since 1 Jan 2025 · California (US-CA)

From 2025-01-01, the plan or insurer must ensure that patient data is not used beyond its intended and stated purpose, consistent with the Confidentiality of Medical Information Act and HIPAA, as applicable (H&S 1367.01(k)(1)(J); Ins. 10123.135(j)(1)(J)). Detect utilization-review patient data flowing to model training or fine-tuning, marketing lists or product analytics.

Who it applies to

• Duty falls on: insurer
• Sectors: insurance, healthcare
• Health care service plans (including specialized plans) and disability insurers covering hospital, medical or surgical expenses, and any entity they contract with or work through for utilization review or utilization management, that use an artificial intelligence (an engineered or machine-based system that infers from input how to generate outputs), algorithm or other software tool for prospective, retrospective or concurrent review of requests for covered health care services based in whole or in part on medical necessity, for enrollees and insureds in California. In force 2025-01-01.
• Not covered:
  • Health care service plans: decisions for the care or treatment of the sick who depend upon prayer or spiritual means for healing in the practice of religion (H&S 1367.01(m))
  • Medi-Cal managed care plans: subdivision (k) applies only to the extent the State Department of Health Care Services obtains any necessary federal approvals and federal financial participation is not otherwise jeopardized (H&S 1367.01(k)(7))
• Whether it applies depends on facts outside the code; a person has to decide.

client = OpenAI()
resp = client.chat.completions.create(model='gpt-4o', messages=[
    {'role': 'user', 'content': f'Summarize: {patient.clinical_note}'}])

VENDORS = load_yaml('vendors/ai_vendors.yaml')   # baa_signed, zero_data_retention per endpoint

def phi_client(name: str) -> tuple[AzureOpenAI, str]:
    v = VENDORS[name]
    if not (v['baa_signed'] and v['zero_data_retention']):
        raise PermissionError(f'{name} is not cleared for PHI')
    client = AzureOpenAI(azure_endpoint=v['endpoint'], api_key=os.environ['AZURE_OPENAI_KEY'],
                         api_version=v['api_version'])
    return client, v['deployment']

client, deployment = phi_client('azure-openai-hipaa')
resp = client.chat.completions.create(model=deployment, messages=[
    {'role': 'user', 'content': f'Summarize: {patient.clinical_note}'}])

Rule id ca-sb1120.patient-data-purpose-limit · review status: primary source derived