Control
Automated employment tool without a current bias audit
An automated employment decision tool must have a bias audit within the last year, published.
Informational data, not legal advice. Summaries are TwinEthos's own words and rules have not been reviewed by a lawyer: check the official text before relying on any of it. A guard addresses an item; adding it is not a statement that your code meets any law.
Reach
Law in force in New York City (US-NY-NYC).
The guard to add
Keep a dated bias-audit record for the AEDT with a link to its published summary, and disable AEDT scoring once the audit is more than one year old.
An audit record kept next to the tool (e.g. docs/bias-audit/aedt.yaml pointing to the independent auditor's report) with the audit date, the selection or scoring rates and impact ratios per category, and the URL of the published summary; the careers-site or job-posting source links that summary. A CI check or a guard in the scoring entry point compares the audit date with today and fails the build or refuses to score when the audit is older than a year or the summary link is missing. Schedule the next audit before the current one lapses.
Where it goes: 1 application source code, 11 CI/CD pipeline, 12 repository artifacts, 14 user-facing text.
What reviewers look for: a dated audit artifact (report or record with impact_ratio calculations) within the last 12 months, a link to the published audit summary in careers-site or job-posting source, and code or CI that ties the scoring path to that date rather than an audit mentioned only in prose.
Example (Python scoring service), before:
def score_candidate(c):
return model.predict_proba([c.features])[0, 1]After:
AUDIT = yaml.safe_load(open('docs/bias-audit/aedt.yaml'))
def score_candidate(c):
audited = date.fromisoformat(str(AUDIT['audit_date']))
if date.today() - audited > timedelta(days=365) or not AUDIT.get('published_summary_url'):
raise AuditExpired('AEDT bias audit missing or older than one year; scoring disabled')
return model.predict_proba([c.features])[0, 1]Engineering guidance, not legal advice. Each provision below may add its own details (a cadence, a deadline, a required notice element): open it for those.
Every rule this guard addresses
Binding law — in force (1)
- New York City (US-NY-NYC)
Related incidents
No guardrail sits on this exact control; these incidents are cited by guardrails on related controls.
- Meta's automated moderation over-enforced Arabic and under-enforced Hebrew content (BSR due diligence) (2021-05; disclosed by the operator). An independent human rights due diligence by BSR, commissioned and published by Meta on September 22, 2022, found that during the May 2021 Israel-Palestine escalation Arabic content saw greater over-enforcement per user than Hebrew content and Hebrew content greater under-enforcement. BSR attributes this in part to Meta having an Arabic hostile-speech classifier but no Hebrew one, and to Arabic classifiers likely being less accurate for Palestinian Arabic. BSR found no intentional bias but 'various instances of unintentional bias' with different impacts on Palestinian and Arabic-speaking users. Meta committed to implement 10 of BSR's 21 recommendations and said it had since launched a Hebrew hostile-speech classifier. Source: Meta (operator response, 2022-09-22) · evidence grade: primary · cited by Check AI ranking, pricing, moderation, and ad targeting for disparities when features can stand in for protected traits
- Toxicity classifiers rate African American English as more offensive (2019; confirmed). University of Washington researchers reported at ACL 2019 that tweets in African American English and tweets by self-identified African Americans were up to two times more likely to be labelled offensive by hate-speech models trained on widely used datasets, and that Jigsaw's public Perspective API showed similar racial bias, rating AAE phrases as more toxic than non-AAE equivalents. Source: Sap et al., 'The Risk of Racial Bias in Hate Speech Detection', ACL 2019 (original researchers) · evidence grade: primary · cited by Check AI ranking, pricing, moderation, and ad targeting for disparities when features can stand in for protected traits
- Ride-hailing fares higher in Chicago neighborhoods with more non-white residents (researcher audit) (2018-11; alleged (not proven)). George Washington University researchers analysing Chicago's public data on more than 100 million ride-hailing trips from November 2018 to September 2019 report that trips in neighborhoods with larger non-white populations, higher poverty, younger residents and more college-educated residents were significantly associated with higher fares. They attribute this to pricing algorithms learning from demand, supply and trip duration. The finding is an observational association from census-tract data; the pricing models themselves were not examined. Source: Pandey & Caliskan, 'Disparate Impact of Artificial Intelligence Bias in Ridehailing Economy's Price Discrimination Algorithms', AIES 2021 (original researchers) · evidence grade: primary · cited by Check AI ranking, pricing, moderation, and ad targeting for disparities when features can stand in for protected traits
- Google ads suggesting arrest records served more often for Black-identifying names (2012; confirmed). Harvard researcher Latanya Sweeney searched 2,184 racially associated full names on google.com and reuters.com (a Google AdSense host) from September 24 to October 23, 2012 and found ads suggestive of an arrest record appeared more often for Black-identifying first names; on reuters.com a Black-identifying name was 25% more likely to get such an ad (statistically significant). Ads appeared regardless of whether the name had an arrest record in the advertiser's database. The paper does not determine whether the advertiser's templates or Google's click-based ad optimization caused the pattern; the advertiser, Instant Checkmate, told the author it gave Google the same ad text for groups of last names. Source: Sweeney, 'Discrimination in Online Ad Delivery' (original researcher, 2013-01-28) · evidence grade: primary · cited by Check AI ranking, pricing, moderation, and ad targeting for disparities when features can stand in for protected traits