Control
Protected-attribute proxies used as AI features without a disparity evaluation
When features that can stand in for protected traits (personal names, postal codes or census tracts, language or dialect, device or IP geolocation) feed AI ranking, pricing, content moderation, or ad targeting, outcomes are compared across the groups those features can proxy for before launch and on a schedule, and any disparity is either mitigated or justified and documented. Narrower than cond.no-documented-proxy-variable-analysis in requiring a measured outcome comparison, and broader than cond.protected-or-proxy-attribute-in-ai-decision in covering ranking, pricing, moderation, and ad targeting as well as decisions about people.
Informational data, not legal advice. Summaries are TwinEthos's own words and rules have not been reviewed by a lawyer: check the official text before relying on any of it. A guard addresses an item; adding it is not a statement that your code meets any law.
Reach
The guard to add
Consider comparing outcomes across the groups that name, postal-code, language, or geolocation features can stand in for, before launch and on a schedule.
Consider adding a disparity evaluation wherever proxy-capable features (personal names, ZIP or postal code or census tract, language, locale or dialect, IP or device geolocation) feed a model that prices, ranks, moderates, or targets ads: in the training or evaluation pipeline and as a scheduled job on live outcomes, compare price, rank position, removal rate, or ad delivery across the proxied groups (for example via census-tract demographics or BISG-inferred race and ethnicity, using fairlearn MetricFrame with sensitive_features). Prefer recording each proxy feature in the model card or a feature-review record with the measured disparity and the outcome: mitigated, coarsened, removed, or justified. Removing or coarsening the feature after a documented review is a reasonable alternative to ongoing measurement.
Where it goes: 1 application source code, 11 CI/CD pipeline, 12 repository artifacts, 13 tests and evals.
What reviewers look for: for each model whose feature list names a proxy-capable column and whose output sets a price, rank, moderation action, or ad audience, a disparity evaluation (MetricFrame, demographic_parity_difference, AIF360 metrics, Fairness Indicators, or an outcome-by-group comparison) in tests or a scheduled job, plus a model card or feature-review entry with the result and the justification or mitigation; or evidence the feature was removed or coarsened after review.
Example (scikit-learn pricing model + fairlearn), before:
features = ['basket_size', 'zip_code', 'device_type', 'visits_30d']
model = GradientBoostingRegressor().fit(train[features], train['price_multiplier'])After:
features = ['basket_size', 'zip_code', 'device_type', 'visits_30d']
model = GradientBoostingRegressor().fit(train[features], train['price_multiplier'])
# compare quoted prices across groups the ZIP feature can stand in for
groups = tract_demographics.majority_group(test['census_tract'])
mf = MetricFrame(metrics={'mean_price': lambda y_t, y_p: y_p.mean()},
y_true=test['price_multiplier'], y_pred=model.predict(test[features]),
sensitive_features=groups)
record_feature_review('zip_code', by_group=mf.by_group, gap=mf.difference())Engineering guidance, not legal advice. Each provision below may add its own details (a cadence, a deadline, a required notice element): open it for those.
Every rule this guard addresses
TwinEthos recommendation (not law) (1)
- Everywhere (*)
- Check AI ranking, pricing, moderation, and ad targeting for disparities when features can stand in for protected traits TwinEthos derivation — guardrail.ethics-evaluate-proxy-features-for-disparity · advisory
Related incidents
- Meta's automated moderation over-enforced Arabic and under-enforced Hebrew content (BSR due diligence) (2021-05; disclosed by the operator). An independent human rights due diligence by BSR, commissioned and published by Meta on September 22, 2022, found that during the May 2021 Israel-Palestine escalation Arabic content saw greater over-enforcement per user than Hebrew content and Hebrew content greater under-enforcement. BSR attributes this in part to Meta having an Arabic hostile-speech classifier but no Hebrew one, and to Arabic classifiers likely being less accurate for Palestinian Arabic. BSR found no intentional bias but 'various instances of unintentional bias' with different impacts on Palestinian and Arabic-speaking users. Meta committed to implement 10 of BSR's 21 recommendations and said it had since launched a Hebrew hostile-speech classifier. Source: Meta (operator response, 2022-09-22) · evidence grade: primary · cited by Check AI ranking, pricing, moderation, and ad targeting for disparities when features can stand in for protected traits
- Toxicity classifiers rate African American English as more offensive (2019; confirmed). University of Washington researchers reported at ACL 2019 that tweets in African American English and tweets by self-identified African Americans were up to two times more likely to be labelled offensive by hate-speech models trained on widely used datasets, and that Jigsaw's public Perspective API showed similar racial bias, rating AAE phrases as more toxic than non-AAE equivalents. Source: Sap et al., 'The Risk of Racial Bias in Hate Speech Detection', ACL 2019 (original researchers) · evidence grade: primary · cited by Check AI ranking, pricing, moderation, and ad targeting for disparities when features can stand in for protected traits
- Ride-hailing fares higher in Chicago neighborhoods with more non-white residents (researcher audit) (2018-11; alleged (not proven)). George Washington University researchers analysing Chicago's public data on more than 100 million ride-hailing trips from November 2018 to September 2019 report that trips in neighborhoods with larger non-white populations, higher poverty, younger residents and more college-educated residents were significantly associated with higher fares. They attribute this to pricing algorithms learning from demand, supply and trip duration. The finding is an observational association from census-tract data; the pricing models themselves were not examined. Source: Pandey & Caliskan, 'Disparate Impact of Artificial Intelligence Bias in Ridehailing Economy's Price Discrimination Algorithms', AIES 2021 (original researchers) · evidence grade: primary · cited by Check AI ranking, pricing, moderation, and ad targeting for disparities when features can stand in for protected traits
- Google ads suggesting arrest records served more often for Black-identifying names (2012; confirmed). Harvard researcher Latanya Sweeney searched 2,184 racially associated full names on google.com and reuters.com (a Google AdSense host) from September 24 to October 23, 2012 and found ads suggestive of an arrest record appeared more often for Black-identifying first names; on reuters.com a Black-identifying name was 25% more likely to get such an ad (statistically significant). Ads appeared regardless of whether the name had an arrest record in the advertiser's database. The paper does not determine whether the advertiser's templates or Google's click-based ad optimization caused the pattern; the advertiser, Instant Checkmate, told the author it gave Google the same ad text for groups of last names. Source: Sweeney, 'Discrimination in Online Ad Delivery' (original researcher, 2013-01-28) · evidence grade: primary · cited by Check AI ranking, pricing, moderation, and ad targeting for disparities when features can stand in for protected traits