NIST AI RMF 1.0

The guard to add

Check a runtime kill switch before each automated AI action, add a circuit breaker and operator override, and emit monitored action metrics with an incident route.

In the agent loop, worker, or scheduled job that applies model output, check a runtime disengage control before every action: a feature flag or config value (for example a LaunchDarkly flag ai-agent-enabled or AI_AGENT_ENABLED) that operators can flip without a deploy, plus a circuit breaker that halts the loop when anomalies cross a limit. Each action emits a span or counter (OpenTelemetry, Prometheus) wired to an alert and an incident route, an operator override endpoint can cancel or reverse queued actions, and a decommission runbook in the repository says how to retire the model and what takes its place.

Where it goes: 15 agent action surface, 3 config and feature flags, 10 logs and telemetry.

for call in response.tool_calls:
    result = TOOLS[call.name](**call.arguments)

ld = ldclient.get()
ctx = Context.builder('support-agent').kind('service').build()
tracer = trace.get_tracer('agent')

for call in response.tool_calls:
    if not ld.variation('ai-agent-enabled', ctx, False):   # operators flip it, no deploy
        raise AgentDisengaged('kill switch off')
    if breaker.is_open():                                   # e.g. error or refund spike
        raise AgentDisengaged('circuit breaker open')
    with tracer.start_as_current_span('agent.tool_call') as span:
        span.set_attribute('tool.name', call.name)
        result = TOOLS[call.name](**call.arguments)
    ACTIONS.labels(tool=call.name).inc()                    # alerted in Prometheus

Control: No override/decommission mechanism for deployed AI. The same guard addresses 1 item. Engineering guidance, not legal advice.

The guard to add

Organizational artifact to keep (not verifiable from code); the guard is the record, its owner and its upkeep.

Run and keep a documented fairness evaluation of the AI decision system, with per-group results, the method used, and the decision taken on each gap.

A fairness evaluation report owned by the system owner and kept with each model version (reports/fairness/<model_version>.md or the model card): which groups were evaluated, the metrics (selection rate, error-rate gaps, disparate-impact ratio), the data used, the results, the thresholds, and the mitigation or acceptance decision with who signed off. It is refreshed at each release and on a set cadence. Where the code can show it, fairness tests in the test suite assert that metrics stay within the agreed thresholds, and a CI check confirms a report exists for the version being deployed.

Where it goes: 11 CI/CD pipeline, 12 repository artifacts, 13 tests and evals.

def test_model_accuracy():
    assert accuracy_score(y_test, model.predict(X_test)) > 0.85

def test_model_accuracy():
    assert accuracy_score(y_test, model.predict(X_test)) > 0.85

def test_selection_rate_ratio_by_group():
    y_pred = model.predict(X_test)
    mf = MetricFrame(metrics=selection_rate, y_true=y_test, y_pred=y_pred,
                     sensitive_features=A_test)
    assert mf.ratio() >= MIN_SELECTION_RATIO, mf.by_group.to_dict()   # threshold from the evaluation plan

Control: No documented fairness/bias evaluation of AI. The same guard addresses 1 item. Engineering guidance, not legal advice.