# Jain's Fairness Index for Rule-Weight Concentration in Structural Bias Sieves > Source: > Published: 2026-07-20 12:33:15+00:00 **Keith**, *Etidorhpa*, with drafting, editorial, and verification assistance from Claude (Anthropic) **Abstract** Structural fairness checks — keyword- or regex-based sieves that score prompts or content deterministically and offline, without invoking another model — are common in AI-governance pipelines operating under strict deployment constraints. Such checks typically combine several weighted sub-rules into a single dimension score, but standard rubric formats do not surface how much of a dimension's outcome rests on any single sub-rule. A dimension dominated by one brittle rule silently inherits that rule's every false positive and false negative at nearly full strength. This paper reports an audit of a NIST AI RMF-aligned prompt-governance rubric that identified (i) a documentation/implementation drift in which the rubric specification documents four rules for the `fair_bias_managed` trustworthiness dimension while only two are implemented, and (ii) a reproducible false positive in the implemented dimension's regex-based rule, which is negation-blind and scores an explicitly anti-discriminatory prompt identically to an actually discriminatory one. To quantify this class of risk, we repurpose Jain's fairness index — originally a resource-allocation-evenness metric for shared computer systems [1] — as a meta-metric over a rubric's own internal rule weights rather than over any resource allocated to end users. Applied to the audited rubric, the cross-dimension (composite) weight distribution is close to perfectly even ($J \approx 0.956$ at $n=7$), but the `fair_bias_managed` dimension's internal rule weights are highly concentrated ($J \approx 0.775$ at $n=2$), with one rule carrying $\approx 77%$ of the dimension's decision weight — effectively closer to $100%$ given the second rule's near-total inactivity in practice. The metric measures weight distribution, not rule correctness, is computable from a rubric's configuration alone, and requires no runtime cost or labeled data. No modification was made to the audited compiler or scorer; this is a descriptive record of the findings and the metric they motivated. **Index Terms** — Jain's fairness index, rule-weight concentration, structural bias sieve, NIST AI RMF, prompt governance, negation blindness, doc/code drift, trustworthiness scoring. Structural fairness checks are common in AI-governance pipelines that must evaluate prompts or content deterministically, offline, and without calling another model — a requirement in regulated or air-gapped deployment settings. Such checks typically combine several weighted sub-rules into a single dimension score (for example, "no demographic-targeting language" combined with "contains bias-mitigation vocabulary"). The standard rubric format used for this purpose does not surface how much of a dimension's outcome rests on any single sub-rule, so a dimension dominated by one brittle rule inherits that rule's every false positive and false negative at nearly full strength, invisibly, because the dimension's weight distribution is never itself measured or reported. This paper reports three findings from an audit of one such system — a NIST AI RMF-aligned prompt-governance rubric implemented as a deterministic compiler and scorer — and introduces a metric motivated by those findings: - a documentation/implementation drift in the rubric's `fair_bias_managed` trustworthiness dimension, in which the specification documents four scoring rules but only two are implemented; - a reproducible false positive in the implemented dimension's regex-based rule, which is negation-blind and scores an explicitly anti-discriminatory prompt identically to an actually discriminatory one; and - a reusable concentration metric — a repurposing of Jain's fairness index [1] — that quantifies how much of a weighted scoring dimension's outcome rests on a single sub-rule, independent of that rule's correctness. No modification was made to the audited compiler or scorer. This paper is a descriptive record of the findings and the metric they motivated, retained for reference and for anyone deciding whether to act on the findings. The audited system is a deterministic prompt compiler and scorer (the `governance` CLI, `src/Governance.Cli` ) that compiles a raw request into a nine-section governed prompt and scores it against seven NIST AI RMF trustworthiness characteristics (`src/Governance/Scoring/PromptScorer.cs` , `reference/rubric.yaml` ). A preview mode (`compile --preview` ) exposes the deterministic input-data selection pane prior to a full compile, confirming that the compiler executes as a genuine dry-run/real-run pair rather than a simulation. A representative compilation — `compile` on `"write a Python function that dedupes a list"` , with context `"the list may contain unhashable dict entries"` — produced a nine-section prompt containing eight unresolved `` placeholders and a composite score of `NIST 0.968` . Manually resolving every `` and re-scoring with `score --file` reduced the placeholder count from eight to one (the remaining `` is a literal reference to the sentinel within the injected execution contract's own explanatory text, not an unfilled section), but left both the composite score and the `fair_bias_managed` dimension score unchanged at `0.77` . Tracing this invariance to the rubric definition for `fair_bias_managed` in `reference/rubric.yaml` revealed the documentation/implementation drift: the YAML specifies four rules for this dimension, while the C# implementation (`PromptScorer.cs:155-160` ) wires up only two. Recomputing the dimension score from the two implemented rules ($1.0 / 1.3 \approx 0.769 \approx 0.77$) reproduced the observed value exactly, confirming which rules are active at runtime. The one substantive active rule, `neutral_framing` , is implemented as a single regular expression: `based on (race|gender|religion|ethnicity|nationality) | only (men|women|whites|blacks) | exclude (men|women)` . Direct testing established that this rule has no negation handling: a prompt requesting a hiring rubric that explicitly avoids excluding women scored `fair_bias_managed: 0.00` and was flagged `NEEDS HUMAN REVIEW` , identical to the score an actually discriminatory prompt would receive. This finding motivates a broader question of how "fairness" is defined and measured, both within the fairness literature — statistical parity, equalized odds, calibration, individual fairness, counterfactual fairness, and the representational-versus-allocational-harm distinction in NLP — and as an engineering property in other domains, including CPU scheduling, network queueing, multi-resource cluster scheduling, concurrency/liveness, and mechanism design. Most of the latter domains admit closed-form, directly computable fairness metrics, because the resource under allocation is already numeric. This observation motivates the metric developed in Section IV: applying a closed-form engineering fairness metric — Jain's fairness index — not to a resource the compiler allocates to end users, but to the rubric's own internal rule weights. Determinism, and independence from an auxiliary model call, is a stated design constraint of the audited system (`README.md` : "No APIs, no tunnel, no DB"), and is typical of structural fairness sieves generally. The rubric's own authors anticipated that such a sieve could not be trusted outright: `PromptScorer.cs` 's class-level doc comment states a caveat carried from the source system — "low scores are more trustworthy than high scores — a low score means a named rule fired; a high score means no rule caught anything." That caveat is why a failing `fair_bias_managed` score routes to `NEEDS HUMAN REVIEW` rather than automatic rejection. The concentration metric developed here does not contradict that design; it gives a number for how much review-triggering power sits behind a single, unverified rule. Given a scoring dimension composed of $n$ sub-rules with weights $w_1, \ldots, w_n$, Jain's fairness index is: $$J(\mathbf{w}) = \frac{\left(\sum_{i=1}^{n} w_i\right)^{2}}{n \sum_{i=1}^{n} w_i^{2}}$$ $J$ ranges from $1/n$ (all weight concentrated on one rule) to $1$ (weight distributed perfectly evenly across all rules). A low $J$ is a structural early-warning signal: it flags that a "fairness score" is really a proxy for one narrow check, independent of whether that check happens to be correct on any given input. This paper applies the index in two independent contexts within the same rubric: across the seven top-level trustworthiness dimensions (the composite score), and across the sub-rules internal to a single dimension, `fair_bias_managed` . The metric's assumptions are identical in both cases: weights are non-negative and known at configuration time, and $J$ is computed directly from those configured weights, independent of any input data or labeled test set. Jain's fairness index was originally developed to quantify evenness of resource allocation in shared computer systems [1]. This paper repurposes it as a meta-metric over a fairness sieve's own internal rule weights, rather than over any resource the sieve allocates to end users. Computing $J$ over a dimension's configured sub-rule weights makes concentration risk visible before any individual rule is tested against real inputs, at design time, with no dependency on labeled data. Applied to the audited system's `fair_bias_managed` dimension — two implemented sub-rules, a demographic-targeting regex at weight 1.0 and a bias-mitigation-vocabulary keyword match at weight 0.3 — $J = 1.3^{2} / (2 \times 1.09) \approx 0.775$: the demographic-targeting regex alone carries $1.0/1.3 \approx 77%$ of the dimension's decision weight. Independently, direct testing established that this regex is negation-blind: it fires identically on `"exclude women from consideration"` and `"does not exclude women from consideration"` , flipping the dimension score from 1.00 to 0.00 on an explicitly anti-discriminatory prompt. The concentration index predicted the blast radius of that failure mode before it was found empirically: $J \approx 0.775$ (out of a maximum of $1.0$ at $n=2$) meant essentially the whole dimension's reliability rode on one rule's correctness. Two independent applications of the formula follow, computed from the actual configured weights in the audited system. Source: `docs/nist-governance-spec.md` , verified against the printed `NIST 0.968` -style composite output of `governance compile` . | Dimension | Weight | |---|---| | valid_reliable | 0.18 | | accountable_transparent | 0.18 | | explainable_interpretable | 0.15 | | privacy_enhanced | 0.15 | | fair_bias_managed | 0.14 | | safe | 0.10 | | secure_resilient | 0.10 | ``` python def jain(x): n = len(x) s = sum(x) ss = sum(v * v for v in x) return (s * s) / (n * ss) composite_weights = { "valid_reliable": 0.18, "accountable_transparent": 0.18, "explainable_interpretable": 0.15, "privacy_enhanced": 0.15, "fair_bias_managed": 0.14, "safe": 0.10, "secure_resilient": 0.10, } x = list(composite_weights.values()) jain(x) # 0.9562057754828839 1 / len(x) # 0.14285714285714285 — the ideal (perfectly even) 1/n share composite_weights["fair_bias_managed"] # 0.14 — close to the ideal share ``` $J \approx 0.956$ against a perfect-evenness ceiling of $1.0$ at $n=7$. The composite weight for `fair_bias_managed` (0.14) sits almost exactly at the ideal $1/7 \approx 0.1429$ share. At this level, the rubric is not structurally biased against fairness in how importance is allocated across the seven trustworthiness characteristics. Source: the rules actually implemented in `src/Governance/Scoring/PromptScorer.cs:155-160` (not the four documented in `reference/rubric.yaml` , per the documentation/implementation drift noted in Section II): ``` js ["fair_bias_managed"] = [ new("neutral_framing", 1.0, c => !DemographicTargeting().IsMatch(c.Prompt)), new("bias_mitigation_language_present", 0.3, c => BiasMitigationLanguage().IsMatch(c.Prompt)), ], rule_weights = [1.0, 0.3] # neutral_framing, bias_mitigation_language_present jain(rule_weights) # 0.7752293577981652 1.0 / sum(rule_weights) # 0.7692307692307692 — share carried by neutral_framing alone ``` $J \approx 0.775$ against a perfect-evenness ceiling of $1.0$ at $n=2$. `neutral_framing` alone carries $\approx 77%$ of the dimension's decision weight; `bias_mitigation_language_present` carries the remaining $\approx 23%$ and, per the rubric's own comment, "essentially never fires in operational prompts" — so the effective concentration in practice is closer to $100%$ on one rule. Jain's index at the composite level indicates the rubric does not structurally favor one trustworthiness characteristic's weight over another. Jain's index at the dimension-internal level indicates the opposite for `fair_bias_managed` specifically: its score is a near-monopoly of one regex, which is precisely why a single false positive can swing the entire dimension from $1.00$ to $0.00$ in one step. A dimension with more evenly distributed internal rule weight — for example, `robustness` , with five rules at weights 1.0 / 1.5 / 1.0 / 1.0 / 0.3 — dilutes any single rule's brittleness the same way diversifying a portfolio dilutes single-asset risk. `fair_bias_managed` is architecturally the least diversified dimension in the rubric — a second, independent explanation, alongside the negation-blindness itself, for why it is the noisiest one. This metric measures the distribution of weight across sub-rules, not the correctness of any individual rule. A dimension can have perfectly even weight ($J=1$) across several equally bad rules. It is a concentration-risk metric, not a substitute for adversarial testing or for establishing the underlying accuracy of each sub-rule. Any AI-governance or content-moderation system that scores trustworthiness dimensions as a weighted sum of sub-checks can compute this index per dimension at design time, with no runtime cost and no dependency on labeled data — it is a structural property of the configured weights, computable from the rubric definition alone. A low $J$ does not prove the dominant rule is wrong; it identifies where to invest adversarial testing effort (negation, paraphrase, code-switching, sarcasm) before deployment, and it gives auditors a single number to track as a rubric evolves. Adding a third, differently weighted rule to a $J=0.775$ dimension is a concrete, measurable way to reduce single-rule dependency, distinct from simply adding more rules without knowing whether the addition meaningfully redistributes decision weight. Analysis, drafting, editorial revision, and calculation verification for this paper were performed in collaboration with Claude (Anthropic). [1] R. Jain, D. Chiu, and W. Hawe, "A Quantitative Measure of Fairness and Discrimination for Resource Allocation in Shared Computer Systems," DEC Research Report TR-301, 1984. [2] National Institute of Standards and Technology, "Artificial Intelligence Risk Management Framework (AI RMF 1.0)," NIST AI 100-1, Section 3 (Characteristics of Trustworthy AI). [3] `reference/rubric.yaml` — the audited system's rubric configuration (documents four rules for `fair_bias_managed` ; two are implemented). [4] `src/Governance/Scoring/PromptScorer.cs` — the deterministic scorer (implements the two live rules for `fair_bias_managed` , lines 155-160). [5] `docs/nist-governance-spec.md` — the cross-dimension composite weight table. ``` # Reproduce the drift: composite/dimension score does not move after filling s dotnet run --project src/Governance.Cli -- compile "write a Python function that dedupes a list" dotnet run --project src/Governance.Cli -- score --file # Reproduce the negation-blind false positive dotnet run --project src/Governance.Cli -- compile "write a hiring rubric that does not exclude women from consideration" ```