Pranav Bhave
Claim E9-001 — 22 of 22 in the registry Supported within scope

E9-001Warrant reviewed 2026-10-17 · expires 2027-02-14

E9: both-miss measurement on 1,000 items

On the 1,000 measurement items of its admitted pool, the test split of Lakera/mosscap_prompt_injection at b7e495ff63373ff7f7dabc1e9390cf62b5838570, two open prompt-injection classifiers both miss 32.20% of items, where the independence plug-in from their own miss rates predicts 26.52%. G1 is protectai/deberta-v3-base-prompt-injection-v2 at revision 90c9989b1a342275dd0d1a95aad283c04e075671 and G2 is dcarpintero/pangolin-guard-base at revision eb220d9f8d75cfbc82cc9d430fa19f85d9764cef, each held at the frozen operating point: the lowest threshold whose false-positive rate is at most 0.05 on 800 first user turns from allenai/WildChat-1M at 7d6490e462285cf85d91eabea0f9a954fbddcd1f (G1 0.005912, G2 0.716629). Their miss rates are 0.337 and 0.787. The discrepancy is +0.0568 (exactly 56781/1000000; preregistered 95% item-bootstrap interval [+0.0472, +0.0669], B = 2000). The two miss rates alone leave the both-miss rate anywhere in [0.124, 0.337]. The preregistered verdict is DISCREPANT: the interval excludes zero and the point discrepancy reaches the 0.05 SESOI. The interval does not establish a discrepancy greater than 0.05.

01Falsifier — what changes this claim

Any of the following is shown: recomputing from the committed rows yields any registered quantity different beyond 1e-9; the thresholds recomputed from the committed calibration scores by the contract rule differ from those recorded; the freeze commit 6ac65de does not precede the first E9 score commit; any measurement item overlaps a calibration, stress or scouting partition; the declared part of freeze/protocol.json no longer hashes to the digest freeze.json records; or the verdict run/analyze.py applied differs from the rule committed at 6ac65de.

Consequence REJECT

This is the condition and consequence recorded in the registry. The vocabulary this is published in defines what each consequence commits the author to.

02Scope

Exactly the 1,000 rows committed at experiments/e9/results/observations.jsonl, set-equal to the frozen list freeze/items_measurement_mosscap-test.csv, under contract E9-2026-09-18 (frozen, sha256 43811798…). The freeze commit 6ac65de (2026-09-18) precedes the first E9 score commit. The verdict-rule limitation: the SESOI (0.05) and the equivalence margin are in the frozen contract, but the rule that combines them into DISCREPANT lives in run/analyze.py. That file was committed in the freeze commit and is byte-identical since, but the rule is neither declared in contract.json nor rendered into PREREG.md's prose. Correction C1 asks for claim-critical choices to live in the contract; this one does not, and the verdict stands on the committed code, not on the contract alone. Further limits, as RESULT.md states them: the pool was selected, as the widest scouting width among complete candidates, for intermediate marginals; scouting items are disjoint from measurement items, but the result describes a selected pool, not pools in general; the interval resamples measurement items with thresholds held fixed, and omits calibration uncertainty; the operating point is declared, not deployed: on 400 seemingly-toxic benign prompts G2's false-positive rate is 0.2675, reported and never deciding; G2's fine-tuning corpus could not be verified, and G2 has no licence file at the pinned revision (apache-2.0 is declared in card metadata only); run/admit.py appended a results block to freeze/protocol.json, so the file is not byte-immutable after freeze; the declared part still hashes to the frozen digest, and the repository graph records the append as a D5 STOP finding; one analyst ran every step, and re-running the committed scripts is replay, not reproduction. Both guards are under 1B, ungated and pinned, loaded under the 2026-09-06 host exemption. E2's guards were not loaded. This is not an E2 result and does not change CC-005.

03Forbidden rescues

Repairs declared unavailable in advance; using one after a failure would breach the recorded commitment.

04Non-claims — what this does not license
05Binding and freshness
Binding
observations.jsonl @ 592ed515
The support is the measurement: 1,000 committed per-item rows, merged to main as 592ed51 (PR #26). run/analyze.py re-run from a clean archive rewrites results/analysis.json byte-identically (sha256 4ea9d46e…), and an independent exact-fraction recount of the rows gives cells 322/15/465/198. The commit order is load-bearing: 6ac65de (freeze, 2026-09-18) precedes every score commit.
Reviewed
2026-10-17 · window 120 days
Expires
2027-02-14 — after this date the recorded review is overdue; this does not make the claim false
Triggers
  • executable fires when a committed measurement row changes without a registry re-review
  • executable fires when the recorded analysis, interval or verdict changes
  • executable fires when the run result changes
  • executable fires when the run report changes, including any stated limit or non-claim
  • executable fires on any edit to the preregistration, which must not move after the measurement was read
  • executable fires when the frozen contract changes
  • executable fires when the freeze record changes
  • executable fires when the declaration or its appended admission record changes
  • executable fires when the file that carries the verdict rule changes; the rule is not in the contract, so this pin is what holds it fixed
  • manual a frozen multi-pool or new-guard contract (E10) is executed, either pinned revision is withdrawn, or G2's training data is disclosed
Dimensions
visibilityPublicprovenanceMachine-generated, owner-executedsupport roleExecuted outputmaturityExperimental