Reproduce this recomputation

The claim, exactly

On the six per-item verdict files of the release's full_run directory at commit fb6f32e6b50b (hash-verified below), the harness stores each guard's native unsafe label as a Boolean blocked bit. The static OR of those bits is 1 on 250/250 benign-labelled image and 32/250 benign-labelled text rows, and on 192/200 harmful-labelled text rows (8 all-zero rows) and 200/200 harmful-labelled image rows. Llama Guard 3 Vision is a multimodal prompt/response classifier; ShieldGemma 2 is image-only and its text rows are deterministic passes as released. The harness uses a blocked item to suppress target generation, so, under a fixed block-on-any rule, this is a valid counterfactual harness-block calculation for these pinned rows. No examined source defines a translation of all native predicates to one shared catch event, so it remains a released-bit aggregate, not a shared-policy safety result, a deployed stack, or a three-independent-catcher result. The full bit-pattern tables, leave-one-out bit ORs, scope, falsifier, and forbidden rescues are registered as claim MC-004.

Run it

From an empty directory — a POSIX shell with git and Python 3.11+ (with its standard venv module). The script downloads the eight pinned files from the bound commit and refuses to count anything until every hash matches.

git clone https://github.com/Cubits11/cubits11.github.io.git
cd cubits11.github.io
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements.txt
python scripts/reanalyze_msbench.py

What you should see

Dozens of ok assertion lines — every recomputed quantity that overlaps the release's own printed metrics is asserted equal to the printed value — then this summary and exit code 0:

benign text  OR(native-unsafe)= 32/250 — all-native-safe 218/250
benign image OR(native-unsafe)=250/250 — all-native-safe 0/250
harmful text  OR(native-unsafe)=192/200 — all-native-safe 8/200
harmful image OR(native-unsafe)=200/200 — all-native-safe 0/200

MC-004 reproduced: the three-guard native-label bit statistics, computed from the bound public release, match the registered claim.

What must fail

A reproduction that cannot fail proves nothing. Save the eight files locally, flip one byte of any of them, and re-run offline:

python scripts/reanalyze_msbench.py --dir path/to/mutated_full_run
FAIL  guard_…jsonl: sha256 …  != recorded …  — the bound artifact changed;
      MC-004 must be re-reviewed, not silently recomputed   (exit code 1)

The same flip smuggled past the hash gate trips nine independent count, pattern, and printed-agreement assertions — that mutation test is part of the claim's record, not a promise.

The eight pinned files

All at PatrickKollman/Multimodal-Safeguard-Bench@fb6f32e6b50b ↗ under results/full_run/ (MIT-licensed upstream; cited and hash-verified here, never redistributed).

File your run — match or mismatch

Both outcomes are wanted. A matching run becomes this claim's first independent reproduction; a mismatching run is a correction, handled under the same-day correction policy and credited where consent permits. Use the reproduction issue form ↗ (evidence class: static-reconstruction or direct-route; environment, commit, command, stdout, match or mismatch) or email. The project-wide qualified-outcome ledger currently records 0 qualified external outcomes; it also records 4 technical interactions, which is diagnostic rather than evidence. That public null is not a growth metric. MC-004 has no recorded independent reproduction — this page is the standing invitation, not evidence that one exists.

Not claimed

• No shared-event catch statistic: the common blocked bit is a harness normalization of native predicates, and no source-defined translation to a single event has been identified.
• No vendor ranking, endorsement, or indictment — the verdicts are at native, unmatched operating rules, and no threshold calibration is documented upstream.
• No population estimate and no interval; the items are the release's own construction, and the counts are about exactly these bytes.
• Not an observed deployed stack; the OR aggregation is arithmetic, not a deployment.
• The harmful-image 200/200 bit-OR count is a fact about these 200 released items, not evidence of image-attack safety in general — and the release's own changelog documents that ShieldGemma 2's image scores are sensitive to the text-rendering stack.
• A recomputation on released files: it changes no census count and adds no new measurement.