Reproduce this recomputation

The claim, exactly

On the six per-item verdict files of the release's full_run directory at commit fb6f32e6b50b (hash-verified below), the OR-union of Llama Guard 4, Llama Guard 3 Vision, and ShieldGemma 2 flags 250/250 benign image and 32/250 benign text items, and catches 192/200 harmful text items (8 caught by none) and 200/200 harmful image items on this file. ShieldGemma 2 is an image-content classifier whose text verdicts are deterministic passes as released — so the harmful-text union is the work of the two Llama guards, not of three independent catchers. The full pattern tables, leave-one-out unions, scope, falsifier, and forbidden rescues are registered as claim MC-004.

Run it

From an empty directory — a POSIX shell with git and Python 3.11+ (with its standard venv module). The script downloads the eight pinned files from the bound commit and refuses to count anything until every hash matches.

git clone https://github.com/Cubits11/cubits11.github.io.git
cd cubits11.github.io
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements.txt
python scripts/reanalyze_msbench.py

What you should see

Dozens of ok assertion lines — every recomputed quantity that overlaps the release's own printed metrics is asserted equal to the printed value — then this summary and exit code 0:

benign text  union flags  32/250 — flagged by none 218/250
benign image union flags 250/250 — flagged by none 0/250
harmful text  union catches 192/200 — all-miss 8/200
harmful image union catches 200/200 — all-miss 0/200

MC-004 reproduced: the three-guard joint statistics, computed from the bound public release, match the registered claim.

What must fail

A reproduction that cannot fail proves nothing. Save the eight files locally, flip one byte of any of them, and re-run offline:

python scripts/reanalyze_msbench.py --dir path/to/mutated_full_run
FAIL  guard_…jsonl: sha256 …  != recorded …  — the bound artifact changed;
      MC-004 must be re-reviewed, not silently recomputed   (exit code 1)

The same flip smuggled past the hash gate trips nine independent count, pattern, and printed-agreement assertions — that mutation test is part of the claim's record, not a promise.

The eight pinned files

All at PatrickKollman/Multimodal-Safeguard-Bench@fb6f32e6b50b ↗ under results/full_run/ (MIT-licensed upstream; cited and hash-verified here, never redistributed).

File your run — match or mismatch

Both outcomes are wanted. A matching run becomes this claim's first independent reproduction; a mismatching run is a correction, handled under the same-day correction policy and credited where consent permits. Use the reproduction issue form ↗ (environment, commit, command, stdout, match or mismatch) or email. As of the last owner review, MC-004 has no recorded independent reproduction — this page is the standing invitation, not evidence that one exists.

Not claimed

• No vendor ranking, endorsement, or indictment — the verdicts are at native, unmatched operating rules, and no threshold calibration is documented upstream.
• No population estimate and no interval; the items are the release's own construction, and the counts are about exactly these bytes.
• Not an observed deployed stack; the OR aggregation is arithmetic, not a deployment.
• The harmful-image 200/200 is a counting fact about these 200 released items, not evidence the union catches image attacks in general — and the release's own changelog documents that ShieldGemma 2's image scores are sensitive to the text-rendering stack.
• A recomputation on released files: it changes no census count and adds no new measurement.