A per-detector leaderboard is a legitimate artifact and this is not a criticism of publishing one. It just cannot be composed. Six things make the difference between an evaluation whose stack behaviour is recoverable and one whose is not — and five of them cost nothing extra if you decide before you run.
1. Score every system on the same items
Different item sets make every downstream comparison a guess. This is the cheapest requirement and the most commonly broken one. Of the 20 evaluations in the census, 14 document a shared item set and a common event definition.
2. Fix one event definition
"Blocked", "flagged", "refused", and "scored above threshold" are different events. If two systems are scored against different notions of what counts as a catch, their union is undefined.
3. Compare at matched operating points, and say so
A detector at a permissive threshold and one at a strict threshold are not comparable, and a threshold-free metric like AUPRC quietly sidesteps the question a deployed stack has to answer. This is the rung almost nobody reaches: 12 of the census's shared-basis evaluations have no stated threshold mismatch, but 0 document matched operating thresholds together with full exposure.
4. Declare the exposure condition
There are three very different worlds, and a results table that does not say which one it is in cannot be interpreted:
- Static full exposure — every system sees every item. Union and all-miss are well defined and directly computable.
- Deployed sequential routing — an upstream block censors what downstream systems ever see. Static composition arithmetic does not apply, and pretending it does inflates the stack.
- Adaptive or agentic — an intervention changes the trajectory, so there is no fixed population to compute a rate over.
The stack study preflight is a browser-local tool that refuses to emit a static joint result when you declare a non-static mode — it is easier to check this before you collect data than after.
5. Publish the joint row
Two numbers over the same denominator: how often the union catches, and how often every system misses the same item. That is the row that per-detector columns cannot reconstruct, and it is one line in the table you are already building. The Minimum Joint Guardrail Disclosure gives the exact template, its preconditions, and a tested reference implementation.
6. Keep the per-item outcomes
One decision per item per system, released alongside the paper, makes every union, all-miss, and pairwise intersection recomputable by anyone — including analyses you did not think to run. If releasing raw items is not possible, leave-one-out unions are a compact, privacy-preserving summary that still identifies each guard's exclusive contribution to the stack.
What you get for it
A results table a reader can deploy from rather than infer from, and a claim that survives someone checking it. If you already ran the evaluation and kept the per-item outcomes, the joint row is a recomputation, not a new experiment.
Next
If your evaluation is already published, the census records where it stands and how to correct the record if a row misreads it.