Pranav Bhave

AI guardrail evaluation · Pranav Bhave

Two AI guards can fail on the same attack.

Their individual scores do not tell you how often that happens.

Correction: E6’s fixed-marginal range and E7B’s preregistration-conforming claim are rejected. Read the dispositions.

Latest full check: UNKNOWN — checking GitHub.

01 / Bounded result

I reviewed 20 public guardrail evaluations within a fixed search scope.

Five preserve some joint evidence; zero document matched thresholds with full exposure.

02 / Demonstration

Move the misses yourself, then inspect the sources.

This is a constructed example, not a measurement of a deployed system.

Inspect the checked 10%-and-10% construction here
Two 10 percent failure rates permit anywhere from 0 to 10 percent joint failure A rectangle of area one represents the population. Two shaded strips each cover ten percent of it, one per guardrail. Their overlap is the probability both fail on the same input. The overlap can be set anywhere from zero to ten percent while neither individual rate ever changes. A vertical cyan scale marks the full supported interval from 0 to 10 percent, with a dashed amber mark at 1 percent — the single value that assuming independence selects. The zero-overlap and full-overlap positions are the lower and upper endpoint witnesses. The population — area = 1 10% upper witness 0% lower witness 1% — what independence picks π(q) = (0.80, 0.10, 0.10, 0.00) · P(A) = P(B) = 10% in every frameP(both fail) = 0% · P(at least one fails) = 20% · the lower endpoint witness0% π(q) = (0.81, 0.09, 0.09, 0.01) · P(A) = P(B) = 10% in every frameP(both fail) = 1% · P(at least one fails) = 19% · the world independence assumes1% π(q) = (0.82, 0.08, 0.08, 0.02) · P(A) = P(B) = 10% in every frameP(both fail) = 2% · P(at least one fails) = 18% · one feasible world2% π(q) = (0.83, 0.07, 0.07, 0.03) · P(A) = P(B) = 10% in every frameP(both fail) = 3% · P(at least one fails) = 17% · one feasible world3% π(q) = (0.84, 0.06, 0.06, 0.04) · P(A) = P(B) = 10% in every frameP(both fail) = 4% · P(at least one fails) = 16% · one feasible world4% π(q) = (0.85, 0.05, 0.05, 0.05) · P(A) = P(B) = 10% in every frameP(both fail) = 5% · P(at least one fails) = 15% · one feasible world5% π(q) = (0.86, 0.04, 0.04, 0.06) · P(A) = P(B) = 10% in every frameP(both fail) = 6% · P(at least one fails) = 14% · one feasible world6% π(q) = (0.87, 0.03, 0.03, 0.07) · P(A) = P(B) = 10% in every frameP(both fail) = 7% · P(at least one fails) = 13% · one feasible world7% π(q) = (0.88, 0.02, 0.02, 0.08) · P(A) = P(B) = 10% in every frameP(both fail) = 8% · P(at least one fails) = 12% · one feasible world8% π(q) = (0.89, 0.01, 0.01, 0.09) · P(A) = P(B) = 10% in every frameP(both fail) = 9% · P(at least one fails) = 11% · one feasible world9% π(q) = (0.90, 0.00, 0.00, 0.10) · P(A) = P(B) = 10% in every frameP(both fail) = 10% · P(at least one fails) = 10% · the upper endpoint witness10%
q = 0% — the lower endpoint witness Dragging changes only the overlap of the two failures. Each guard's marginal failure rate remains ten percent.
Illustration — no measured system. Each guard misses 10% of the same population. Shared misses range from 0% to 10%; independence selects 1%. The endpoints and each frame’s geometry are checked by the figure verifier. Read the argument and assumptions.

03 / Relevant work

I’m Pranav Bhave, a security engineer and Penn State CS graduate.

I measure what AI guardrail stacks miss together, and build evidence systems that show exactly what data can and cannot establish.

04 / Contact

Bring one results table, claim, or receipt: email me.

bhavepranavwork@gmail.com