How many outages rule 1 actually needs

Rule 1 ends probation when the 95% upper bound on p99 outage length is within 25% of the estimate. This note works out how many *observed outages* (not intervals) that takes, because that count, not the calendar, sets probation length.

Without a model: 299 outages before a bound exists

A distribution-free upper bound on p99 uses the largest observed outage. It covers p99 with 95% confidence only if 1 - 0.99^n >= 0.95, so n >= 299 outages. Below that there is no nonparametric upper bound at all, so the 25% test cannot even be evaluated. For a guardian with one outage a week that is almost six years.

With a lognormal fit: about 200·σ² outages

Outage lengths are usually right-skewed, so fit ln(length) ~ Normal(μ, σ). Then p99 = exp(μ + 2.326σ) and the standard error of μ̂ + 2.326σ̂ is about σ·sqrt((1 + 2.326²/2)/n) ≈ 1.925σ/sqrt(n).

"Upper bound within 25%" means 1.645 · 1.925σ/sqrt(n) <= ln(1.25) = 0.223, so n >= 203·σ².

What this means for the direction

• Even with a model, a guardian with weekly outages and σ = 0.5 needs about a year. 30 days is not close for anyone except guardians that fail several times a day.
• The 25% tolerance drives the count: n scales as 1/ln(1+tol)². At 50% tolerance the counts drop to about a third (σ = 1 → 61 outages).
• Rule 2's spacing rule makes this worse, since clustered outages count once.
• Until a guardian reaches its n, the pooled floor from rare-failure-floor.md has to apply to guardians that fail often too, not only to rare ones.

Open question: the lognormal fit must be checked (for example by a QQ plot of log lengths) before the 200·σ² count is trusted. A heavy tail beyond lognormal would make the bound too optimistic.

How long probation must run (rules 1 and 2 worked through)

Rule 1 ends probation when the 95% upper bound on a guardian's p99 outage is within 25% of the estimate. Rule 2 only counts outages whose check windows sit at least one p99-outage length from other counted windows. This note works out what those two rules imply for probation length, so the stopping point comes from data and not from "30 days".

Distribution-free bound: at least 299 intervals

With no assumption about the outage distribution, a 95% upper bound on the 99th percentile exists only if the largest of n observations has a 95% chance of exceeding it:

1 - 0.99^n >= 0.95 => n >= ln(0.05) / ln(0.99) = 298.1 => n >= 299

So with one check interval per day, a distribution-free grace floor needs at least 299 days. That bound is just the worst observed outage, so the "within 25%" test cannot usually be met. A 30-day probation gives no p99 bound at all under this method.

Exponential outage durations: about 68 counted outages

If outage durations are exponential with mean m, then p99 = m * ln(100), and the ratio of upper bound to estimate is the same for p99 as for m. With k counted outages, the 95% upper bound on m is 2k*m_hat / chi2_{0.05}(2k). Using chi2_{0.05}(2k) ~ 2k - 1.645*sqrt(4k):

upper / estimate ~ 1 / (1 - 1.645 / sqrt(k)) <= 1.25 => sqrt(k) >= 8.22 => k >= 68

The probation length is then 68 divided by the rate of *counted* outages. Rule 2 lowers that rate: outages that cluster inside one p99 length of each other count once, so a guardian that fails in bursts takes longer to graduate. That is intended, because a burst is one incident and not 68 pieces of evidence.

Rule of thumb

• Count spaced outages (rule 2), not days.
• Under an exponential fit, stop at about 68 counted outages plus 30 outage-free intervals (rule 1). The grace floor is the upper bound at that point, about 1.25x the p99 estimate.
• A guardian that almost never fails will not reach 68 outages. It needs the distribution-free path (299+ intervals) or a fixed default floor, not an early graduation.
• Check the exponential fit before relying on it. A heavy tail (for example, rare multi-day outages) makes the 68 figure too optimistic.

Grace floor for guardians that rarely fail

Rule 1 ends probation when the 95% upper bound on p99 outage is within 25% of the estimate. A guardian with almost no outages never gets there: you can't estimate the size of outages you haven't seen.

What zero outages does tell you

If there are 0 outages in N independent counted intervals (spaced per rule 2), the 95% upper bound on the per-interval outage *rate* is about 3/N (the rule of three). For example:

At N = 299 the rate bound reaches 1%. That means p99 outage is 0 at 95% confidence: in 99% of intervals there is no outage.

Proposed floor before that point

Before N = 299, set the grace floor to the fleet-pooled p99 outage length, which is the p99 over all graduated guardians' counted outages. Don't use the guardian's own data. The guardian graduates early only if both of these hold:

• it reaches the 299-interval bound, so its floor can be its own measured value, or
• it records enough outages to meet rule 1's 25% test on its own data.

Why not just keep 30 days

With daily intervals, 30 days gives a rate bound of 10%, which says nothing about p99. A reliable guardian on a hand-picked 30 days gets a floor that has no evidence behind it. The pooled floor at least uses real outages from similar guardians.

Open question

Pooling assumes the fleet is homogeneous. If a few guardians fail often and with long outages, they push the pooled p99 up, and reliable guardians end up with floors that are too long. A pooled value cut down to guardians of the same type or provider would fix that, but each type then needs enough guardians to pool from.