Cycle 93, stuff 2 (member-3): calibration and paired-SE check of the tail-signed S2 max (M), new seed 93302

Independent replication of stuff 1 under default_rng(93302), plus two checks that stuff 1 could not make from its own output: an out-of-sample size test of every critical value, and a closed-form cross check of the paired SE. Runner: stuff/cycle93_m_check.py (uv run --with numpy --with scipy python stuff/cycle93_m_check.py), wall clock 40 s. 20000 sims per cell, stream consumed in order n=25, 50, 100; per n: null (calibration), null again (out-of-sample size), then chi2(5), t3, t5, Laplace, cont10 (0.9 N(0,1) + 0.1 N(0,9)), cont5 (0.95 N(0,1) + 0.05 N(0,25)). All four tests are scored on the same draws in every cell. No cell edited by hand.

Definitions (rule 1, before any table)

z = sample-standardized order statistics (sample mean, n-1 sd, sorted ascending), i = 1..n. Lower-tail AD term l_i = -((2i-1)/n) ln Phi(z_(i)); upper-tail term u_i = -((2i-1)/n) ln(1 - Phi(z_(n+1-i))); plain t_i = l_i + u_i. S2+ = sum log(1 + 2 u_i)/2, S2- = sum log(1 + 2 l_i)/2 (c = 2); M = max(S2+, S2-), reject large. Comparators: A2 = sum t_i - n (plain AD, reject large); W = Shapiro-Wilk (scipy.stats.shapiro, reject small); S2 = sum log(1 + 2 t_i)/2 (reject large). Each 5% critical value is the empirical 95% (W: 5%) quantile of that statistic's 20000 null values.

Null calibration (rule 1): 20000 N(0,1) draws per n, seed 93302

The in-sample null rate is 0.0500 by construction (the critical value is the sample quantile), so it does not test calibration. The check below does: a second, independent 20000 null draws per n scored at the critical values above. The MC SE of a size estimate at 0.05 is 0.0015, so 0.047 to 0.053 is within two SE.

All 12 out-of-sample sizes are within 0.0041 of 0.05, at most 2.7 SE (M at n=50, A2 at n=100, both on the conservative side). Critical values agree with stuff 1 (seed 93201) to the third decimal for M (8.8442 vs 8.8417, 17.3573 vs 17.3666, 34.2755 vs 34.2751). Calibration holds.

Comparison (rule 2): power at exact 5% size, 20000 paired draws per cell

Gap = M minus other; paired SE = sd of the per-draw rejection-indicator difference / sqrt(20000). Verdict: beats if gap > 2 SE, loses if gap < -2 SE, else tie. "Within 2 SE of S2?" is the tie verdict on M-S2.

Paired SE check

The script's paired SE is the sd of the per-draw indicator difference d = 1[M rejects] - 1[other rejects], divided by sqrt(20000). The closed form for a difference of paired proportions is sqrt((p10 + p01 - (p10 - p01)^2) / N), with p10 and p01 the two discordant-pair rates. Both are listed per cell with the unpaired SE sqrt(p(1-p)/N + q(1-q)/N) for scale.

The indicator-sd SE and the closed form agree to five decimals in all 18 cells (they are the same quantity up to the N/(N-1) factor). The paired SE is 0.6 to 0.8 of the unpaired SE in most cells and 0.38 for M-S2 under chi2(5) at n=25, so the paired design is doing real work and no verdict above would flip if the unpaired SE were used instead except the two near-zero M-A2 gaps at n=50 (t5 and cont10), which are ties or marginal under either SE.

Agreement with stuff 1 (seed 93201)

Every verdict matches stuff 1 in 52 of 54 cell-comparisons. The two that moved are M-A2 at n=50 under t5 (stuff 1 tie at +0.0013, here loses at -0.0091) and M-A2 at n=50 under cont10 (tie at +0.0014, here tie at -0.0031): both gaps are within 2 cross-seed SE of zero, so the true gap is near zero and the verdict is seed-sensitive, as expected. No chi2(5) or M-S2 verdict changed. The largest cross-seed difference in any power estimate is 0.016 (t5, n=50, M), under 2.5 combined SE.

Closing line (rule 2)

No: M beats AD under chi-square(5) at every n but loses to Shapiro-Wilk at every n by more than two paired SE (-0.019, -0.036, -0.006), and M is not within two paired SE of plain S2 on any of the 15 heavy-tail and contamination cells (below on all 15, by 0.025 to 0.388); the direction fails on both clauses, under a second seed, with calibration and paired SE confirmed independently.

Cycle 93, stuff 3 (member-1): third-seed replication of the tail-signed S2 max (M), seed 93403

Same runner as stuff 2 with only the seed changed (stuff/cycle93_m_seed3.py, uv run --with numpy --with scipy python stuff/cycle93_m_seed3.py, wall clock 39 s). 20000 sims per cell; per n: null (calibration), independent null (out-of-sample size), then chi2(5), t3, t5, Laplace, cont10 (0.9 N(0,1) + 0.1 N(0,9)), cont5 (0.95 N(0,1) + 0.05 N(0,25)). All four tests are scored on the same draws in every cell. Tables are the script's output, unedited.

Definitions (rule 1, before any table)

z = sample-standardized order statistics (sample mean, n-1 sd, sorted ascending), i = 1..n. Lower-tail AD term l_i = -((2i-1)/n) ln Phi(z_(i)); upper-tail term u_i = -((2i-1)/n) ln(1 - Phi(z_(n+1-i))); plain t_i = l_i + u_i. S2+ = sum log(1 + 2 u_i)/2, S2- = sum log(1 + 2 l_i)/2 (c = 2); M = max(S2+, S2-), reject large. Comparators: A2 = sum t_i - n (plain AD, reject large); W = Shapiro-Wilk (reject small); S2 = sum log(1 + 2 t_i)/2 (reject large). Each 5% critical value is the empirical 95% (W: 5%) quantile of that statistic's 20000 null values, so in-sample null rates are 0.0500 by construction; the out-of-sample table tests calibration (MC SE 0.0015 at 0.05).

Calibration, out-of-sample size, power with paired SE (rules 1 and 2), and paired SE cross-check

Gap = M minus other; paired SE = sd of the per-draw rejection-indicator difference / sqrt(20000); beats if gap > 2 SE, loses if gap < -2 SE, else tie.

Three-seed agreement (seeds 93201, 93302, 93403)

Out-of-sample sizes are all within 0.0050 of 0.05 (worst A2 at n=50, 0.0450, 3.3 SE conservative; M worst 0.0525). M critical values agree with both earlier seeds to within 0.012 at every n. Every chi-square(5) verdict is the same on all three seeds: M beats AD (+0.057, +0.052, +0.005) and loses to SW (-0.012, -0.039, -0.006) at every n, each beyond two paired SE. Every M-S2 verdict on the 15 heavy-tail and contamination cells is the same on all three seeds: below, by 0.023 to 0.384. The only seed-sensitive verdicts are M vs AD at n=50 under t5 and cont10 (tie, loses/tie, and now beats at +0.013 and +0.008): the true gap there is near zero.

Why the max fails: on symmetric heavy tails both S2+ and S2- carry evidence, and the max keeps only one while its critical value (about 0.65 of S2's) is set for the larger of two near-independent sums, so M discards roughly half the signal S2 pools. Under chi2(5) only the upper tail carries signal and M beats S2 and AD but still not W, whose skew sensitivity is not confined to the extreme order statistics. Tail-signing by max is exhausted; any further S2 variant needs a pooled statistic, not a selected one.

Closing line (rule 2)

No: under a third seed M again beats AD but loses to Shapiro-Wilk under chi-square(5) at every n by more than two paired SE, and falls below plain S2 by more than two paired SE on all 15 heavy-tail and contamination cells; the direction fails on both clauses, identically on all three seeds.

Cycle 93, stuff 1 (member-2): tail-signed S2 max (M) vs AD, Shapiro-Wilk and plain S2, exact 5% size, 20000 sims per n, seed 93201

One default_rng(93201) stream consumed in order n=25, 50, 100; per n the laws normal, chi2(5), t3, t5, Laplace, cont10 (0.9 N(0,1) + 0.1 N(0,3^2)), cont5 (0.95 N(0,1) + 0.05 N(0,5^2)). All four tests are scored on the same draws in every cell. Runner: stuff/cycle93_m_tailsigned.py (uv run --with numpy --with scipy python stuff/cycle93_m_tailsigned.py), wall clock 34 s. No cell edited by hand; the tables below are the script's output.

Definitions (rule 1, before any table)

z = sample-standardized order statistics (sample mean, n-1 sd, sorted ascending), i = 1..n. Lower-tail AD term l_i = -((2i-1)/n) ln Phi(z_(i)); upper-tail term u_i = -((2i-1)/n) ln(1 - Phi(z_(n+1-i))); plain t_i = l_i + u_i. S2+ = sum log(1 + 2 u_i)/2, S2- = sum log(1 + 2 l_i)/2 (c = 2); M = max(S2+, S2-), reject large. Comparators: A2 = sum t_i - n (plain AD, reject large); W = Shapiro-Wilk (scipy.stats.shapiro, reject small); S2 = sum log(1 + 2 t_i)/2 (reject large). Each 5% critical value is the empirical 95% (W: 5%) quantile of that statistic's 20000 null values, so every power row is at matched exact size. MC SE of a null rate at 0.05 is 0.0015.

Null calibration (rule 1): 20000 N(0,1) draws per n

Comparison (rule 2): power at exact 5% size, 20000 paired draws per cell

Gap = M minus other; paired SE = sd of the per-draw rejection-indicator difference / sqrt(20000). Verdict: beats if gap > 2 SE, loses if gap < -2 SE, else tie. "Within 2 SE of S2?" is the tie verdict on M-S2.

Reading the table

1. Chi-square(5) vs AD: M beats AD at every n (+5.6, +5.5, +0.4 points). Vs Shapiro-Wilk: M loses at every n (-1.4, -3.7, -0.6 points, each more than 2 paired SE). Keeping the tail sign buys about the same skew power as the min-p S2+W combination of cycle 92 did, and still does not reach W alone.
2. Heavy tails and contamination vs plain S2: M is below S2 by more than 2 paired SE on all 15 cells, and the loss grows with n: 2 to 9 points at n=25, 6 to 26 points at n=50, 10 to 39 points at n=100. Taking the larger of two one-sided sums discards the half of the evidence that a symmetric alternative puts in the other tail, and S2's power came from summing both tails.
3. Heavy tails and contamination vs AD and SW: at n=25 M still beats AD on 5 of 6 cells, but at n=50 and 100 M loses to AD on Laplace, t3 (n=50) and on every cell at n=100, and loses to SW on 16 of 18 off-skew cells. M is not an omnibus improvement over anything; it trades S2's whole tail advantage for a skew gain that still falls short of W.

Answer to the direction (seed 93201)

No. M beats AD under chi-square(5) at every n but loses to Shapiro-Wilk at every n (n=25, 50, 100), and it does not stay within two paired SE of plain S2 on any of the 15 heavy-tail and contamination cells: it loses to S2 by 2 to 39 points on all of them, so neither condition of the direction holds.