Between-condition contrasts (#269)

The toolbox produces classification accuracy per run. Until now it had no way to contrast runs — a protocol built as deliberate A/B pairs (the teaching-lab worked example: shape, colour, pattern) could only be compared by eye across separate HTML reports. This is that contrast: a declared design, a stated test, and an honest account of what the test can and cannot detect.

All the statistics live in ssvep.analysis.contrast; the report page in ssvep.analysis.contrast_report; both are read from BIDS/code/03_run_contrasts.py and the GUI's Compare tab, which are thin callers over the same function. See the API reference for the full docstrings — this page is the narrative version.

What this is, and what it isn't

It is not a group report. A report that puts numbers side by side invites exactly the eyeballing this is meant to replace. A contrast has a factor, two named levels (or, for a between-subject factor, two levels of a participants.tsv column), a declared confirmatory window, and one pre-declared primary dependent variable — all of it stated before the data are read, in the protocol itself.

Three designs. "within" compares two runs the same participant did (stage 1). "between" compares two groups of different participants, split on a documented participants.tsv column, on the one run every participant contributes (stage 2). "mixed" asks whether a within-subject effect itself differs between two such groups — the interaction, and only the interaction (stage 2). See Between-subject contrasts and Mixed designs below.

Declaring a contrast

A contrast is declared in the session protocol's contrasts block (schemas/session_protocol.schema.json, $defs.contrast), which arrived with protocol_version 1.3 (see Protocols and sessions); design: "between"/"mixed" need 1.4. A protocol with no contrasts block offers none — nothing is contrasted implicitly, and a protocol whose contrasts are all "within" round-trips byte-identically as 1.3.

Every design shares these fields:

Field	Meaning
name	Stable identifier, unique within the protocol; used in the report's filename.
factor	The manipulated factor this contrast isolates, e.g. "shape". For "mixed", the within factor — the between factor is named by between.column, not here.
design	"within", "between", or "mixed" — see above.
confirmatory_window_s	The one pre-declared decision window carrying the confirmatory test — pre-declared so the window that happens to look best cannot be picked after the fact.
primary_dv	"accuracy" is the only DV that can be tested. "snr_db" and "itr" are accepted by the schema so each can be refused with its reason stated — see Refusals.
min_trials	Optional DV-independent floor on decoded trials per run; below it a run is excluded regardless of primary_dv.
description	Free text: what the contrast isolates and what it doesn't.

Which of the remaining fields apply depends on design:

Field	within	between	mixed
levels	Required: exactly two run labels. Difference is levels[1] − levels[0].	Forbidden.	Required: the two within run labels.
run	Forbidden.	Required: the one run label every participant contributes.	Forbidden.
between	Forbidden.	Required: {"column": ..., "levels": [a, b]} — see Between-subject contrasts.	Required: same shape as between.

The teaching protocol's own block (session-protocols/example_for_teaching_lab_cyton.json) declares its three contrasts this way:

{
  "name": "shape",
  "factor": "shape",
  "levels": ["white-square", "white-circle"],
  "design": "within",
  "confirmatory_window_s": 2.0,
  "primary_dv": "accuracy",
  "min_trials": 18,
  "description": "White square vs white circle, both solid, same bounding box. The circle covers ~78.5% of the square's area and emits ~21% less light (docs/EXAMPLE_TEACHING_LAB.md §4.1), so this contrast is shape confounded with luminous flux, not shape alone."
}

— and likewise for colour and pattern. See the worked example for what each of the three isolates and where it leaks.

What is tested, and why

Accuracy: sign-flip permutation, at every n

The confirmatory test is a two-sided sign-flip permutation test on subject-level paired differences. Under the null each subject's difference is equally likely to have either sign, so the reference distribution is every re-signing of the observed differences — exact when all 2**n arrangements can be enumerated, Monte Carlo above that. It has no cluster-count cliff and makes no distributional assumption, unlike a paired t-test, which would return a p-value below .05 at n = 5 but only by importing a normality assumption five points cannot verify.

The report states the minimum attainable p for the current n, next to the result, and before it — a reader who meets the p-value first reads a floor as "nearly significant" when it is a floor. At n = 5, two-sided, that floor is 2 / 2**5 = 0.0625: significance at α = .05 is unreachable by construction, whatever the data show. A subject whose paired difference is exactly zero cannot change sign, so each such tie removes arrangements and raises the floor (to 2 / 2**(n − ties)). With accuracies over a few dozen trials, ties are common at small n, and the page states the floor the test actually had. The worked example below confirms this on real output, not just in prose.

GEE: estimation, not the test

A GEE fit (binomial for accuracy, Gaussian for SNR; exchangeable working correlation; cov_type="bias_reduced", the Mancl–DeRouen correction; groups=subject) is reported alongside the permutation test, on the identical complete-case sample. It gives an effect size (percentage points for accuracy, an odds ratio beside it; dB for SNR) with a confidence interval — the estimation half of the report, distinct from the permutation test's confirmatory role.

Its p-value is suppressed below 15 clusters (subjects). GEE's robust sandwich standard error is asymptotic in the number of clusters, and with few clusters it is too small — anti-conservative, in the direction that manufactures findings. The bias-reduced correction helps but does not remove this below roughly 10–15 clusters. Below the floor the page states why and shows only the estimate and CI, with a note that the CI is likely too narrow; at or above it, GEE's p-value is shown too.

Order covariates: in the model where they can be estimated, left out with the reason where they can't (#31)

Two kinds of order can masquerade as the manipulation, and GEE adjusts for each wherever the data can tell it apart from the compared factor:

• Run position — which of a participant's two runs came first (second_in_group; in a between design, the counterbalancing arm, below). It enters the model when both orders occur in the sample, and is left out with the reason stated when every participant ran the levels in the same order (it is then the level, under another name) or, in a mixed design, when the arm coincides with the group (below).
• Time in run (time_in_run, accuracy only) — where in its own run each trial fell, 0 for the run's first planned trial and 1 for its last. It reads each trial's own outcome at the declared window, which metrics.json records per sweep window (window_sweep[].trial_outcomes: block, trial, correct), and places the trial through the sidecar's planned order. A run stopped early therefore holds only early trials, which is exactly the case it matters most for: a level whose runs were stopped early holds only fresh trials, and without the term a downward drift reads as an advantage for that level. Its coefficient is reported as an odds ratio for a run's last planned trial against its first, with a CI, and with a p-value under the same 15-cluster rule as the level effect. SNR is one value per run, so it has no time term. The model is additive: one time slope for both levels, no level × time interaction, so a level-specific drift is not modelled but averaged into that slope (an interaction would be a diagnostic only, never part of the estimate). The confirmatory permutation test is not adjusted for time in run; the report shows each level's time coverage and says so when the levels differ.

Time in run varies inside every run, so in a run-level contrast it can be separated from the level whenever the runs have per-trial outcomes; the model still checks, and leaves it out with the reason when the trial-level design is rank-deficient (every trial at one place in its run, for instance). It enters for every run or for none: if any run in the contrast lacks per-trial outcomes at the declared window — a metrics.json written before they were recorded, a synthetic record, outcomes that do not reproduce the run's accuracy at that window, or a played order the trials cannot be placed in — it is left out for all, and the page says how many runs lacked them and why. A model in which some runs are adjusted and others are not would be neither analysis. The fit is then exactly the one made before the term existed, so every existing result is unchanged.

What was adjusted for is part of the result. GeeResult.covariates lists the model's terms after the intercept, in order (for example level, second_in_group, time_in_run), and the page's methods paragraph prints them. An estimate with time_in_run and one without it are different analyses: compare covariates before putting two results side by side, and never average across results whose covariates differ. This is the rule D7 applies to decode settings (Refusals), applied to the model; and because the term is all or nothing within a contrast, no single result mixes the two.

Inside a single multi-condition run the same rule holds, with the order a design declares: time in run is adjusted for under a randomised or counterbalanced order, and withheld under a fixed one, where it is collinear with condition (below).

Holm across the protocol's contrasts

Accuracy is the only confirmatory dependent variable (the maintainer, 2026-09-24). Its p-values are combined across every contrast the protocol declares with a Holm step-down correction — the family is the protocol, not the whole toolbox. Target SNR is descriptive only: it is never tested and has no p-value to correct, Holm or otherwise. A contrast that declares primary_dv: "snr_db" is refused outright, with its reason stated (ssvep.analysis.contrast.SNR_REFUSAL) — see Refusals.

The family spans every design. A protocol that declares a mix of "within", "between", and "mixed" contrasts corrects all of their confirmatory p-values together, in one Holm family — run_protocol_contrasts collects the primary permutation p-value from whichever test each contrast actually ran (sign-flip for "within", label permutation for "between"/"mixed") and corrects across the whole list, because the family is "this protocol's declared confirmatory tests", not "this protocol's within-subject tests".

SNR: descriptive only, and why it's a mean

Accuracy alone cannot tell "the brain responded less" from "the decoder handled it worse". The pattern contrast in the worked example below is exactly this case: a contrast-reversing checkerboard's strongest response lands at 2f, not the fundamental the decoder's candidate frequencies sit at, so lower accuracy for checkerboards may be a decoder artifact rather than a weaker cortical response. Target SNR in dB is decoder-independent and helps tell the two apart — which is why it is still shown, beside accuracy, on every page — but that motivation does not make it a confirmatory measure. It is shown as each participant's value, the difference (paired / group / interaction, matching the design), and a GEE estimate with its confidence interval, always labelled "descriptive, never tested": no permutation test is run on it and no p-value, GEE or otherwise, is ever reported for it (the maintainer, 2026-09-24).

SNR is the mean over the decode channel set, not the max. mean_target_snr_db in the existing per-run pipeline takes np.nanmax across channels — a "best-channel" SNR that stays useful for that report, but a max is biased upward, and the bias grows with noise, so a noisier run could post a higher value. ssvep.analysis.contrast computes the mean instead, from the channel_mean_snr_db / channels / decode_channels every metrics.json already carries. No pipeline change was needed for this.

SNR is guarded against the mains line too (#309). target_snr_db / per_channel_snr_db / channel_mean_snr_db sum a target's response over its fundamental and, by default, its 2nd harmonic. A harmonic that lands on the mains line (or one of its own harmonics) is a large, narrow, stationary line in unfiltered EEG — a 30 Hz target's 2nd harmonic is 60 Hz — and summing it into the SNR at a 60 Hz site would report line noise as signal, not response. mean_target_snr_db / per_channel_target_snr_db drop that harmonic from the sum (ssvep.classification.spectral.snr_safe_harmonics) with the decoder's own collision rule (ssvep.classification.decoding.mains_safe_harmonics, #280) — but with a wider tolerance, and per target, not shared across every target the way the decoder's guard is.

The tolerance is wider because the SNR estimator reaches further than the decoder's frequency resolution. A harmonic's SNR divides its bin by the mean of its neighbouring bins, so a mains line anywhere in that neighbourhood inflates the noise estimate and pulls the SNR down as surely as a line in the signal bin pushes it up; and a line leaks into the Welch window's main lobe, two bins either side of its own. So a harmonic counts as colliding within ssvep.classification.spectral.snr_reach_hz: (neighbour bins + guard bins + 2) × the bin width, computed from the estimator's own parameters — 1.75 Hz at 250 Hz with 4 s trials, where the decoder's 1/T is 0.25 Hz. Measured on synthetic data: a 29.6 Hz target's 2nd harmonic (59.2 Hz) sits well outside the decoder's tolerance, but with the line in its neighbourhood its own SNR fell by about 40 dB and the target's sum by 1.8 dB. The main lobe is covered, the side lobes are not: an off-bin line tens of dB above the noise still leaks a smaller, decaying error beyond that reach. At a site whose mains frequency is not configured both 50 and 60 Hz are guarded, as the decoder does, so a 25 Hz target loses its 2nd harmonic there.

On the "per target" part: The decoder drops a colliding order from every decodable candidate, even ones that don't collide, because a candidate left with fewer reference terms scores systematically lower in the canonical correlation the decoder argmaxes over — a bias in which candidate wins. SNR has no such competition: each target's value is computed independently, so a target whose own harmonics never collide is not charged for a different target's collision. A target whose own fundamental sits on the mains line gets no SNR at all — nothing is left to sum, and it is left out of the report rather than shown as a number that is really the mains line.

A trial too short is left out of the SNR and recorded in target_snr_harmonics.skipped_trials, rather than failing the analysis or silently degrading every target's guard (#362). "Too short" is max( 8 samples — the estimator's own absolute floor, below which it cannot resolve a spectrum at all — , half (ssvep.analysis.pipeline.SNR_MIN_TRIAL_FRACTION) of that trial's own design-declared stimulus duration after the onset lead-in (trial.offset_s - trial.onset_s, once the lead-in trim has shifted onset_s forward by the lead-in it dropped — from the recording's stim_on/stim_off markers, never inferred from the EEG data itself) ). The 8-sample floor alone is not enough for a real design: an 8–12 sample trial out of a 4 s (1000-sample) recording is 1–2% of the intended stimulation, but the fixed floor kept it, and using it as "the shortest kept trial" then inflated the mains-guard tolerance to tens of Hz and could silently empty every target's SNR, not just the truncated trial's own. The fraction is a stated, defensible choice, not a measured line: a trial that lost more than half its design's own stimulation after the lead-in is a different, shorter measurement than a full trial, not merely a noisier one.

That tolerance is also resolved per target, from that target's own shortest kept trial, not the recording's global minimum (#362): one short trial for target A must not widen the guard — and so drop harmonics — for an untouched target B. The Run tab's live SNR readout applies the same guard to its own window (against the site's configured mains frequency, the same setting the offline decode and quality checks use — an unconfigured site guards both 50 and 60 Hz, as offline does), so a 30 Hz target no longer reads the 60 Hz hum as a strong response.

A contrast compares SNR on the harmonic orders both arms kept (#309 review). Per target is right for one run's report and biased between runs: a target that lost its 2nd harmonic reports its fundamental's SNR where the others report the mean of f and 2f, and since 2f is usually weaker, the arm holding the colliding target reads spuriously higher. Measured on synthetic data (white noise, a 60 Hz line, 16 seeds): +0.5 to +1.3 dB for two-target arms [12, 30] vs [12, 15] as 2f goes from −3 dB to absent, and +1.0 to +2.6 dB for a single 15 Hz target solid (response 15 Hz) against contrast-reversing (response 30 Hz). So every metrics.json also records each order's own per-channel SNR (target_snr_by_order, keyed by response frequency then order), and the contrast re-sums each arm's SNR on the orders every target in the compared runs kept (ssvep.analysis.contrast.common_snr_orders / snr_db_on_orders): the two runs of each subject for a within-subject contrast, every compared run for a between-subject or mixed one. On the same synthetic data the spurious difference is then under 0.1 dB in every case. An arm whose targets all summed exactly those orders keeps its recorded value, so a contrast no harmonic was dropped from reads the numbers it always did; the page's methods paragraph names the orders used, and a note names who was re-summed. A run analysed with the SNR guard but without target_snr_by_order is refused and needs re-analysing; a pre-guard run is read as recorded, since it summed the same orders in both arms, and D7 already keeps it apart from guarded ones.

What the guard used is recorded in target_snr_harmonics (mirroring the decoder's own mains_guard): the setting (line_hz / line_hz_source), the response frequency of each entry (response_hz), which orders the mains guard kept for each target (harmonics_used_by_freq, [] for a target with no SNR; since #387 an order listed here is summed only if its kernel was also fully real — edge_affected below names any it left out, and target_snr_by_order holds exactly the orders the sum kept), each target's own collision tolerance at its own window (tolerance_hz, one entry per response_hz, None for a target with no kept trials) and any trials left out (skipped_trials, each with n_samples, design_samples and floor_samples so the reason is checkable). A metrics.json from before this guard existed reads as pre-guard (ssvep.analysis.contrast.PRE_SNR_GUARD), mirroring mains_guard_key's own reading of an old record, and D7 (below) refuses to pool a pre-guard run beside a guarded one, or two runs whose SNR mains-guard setting differed — the setting, not which orders it kept per target, since that legitimately differs by design the same way the decoder's own dropped harmonics do. Since #387 the block also carries edge_affected: each target order whose SNR the real-neighbours edge fix touched — "unavailable" (its bin sits within the SNR kernel's half-width of 0 Hz or Nyquist at that target's window, so it has no real noise neighbourhood and is left out of the sum rather than reported inflated; a target left with no order gets no SNR) or "remeasured" (summed on real neighbours where the PSD fetch used to stop at 1 Hz and zero-pad the rest, which read high; or, at a rounding-tie window, summed with a differently sized noise kernel — below). It is [] for every run whose targets are clear of the edges: every target from about 2.25 Hz at a 4 s window, and from about 3 Hz at any window from 0.5 s, so every shipped design — except at a rounding-tie window. The SNR's noise kernel is sized as round(1 Hz / bin width) neighbour bins and round(0.2 Hz / bin width) guard bins, the bin width read off the frequency axis it is given; at a window where one of those quotients is exactly a half (1.5, 2.5 or 3.5 s at a whole number of samples), the old fetch's narrower axis could carry a bin width one rounding error away from the full spectrum's and round to a different kernel (a 1-bin rather than 2-bin neighbourhood at 1.5 s; about 2.5 dB apart on a real response). Whether it did depended on the target. Such orders are "remeasured" too, however far from both edges; the fixed path uses one axis per window, so its kernel no longer depends on the target.

ITR is refused, not silently absent

wolpaw_itr(accuracy, n_classes, selection_time_s) is a deterministic function of accuracy, and the two arms of a within-protocol contrast share n_classes and selection time. An ITR contrast would therefore be a monotone restatement of the accuracy contrast — zero additional information, worse sampling behaviour. Asking for primary_dv: "itr" raises with that reason stated rather than quietly returning nothing. ITR is still shown in the per-subject table and on the sweep plot, as a description.

SNR is refused as a primary DV, not silently descriptive

The same pattern applies to SNR, for a different reason: it isn't that it carries no information, but that it is never tested at all (the maintainer, 2026-09-24 — see above). Asking for primary_dv: "snr_db" raises ssvep.analysis.contrast.SNR_REFUSAL rather than silently computing a test nobody wanted run. SNR still appears on every page, unconditionally, as the descriptive companion to accuracy.

The exploratory sweep

The full decision-window sweep gets a cluster-based sign-flip permutation test (adjacent windows, paired t as the cluster-forming statistic, corrected on maximum cluster mass), reported under an explicit exploratory heading. Only the one pre-declared window above carries the confirmatory test; the sweep is shown so that window can be read in context, and its cluster test answers "is there a difference anywhere in the sweep", nothing more precise about where.

n = 1 is descriptive only

A single subject's difference between two runs mixes the manipulation with everything else that differed between those two runs. With one subject the report shows overlaid sweep curves with Wilson confidence intervals and no p-value, ever — not a test with reduced power, but no test at all.

Between-subject contrasts

A "between" contrast compares two groups of different participants on the one run every participant contributes (run), split on a documented participants.tsv column (between.column, with between.levels naming the two values compared):

{
  "name": "cohort-square",
  "factor": "cohort",
  "design": "between",
  "run": "white-square",
  "between": { "column": "cohort", "levels": ["novice", "experienced"] },
  "confirmatory_window_s": 2.0,
  "primary_dv": "accuracy",
  "description": "Novice vs experienced cohort, on the shape contrast's white-square run."
}

The factor must be a documented column

The column named by between.column must exist in participants.tsv and be documented in participants.json with a Description and a Levels entry for both between.levels (keyed by each level's closed token: a level written as a label such as 18-25 is keyed L18_25, a one-word level is its own key; docs/BIDS.md §"Participant declarations") (docs/BIDS.md §"Adding a participant column"). An undocumented column, a missing column, or a participant whose value is n/a (undeclared) is refused or excluded rather than guessed at — a between-subject factor must say what its values mean before it is compared:

• sourcedata/ is a link (symlink or junction), so no participant's declarations can be read: the whole contrast refuses, because any participant could be a declared pilot (BIDS.md).
• The column doesn't exist in participants.tsv at all, and no participant's declarations name it with a level participants.json documents: the whole contrast refuses. Declare each participant's level with their recordings (python -m ssvep.io.declarations, BIDS.md) so it reaches every machine the contrast runs on; a value typed only into participants.tsv exists only on the machine it was typed on.
• The column exists but isn't documented (no Description, or a Levels entry missing for one of between.levels): the whole contrast refuses.
• A participant's value on an otherwise-documented column is n/a: that participant alone is excluded (MISSING_FACTOR).
• A participant's declared value is neither of between.levels: that participant alone is excluded (OTHER_FACTOR_LEVEL) — e.g. a third cohort the contrast doesn't compare.

group, participant_id, and species can never be a between-subject factor. group in particular is the exclusion declaration — test and pilot participants are dropped through it (see Who is included) — so using it as a compared factor would conflate "excluded from every contrast" with "a level of this one". A contrast naming any of the three is refused at the schema itself (not: {enum: [...]} on between.column), and again — redundantly, on purpose — by SessionProtocol.contrast_problems() and the analysis layer.

Nor can sex or hand, for a different reason: they're converter-owned. The BIDS converter re-derives participant_id/species/group/age/sex/hand/weight/height from its own defaults on every run of 01_xdf_to_bids.py (docs/BIDS.md "Adding a participant column"), so a Levels entry hand-added under sex or hand in participants.json is reverted by the very next conversion, not preserved like a genuine custom column. Use a custom column instead — e.g. handedness_group rather than hand.

The test: label permutation, on the Welch t

The confirmatory test permutes the group labels across participants and recomputes a statistic each time — the between-subject analogue of the within design's sign-flip test, with the same purpose: no distributional assumption, and a stated floor. The reference set is every relabelling either way, so the test is exact in the permutation sense whichever statistic it's built on; what the choice of statistic changes is how well that exact test's size holds up when the two groups differ in both count and spread. The statistic is Welch's t, not the raw mean difference, because studentizing removes most — not all — of the size inflation a raw mean difference shows under unequal group sizes crossed with unequal variances (the generalized Behrens–Fisher problem; asymptotic justification in Janssen, 1997, "Studentized permutation tests for non-i.i.d. hypotheses and the generalized Behrens–Fisher problem", Statistics & Probability Letters 36:9–21). Measured directly at α = .05 under the null, with the smaller group (3–5 participants) at 2–3× the larger group's SD: the raw mean difference rejects 14–26% of the time, Welch's t about 7–13%, better but still above nominal. With the smaller group the less variable one, the studentized test is conservative (about 2–4%), and with comparable variances or equal group sizes it is close to nominal. No permutation statistic fully repairs Behrens–Fisher with three people in a group, and the page says so rather than implying the studentizing step has made group-size/variance imbalance a non-issue: when the groups are unequal in size, the smaller one has 5 or fewer participants and is at least 1.5× as variable as the larger, and the test can reject at all at this n, the report adds a caution naming the numbers. The same reasoning is why the mixed design's interaction test below is also studentized.

The smallest attainable two-sided p with groups of size n0 and n1 is 1 / C(n0 + n1, n1), doubled when the two groups are the same size (swapping which group is which gives the same absolute difference, so half the arrangements are redundant for a two-sided test) — the floor is a property of the label arrangement, not of the statistic, so studentizing does not change it. Two worked cases, both checked against ssvep.analysis.contrast.min_attainable_p_between directly rather than by hand:

• 3 vs 2: C(5, 2) = 10 arrangements, unequal groups, floor 1 / 10 = 0.1.
• 5 vs 5: C(10, 5) = 252 arrangements, equal groups, floor 2 / 252 ≈ 0.0079.

Exact enumeration holds up to the same 2**16 arrangement ceiling the within test uses; Monte Carlo takes over above it, with the observed labelling counted among the draws.

Run order

In a "between" contrast, the declared run's position in the session is set by its group's counterbalancing arm (AGENTS.md §2.1's arm/order distinction), and that arm can happen to coincide with the between-subject grouping — for instance if arm is assigned by subject number and cohort is too. When it does, the page warns that the group difference is confounded with run order: nothing in the design can tell "the group differs" from "whichever group happened to run this arm more often differs". When both arms occur in both groups — the ordinary case, and the one every worked example on this page is built to be — GEE adjusts for arm as a covariate, the same way it does for a within contrast's run-position term.

GEE: the group main effect

ssvep.analysis.contrast.gee_between fits the same family of GEE (binomial for accuracy, Gaussian for SNR, groups=participant) as the within design, with the between-subject factor as the sole covariate. Its coefficient is the group main effect — an average marginal effect in percentage points (with an odds ratio beside it for accuracy) or dB for SNR. For accuracy, the p-value is suppressed below the same 15-cluster floor, for the same reason as the within design. SNR's p-value is never shown at all, at any cluster count — it is descriptive, not suppressed only at small n — and the estimate and CI are what the page reports for it.

One participant in a group is descriptive only

The between-subject analogue of n = 1: with one participant in a group, a "group mean" is one person, and a "group difference" is a difference between two people. No permutation test is run; the strip plot and the estimate stand alone, with no p-value.

The decoder-configuration confound

All included participants must share one decoder configuration. ssvep.analysis.contrast._refuse_group_confound checks that every included run — both groups together, not group-by-group — was analysed with the same decoder method, re-referencing, decode-channel set, mains-guard setting and FBCCA filter-bank setting, and recorded at the same sampling rate. It is not enough for each group to be internally consistent: any difference in how the two groups were decoded could align with the factor being compared, whichever side of the groups it actually falls on. If one group's runs were all decoded one way and the other group's a different way — say, every experienced participant was re-analysed with a newer decode setting the novice group never got — a between-subject "difference" could really be a decoder difference wearing the factor's label. The contrast refuses, naming one subject from each side and the configuration that differs, rather than reporting a number that answers the wrong question.

A duplicated participant_id refuses

If participants.tsv has more than one row for the same participant_id, a "between"/"mixed" contrast has no way to pick which row's factor value applies to that subject's recordings, and picking one would be an unstated selection decision. The whole contrast refuses, naming the duplicated id, rather than silently taking the first or last row.

Mixed designs

A "mixed" contrast asks a narrower question than "is there a within effect" or "is there a between-subject difference": it asks whether the within-subject effect itself differs by group — the interaction, and only the interaction. Every participant contributes both levels (as in a within contrast), and is also split by a documented participants.tsv column (between, same shape as the between design):

{
  "name": "shape-by-cohort",
  "factor": "shape",
  "design": "mixed",
  "levels": ["white-square", "white-circle"],
  "between": { "column": "cohort", "levels": ["novice", "experienced"] },
  "confirmatory_window_s": 2.0,
  "primary_dv": "accuracy",
  "description": "Does the shape effect itself differ by cohort?"
}

The test: labels permuted over each participant's within-difference

Each participant contributes one number — their own levels[1] − levels[0] difference — exactly as in the within design. The confirmatory test then permutes the group labels over those within-differences, using the same label_permutation_test and the same min_attainable_p_between floor the between design uses (§ above), because that is what it is: a between-subject test, just run on within-differences instead of raw values. Simple effects are shown descriptively, never tested — the per-group mean within-difference (fig_paired_mixed, fig_strip_mixed) is on the page so a reader can see which group the interaction is driven by, but only the interaction itself carries a confirmatory p-value; a reader who eyeballs one group's simple effect as "significant" on its own is doing exactly the uncorrected multiple comparison the design is built to avoid.

GEE: the interaction term

ssvep.analysis.contrast.gee_mixed fits level, group, and their interaction (plus run position within the pair, when estimable). The reported estimate is the interaction coefficient — a difference of within-differences — with the accuracy version reported as an average marginal effect in percentage points, and a ratio of odds ratios beside it. The same 15-cluster suppression applies.

The interaction that is tested is additive in accuracy (percentage points), and the ratio of odds ratios is a different question. The permutation test above is run on each participant's own levels[1] − levels[0] accuracy difference, in percentage points — an additive scale. GEE's ratio-of-odds-ratios is a multiplicative, logit-scale statement, and the two need not agree: a group difference concentrated near ceiling (accuracies close to 100%) compresses the logit scale much more than the pp scale, so a real, permutation-detectable pp interaction can sit beside a GEE ratio of odds ratios close to 1, and vice versa. Neither is wrong; they answer different questions about the same data, and the page reports both rather than picking one.

The order term is sometimes not estimable, and the page says why. When the counterbalancing arm (which run a participant did first) coincides with the between-subject grouping — for instance, both happen to follow subject number — run position becomes an exact linear function of level and group, and order cannot be separated from the interaction. When that happens, the order covariate is dropped and the page states the reason instead of reporting an uninterpretable coefficient; the interaction estimate itself is unaffected, because it is still identified from level × group.

Exclusions apply the same way

A mixed contrast excludes on every reason a within contrast can (missing level, ambiguous or split session, fingerprint mismatch, aborted run, and so on) and on the between-subject factor reasons above (MISSING_FACTOR, OTHER_FACTOR_LEVEL), and refuses on the same decoder-configuration confound, checked across the two groups' within-pairs.

Who is included

Everyone on disk. There is no subject-selection argument on any confirmatory entry point (collect, run_contrast, run_protocol_contrasts, build_contrast_reports, the declared list in the Compare tab, 03_run_contrasts.py without --exploratory). The one place people can be chosen is the exploratory path, which reports no p-value. Every exclusion is DV-independent, listed by subject and reason, and shown above the results:

Exclusion	Meaning
EXCLUDED_GROUP	participants.tsv group excludes this subject.
MISSING_LEVEL	A level of the contrast was not recorded (complete-case).
AMBIGUOUS_SESSION	More than one recording exists for a level. The contrast does not choose one, because choosing is a selection degree of freedom; retire the recording that should not count (move it out of sourcedata/).
SPLIT_SESSION	The two levels were recorded in different sessions. A within-subject contrast pairs two runs of one session; across visits, visit would be confounded with level.
FINGERPRINT_MISMATCH	The presented stimulus does not match the protocol's run.
RUN_ABORTED	The run was stopped early, with Abort or with Esc, or lost its EEG stream. A run that merely recorded fewer trials than planned is judged by min_trials, not called aborted.
ANALYSIS_FAILED	No usable analysis output for this run.
TOO_FEW_TRIALS	Decoded trial count below the declared floor.
WINDOW_UNAVAILABLE	The declared confirmatory window is not in this run's sweep.
MISSING_FACTOR	The between-subject column is not declared for this participant.
OTHER_FACTOR_LEVEL	This participant's between-subject value is not one of the contrast's levels.
QC_FAILED	The recording's QC verdict is failed.
PILOT_SESSION	The recording was declared a pilot / development session at lock (its sidecar's data-use purpose pilot, or the older session_position.pilot). Per recording: the participant's study recordings stay eligible.
DECLARATION_CONFLICT	This participant's declarations of a between-subject factor disagree (participants.tsv against the declarations recorded with the recordings, or two sessions' declarations against each other), or one of their declarations files cannot be read. The contrast picks neither side, and excludes as little as it can: a disagreement counts only in a contrast on that factor; an unreadable file leaves out its own session's recordings (it may say pilot), and the participant from any contrast on a between-subject factor (it may declare another level). See BIDS.md.
DECLARED_GROUP_SESSION	The session's declarations, recorded with the recordings after it was made, say group pilot or test. Per recording, like PILOT_SESSION: only the sessions that carry the declaration are left out, and the participant's other sessions stay eligible. See BIDS.md.
UNSCORED_FREE_CHOICE	The recording is a free-choice run (design.cue_mode: "free", #268): the participant chose where to look and no intended target was recorded, so it has no ground truth and no accuracy. Per recording, decided from the run's own manifest before any metrics are opened.
PURPOSE_NOT_POOLED	The recording is not of the one purpose this contrast pools (below): another purpose, no readable data-use stamp, or a malformed one. Per recording, with the reason.

MISSING_FACTOR and OTHER_FACTOR_LEVEL only apply to "between"/"mixed" contrasts (see Between-subject contrasts) — a "within" contrast never reads a participants.tsv column and can't produce either.

MISSING_FACTOR covers two different situations on disk and does not tell them apart in its name, only in its message: a participant with no row at all in participants.tsv (never entered), and a participant with a row whose between.column cell is explicitly n/a (entered, and declared undecided). Both are excluded the same way — a contrast that can't read a factor value can't use that participant — but the page's exclusion message names which of the two it was, because "never recorded" and "recorded as undeclared" call for different fixes on the acquisition side.

That list is exhaustive by construction (ssvep.analysis.contrast.Exclusion): the collection code can only exclude by one of these reasons, and nothing anywhere accepts an accuracy- or SNR-based exclusion argument. Participants whose declared participants.tsv group is test or pilot are excluded via EXCLUDED_GROUP, keyed off that declared metadata — never off a hardcoded subject number. (ANALYST_SELECTION is the one exception in the enum: it is only ever produced by an exploratory comparison's own subject selection, never by a declared contrast.)

A contrast pools exactly one purpose (PURPOSE_NOT_POOLED). Every recording carries a data-use stamp written at lock (docs/DATA_GOVERNANCE.md), and a contrast pools recordings of one purpose only: research on a lab or generic machine, teaching on a teaching machine (where an unstamped class recording counts as teaching), and bench on a dataset the synthetic generator made, which stamps every recording bench because nobody was recorded. Every other recording (another purpose, unstamped, or a stamp that cannot be read) is left out one by one and listed with its session, run and reason, so a bench test or another purpose's data is never aggregated into a claim about participants. The exclusions are decided in one order: first the one rule for what is not study data (ssvep.io.bids.not_study_sessions, which export, the demographics fill and session_type read too: a pilot recording is PILOT_SESSION, a session declared pilot or test afterwards DECLARED_GROUP_SESSION, one with unreadable declarations DECLARATION_CONFLICT), then the purpose rule for what is left.

Pilot / development sessions are excluded per recording (PILOT_SESSION). A recording whose own sidecar says it is a pilot (its data-use stamp's purpose pilot, or the older session_position.pilot: true) — the operator declared it not study data at lock (docs/SITE_PROFILE.md §3a, "Piloting before approval") — never enters a contrast, and is listed with its session and run. It is read from the sidecar, so no conversion step and no participants.tsv edit is needed for it to hold. The participant's other, study recordings are still eligible: if they cover every level of the contrast, the participant is included from those; if not, the missing level is listed too.

group is a declaration. The BIDS converter never overwrites it. It fills an empty cell with test for the reserved sub-000 bench code and n/a (undeclared) for everyone else, and contrasts include undeclared participants with a note. So marking someone pilot is a deliberate edit. Before #269 the converter stamped everyone except sub-000 pilot on every conversion (and Analyze converts on the way in), which would have excluded every participant ever analysed. The first conversion under the new rule rewrites those stamps (a pilot with nothing else declared on its row) to n/a, once, and records that in the dataset's dataset_description.json (SSVEPParticipantsGroupRule). Until then, contrasts read such a stamp as undeclared, so a dataset is read correctly without anything being written. The one limit: before the marker exists, a pilot set by hand looks exactly like the old stamp, because no demographics are recorded to tell them apart. So hand edits are honoured once the dataset is marked, and Analyze marks it at the start of every batch, whether or not anything needed converting.

The window every contrast reads at (#457)

Every dependent variable in a contrast (accuracy, trial count and ITR) is read at the declared confirmatory_window_s, from each run's window_sweep. A run's own operating_window_s (the window its per-run report headlines) is never a contrast input, so two runs whose operating windows differ are still compared on the same window. A run whose sweep has no row at the declared window is excluded as WINDOW_UNAVAILABLE, named with its reason on the page above the results (a live-stopped run is the exception, see the live-stopped bullet under Matching recordings to arms). Both properties are pinned by tests.

The window-length confound that #457 describes therefore belongs to the per-run exploratory report, whose headline accuracy is "at its own operating window": two runs reported at different operating windows are not comparable by eye. It is not a property of a contrast.

Matching recordings to arms

A run is matched to a contrast's level on protocol name + run label. Its presented stimulus is then verified against the current protocol by a fingerprint of the run manifest with the acquisition block excluded — that block is legitimately rewritten at record time (an as-run headset override, the measured impedance mirrored into the montage), and everything a participant actually saw sits outside it.

The protocol-level fingerprint is deliberately not required to match. Adding a contrasts block changes SessionProtocol.fingerprint(), because it hashes the whole protocol dict — so a recording made before the block existed would otherwise look like drift, and would need editing to be includable. That is not acceptable: a contrast declaration changes what is analysed, not what was presented. A protocol-fingerprint mismatch is therefore reported as a note on the page, never as drift.

This is a deliberate, narrow weakening, stated as one: the protocol-level fingerprint also covers run ordering and the counterbalancing scheme, and a change to either would not be caught by the per-run check alone.

Refusals

A contrast refuses, with the mismatch or reason named on the page, rather than computing something that doesn't answer the question asked:

• Method, re-reference, decode-channel-set, mains-guard, filter-bank, decision-rule, sampling-rate, TRCA-band, SNR-mains-guard, Nyquist-alias or SNR-edge-fix mismatch. If a subject's two arms were analysed with different settings — decoder method, re-referencing, a different decode channel set, a different mains-guard setting, a different FBCCA filter-bank setting, a different decision-window rule, a different sampling rate, a different TRCA band, a different SNR-DV mains-guard setting, a different Nyquist-alias-guard outcome, or (only for a run the fix changed) target SNR from either side of the edge fix — the contrast refuses for that subject's inclusion to even be attempted. Comparing two different analyses is not a contrast of the manipulation.
  • The SNR DV's own mains guard is compared by its setting too, not by which harmonics it kept per target (#309). target_snr_harmonics (mirroring mains_guard) records line_hz / line_hz_source and, per target, harmonics_used_by_freq — ssvep.analysis.contrast.snr_harmonics_key reads only the setting, the same "setting, not outcome" rule mains_guard_key follows, because which orders a target's own sum keeps is per target (see above) and legitimately differs by design. A record from before this guard existed reads as pre-guard (ssvep.analysis.contrast.PRE_SNR_GUARD) and never pools with a guarded one; a target_snr_harmonics block that is present but unreadable refuses on its own, matching an unreadable filter_bank.
  • The filter bank is compared by its setting, not its sub-bands. metrics.json records the FBCCA filter bank a decode ran with (filter_bank: the sub-bands actually filtered, their weights, and the setting that produced them — at most n_subbands_requested sub-bands, sub-band k weighted k^−a + b). The sub-bands scale with the candidate frequencies, so a checkerboard decoded at 2f has different sub-band edges from a solid square decoded at f, and a high-frequency run can fit fewer sub-bands below Nyquist. Those differences follow from the design and do not refuse; a different setting does. The sampling rate moves the edges too (it sets where Nyquist is), but it is acquisition, not design, so it is compared on its own: two runs recorded at different rates refuse, naming the rates. Within a subject that cannot happen in practice, since both arms come from one session on one amplifier; between people recorded on different amplifiers it can. The mains guard is compared the same way: its setting, not which harmonics it dropped.
  • A metrics.json from before the filter bank was recorded reads as the setting it was decoded with: 5 sub-bands, a = 1.25, b = 0.25 (ssvep.analysis.contrast.PRE_RECORD_FILTER_BANK), the only setting the decoder had until then. So it pools with a new record decoded with that same default and refuses beside any other. A record from before the mains guard, by contrast, reads as pre-guard and never pools with a guarded one, because it was decoded without the guard: a different analysis, where the filter bank's was the same one. Only a missing filter_bank key reads this way: a key that is present but null or empty says nothing about the decode, so the contrast refuses that run until it is re-analysed.
  • Scope: filter_bank covers FBCCA (and plain CCA) only. TRCA is reported beside them but band-passed separately, and its band lives in its own key, trca.band_hz (the edges actually built, post-clamp since #325 — trca.band_hz_requested is the unclamped candidate). D7 compares it too (trca_band_key), and the same way it compares the filter bank: by the frequency-invariant margin TRCA band-passes either side of the response range (today always 2 Hz), not the edges themselves — so a within-subject contrast whose two arms decode different response frequencies (the pattern contrast: solid at f, checkerboard at 2f) still pools for TRCA, exactly as it does for FBCCA. A record from before #325 has only the unclamped value under band_hz itself; the margin is recovered from it and the record's response_freqs_hz the same way (the clamp arithmetic never changed, only the record was wrong), so no "old default" is needed the way filter_bank's is. A trca block that is present but unparseable refuses on its own, matching a null filter_bank.
  • The Nyquist-alias guard is compared by its outcome, the one exception to "setting, not outcome" above — because it has no setting (#347). A reference harmonic that aliases at or above Nyquist for any decodable candidate is dropped from every decodable candidate, regardless of the mains-guard setting (#310, #346); the record is nyquist_guard (mirroring mains_guard) plus harmonics_used_by_freq on every sweep row. Unlike the mains guard, the filter bank, TRCA's band or the SNR guard, this guard has no on/off switch and nothing configures it: what it drops is a fixed fact of the candidate frequencies and the sampling rate, not a knob two analyses could have set differently, so there is no setting for ssvep.analysis.contrast.nyquist_guard_key to fall back on comparing. Comparing the raw shared set (nyquist_guard.harmonics_used) directly would still break the reversal contrast, though, because that set folds the mains guard's own (legitimately design-driven) drops in too — so this key first subtracts out the mains guard's own contribution (mains_guard.harmonics_used minus the merged set), leaving only what aliasing itself removed. Two arms that legitimately differ in which order collides with mains, but whose designs never cross Nyquist, both read () and still pool; the same design decoded at two sampling rates where a harmonic aliases at one and not the other (a 3rd harmonic aliasing at 250 Hz sampling but not at 500 Hz) reads two different sets and refuses, naming them. A record from before #310 reads () (nothing was ever dropped for aliasing, known deterministically, since the pre-#310 decoder had no alias awareness at all) whenever its own mains_guard is readable; a nyquist_guard (or, for a pre-#310 record, mains_guard) present but unreadable refuses on its own, matching an unreadable filter_bank. The subtraction alone can fail open: mains_guard.harmonics_used is itself computed on each candidate's folded frequency, alias-unaware, so when an aliased harmonic's fold happens to land within the mains guard's own (window-dependent) tolerance of a mains harmonic, the mains guard drops that order too, for the wrong reason, and the subtraction reads () — hiding a real alias drop, and only for as long as the window's tolerance keeps reaching the fold. So the key also computes the alias-only set directly, from _is_alias over the record's own decodable response_freqs_hz at its own sfreq, and unions it in — a fact of the design and sampling rate alone, independent of the mains tolerance and the window length (review of #363).
  • The target SNR's real-neighbours edge fix is keyed so that only a run it changed refuses (#387). The fix changed the snr_db DV for target orders near 0 Hz or Nyquist, and, at a rounding-tie window (1.5, 2.5 or 3.5 s; above), for orders whose old, narrower frequency axis sized the noise kernel differently; every other order is bit-identical across it. So ssvep.analysis.contrast.snr_edge_key reads () for every record written after the fix (its target_snr_harmonics carries edge_affected, whatever its contents — two post-fix records pool whatever their designs), and () too for a record written before it none of whose target orders the fix changes at any window that record could have been analysed at. Which orders a pre-fix record had affected is derived deterministically from what that record already carries — sfreq, each target's response_hz and harmonics_used_by_freq, its tolerance_hz (which pins each target's window) and the snr_spectrum curve's bin spacing (the run's shortest kept trial, a floor on every target's window) — through the same rule the fix applies (ssvep.classification.spectral.snr_edge_status); where the window is only bounded, every window it could have been is tried and an order affected at any of them counts, so the uncertainty can only refuse, never pool. Nothing in a pre-fix record bounds the window from above, so that search refuses more often than "only near the edges" suggests, and it is meant to: refusing a pooling that would have been safe costs a re-analysis, pooling one that was not costs a wrong result. Concretely, at 250, 500 or 1000 Hz sampling a pre-fix record refuses at a rounding-tie window; with the SNR mains guard on, its tolerance_hz pins the window to those that reproduce it, but 2.0 and 3.0 s share the 3.5 s tie's tolerance at 250 Hz, so a design with a target the 3.5 s tie touches (the teaching grid's 8.5 and 10.5 Hz, for one) refuses at those too; with the guard off nothing pins it, so a pre-fix record refuses whenever its shortest kept trial was 3.5 s or less (2.5 s for a design only the 1.5 and 2.5 s ties touch). At 125 Hz there are no rounding ties. Separately, the "about 3 Hz at any window" threshold holds for windows of 0.5 s and up, and the pipeline keeps a trial down to 8 samples or half its design length, so a pre-fix record with one kept trial truncated to 0.3 s or less has candidate bins wide enough that targets well above 3 Hz count as affected, and it refuses at any rate. In every case the remedy is the one the refusal names: re-analyse the run with the current toolbox. A pre-fix record with an affected order reads ("pre-fix", "<f> Hz order <k>", …) and refuses beside a post-fix one (two pre-fix records naming the same orders still pool: they carry the same inflation). A pre-fix record with no SNR block or no orders reads () (the first is already refused as pre-guard by the SNR mains-guard key; the second has nothing the fix could change). A pre-fix record with orders but no readable sfreq, no snr_spectrum curve of at least two bins, or orders that cannot be parsed reads "pre-fix, window unknown" (ssvep.analysis.contrast.UNKNOWN_SNR_EDGE_WINDOW) and refuses on its own, even beside an identical record: run_analysis always writes both with kept trials, so it is a hand-built or stripped record, and whether the fix changed it cannot be told — so it fails closed. An edge_affected present but not a list refuses on its own, matching an unreadable filter_bank.
  • The SNR kernel's size and the Nyquist bin are decided exactly, and keyed the same way (#392). The noise kernel used to be sized from the spacing of the first two bins of whatever frequency axis a caller passed, so at a rounding-tie window (1.5, 2.5 or 3.5 s) two slicings of one bin grid could round to different kernels, and whether the spectrum had a Nyquist bin at all depended on rfftfreq putting it one ulp above sfreq / 2 (about one window in fourteen at 125–1000 Hz). Now ssvep.classification.spectral.snr_kernel_bins chooses the kernel from sfreq and the window as exact rationals, rounding a tie half to even, and every SNR (the DV, the live readout, the plotted and the live spectrum, the mains guard's reach, the edge rule) takes it from there; compute_psd keeps the Nyquist bin by its index. Half to even is what the DV's own full-grid quotient and the mains reach already rounded to at every window at 125, 250, 500 and 1000 Hz, so the tie rule moves no DV value there; round half up would have moved every 2.5 s window. What does move is an order against the top of the band — within 3.5 Hz of Nyquist — at a window where the old mask dropped the Nyquist bin: it had no full kernel then and is summed now (503 of the 7 034 windows 0.25–4 s at those four rates, one bin each). The plotted and live SNR spectrum also change at a rounding-tie window (1.5, 2.5 or 3.5 s at 250, 500 or 1000 Hz, and at other sampling rates' own ties), depending on where the band's cut fell, because their sub-band axes had rounded the other way; that curve is never pooled. A record written since carries target_snr_harmonics.exact_bins: true and reads (). One with edge_affected but no exact_bins was written between #387 and #392: snr_edge_key derives, from its own sfreq, orders, tolerance_hz and snr_spectrum bin spacing, which of its orders #392 moved at any window it could have been analysed at, and reads () (pools) when none did or ("pre-exact-bins", "<f> Hz order <k>", …) (refuses beside a current record) otherwise — the same derivation, window bounding and fail-closed rules as the pre-#387 key, in the same field, with its own tag so records of the two eras never pool by accident. No shipped design has an order that close to Nyquist, so none refuses. A pre-#387 record is now compared with the current toolbox, so an order that both the pre-#387 fetch and the current axis give a full kernel at the Nyquist edge no longer refuses. exact_bins present but not true refuses on its own.
  • The decision-window rule is compared the same way (#23): the setting, not which window a trial actually decided at. metrics.json records decision_rule: "fixed" for ordinary single-operating-window decoding, or, for an opted-in run, "early_stop" with the stopping statistic by name (statistic, currently rho_margin) and the schedule it used (min_window_s/max_window_s/step_s/confidence_margin; decoding.decision_rule_summary). A different statistic, threshold or schedule is a different rule and refuses beside this one. Which window each trial stopped at (dynamic_decode.decided_window_s) is a result, which legitimately differs trial to trial and arm to arm, and is never what pooling compares.
  • Live-stopped runs never pool with fixed-length runs (#23). A run whose trials the live decoder stopped early records decision_rule.applied: "live", and its key ends applied=live, so it never equals an offline rule or "fixed": the participant saw different trials, whose length was itself feedback. Within a pair that refuses; across subjects, where a configuration difference is otherwise only a note, a contrast that mixes the two stopping modes refuses too. The exception is a contrast that declares the stopping mode as its factor, "factor": "stopping": it compares the two, its accuracy is read at the declared window from each run's sweep, and its notes give each mode's mean decision time and ITR, each under its own rule, because the arms differ in decision time. A live run's sweep ends at its first look (window_sweep_truncated.at_s, the last window every trial has), so a declared window past it refuses the contrast, naming the run and asking for a window at or below the first look; a live run is never excluded as WINDOW_UNAVAILABLE, which would drop the very arm the contrast is about. The same holds in a contrast that does not declare the stopping mode: a live run past its first look refuses by name, and at or below it the run still never pools with fixed-length runs (a contrast of live-stopped runs only pools them). A run whose manifest declares design.early_stop but whose metrics are not the live analysis refuses by name too. Two things in a stopping contrast are not the decisions the participants experienced, and are said so in its note: its accuracy, the GEE's time_in_run covariate and every per-trial outcome are the live arm's fixed-window outcomes at the declared window, not its live decisions; and the SNR beside accuracy is on two windows (the live arm's trials cut to its first look, the fixed arm's whole trials), so it is descriptive only. ITR is charged at the presented time plus the ITI in both modes (cue and feedback slots in neither), so the two are on one convention. A contrast of live-stopped runs only carries a note that its accuracy and outcomes are the fixed first-look re-decode, not the live decisions, and that decision time is not compared; factor: stopping needs both modes present and refuses when every compared run is of one mode.
  • Only an absent decision_rule reads as "fixed". A metrics.json from before the record existed can only have been decoded at a fixed window, so it pools with an explicit "fixed" record and refuses beside an early-stopped one. A decision_rule that is present but null, empty or not an object, names a rule this version does not know (smoothing, the other half of #23's title, is not one yet), names an unknown statistic, or is an early_stop without four finite numbers, or carries an applied other than offline or live (or live on a fixed rule), is unreadable: the contrast refuses that run until it is re-analysed, even beside an identical record, as it does for an unreadable filter_bank.
  • dynamic_decode.calibration_caveats is diagnostic, not a setting: it never enters D7. It names which of the false-stop calibration's own conditions (the roadmap doc's increment-3 record) this particular run falls outside of — fewer than 3 candidates, fewer than 4 decode channels, a schedule other than the calibrated one, or a CCA candidate in the 8–13 Hz alpha band — and a non-empty list is also raised as a pipeline.CalibrationCaveatWarning at analysis time. Two runs sharing one decision_rule setting still pool even when their caveats differ (one has 8 decode channels and no caveat, the other 2 and one): the setting is what D7 compares, and a caveat says something about how much the false-stop bound can be trusted for this particular run, not about what it was decoded with. The shipped defaults (CCA 0.6, FBCCA 0.45) hold the false-stop rate to about 1 % / 0.7 % under the calibration's own noise model — not a guarantee for every deployment. See the roadmap doc and ssvep.classification.decoding.DEFAULT_CONFIDENCE_MARGINS for the harsher numbers a later, independent re-calibration measured outside that model, including on the teaching lab's own 8–12 Hz grid with CCA, and for why the maintainer chose the tighter margins over the faster, looser pair that calibration also measured.
• The two between-subject groups were not analysed identically. A "between"/"mixed" contrast refuses when the two groups' runs were decoded with different settings — see The decoder-configuration confound. Comparing groups analysed differently is not a contrast of the factor.
• between.column is undocumented, missing, or reserved. A "between"/"mixed" contrast whose factor column doesn't exist in participants.tsv, isn't documented in participants.json, or names group, participant_id, or species refuses outright, rather than excluding participants one at a time — see The factor must be a documented column.
• primary_dv: "itr". See ITR is refused above.
• primary_dv: "snr_db". See SNR is refused above.

Free-choice runs never enter a contrast (#268)

A free-choice run is not scored, so it has nothing an accuracy contrast could compare. It is excluded by name (UNSCORED_FREE_CHOICE) from the manifest's design.cue_mode, never from what its metrics file happens to contain, and the exclusion is listed with the run like every other. Pooling a free run with cued runs on accuracy is therefore impossible, not merely discouraged.

Runs that differ in participant feedback (#268)

A run can show the participant a check mark or cross after each trial (design.feedback). Feedback changes motivation and strategy, so a run with it and a run without it are not the same task. A contrast that pools them does not refuse: it records a caveat in the result's notes naming the modes present. Each run's mode is read from its own manifest when it is analysed and stored in its metrics.json as feedback_mode (absent for a run with no feedback, which is every run from before the feature). A contrast between a feedback level and a no-feedback level is a legitimate design, and carries the same caveat.

Within-run conditions and their order (#31)

The contrasts above compare runs. A run with several conditions inside it (a factorial sweep) cannot be contrasted yet: a multi-condition run is refused (N decoded conditions in metrics.json; expected exactly one), and the within-run contrast, with its k-level and crossed-factor tests, is the rest of #31. What is settled now is the part that decides whether such a contrast can ever be confirmatory: the order the conditions were played in (docs/PROTOCOLS.md, The order of conditions inside a run).

• Confirmatory only under a randomised or counterbalanced order. ssvep.analysis.order_effects.require_within_run_order(manifest) passes a run whose design.order.scheme is blocked_randomized or counterbalanced. On a fixed-order run (including every run that declares no order) or one with an explicit, hand-written order it raises OrderConfounded (a ContrastRefused), saying that condition is confounded with time-in-run for every participant, so fatigue or drying gel would read as a condition effect. With exploratory=True it lets the run through labelled exploratory instead; what an exploratory comparison may report is #284's to decide. The within-run contrast, when it is built, goes through this gate.
• Time-in-run is always reported; it is adjusted for only under a randomised or counterbalanced order (the maintainer's request on #31). order_effects.order_confound is a design-only diagnostic from the order the run played: the share of trial-position variance explained by condition (η²; about 1 − 1/k² for a fixed blocked order over k conditions, 0 for a complete counterbalanced one) and each condition's mean position. Every multi-condition run gets it. order_effects.position_adjusted_accuracy is the covariate: a least-squares linear-probability fit, correct ~ condition + position (position 0 = first trial, 1 = last), which gives each condition's accuracy at the run's mean position next to its raw accuracy, plus the position slope. It is estimation, never a test: no p-value, for the reasons GEE's is withheld at small n (above). It reads the per-trial outcomes metrics.json now records (trial_outcomes: block, trial and whether the operating-window decision was right) and places each trial in the run through the sidecar's plan.trial_order; a record from before trial_outcomes existed, or a synthetic one, gets the diagnostic but no adjusted values.
• Why a fixed order gets no adjusted values. Under a fixed blocked order each condition owns its own stretch of the run. The position slope can then only be estimated inside each block, and "adjusting" means extrapolating it across stretches the other conditions never played in. With the trial counts a run has, that manufactures differences: three conditions with the same true accuracy and no drift at all, 20 trials each, can come out raw 0.80 / 0.65 / 0.70 and "adjusted" 0.98 / 0.65 / 0.52, from noise alone. So on a fixed or hand-written order the adjusted values are withheld, with the reason stated (order_effects.ADJUSTMENT_WITHHELD), and η² with a plain warning is what the run reports. A linear-probability coefficient can also fall outside [0, 1]; under a deconfounded order such a value is clipped to the bound and flagged (clipped, adjusted_unclipped, out_of_range), because it says the linear model does not fit that condition, not that the accuracy is 1.5.
• Where it shows. The per-participant sweep ranking, ssvep.analysis.optimization.optimization_report (descriptive under #269 D1), carries an order block with the status, the diagnostic and, when the records have outcomes and the order allows it, the adjusted accuracies; its text table (format_ranking_table) says "order-confounded" under a fixed-order run's ranking and why it is not adjusted. The function is available to any caller that has a run's per-condition records, but no pipeline stage or GUI page calls it yet, so nothing produces this report on its own today.
• In the run-level contrasts. Their order is the order of runs in a session, which the protocol already declares and counterbalances, and the permutation test is the confirmatory test. Their GEE estimate adjusts for run position and for time in run wherever each can be estimated, and says why where it can't (Order covariates).

Exploratory comparisons: asking an undeclared question (#284)

A contrast exists only if the protocol declares it, before the data are read. A question nobody declared can still be asked, on its own path, under its own label. An exploratory comparison is never a contrast, never confirmatory, and never silently either.

• The p-value is shown, uncorrected, labelled, and cautioned (maintainer, 2026-10-09). Accuracy gets the same test a declared contrast gets, computed the same way (the permutation test, GEE, the sweep's cluster test). Wherever its p appears (page, status line, CLI, Compare tab) it is named p (exploratory, uncorrected), with a caution beside it: the comparison was not declared before the data were seen, so the p-value is descriptive and cannot support a confirmatory claim, and trying several comparisons inflates false positives. There is no Holm adjustment, because there is no declared family to correct against. SNR stays descriptive and never tested, as in a declared contrast. A result that matters is declared in the protocol and recorded again.
• Labelled everywhere. The page opens with an EXPLORATORY banner and its title and filename say so; the status text, the CLI output and the Compare tab's own box say so. The page's methods say what was and was not done.
• The same gates as a declared contrast. The same collection runs: pilot and test groups, each recording's data-use stamp (a stamp the declared path refuses is refused here, per recording), aborted or failed runs, the presented-stimulus check, identical decoder configuration across arms, and live-stopped runs never pooled with fixed-length ones. Nothing about being exploratory loosens any of them.
• One protocol. The comparison names a protocol and runs of it (or a between-subject column of its participants). Across protocols the manifests, headsets and decoder settings can differ in ways nothing here can vouch for, so it is refused.
• Subject selection, the one added freedom. --subjects (or the Compare tab's participants field) narrows the pool to the people you name. It only narrows: the stamps and every other check still apply to them, a name with no recording of the protocol is refused, and the page states that the selection was the analyst's choice and lists everyone left out as such. A declared contrast has no such option (D12).
• Its own files. The page is derivatives/ssvep-analysis/contrasts/_exploratory/<protocol>/exploratory-<name>.html. It is rebuilt in place, listed nowhere in the declared contrasts index, carries the ssvep-provenance meta tag and the pooled-sessions list (so deleting a session deletes it), and nothing else is recorded: there is no log of what was tried.
• A hand-built spec cannot get a confirmatory p-value. contrast.run_contrast hands any spec the protocol does not declare to the exploratory path, and an exploratory comparison cannot name the stopping factor (that word is what lets a declared contrast compare live-stopped with fixed-length runs). build_contrast_reports, run_protocol_contrasts and collect take no spec choice and no subject selection; the selection lives in collect_exploratory only.

python code/03_run_contrasts.py --exploratory --protocol <protocol> --name <label> \
    --factor <what differed> --design within --levels <runA> <runB> --window 2.0 \
    [--subjects sub-101 sub-102]     # add --check to see who it would include, writing nothing

Running it

From the GUI: the Compare tab, after Analyze has produced metrics.json for the runs involved (contrasts read derivatives only and never decode). From the command line, in BIDS/code/:

conda activate ssvep
python code/02_run_offline_pipeline.py     # analyse first — contrasts read metrics.json, never decode
python code/03_run_contrasts.py            # every contrast of every protocol recorded here
python code/03_run_contrasts.py --check    # who each contrast would include/exclude, and why; writes nothing

--check is the verification step to run on the machine holding the recordings before relying on the protocol-fingerprint rule above: it prints each recording's session_position (group, label, arm, protocol fingerprint) and its presented-stimulus fingerprint, then what each declared contrast would include and exclude.

Where the report lives

One HTML page per (protocol, contrast), under derivatives/ssvep-analysis/contrasts/<protocol>/<contrast>.html, with contrasts/index.html listing every one and the root derivatives index linking to it. Like the rest of the derivatives tree, every page is rebuilt in place, never accumulated — re-running always covers everyone currently on disk. These pages are participant-derived and stay inside the git-ignored BIDS/ tree; each carries an ssvep-provenance meta tag (participant-derived, or synthetic when the dataset is ssvep.analysis.synthetic's own output), which is what lets the built-site scan (scripts/scan_built_site.py) refuse to publish one that was ever copied into docs/.

A worked example, on synthetic data

Every figure and number below comes from ssvep.analysis.synthetic — no participant, recording, or measurement here is real (AGENTS.md §4). Reproduce all of it with:

conda activate ssvep
python scripts/build_contrast_figures.py

The script generates two synthetic datasets from the shipped teaching protocol, with a known, injected effect per contrast, and runs every declared contrast over each with ssvep.analysis.contrast.run_protocol_contrasts. Try the generator yourself with scripts/generate_synthetic_contrasts.py --out <dir> --effect shape=0.8,1.0 --n-subjects 5. The same script also builds the between/mixed worked example below, declaring a "between" and a "mixed" contrast in memory over the teaching protocol's own runs, rather than in the shipped protocol file — see below.

n = 5 — the protocol's actual class size

At n = 5, shape was injected with a real effect on both accuracy and SNR (Effect(log_or=0.8, snr_db=1.0)):

The permutation test reports p = 0.0625 — which is exactly min_attainable_p(5), the floor, not a near-miss. GEE's estimate (≈ +10 percentage points, 95% CI roughly [+8, +12]) is shown; its p-value is suppressed, with the reason stated on the page, because five subjects is below the 15-cluster floor. Notice how narrow that CI is for five subjects: it is the anti-conservative sandwich standard error the floor exists to warn about, which is why the estimate is shown as estimation and not as evidence.

pattern was injected to disambiguate the decoder from the brain (Effect(log_or=-0.4, snr_db=0.0)): accuracy moves down, while SNR's descriptive difference is +0.55 dB (GEE 95% CI [−0.17, +1.06]) — a CI that includes zero is not evidence of an effect, and none was injected here; the number is the noise a descriptive measure carries at five subjects, not a real movement.

At n = 5, accuracy's permutation test does not reach its own floor of 0.0625 either — five subjects is not enough to confirm a real difference of this size, which is itself part of what the power section below is about. SNR is descriptive at every n, so there is no floor or p-value for it to reach or miss; its GEE difference is shown with its CI under a "descriptive, never tested" heading, with no p-value slot at all, at every n, not only below the 15-cluster floor.

colour was injected as a true null on both DVs (Effect()), matching the isoluminant pair it is built from (docs/EXAMPLE_TEACHING_LAB.md §4.2).

n = 40 — what changes with more subjects

The same three effects, at n = 40: the permutation floor drops from 0.0625 to 1 / 20000 (this analysis's Monte-Carlo arrangement count above EXACT_MAX_N = 16), and GEE's p-value now appears, because 40 subjects clears the 15-cluster floor.

For shape, accuracy comes back p = 0.0001 (the permutation floor at this n is 1/20000, so this is close to it but not at it; Holm-adjusted across the protocol's three contrasts, p = 0.0003), GEE estimate ≈ +13 percentage points [+8, +19], GEE p < 0.0001. For pattern, accuracy is p = 0.0002 in the direction the injected effect predicts (GEE estimate ≈ −10 percentage points) — while its SNR difference, shown descriptively, is close to zero and its CI straddles zero, no p-value reported. That is the decoder-vs-brain disambiguation this design exists to demonstrate: a confidently detected accuracy drop, alongside a decoder-independent measure whose difference shows nothing moved. For colour, accuracy shows nothing at n = 40 either, consistent with the true null it was built from, and SNR's difference is likewise near zero.

Between and mixed, n = 10

Neither of "between" or "mixed" is in the shipped teaching protocol — that protocol is in live class use and under a change hold — so this example declares its own contrasts over the teaching protocol's runs, in memory, the same way tests/test_contrast_between.py does. A synthetic cohort column (novice/experienced) splits ten synthetic participants 5/5; Effect(log_or=0.5, group_log_or=1.0, interaction_log_or=1.3) injects a real within-subject shape effect, a real between-subject group effect on top of it, and a real interaction. Cohort is deliberately crossed with the counterbalancing arm, not assigned in lockstep with it: each cohort contains participants who ran both counterbalancing arms (scripts/build_contrast_figures.py's COHORT_PATTERN_N10), because a docs example that let cohort track run order would be demonstrating exactly the confound Run order above warns about, in the one place a reader is meant to trust the numbers.

cohort-square (design: "between", run: "white-square"): group sizes (5, 5), so the permutation floor is 2 / 252 ≈ 0.0079 — the 5-vs-5 case worked above. The observed group difference is +30.4 percentage points, permutation p = 0.0159, Holm-adjusted p = 0.0317 (the family here is two contrasts: cohort-square and shape-by-cohort), below .05 even after correction. GEE's estimate agrees, +30.9pp (95% CI roughly [+3.9, +57.8]); its p-value is suppressed, with the reason stated, because ten participants is below the 15-cluster floor, the same suppression the within design shows at small n. Target SNR, descriptive only, shows a group difference close to zero (−0.33 dB), with a CI that straddles zero — no test, no p-value, at any n.

At ten participants split 5/5, this injected effect is large enough that the permutation test clears its own floor and survives the Holm correction — unlike the within design's n = 5 example, which is built to show the opposite case. Read together, the two examples are the same lesson from both sides: the floor is about the design's own resolution, not about whether any particular run finds something.

shape-by-cohort (design: "mixed", levels: ["white-square", "white-circle"]): the same 5/5 groups, the same floor. The interaction — the difference between the experienced group's own shape effect and the novice group's own shape effect — is +15.6 percentage points, permutation p = 0.0635, Holm-adjusted p = 0.0635 (this contrast's own p was already the larger of the two, so Holm leaves it unchanged) — just above .05, where cohort-square's main effect was just below it. GEE's interaction estimate is +16.3pp [+1.4, +31.1], again suppressed below the 15-cluster floor.

The second figure is literally what the permutation test permutes: each participant's own white-circle − white-square difference, grouped by cohort. Reading it against the first figure is the point of showing both — the first shows each group's simple effect (descriptive only, never tested); the second isolates the one number, per participant, that the interaction test actually uses.

Power, stated in advance

Five subjects, one paired contrast. For a paired design at α = .05, 80% power needs a within-subject standardized effect d_z ≈ 1.66 — a very large effect, well above what most manipulations of this kind produce. Every run of this protocol will produce a difference, a confidence interval, five visible per-subject lines, and a permutation p-value that cannot fall below 0.0625 whatever the true effect is, because five subjects bounds the test's own resolution before any data are collected.

The teaching content is the power analysis, not the result. A class that runs this protocol is not expected to detect the shape or pattern effects with confidence at n = 5 — the worked example above shows exactly that gap between what a real effect of this size looks like at n = 5 and at n = 40. What the exercise teaches is reading a floor as a floor, not as "almost significant", and recognising when a design's own sample size has already decided the answer to "was that significant" before anyone sat down. The feature itself is built for general use at whatever n a study actually has; this protocol is the shipped teaching example of it.

Protocol files

The contrast declaration defines the confirmatory test, so it must come from exactly one file. If two protocol files under session-protocols/ share a name and differ, every contrast of that protocol is refused, naming both files, rather than resolved to whichever sorts first. Identical copies are fine.