BIDS conversion & the `BIDS/` dataset¶

How raw acquisitions become a BIDS-EEG dataset, and how to regenerate/analyze it. The BIDS/ tree is git-ignored (participant data can never enter git by construction — see docs/DATA_GOVERNANCE.md; each deployment binds those rules to its own governance documents); only the code below is version-controlled.

Layout¶

BIDS/
├── CHANGES, .bidsignore                            # static, tracked
├── dataset_description.json, README                # built from the SITE PROFILE
│                                                   #   GIT-IGNORED: they take on this
│                                                   #   deployment's institution + ethics
│                                                   #   reference. BIDS/*.template ship instead.
├── participants.json                              # the column dictionary
│                                                   #   GIT-IGNORED: merged by the converter from
│                                                   #   its own defaults + any custom column a
│                                                   #   deployment documented by hand (e.g. a
│                                                   #   between-subject contrast factor).
│                                                   #   BIDS/participants.template.json ships instead.
├── participants.tsv                               # sub-XXX + species/group; age/sex/hand/weight/
│                                                  #   height = n/a unless entered by hand;
│                                                  #   age_range/gender from the background
│                                                  #   questionnaire once administered (#290)
│                                                  #   GIT-IGNORED: grows a row per participant,
│                                                  #   regenerated by the converter
├── task-ssvep_events.json, channels.json         # inherited sidecars (event + impedance columns)
├── code/                                          # dataset-local runners (see below)
├── stimuli/<sha256>/<name>                        # pictures image-target runs showed (#258);
│                                                  #   events.tsv:stim_file names them
├── sourcedata/sub-XXX/ses-YYY/eeg/                # ← RAW: *.xdf + *.session.json + *_impedance.json
├── sourcedata/sub-XXX/ses-YYY/stimuli/<sha256>/   # ← the pictures, as saved with the recording
├── sourcedata/sub-XXX/ses-YYY/declarations/      # ← the session's group / the participant's factor
│                                                  #   level, declared with the recordings (#276)
└── sub-XXX/ses-YYY/eeg/                           # ← DERIVED (BIDS): BrainVision + sidecars
        *_eeg.vhdr/.vmrk/.eeg  *_eeg.json  *_channels.tsv  *_events.tsv
        (*_electrodes.tsv + *_coordsystem.json only where the montage has 10-20 positions)

The XDF + embedded manifest in sourcedata/ are the source of truth. The subject-level BrainVision is derived and can be regenerated at any time; never hand-edit it.

Participant feedback rows (#268)¶

A run whose manifest declares design.feedback emits a feedback marker at the frame the check mark, cross or no-decision clock appears. The converter writes it as a trial_type = feedback row, and only such a run gets two extra events.tsv columns: shown (correct, incorrect, late or undecided) and latency_s (seconds from stim_off to the live answer reaching the stimulus window; n/a when late). The row's target_freq_hz and target_id are the cued target. A run without feedback converts exactly as before.

A free-choice run (design.cue_mode: "free", #268) has no label: every row of its events.tsv has target_freq_hz, value and target_id as n/a, and its feedback rows carry shown = selected plus a third column, shown_target_id (the target the participant was shown as decoded). The same columns appear on a cued run that uses the selection feedback mode.

A run whose manifest declares live early stopping (design.early_stop, #23, experimental) gets five more events.tsv columns, filled on stim_off rows from the marker's own fields and n/a elsewhere: presented_s (how long the flicker was presented, lead-in included), stopped (early or max), and for an early stop decided_window_s, margin and stop_latency_s. They are described in that run's own *_events.json, not the task-level one, so every other dataset converts exactly as before. The stim_on row's duration already runs to the real stim_off, so it is the presented interval on the EEG clock.

Image targets: `stimuli/` and `stim_file` (#258)¶

In sourcedata/. When a run with image targets is saved, the recorder copies each picture it showed into sourcedata/sub-XXX/ses-YYY/stimuli/<sha256>/<name>, verified against the manifest's pin, and lists it in the sidecar's stimulus_files (path, sha256, targets, and file, relative to the sidecar). The directory is named by the picture's content, so two different pictures that share a name never collide and one picture shown in many runs is stored once. A picture that could not be copied has "file": null and a problem saying why; the recording itself is never refused for it. Nothing is written for an observer-only participant.
In the BIDS view. The converter copies each picture from there to the dataset-root stimuli/<sha256>/<name>, again only if it hashes to the pin and only from inside that session's stimuli/. Each events.tsv row whose cued target (target_id) is an image target gets stim_file = <sha256>/<name> (BIDS reads it relative to stimuli/).
What stim_file means here. An SSVEP run shows every target at once, and BIDS gives a row one stimulus file, so stim_file names the cued target's picture. The run-level *_events.json says so. Every target's picture, with its pin, fit and luminance statistics, is in the run manifest.
When a picture is missing. A run without image targets gets no stim_file column and converts exactly as before. A picture that never reached sourcedata/ gets n/a, and ConversionReport.stimuli_missing lists it (the batch log warns).
Git. Both directories are already git-ignored (BIDS/* and sourcedata/), like everything else in the data tree. Export never ships a session's sourcedata/ copies, and ships the dataset-root stimuli/ pictures only when asked for that export: see docs/EXPORT.md §8.

The converter (`ssvep.io.bids`)¶

convert_dataset(BIDS_ROOT) (or python scripts/bids_convert.py):

Migrates any raw files still at the subject level down into sourcedata/ (idempotent). As of 2026-07-15 the runner writes into sourcedata/ directly (RunRecorder.output_base), so this step only matters for legacy trees recorded before that change — a raw .xdf at the subject level is not a BIDS datatype and makes the root fail validation.
For each run: reads the XDF (EEG µV + structured markers) and its session.json manifest → mne.io.RawArray (µV→V) → BrainVision via mne-bids, with:
channel names = the manifest montage's scalp labels;
a standard_1020 montage only when all labels are 10-20 sites (so the g.tec Unicorn runs get electrodes.tsv/coordsystem.json; the OpenBCI colour-labelled dry runs do not);
sample-accurate events: each marker is snapped to its nearest EEG sample via the recorded LSL timestamps (not time×nominal_rate), so events stay aligned despite clock drift;
a rich events.tsv (trial_type, target_freq_hz = decode label, condition/block/trial/ target_id), an impedance column in channels.tsv, and device/SSVEP fields in _eeg.json;
meas_date (and so scans.tsv acq_time) = the run's date-shifted start, in UTC (bids.measurement_date). A start time that isn't shifted (a sidecar from before date-shifting) or has no timezone gives no date at all (n/a), never the real one or a guessed one. See DATA_GOVERNANCE.md, "Recording dates are shifted when they are written".
Writes the dataset-level files and a participants.tsv (no demographics are invented — see "Demographics" below for the one exception, and it is still never invented, only carried over from an answer someone actually gave).

The `task-<x>` entity comes from the manifest, per run¶

Each run is labelled from its own manifest.run.task (bids.task_of; legacy experiment.task is still read), and a task-<x>_events.json is written for every task present in the dataset. A dataset holds more than one paradigm: task-ssvep and task-rest live side by side.

Why this is spelled out. convert_dataset(task=…) used to apply one caller-wide default to every run, so a resting run once landed as task-ssvep despite its manifest saying rest (2026-07-15). That is precisely the "analysis infers design instead of reading the manifest" failure the manifest spine exists to prevent — AGENTS.md §2. The task= argument still exists, but only for tests/legacy callers; leave it None.

The `acq-<board>` entity¶

Runs in one session used different front-ends with different montages (OpenBCI Cyton colour-labels vs Unicorn 10-20). Because electrodes.tsv is scoped by [ses][acq][space] (not by run), each run carries acq-<board> (cyton, cytonwifi, unicorn) so the montage sidecars attach to the headset that produced them instead of one wrong session-level file. It also self-documents the amplifier in every filename.

Adding a participant column¶

A between-subject or mixed contrast (schemas/session_protocol.schema.json $defs.contrast, design: "between"/"mixed") needs its grouping factor to be a documented participants.tsv column — e.g. cohort, with values like novice/experienced. This is what lets such a column become a between-subject factor — see Between-subject contrasts for how it's declared and tested. Two steps, neither of them a git commit:

Declare each participant's value with their recordings: python -m ssvep.io.declarations sub-XXX --set cohort=novice --protocol <the protocol> (see Participant declarations below). The value must be one of the between.levels that protocol's contrast declares for the column, and conversion renders it into participants.tsv. Typing the value straight into BIDS/participants.tsv (the per-dataset file, git-ignored — see above) still works, but only on that machine: the table is never pushed, so a contrast run anywhere else does not see it.
Document it in BIDS/participants.json: add an entry keyed by the column name, with a Description and, for a categorical column, a Levels map from each value to what it means — the same shape the converter's own group/species entries already use. This file is per-machine too, and is not pushed: document the column on every machine a contrast runs on.

The converter preserves any column entry it doesn't own (ssvep.io.participants. render_participants_json, called from write_dataset_metadata): re-running 01_xdf_to_bids.py never erases a hand-written entry, it only re-renders the columns it generates itself (participant_id, species, group, age, sex, hand, weight, height, age_range, gender). An entry for any other column — cohort included — round-trips through every future conversion untouched.

Which recordings are converted, and what the view says about them¶

Every recording carries a data-use stamp written at lock: why the session was run (its purpose: research, pilot, teaching or bench) and what the participant agreed to (its tier). One matrix in ssvep.runtime.data_use says what each stamp allows (docs/DATA_GOVERNANCE.md), and conversion reads its convert column (#286 P3d):

Converted: every stamp with something saved, so research from tier 1, pilot, teaching and bench. Tier 0 (observer only) saved nothing, so there is nothing to convert.
Refused, by name: a recording with no stamp, a malformed one, or only the older three-flag consent record. There is no migration, so it stays in sourcedata/ and is never converted or analysed. 01_xdf_to_bids.py and scripts/bids_convert.py print each refused run with its reason; Analyze's run log and its conversion report (ConversionReport.refused) do too. A BIDS view of it written before the gate existed is named, never deleted or updated.
The one exception: a teaching machine. There, an unstamped recording counts as teaching unless it says it was made on a lab machine (or carries the older pilot flag). Teaching is the most restrictive purpose (converted and analysed locally, never pushed or exported), so inferring it widens nothing.

The purpose is then written into the view in two places:

session_type in sub-XXX/sub-XXX_sessions.tsv, for every converted participant: each session's stamped purpose (research, teaching, bench, unstamped for one with no readable stamp). A session with any pilot recording is pilot; one locked more than once with other purposes lists them joined by + (e.g. bench+research). A session declared pilot or test after it was recorded has _declared_<group> appended, so its locked purpose stays visible beside the label (e.g. research_declared_pilot); a held-back session with no recording (a pilot questionnaire answer saved before the first run, or only a declaration) is pilot or declared_<group>. The column is derived from the recordings and declarations on every conversion, so a different value typed into it is replaced and the conversion says so, except by unstamped: that says nothing was derived (the recording is not even converted), so a value typed into the cell stays. sub-XXX_sessions.json documents it.
SSVEPStudy in dataset_description.json (raw and derivatives): the id of the study the dataset is (docs/SITE_PROFILE.md §10a), when it is one; absent for an install with no studies, so its datasets are unchanged.
SSVEPPurposes in dataset_description.json: the sorted purposes of every recording under sourcedata/ (with unstamped among them when any has no readable stamp). The derivatives dataset's dataset_description.json carries the same key for the runs analysed there. Purpose words only: no dates, no codes, no counts.

Neither is what any gate reads. Export, the demographics fill and every contrast read each recording's own stamp and each session's declarations themselves, through one rule for which sessions are not study data (ssvep.io.bids.not_study_sessions): a session with a pilot recording, one declared pilot or test afterwards, and one whose declarations cannot be read.

Pilot / development sessions (a sidecar's data-use stamp says "purpose": "pilot", chosen at lock on Set up Session; or, in a recording made before the stamp existed, its session_position.pilot is true; only the string "pilot" and the JSON true count, so a hand-edited "true" is not a declaration) are declared by every conversion path identically (#161; docs/SITE_PROFILE.md §3a). Each such session gets session_type pilot in sub-XXX/sub-XXX_sessions.tsv (see Which recordings are converted, and what the view says about them for the column's other values); other columns and rows of that file are kept. A participant's group becomes pilot only when every one of their recordings is a pilot session and no group is declared. A hand-set group is never overwritten (the conversion warns instead), and a pilot the converter set itself is cleared to n/a when a study recording appears (it keeps a list of those in dataset_description.json, SSVEPConverterPilotGroups). Neither file is what keeps pilot data out of a contrast: the contrast excludes each pilot recording from its own sidecar.

group is reserved and can never be a between-subject factor. It is the exclusion declaration — contrasts drop pilot and test participants through it (#269 D13) — so using it as a manipulated factor would conflate "excluded from every contrast" with "a level of this one". A contrast whose between.column is group, participant_id or species is refused, both by the schema (schemas/session_protocol.schema.json $defs.contrast.properties.between.properties. column) and by SessionProtocol.contrast_problems() and the analysis layer as a second, later gate.

sex and hand can't be a between-subject factor either, for a different reason: they're converter-owned. render_participants_json always re-derives participant_id, species, group, age, sex, hand, weight, height, age_range, gender from its own defaults on every conversion (the paragraph above) — a Levels entry hand-added under any of these is not preserved, it is overwritten the next time 01_xdf_to_bids.py runs, silently reverting the very documentation a between/mixed contrast needs to stay valid. Use a custom column instead, e.g. handedness_group rather than hand — a name the converter has never heard of and will never touch.

Participant declarations¶

A participant's group and their level of a between-subject factor are declarations — someone decides them, nothing derives them — and a declaration belongs with the data it is about (AGENTS.md §2.2). participants.tsv cannot be that place: it is git-ignored and never pushed, so a pilot typed into it on the acquisition PC excludes nobody in a contrast run on the compute server, which reconverts from sourcedata/ and gets a fresh table (#276). So they are recorded beside the recordings:

python -m ssvep.io.declarations sub-005 --group pilot
python -m ssvep.io.declarations sub-005 --group pilot --session ses-002   # only that session
python -m ssvep.io.declarations sub-005 --set cohort=novice --protocol session-protocols/<protocol>.json
python -m ssvep.io.declarations sub-005 --group n/a          # clears it

--root names the BIDS root (default: the site profile's data root, else the checkout's BIDS/). Run with no option, it shows what each session declares.

Where. sourcedata/sub-XXX/ses-YYY/declarations/sub-XXX_ses-YYY_declarations.json, written into every session of the participant on that machine, or only the ones named with --session. Per session, not per participant, because push works per session: the file is one more row of push's file list (the push guide, Which files count as data) and travels or is held under its session's data-use stamp, link refusals and all. It carries no stamp of its own, and the push gate never reads it: the declarations of a teaching session, or of one recorded before the stamp existed, are held with the rest of that session. An observer-only session saved nothing, so there is no folder to declare into. A participant with no session on the machine cannot be declared there.
A group binds only its own session. Exclude as little as possible: a session whose file says pilot or test is treated exactly as a session locked as a pilot is (#400): left out of every contrast, recording by recording (DECLARED_GROUP_SESSION), held back from export under both purposes, skipped by the demographics fill, and labelled in session_type (<purpose>_declared_<group>). The participant's other sessions stay eligible. To mark one session, name it with --session. Because every value it can take excludes, it is never compared with participants.tsv or between sessions, and it is never rendered into participants.tsv: that cell is participant-wide and would also exclude sessions that do not carry the declaration, a later study session included. Only when every one of the participant's sessions on the machine carries one is the participant reported as that group by the shared reading (contrasts, export).
What it may hold. Closed declarations only. group is pilot or test: it is the exclusion declaration, so those are the only values that mean anything to a contrast, and a grouping you want to compare is a factor column, never group. A factor value must be one of the between.levels the named protocol's between/mixed contrast declares for that column. A level written as a label (18-25, high dose) is recorded as its closed token: runs of characters outside letters and digits become one underscore, a leading digit gets an L prefix (L18_25, high_dose), and a one-word level is its own token, unchanged. You may type either the label, exactly as the protocol spells it, or the token; only the token is written, to the declaration, participants.tsv and the Levels keys in participants.json (put the label in the Levels description, which is where it is shown). Two levels that make the same token, a label with no letter or digit, and a label shaped like an identifier (a date in any common form, an e-mail address, a phone number, or six or more digits in a row) are refused when the protocol is loaded, and by the contrast that names them; a numeric range such as 18-25 or 100-150 is a category and is fine. Every value is one token (starts with a letter; letters, digits, underscore; at most 32 characters), so a name with a space, an e-mail, a date or a phone number cannot be recorded even by hand-editing the file: the reader refuses any file that departs from the format. No free text, no identity, no date. Converter-owned and identity columns (participant_id, species, age, sex, hand, weight, height, age_range, gender) cannot be declared.
Only documented factors are used. A factor declaration is rendered into participants.tsv (and compared on) only when participants.json documents that column with a Levels map naming the value, the same documentation a between contrast already requires. Anything else in a file (a hand-edited column such as notes, or a value the Levels don't list) is ignored and named in the conversion log, never rendered; the warning names the file, not the value. An undocumented value still counts when two sessions are compared, so it can make a disagreement but never hide one. group is always documented: its two values are fixed.
The documentation travels with the declaration. When a factor is declared against a protocol, the file also records a factors block: for each declared column, the closed tokens of that column's levels ({"cohort": {"Levels": ["expert", "novice"]}}), never the labels and never group. Conversion and Analyze render any column participants.json does not already document from it (a fixed Description, each token its own Levels entry), so a machine that only received the sessions, such as the compute server, can document the column and run the between contrast without anyone authoring participants.json there; the declarations count from the first conversion or Analyze, before participants.json exists. Files that list different levels for a column (a protocol that gained a level does this) document every level any of them lists, and the conversion says so. An entry the converter rendered is extended when a new level appears; once you edit it (put a readable label there, say) it is yours and is never touched again, and a level the files list that your entry lacks is named in the conversion's notes. Clearing a factor stops its documentation being used (the block stays in the file, which only grows). The reader refuses a factors block that holds anything but distinct closed tokens, a token shaped like an identifier (the same date, contact and six-digit shapes as a level label), or a column the file never declares. The block only extends what a file may say about which values exist; the compliance allowlist still accepts a participants.tsv factor column only when every cell is n/a or a documented token.
Format versions. A file with a factors block is format version 2 ("SSVEPParticipantDeclarations": 2); every other file is version 1, byte for byte what toolbox 0.9.x and older wrote, so a group declaration or a factor declared without a protocol still reads everywhere. Toolbox 0.9.x and older refuse a version-2 file with "it must hold exactly the keys …" and flag that session as unreadable (its recordings are left out of every contrast there): update the toolbox on the machine that runs the contrast before pushing a session declared against a protocol.
Session folders. Declarations are written into, and read from, session folders named ses- plus letters and digits only (the ones the toolbox creates). A declarations folder inside any other ses-* folder is not read; push would still carry it, so that session is flagged (as for an unreadable file) with a reason naming the folder rule.
Append-only. Each change appends the participant's whole current declaration to the file's history; the last entry holds. Push decides whether the server's copy is out of date by size, and an in-place pilot → test edit would keep the size. No timestamp is written.
Push them. After declaring, the files show in the push Review backlog as new or changed, ticked by default. Re-run conversion (or Analyze) where the contrast runs. A file whose server copy is longer than this machine's is held, unticked, because the server has declarations this machine lacks (the push guide, Which files count as data).
Links. No link is followed. A ses-* or declarations/ folder that is a symlink or junction flags its session, as an unreadable file does; a sub-* folder that is one flags the participant, who is left out of every contrast. If sourcedata/ itself is a link, no declaration can be read at all, so every contrast refuses (it does not exclude everyone, which would look like a result) and conversion renders nothing and says why; point the data root at the real folder instead. The root itself may be reached through a link.

How a factor meets participants.tsv — the precedence rule, chosen so that a disagreement is never resolved by picking a side, and excludes as little as it can:

`participants.tsv` cell	Declarations	Result
`n/a` (undeclared), or no row yet	declare a value	Conversion writes the value into the cell (a factor column is added if missing). A between contrast uses it even before that conversion.
a value	none	The cell is kept exactly as before (#272's rule, unchanged), and so is a hand-set `group`. This is the migration path. To move a hand-set value into a declaration, declare it; once the two agree, nothing changes.
the same value	the same value	Nothing to do.
a different value	a value	Conflict. The cell is kept (the converter never overwrites a declaration), the conversion warns, and every contrast on that factor leaves the participant out (`DECLARATION_CONFLICT`); other contrasts are unaffected. Fix it by clearing the cell or changing the declaration.
—	two sessions declare different values	Conflict, as above. Nothing is rendered.
—	a declarations file can't be read (malformed, misnamed, a link)	What it declared is unknown: it may say `pilot`, so that session's recordings are left out of every contrast, and it may declare another level, so the participant is left out of every contrast on a between-subject factor. Their other sessions stay eligible for a within contrast. Nothing is rendered for them, and the conversion names the file. The writer refuses to touch such a file.

group is not in this table: it binds only its own session (above). The age/gender columns are not affected by any of this.

Nothing from the file is ever exported (an export carries nothing from sourcedata/); the group it declares reaches an exported participants.tsv through the same shared reading, checked like any other cell, and a factor column is not on the export allow-list, so it is dropped and named as before.

Demographics: `age_range`/`gender` (#290)¶

Two more converter-owned participants.tsv columns, filled from the built-in background questionnaire (ssvep.runtime.questionnaire.demographics) — closed-choice age bin and gender, never an exact age or date of birth. Machinery only, as of #290: nothing in this toolbox attaches the instrument to a protocol or a session flow yet, so on a fresh checkout these columns stay n/a — administering it (holding the answers in memory with the safety statements until they pass, per the drafted procedure) is separate work (#286 P4). Once a deployment's own screens write a sourcedata/sub-XXX/ses-YYY/beh/..._task-demographics_beh.json, enrich_participants picks it up on the next conversion, same as group.

Only a value from the item's own closed option list is ever copied in — free text, an exact age, or anything else that isn't one of the documented Levels keys is skipped, on both the writer side (unanswered/is_complete/save_response in ssvep.runtime.questionnaire fail closed on an off-list value) and the reader side (resolved_demographics re-checks against the sidecar's own embedded instrument, or the shipped bins/options, and warns — naming the subject/session, never the value — rather than copying it). This is a second, independent guard, not a substitute for the first: a _beh.json from an older toolbox build might not have been through the writer-side check.
Asked every session, not once at enrolment, because there is no cross-machine state to look up (#290). A participant can therefore have more than one answer on file, and they can disagree — a later session, or a change of mind. Sessions are ordered by a natural sort of the session label (ses-2 before ses-10; a plain text/path sort gets this backwards) — never by the timestamp inside the file, because ssvep.runtime.dates draws an independent random offset per (subject, session) pair, so no two sessions' own date-shifted timestamps are ever comparable to each other, by construction. A session label with no digits at all (ses-pre/ses-post) has no numeric signal either; the toolbox falls back to alphabetical order for those and warns, since that fallback is not guaranteed to be visit order.
The multi-session rule (#290). For each column independently, the most recent real answer wins: the last completed administration whose answer is not "prefer not to say". "Prefer not to say" is recorded only when the participant never gave a real answer in any counted session. Isolated in one function, ssvep.io.participants._pick_demographic_value (its docstring is the authoritative statement).
A later "prefer not to say" does not withdraw an earlier answer (maintainer decision, 2026-09-30). If a participant answers in one session and chooses "prefer not to say" in a later one, the earlier answer stays in participants.tsv. It was freely given, and a later decline usually means "not answering again today", not "take back what I said". Withdrawing data is not done through the questionnaire; it goes through the study's formal withdrawal route. This replaces the rule PR #343 shipped, under which a later decline won outright and withdrew the earlier answer; the derived-column rule below is unchanged.
Answers from pilot sessions are never used. A session declared pilot / development at lock (its recordings' data-use purpose pilot, or the older session_position.pilot, the declaration session_type in sub-XXX_sessions.tsv mirrors and every contrast excludes by, #161), or declared pilot or test after it was recorded (its declarations file, #400), or whose declarations cannot be read, is not study data, so its background-questionnaire answer is skipped exactly as a declined one is. Otherwise a pilot answer could label a participant's study recordings in a between-subject contrast on age_range or gender. An answer is excluded if either its own sidecar or any recording in its session declares pilot. The answer's own _beh.json carries the session's data-use stamp (purpose pilot when it was given in a session locked as pilot, the same stamp every run of that session carries; older answers carried session_position.pilot), which covers an answer with no recording beside it: a questionnaire given before the first run of a session that was then aborted, a run that failed to save, a run sidecar deleted or unreadable, or a run rescued to a .rescued-NN pair the converter does not discover. The session's declaration (ssvep.io.bids.not_study_sessions, the same any-recording rule session_type, export and contrasts use) covers answers saved before that stamp existed, and a session declared pilot afterwards; such an answer in a session with no readable recording has nothing declaring it pilot, so it counts. A session with both pilot and study recordings (locked more than once) is pilot for this purpose, and the conversion says so in a note. A participant whose only answers are pilot gets n/a in both columns, and a dataset where nothing but pilot answers exists gets no demographic columns at all (if the columns are already there, from an earlier conversion, they are recomputed to n/a rather than left standing).
age_range/gender are DERIVED, not a declaration — unlike group. Once any background-questionnaire beh/ file exists in the dataset, every conversion recomputes both columns from every completed administration on record and overwrites whatever is currently in the cell, including a hand edit or an earlier conversion's answer. A participant with no completed administration gets n/a, not a blank or a stale value. This is what makes the result independent of conversion order and timing: converting after every session and converting once at the end from the same sourcedata/ produce an identical table — the contradiction the review found (a "never overwritten" rule combined with "most recent wins" meant the first converted session actually won) is resolved by dropping "never overwritten" for this column pair only. group keeps its own, different rule (a declaration, filled once and then protected) — the two columns are governed independently, and nothing here changes group.
A hand-entered gender (or age_range) column survives until the first administration. Coarse, non-identifying demographics may be entered in participants.tsv by hand (AGENTS.md §4), and gender has been on the compliance allow-list since #45. With no background-questionnaire beh/ file anywhere in the dataset there is nothing to derive from, so a conversion leaves such a column exactly as it is. Once the first file appears (counted or not), the derived rule above applies to the whole column: every row is recomputed from the questionnaire, and a hand-entered value on a row with no counted answer becomes n/a.
Small-sample identifiability. The age bins are ten years wide specifically so a small, mostly-adult sample doesn't put one or two people alone in a top bin — but even a ten-year bin can end up with just one or two participants in it. In a participants.tsv that is shared or made public, a lone occupant of a bin, combined with gender, can be identifying on its own. Before sharing, a site should look at the cross-tabulation of age_range × gender and merge bins by hand (e.g. combine the top two) if any cell has only one or two people in it — the toolbox does not do this automatically, because collapsing bins is an analysis decision, not a conversion one. The toolbox's own export carries these two columns for future use only and never in a public share, whatever the bins (docs/EXPORT.md §3).

Running it — the standard procedure for new data¶

Whenever new recordings arrive in sourcedata/, run both runners in order — this is the reproducible, documented way to "review new data", not eyeballing a raw XDF:

conda activate ssvep
python code/01_xdf_to_bids.py          # or: python scripts/bids_convert.py BIDS
python code/02_run_offline_pipeline.py # epoch → FBCCA decode → metrics.json → report → derivatives/

01 creates/updates the top-level sub-XXX/ BIDS folders + sidecars from sourcedata/. 02 writes each run's metrics.json + HTML report to derivatives/ssvep-analysis/, applies the per-recording spatial decode policy from the manifest (ssvep.spatial.occipital_decode_plan — large reference-free caps decode from an occipital ROI with CAR; stored as reref/decode_channels in the metrics), and writes a timestamped run log to derivatives/ssvep-analysis/logs/ recording toolbox + dependency versions, the git commit, and the spatial policy + result per run. The log + derivatives are the review record. --no-auto-spatial reverts 02 to the legacy all-channels/no-reref decode (which reads at chance on the 64-ch actiCAP — see docs/ACTICHAMP.md, DESIGN_PRINCIPLES #8).

`EEGReference` / `EEGGround` (#19)¶

Both are filled from the best answer available, in four steps, and never from the signal:

What the run recorded in acquisition.reference / acquisition.ground. This wins even when it is a vague legacy value like occipital: overriding a recorded field because it looks unhelpful is the design-inference AGENTS.md §2.2 forbids, and the fix for a bad value is to record a better one.
What the carrier implies (builder.headset_reference_ground, #147), when the run recorded nothing. A carrier whose wiring is part of the montage states an actual scalp site — the free-electrode array and the occipital 10-10 montage both give A1 (left earlobe) — Cyton SRB, white lead / A2 (right earlobe) — Cyton BIAS, black lead. A run that names a known headset has told us where the electrodes went, even without a reference string.
What the amplifier fixes (builder.board_reference_ground), when the carrier said nothing. A Cyton has one reference input and one driven-bias output whatever is clipped to them, so the sidecar records Cyton SRB pin (white lead); electrode site not recorded rather than a shrug. The actiCHamp streams ground-only with no true reference, so its EEGReference says exactly that — which is what tells an analyst the data rides on the common mode and needs re-referencing (and is why occipital_decode_plan applies a common average automatically).
unknown (amplifier default), when none of the above answers. This is a real outcome, not a bug: the g.tec Unicorn's reference and ground are operator-placed clips and nothing in this repository records which site each takes. An invented value in a recording's metadata is worse than an honest unknown, because nothing downstream can tell the two apart.

Each half falls back independently — a board may fix the reference and not the ground, which is exactly the actiCHamp's case.

BIDS/code/ holds thin, dataset-local runners that call the version-controlled, unit-tested package (tests/test_bids.py), so the dataset is self-describing on the NAS while the logic stays in git.

`02` does `01`'s work for the runs it analyses — it no longer assumes you ran it¶

Analysis reads the XDF (sourcedata/) directly, so nothing in the pipeline ever needed the BrainVision copy — and so it got skipped. Analysing from the GUI (which never invoked 01) filled derivatives/ for subjects that had no sub-XXX/…/eeg/*.vhdr at all: a dataset of derivatives with no BIDS-standard raw data to be a derivative of.

run_batch now derives the BIDS view of its selected runs first (ensure_bids=True, the default), converting only runs that lack an up-to-date one:

"Already there" is checked, not assumed — ssvep.io.bids.is_converted wants the run's .vhdr + .eeg to exist and to be no older than its XDF. A source newer than the copy derived from it means the copy describes data that has since changed, so it is re-derived. Conversion is idempotent; the cost is paid once per recording.
A converter code change is also checked, not just the data (#113). The mtime check above only catches a source XDF that changed — it cannot see that the converter changed, so a bug fixed in the conversion logic never used to reach a run that was already converted (its XDF hadn't moved, so is_converted reported it up to date forever). Every conversion now also stamps a *.converter.json marker beside the BrainVision header with a hash of ssvep/io/bids.py's own source (ssvep.io.bids._converter_fingerprint — not ssvep.__version__, which is a static "0.0.0" and would never move); is_converted compares that stamp against the fingerprint of the code running now, and a run with no marker at all (converted before this existed) is treated the same as a mismatch. The marker is a plain file next to the derived output, not a new key in the *_eeg.json BIDS sidecar, so it never reaches the schema the official validator checks — it is listed in .bidsignore like the other convenience sidecars. ⚠️ Operational consequence: the first 01_xdf_to_bids.py run after this lands reconverts every already-converted run in the tree once, because none of them yet carry the marker — expected and intended, not a regression.
Whole-tree metadata stays whole-tree. When anything is converted, the dataset-level files (dataset_description.json, README, participants.tsv, task-<x>_events.json) are rewritten from every run on disk, not just the selection — otherwise analysing one session would drop the task labels of the runs it didn't touch.
A failed conversion does not withhold an analysis. The XDF is the source of truth; a run whose BIDS view can't be derived is a broken export, logged as BIDS: … could not be converted, and still analysed.

01 remains the way to (re)derive the whole tree in one go — including migrating any legacy top-level raw files down into sourcedata/, which 02 does not do.

Analysing one session, not the whole dataset¶

02 takes a selector, so finishing a visit does not mean re-analysing every participant ever collected (#67):

python code/02_run_offline_pipeline.py --sub 001 --ses 002   # one session
python code/02_run_offline_pipeline.py --sub sub-001         # one subject, all sessions
python code/02_run_offline_pipeline.py                       # the whole dataset (unchanged default)

Both entity forms are accepted (001 and sub-001) because operators type both. --ses without --sub is refused — "session 002" is only meaningful inside a participant — and a selector matching nothing exits non-zero listing the subjects/sessions that do exist, because a silent "0 analysed" reads as success and is how a typo'd ID becomes "my data is fine". The run log records the selector used, so a log in logs/ is never ambiguous about its coverage.

The selection and the batch live in the package (ssvep.analysis.batch), not in the script: the script is argparse over run_batch, and the GUI's Analyze tab (below) calls the same function.

From the GUI: the Analyze tab¶

ssvep-toolbox's Analyze tab is the same batch with a selector you cannot typo. It scans the BIDS root's sourcedata/ and shows a checkable sub-XXX → ses-YYY → run tree with each run's paradigm (read from its manifest), whether its BIDS raw data has been derived, and whether it already has a Report; tick a session, press Analyze, and it runs ssvep.analysis.batch.run_batch on a worker thread — same derivatives/ssvep-analysis/ output, same run log. The tab's BIDS root follows the Set up Session tab's output directory, so "analyse the session I just recorded" needs no path typed.

Because the tab calls run_batch, analysing also converts: any ticked run without an up-to-date BIDS view is converted first (status: Converting to BIDS i/n: <run>…, which is where the first minutes of a fresh session go), and the finished status says how many were converted. This is the whole point of the BIDS column — the conversion state is visible before you press anything, rather than something you have to remember to have done from a terminal.

When it finishes, the index opens by itself. Every batch rebuilds derivatives/ssvep-analysis/index.html and the GUI opens it in your browser (Open index opens it any time — see below; the CLI logs its path). One tab, not one per run: an 8-run session would otherwise bury the browser. Open report, or double-clicking a run in the tree, opens a single run's *_report.html directly.

The derivatives index (#80)¶

Analysis outputs are navigable, not a flat pile. Each run's report and metrics are written to derivatives/ssvep-analysis/sub-XXX/ses-YYY/ — the relative path of the raw recording they came from — and three levels of generated index.html link them:

derivatives/ssvep-analysis/
├── dataset_description.json      # this is a derivatives dataset (#18) — see below
├── index.html                    # every subject → its sessions
├── sub-001/
│   ├── index.html                # every session → its runs
│   └── ses-002/
│       ├── index.html            # every run, in protocol order, + its headline result
│       ├── sub-001_ses-002_task-ssvep_run-001_report.html
│       └── sub-001_ses-002_task-ssvep_run-001_metrics.json
└── logs/offline-pipeline_<stamp>.log

Three things about that page are deliberate:

The window is part of every headline number (#457). The session index has a "Headline result" column and an "Operating window" column. For an SSVEP run the headline is the best window of the sweep, picked after the fact; the operating-window column says which window the run was actually decoded at, and a note appears when the runs listed were decoded at different windows, since their accuracy and ITR are then not on the same footing. A live-stopped run has no fixed operating window (its headline is the live decisions and mean stop time), and neither does an FPVS, resting or free-choice run. The per-run report states the same in an "At operating window" row (and says so when the operating window is not a point of the sweep).
It is rebuilt from disk, not from the batch. ssvep.analysis.derivatives.build_index scans the tree for *_report.html + *_metrics.json and rewrites every level. So it covers everything analysed so far rather than the last selection, a batch that died part-way still leaves it truthful, and re-analysing a run updates its row instead of adding a page. It replaces the old per-batch summary_<stamp>.html, which described one invocation and accumulated one file per run of the pipeline — three pages that each looked like the entry point.
Run order is declared, not inferred. The # column is the run's run_index from its session_position sidecar stamp (#43) — its derived place in the session protocol — so the page reflects the order the participant actually experienced, not alphabetical filenames. A run with no sidecar (ad-hoc, or pre-#43) is still listed, tagged ad-hoc, ordered after the declared ones.
Links are relative. The pages open by double-clicking off local disk or the NAS; no server.

A first analysis on an existing tree migrates the old flat layout: reports and metrics are moved into sub-XXX/ses-YYY/ (moved, not regenerated) and the obsolete summary_*.html are removed. It is idempotent and logged.

It is a formal derivatives dataset (#18)¶

derivatives/ssvep-analysis/dataset_description.json declares DatasetType: "derived", a GeneratedBy block naming the pipeline, the toolbox version, and the git commit that produced the outputs, and a SourceDatasets link back to the raw dataset. Previously the directory held analysis outputs with nothing but its name to say what had made them or from what.

One deliberate deviation from strict BIDS-derivatives filenaming, recorded in the description itself: outputs sit at sub-XXX/ses-YYY/ without the eeg/ datatype level, because an HTML report spans a whole run rather than describing one datatype's data.

Notes / limits¶

BrainVision stores samples on the nominal clock (250/500 Hz); events are sample-indexed, so epoching is exact even though the wall-clock length differs from the effective rate by ~0.2 %.
The full BIDS validator now runs via bids-validator-deno (in environment.yml's pip section — ships the JS bundle plus a deno wheel that vendors the Deno runtime, so no separate Deno/npm/node install and nothing fetched to obtain the tool itself; the launcher grants Deno --allow-net unconditionally, and the bundled validator can still reach out — for remote HED schemas when a dataset declares a HED version, or for the BIDS schema itself under --schema stable/latest):
```
conda activate ssvep
bids-validator-deno BIDS/
```
Before this, this repo relied only on mne-bids' writer + a read_raw_bids round-trip test. Run this before any release/share.
On a checkout with no local data, a non-zero exit is expected and is not a conformance failure. The tracked scaffold is de-identified metadata only — sub-*/ is git-ignored — so the validator reports SUBJECT_FOLDERS ("no subject directories") and, at [ERROR] level, SIDECAR_WITHOUT_DATAFILE for channels.json / participants.json. Those are sidecars whose data files are not on this machine, which is the compliance design working, not a defect.
⚠️ Run it locally and keep its output local. The validator prints sub-* paths; summarising findings elsewhere means rules, counts and sub-XXX codes only (COMPLIANCE R2/R4).

Validate from the GUI (#98)¶

The Analyze tab has a Validate BIDS… button beside Export…, so checking a dataset before a share doesn't need a terminal. It runs the validator against the tab's current BIDS root, off the GUI thread, with a Cancel button, a result split into an Errors/Warnings tree (grouped by rule code, each with its file locations as children) plus the full raw output, and a Save report… button.

Unlike running bids-validator-deno by hand (above), the dialog does not exec the pip console-script launcher: it resolves Deno's own binary and the validator's JS bundle (bids_validator_deno's package data) and starts Deno directly, deno run --no-prompt with a minimal permission list — --allow-read and --allow-env only, not the launcher's full list. A #329 round-2 review measured each of the launcher's other permissions (--allow-net, --allow-write, --allow-run=git, --allow-sys=osRelease) against six dataset shapes (plain, HED-tagged, derivatives, a raw dataset with a nested derivatives/, one with a tracked .git/, and empty) and found the validator's output byte-identical without them: it reads the dataset and a couple of environment values (e.g. IGNORE_TEST_WIN32) but never writes (its --outfile flag is never passed here), never shells out (its own git describe self-version-stamp is try/except'd and silently skipped if it fails), and never queries the OS release. The dialog also strips any inherited BIDS_SCHEMA from the child's environment, so a value set on the operator's machine can't silently swap the bundled offline schema. Two things this buys: - The validator runs with no network permission at all, not merely "no --schema flag" — --allow-net is never granted in the first place, closing the path a dataset's own HEDVersion would otherwise open (above), proven behaviourally identical (same findings, same exit code) against a dataset that declares one, since --no-prompt makes any network attempt raise instead of silently degrading. - Cancel and closing the dialog actually stop it, without freezing the GUI. The launcher's real work (deno.exe) was a grandchild it spawned internally; killing the launcher left that grandchild running to completion, still reading the dataset, while the dialog claimed "Cancelled." Running Deno directly makes it the process the dialog owns, so Cancel/Close's kill() reaches the thing actually doing the work — and that is the only thing it does. An earlier fix additionally shelled out to taskkill /T /F as a tree-kill; a #329 round-2 review found that call ordered before kill() and costing ~11 seconds on a real Windows box (a WMI process lookup, paid whether or not anything needed killing) — freezing the dialog for that whole window while the validator kept running. With the permission list above, the bundle spawns no child processes at all (dropping --allow-run=git removes the one thing that could have spawned one), so there is no tree left to kill: kill() alone, called first, is both necessary and sufficient, and Cancel/Close now return in well under a second.

It parses --format json (the stable, structured output; plain --json is deprecated), so counts and locations come from the tool's own per-finding data rather than a [ERROR]/[WARNING] text heuristic — the exit code decides what those counts mean: 0 = passed, 16 = validation errors present, anything else = the validator failed to complete a run (shown as such, with the raw output below it — never read as "0 errors"). If parsing that JSON output ever fails (an unexpected shape, or an interrupted run), the dialog falls back to the old text-tag heuristic on the raw output, with a note saying it could not parse the structured form, rather than losing the finding entirely. If the validator isn't installed (e.g. the env hasn't been reinstalled against the current environment.yml yet), the dialog says so and disables Run rather than crashing.

Save defaults to the repo's qc-reports/ (git-ignored, same convention as Measure flicker's reports) — never the BIDS root: a report saved inside it is not itself a BIDS file, so the validator would flag it NOT_INCLUDED — an error — on the dataset's next run, turning a clean dataset red the moment the operator keeps a copy of its own report. The operator can still save anywhere, and the saved report's header names only the dataset's folder, never the machine's absolute path to it.

What the validator found, and what we did about it¶

Validated against the real tree: 2026-08-06, 13 subjects / 738 files / 145 MB, exit 0 — zero errors, the converter's output is valid BIDS-EEG (#110). Every remaining finding is a warning; here is the disposition of each, so a future run can tell a known-and-accepted warning from a new one. The first three rows were fixed in the converter and the tree re-derived from sourcedata/; the rest are deliberate or open.

⚠️ A converter fix used to not reach runs that are already converted. is_converted() compared only mtimes, so it couldn't see that the converter changed — clearing these findings needed a forced re-derive by hand. Fixed by #113: is_converted() now also stamps and checks a converter-version marker (see above), so a future converter fix reconverts every affected run automatically on the next 01/02 run, with no manual re-derive.

Finding	Disposition
`SIDECAR_FIELD_OVERRIDE` (5 keys × 56 files)	Fixed — and it was not cosmetic. mne-bids writes a run-level `_events.json` repeating every column with generic boilerplate, and BIDS inheritance is nearest-wins key by key* — so our real column documentation in `task-<x>_events.json` (the `trial_type` level map, `target_freq_hz` as the decoding class label) was written to disk and then shadowed on every run. A BIDS reader got "The type, category, or name of the event" instead of the design. The run-level file now carries only `StimulusPresentation`, which genuinely varies per run.
`SIDECAR_KEY_RECOMMENDED` (12 keys × 173 files)	Mostly fixed, from the manifest — `TaskDescription`, `Instructions`, `SoftwareVersions`, `CapManufacturersModelName`, `MISCChannelCount`, `Institution`, and `StimulusPresentation` (with the display's refresh rate, which decides which square-wave frequencies were presentable at all — and prefers the measured* rate over the nominal one when the builder recorded it).
`EVENTS_TSV_MISSING` (2 runs)	Not a converter defect — aborted runs (one is ~8 s of EEG with the marker stream present and nothing ever pushed). The data is real, so it is kept. What was a defect is that they converted silently; `ConversionReport.eventless` now reports them, on every pass and not just the first, and the pipeline logs a warning. Currently two aborted pilot/QA runs (one ssvep, one rest).
`TOO_FEW_AUTHORS`	Open — the maintainer's call. `Authors` lists the lab as a single collective author. Naming individuals is an authorship decision, not a conversion one, and investigator names are not participant data either way.
`JSON_KEY_RECOMMENDED` → `HEDVersion`	Deliberate — do not "fix" it. We use no HED tags, and declaring a HED version is exactly what makes the validator fetch a remote HED schema at validation time. Leaving it unset is why validation runs with no network activity despite the launcher's blanket `--allow-net`. Silencing a cosmetic warning would buy an outbound request while the validator reads a tree of participant data.
`SIDECAR_KEY_RECOMMENDED` → `HeadCircumference`	Deliberate. A physical measurement of the participant — the kind of indirect identifier COMPLIANCE R3 keeps out of the tree — and it buys no analysis here.
`SIDECAR_KEY_RECOMMENDED` → `SubjectArtefactDescription`	Deliberate. Per-participant freeform that nothing records. A fabricated `"n/a"` is not neutral: BIDS reads it as absence of major artifacts, a claim nobody has made.
`SIDECAR_KEY_RECOMMENDED` → `CogAtlasID` / `CogPOID`	Deliberate. No Cognitive Atlas or CogPO term corresponds to this paradigm; a wrong URI is worse than an absent one.

The same rule governs the whole table: a recommended key gets filled when the manifest already knows the answer, and is left absent when filling it would mean inventing one. "n/a" is a claim, not a blank — SoftwareRRID and OperatingSystem are omitted for exactly that reason (and an "n/a" RRID is a schema violation, which the validator does catch as an error).

Reference/ground are unknown for these amplifiers (n/a); fill in if the hardware specifies them.

BIDS conversion & the BIDS/ dataset¶