Auto-push — mirroring recordings to the lab server¶

The acquisition PC is not the archive. This is the local disk → NCIL server → compute server leg of the data flow (docs/ARCHITECTURE.md): when a run ends, the toolbox copies it to ncil.psychology.dal.ca over SFTP so it survives the bench and is reachable for analysis.

It is opt-in and off by default, it never deletes anything locally, it only sends the run (or questionnaire) that just finished, and it never pushes observer-only data (COMPLIANCE R5). Code: src/ssvep/runtime/push.py; UI: the Auto-push block on the Set up Session tab of ssvep-run.

1. One-time setup (per acquisition PC)¶

Authentication is this machine's SSH key — there is no password anywhere in the toolbox, and no secret in any config file.

Make a key, if this machine has none:

ssh-keygen -t ed25519          # accept the default path; a passphrase means you must load an agent

Give the public key to the server — send ~/.ssh/id_ed25519.pub to whoever administers the ncildata account, to be appended to ~/.ssh/authorized_keys there.
Connect once by hand. This is not optional:
```
ssh ncildata@ncil.psychology.dal.ca
```
The first connection asks you to trust the server's host key. The toolbox deliberately never auto-accepts a host key (StrictHostKeyChecking is left alone — auto-accepting would weaken the server's own access controls, COMPLIANCE R4), so until you have done this once, every push fails with "host key not trusted".
In ssvep-run → Setup → Auto-push: click Test connection. Green ✓ means the key works. Then tick Push recordings to the server after each run.

Settings are written to config/push.local.json (git-ignored) and reload on the next launch, so this is a once-per-machine job. config/push.example.json documents the fields.

If you are not at NCIL: change User, Host, and Path to your own server. The only hard rule is the destination's location — see §6.

2. Paths mirror 1:1 — there is no rearranging¶

Remote base is the BIDS root, not sourcedata/. A file's path relative to the local BIDS root is its path relative to the remote base, verbatim:

local :  BIDS/sourcedata/sub-903/ses-001/eeg/sub-903_ses-001_task-ssvep_run-001_eeg.xdf
remote:  ~/NCILscience/Gambit/ssvep-toolbox/BIDS/sourcedata/sub-903/ses-001/eeg/<same name>
         └────────────── remote_base ──────────────┘└──── identical to local ────┘

sourcedata/ appears on the server because it is in the local path, not because the push puts it there. RunRecorder.output_base() writes into sourcedata/ directly, which is both correct BIDS (a raw .xdf is source data, not a BIDS datatype — at the subject level it fails validation) and what makes the mirror trivial to reason about: what you see locally is what lands remotely.

Older trees with raw files at the subject level (BIDS/sub-902/...) are the pre-BIDS layout. Run scripts/bids_convert.py (or io.bids.migrate_toplevel_to_sourcedata) once to relocate them; the push only looks under sourcedata/ and will silently ignore anything still at the top level.

3. What gets pushed, and when¶

Automatically, at the end of a run¶

Only the run that just finished:

File	Written by	Scope
`sub-XXX_ses-YYY_task-*_run-NNN_eeg.xdf`	the runner	per run
`…_eeg.session.json` (manifest + provenance + consent + impedance)	the runner	per run
`sub-XXX_ses-YYY_impedance.json`	Run Session's QA-check pane, on every Done	per session

The impedance log is per-session and is rewritten on every check, so it rides along with each run — otherwise it would never be current on the server.

Automatically, the moment a questionnaire is saved¶

A break-time checkpoint or the end-of-session instrument has no run of its own to ride along with — it can land between runs, or after the very last one — so the end-of-run push above cannot see it. SessionRecordPanel.administer() pushes its own pair the instant save_response writes them:

File	Written by	Scope
`sub-XXX_ses-YYY_task-{postcondition,endofsession}_run-NNN_beh.tsv`	`ssvep.runtime.questionnaire.save_response`	per administration
`…_beh.json` (instrument + answers + flags + operator acknowledgement)	`save_response` (+ `record_acknowledgment`, if a flag was acknowledged)	per administration

Before this existed, every questionnaire file needed a manual Review / push… — which is exactly what happened in the 2026-08-28 pilot (sub-994): the run's own xdf/sidecar/impedance mirrored fine, and all five beh files sat unpushed until pushed by hand.

Everything else: Review / push…¶

Nothing else is ever sent automatically. A file being absent from the server is not evidence it belongs there — it may be a bench test, or a subject someone else is handling. So Review / push… lists the difference and lets you choose. It ticks only rows it is confident about, and each row shows its status:

Status	Meaning	Ticked by default?
`new`	not on the server at all	✅ yes
`changed`	on the server, but a different size	✅ yes
`same`	already there, byte-count matches	❌ no — nothing to do

Rows held back by consent are shown greyed and cannot be ticked (§5). sub-000 rows are shown unticked but are selectable — see §4.

Which files count as data¶

Explicitly listed per BIDS modality directory, not "everything in the folder":

Modality	Patterns
`eeg/`	`_eeg.xdf`, `_eeg.session.json`, `*_impedance.json`
`beh/`	`.csv`, `.log`, `.psydat`, `.tsv`, `*_beh.json`

A stray note or half-exported figure in an eeg/ folder is not data and is not sent. Adding a new modality means adding a row to push.MODALITY_PATTERNS — until you do, its files are silently skipped. Add the row in the same change that starts writing them.

4. The server is the source of truth¶

The toolbox asks the server what it already has (one ssh … find round-trip) and diffs against local. It keeps no local ledger of what it has pushed.

This is deliberate, and it is a correction. A ledger cannot know about files an operator copied up by hand — which is exactly how the sub-902/903/904 pilots got there, over WebDAV from the Synology. A ledger-based version reported a 37-file "backlog" that was already on the server in full. Ask the server; it knows.

Consequences worth knowing: - Comparison is name + size, not a checksum. An XDF is large and the point is to catch a file that was re-recorded, not to defend against tampering. - If the server is unreachable, you get an error, not an empty list. We don't know what's up there, so we must not present a backlog we can't vouch for. - A missing remote base is not an error — it means nothing has been pushed yet, so everything reads as new. - An unreadable base is an error — and it is not the same thing as a missing one, even though find exits 1 for both. If the account cannot enter the remote directory, we do not know what is up there, so the scan refuses rather than showing a backlog. Only "No such file or directory" counts as an empty server. This was a real failure (#76): the index command ended in 2>/dev/null || true, which discarded the difference and forced exit 0, so a project folder the push account had no permission on looked exactly like a fresh server — every file reported new, the dialog looked healthy, and no push could ever have succeeded. If you are diffing against a directory somebody else owns, this is the message you want to see. - The NAS scatters @eaDir/ and ._DAV/ sidecar directories through the tree; those are pruned from the index and never mirrored.

`sub-000`¶

sub-000 is the Set up Session tab's pre-filled subject, so a recording under it means nobody set the subject — a bench test, not a participant. It is never ticked by default, but it is listed and is selectable, so you can push it deliberately. This is a default, not a prohibition (unlike the consent gate). To change the rule, edit push.SKIP_SUBJECTS.

R5 is explicit that observer_only_delete data must never be pushed. Three things enforce it:

Observer-only sessions never reach the disk at all — save_outputs discards them, so there is nothing for the push to find.
The push re-reads consent off each sidecar anyway, rather than trusting the caller. This matters for a file pushed days later by someone who never saw the consent form.
The gate applies to the automatic end-of-run path too, not just the dialog — that path runs with nobody looking at it, so it must not depend on a human noticing.

A session is held back whenever consent cannot be positively confirmed:

Situation	Pushed?
sidecar says `observer_only_delete: false`	✅ yes
sidecar says `observer_only_delete: true`	❌ never
any run in the session is observer-only	❌ the whole session is held back
`observer_only_delete` flag absent	❌ held back — R5: most restrictive when a flag is missing
no sidecar / unreadable sidecar	❌ held back — consent unknown

Two practical consequences of that last row. An impedance check with no session after it has no sidecar, so it is never pushed — the correct reading of R5, though it means a contact-quality record from an abandoned setup stays on the acquisition PC. And legacy recordings with no *.session.json (e.g. the PsychoPy-suite pilots sub-005–008) can't be pushed by the toolbox at all; they must be placed by hand, as they were.

future_use and public_share do not gate this push. They govern sharing — a public release or reuse beyond the study. Pushing to the NCIL server is internal storage on the lab's own Canadian machine, i.e. the sanctioned local → server → compute chain of R1, not a release. public_share remains the gate for anything leaving that chain.

6. When it fails¶

Nothing is lost, and nothing is silent. The push is a copy of files already safely on local disk, so a failure is a delay, not a hole. The Run Session tab shows a red warning:

⚠ push: 3 failed — ncil.psychology.dal.ca is not answering on port 22 — server down, or VPN needed.
   Data is safe on this machine — use Review / push… to retry.

Retry with Review / push… when the server is back; the failed files still show as new, because the server is what's being asked.

Message	Meaning	Fix
host key not trusted	this machine has never talked to the server	run `ssh ncildata@…` once and accept the key (§1.3)
server refused the key	your public key isn't on the account	get it added to `~/.ssh/authorized_keys`
cannot read `<path>` … no permission on the directory	the key works and you are logged in, but the push account cannot enter the remote project folder	whoever owns it must grant that account access — not an SSH problem, so don't touch `authorized_keys`
cannot resolve …	DNS / not on the network	connect to the network or VPN
… is not answering on port 22	server down, or firewall/VPN	check the server; retry
OpenSSH client not found	Windows OpenSSH feature missing	Settings → Optional features → OpenSSH Client

Each file uploads to a .part name and is renamed into place, so an interrupted transfer leaves visible debris rather than a truncated file that looks like a complete recording. If a batch fails but the server is reachable, the toolbox retries file-by-file so one bad file cannot hold the rest behind it — and if the server is unreachable it skips that fallback, so a down host costs one timeout rather than N.

7. Compliance notes¶

Residency (R1/R3). ncil.psychology.dal.ca is the lab's Canadian server, named in R3 as the sanctioned destination. The host is configurable for other labs, and the destination must stay in Canada — pointing this at a foreign/US cloud host (S3, Dropbox, Drive, …) violates R3, regardless of encryption. Nothing in the toolbox can check where a hostname resolves; that is on whoever configures it.

Transport (R4) — a noted divergence. R4 enumerates the server's approved encrypted external access as HTTPS/AFP/WebDAV; SFTP is not in that list. It is encrypted, key-authenticated, and sits on top of the server's existing firewall/IP controls rather than around them — arguably equivalent-or-stronger — but it is a protocol the written controls do not name. Logged as A11 in docs/REB_AMENDMENTS.md; it is not collection-blocking (it moves already-collected data by an alternative route to the same destination), but the wording should be confirmed.

Why the system OpenSSH client, not paramiko. Windows ships sftp.exe in System32\OpenSSH, so shelling out to it adds no dependency and no license review, and it reuses the operator's existing key, ~/.ssh/config, and known_hosts — the same credentials that already work when they type ssh by hand. paramiko would have been licence-compatible (LGPL-2.1+) but pulls in a crypto stack for no gain here.

No PII. Only de-identified sub-XXX paths ever travel (R2), and the config file holds no secret — just a hostname, an account name, and a path.