Media is data with a time base

A picture in a folder is a file. The same picture with an onset — 3.00 s into the recording, 1.00 s long, on trial 7 — is data, because now it can be epoched with the signal, split with the signal, and scored against what the signal predicted. Everything on this page exists to turn the first thing into the second, and then to keep them together for the rest of the analysis.

Two objects do that work. A media set is the content: the items themselves, stored once, hashed. A binding is the time base: which item was shown when, to whom, and how confidently we know that. Sets are large and belong to the study; bindings are small and belong to the experiment. Branching an experiment does not fork your stimuli.

study (.bltd)                       experiment (.blts)
  /media/sets/<set_id>                media_bindings
     manifest   ← the item table         [ { set_id, role, regime,
     /blobs/…   ← embedded items             method, events[], coverage } ]

The manager lives as a Media tab inside Load Dataset, next to the rest of the study’s data. It lists manifest rows; it does not decode. Decoding happens on an explicit Run, never on a page or tab switch.

Media sets

Every item is its own hash

An item’s id is the first 16 hex characters of the SHA-256 of its bytes. Two identical files are one item. An edited file is a different item, with a different id, so a year later the provenance record can still prove which bytes were on the screen. The filename is display only and is never used as an identity or as a caption.

Embedded or linked

A set is stored one of two ways, and the app picks the default from the size rather than from the kind alone. The thresholds are one definition in src/media/limits.py.

Crossing a threshold flips the default and the import dialog says which threshold and by how much — “linked because this set is 2.1 GB” reads as a fact, where “linked” alone reads as an arbitrary app decision. You can override either way.

Licence, and what publishing sends

Every set carries license and license_url. A set left at unknown blocks publish, with a picker offering own-work, CC0, CC-BY, CC-BY-NC, research-only, proprietary, or a custom string.

research-only and proprietary sets are never uploaded. What the published study carries instead is the manifest — hashes, names, counts, licence — so a reader can obtain the originals from wherever they are licensed and verify byte-for-byte that they had the same ones. The stimulus licence is displayed on the study page beside the code and data licences.

Bindings: media on the recording clock

Four roles, two regimes

Roles are never merged, and a wire between mismatched roles is refused by the graph IR.

Four ways in

Route 2 auto-detects these column names before it asks you anything:

Route 4 anchors, in order of how much they can be trusted:

Clock drift is estimated when two anchors exist and reported as "unmeasured" when only one does. A 12-minute media file played from a separate clock is typically 10–200 ms out by the end, which is enough to smear a response into nothing.

Coverage, and what a partial binding may be used for

Every binding carries a coverage record: trials total, trials bound, the unbound indices, overlaps and out-of-range events. What happens next depends only on that record.

One index vector

This is the invariant the whole layer is built around. For a trial-locked source, Stimulus[i] is the media of the trial that produced X[i] and Y[i] — after every filter, every rejection and every reorder.

         media          X (epochs)        Y (labels)
  i=0    face      ↔   [ 62 × 250 ]   ↔   face
  i=1    tone      ↔   [ 62 × 250 ]   ↔   tone
  i=2    "cat"     ↔   [ 62 × 250 ]   ↔   word
  i=3    ✗ unbound     ✗ dropped          ✗ dropped     ← leaves all three
  i=4    house     ↔   [ 62 × 250 ]   ↔   house

The alignment layer returns media with the X and Y it belongs to; there is deliberately no API that returns media alone. The three-separate-lookups version agrees with this one right up until a trial drops or the order changes, and then disagrees silently, because every array is still exactly the right length.

Designing with media

Modality is a property, not a block type

A Visual Cue that shows a picture is still a Visual Cue. Every NeuralFlow block gains one optional Stimulus card in the inspector, below Duration: a set picker, the modality (read-only, from the set), presentation order, a tag subset, a seed, a thumbnail strip, and a live occurrences × items line.

All defaults means the group is omitted on save, so a paradigm written before any of this existed still loads and re-saves byte-identically.

Stimulus Set and Media Playback

What refuses at compile

All of these fail closed, before a participant is ever in the chair:

• random_no_repeat with fewer items than occurrences — raises, naming both counts.
• Media Playback media longer than the block duration — raises unless you asked for truncation.
• A set id that is not in this study — raises, with the id and every block label that references it.
• A modality that does not match the block type — warns and stamps, but does not raise.

Presenting it

Without a runner, media on a block is metadata about data collected somewhere else. That is genuinely useful — it covers every public dataset — but it is not “run my experiment”. The Paradigm Runner is a separate process with its own timing loop, full-screen on the presentation display, emitting a marker per onset over LSL or a serial/parallel TTL line, abortable at any point and leaving a valid partial log.

The onset band is measured, not claimed

Realistic onset accuracy for a stimulus drawn in a desktop window is 5–20 ms, and worse for audio. The runner states that band in the UI before your first session, then measures its own performance and reports what it actually achieved.

For the cases in the right column, use a photodiode patch and the Route 4 channel anchor to verify against a real channel — or export the paradigm and run it on hardware built for that.

Export to PsychoPy or BIDS

Run it on your own rig, bring the events back, and import them through Route 2. BIDS export is free deliberately: putting the standard interchange format behind a paywall makes a tool untrusted in a teaching lab.

Response Capture and consent

Response Capture records microphone, webcam, keystrokes or typed text, timestamped on the recording clock — the basis for overt-speech decoding and behavioural annotation. It is the most sensitive data this app will ever hold: a webcam clip is identifiable in a way an EEG trace is not, and voice is biometric under regimes the compliance layer already tracks. Four rules are enforced in code, not documented as guidance:

Training on it

Media ports on Select Files

Media is an orthogonal selector, not another entry in the Data Source dropdown. That dropdown means “which container stage does this node read”, one entry to one stage; a media entry there would force you to choose between Epochs and Stimulus — Images when the graph needs both.

Data Source  [ Epochs                  ▾ ]   unchanged
Media        [ ☑ Stimulus   ☐ Response   ]   new
  Binding    [ faces · designed · 40/40  ▾ ]

The ports append after File Paths, so no existing port index moves:

The six Media blocks

Three things you might expect as separate blocks are folded in on purpose: resampling is a Media Loader parameter; raw pixels, patch grids and colour histograms are Media Loader output options, because they are decode choices rather than a feature family; and both TRF directions are one parameter on one block.

Encoding Model (TRF), both directions

One block rather than two, because both share the lagged design matrix, the ridge solve and the null. Parameters are the lag window (default −100 to 400 ms), the ridge alphas and the CV folds; it returns per-channel prediction correlation, the kernel and the selected alpha. Auditory attention decoding is this block in backward with two bindings — attended and unattended — scored by which one reconstructs better.

Stimulus leakage is reported, never enforced

The same image appearing in train and test is a new class of leakage, and it does not fit the existing one. For a reconstruction or generation model it is real leakage even when the trials differ: the model can memorise the picture and then be scored on recalling it. For a classifier it is not leakage at all — and grouping by item would be actively wrong, because it would put every “cat” trial in one group and destroy the split.

The splitter cannot see which of those you are doing. So it computes item-level overlap and reports it as its own field, and leaves the existing group_leakage and publication_grade verdicts completely untouched. A reader of a reconstruction result must look at it; a reader of a classification result may ignore it.

After training

Depth: patterns, never raw weights

The Depth tab of Model Analysis reads artifacts that already exist, so it is cheap to open. It shows learned spatial filters as topomaps (channel positions are already in the study), learned temporal filters as frequency responses, per-layer activation statistics on a chosen batch — mean, spread, saturation, dead-unit fraction — and per-layer output distributions.

Breadth: the same statistics as everywhere else

Breadth renders the group statistics the rest of the app computes; it does not reimplement them, and a test greps its own source to prove it computes none. It shows a subject × class accuracy heatmap, fold-to-fold spread beside every mean, a ranked leave-one-subject-out delta, session and run drift over time, and confusion matrices side by side. See Results Validation for the statistics themselves.

Inference gates

Running a saved model on new data is the most likely place in the whole app for a silent mismatch, because the model and the data now come from different places. Five gates run before it; four block, and the fifth deliberately does not.

Ground truth is optional. With labels, metrics are wrapped exactly as a training run’s are. Without labels you get predictions, confidence and the prediction distribution against training — and no accuracy number is invented: there is no key in the output a reader could mistake for a score, plus a sentence saying why.

Storage and plans

The rule is the same one that governs the rest of the app: charge for management burden, not for capability. Reading your own data is never paid, and all four binding routes are free.

Two sets per study is the same “two of everything, free” shape used across the app, and the refusal message says per study so the limit cannot be misread as per account. On storage: embedded media counts against your cloud quota and linked media does not, which is exactly why the size thresholds default the expensive kinds to linked. One 40-image set is around 2 MB; one movie clip is around 400 MB.

What happens when something is wrong

Each row here is a test, not an intention.

What this does not do