One run, one validation design

A score means nothing until you know how it was tested. 0.977 from a random 20% of the rows and 0.634 from a model trained on the indoor session and tested on the outdoor session answer two different questions — the first is how well the model fits one recording, the second whether it survives a new session. Averaged together they answer neither.

So every run carries its validation design, written when the trainer runs. Every page reads it: the Results page summarises one design at a time, the Leaderboard ranks within a design, and Paired Statistics refuses to pair runs validated differently.

The three designs

The design is set by what is wired into the trainer's Val/Eval input (named Validation until October 2026; a graph saved then opens under the new name with its wire).

A protocol is also recorded with how its trainer used it: plain evaluation, adapted (DANN Trainer), fine-tuned (Fine-Tuning Trainer) or learned in sequence (the continual trainers). An adapted score is not a plain transfer score, so these never share a ranking either.

On the trainer card

The trainer's card on the canvas states its design in one line, and updates as soon as you wire or unwire Val/Eval:

The other two read, for example, validation: hold-out, stratified 20% and validation: 5-fold cross-validation, stratified (Validation Scheme).

What the Val/Eval input accepts

Only a Validation Scheme or an Evaluation Protocol. A scikit-learn splitter object or a bare number of folds is refused, because neither says on the canvas — or in the run record — how the run was validated:

What each scenario shows

One rule set decides what leads, what qualifies it and what a reader is warned of. It reads only what the run recorded — it never asks for a metric your graph did not compute.

Which number leads

The chance bar at your n

At the trial counts a BCI study has, 1/k is not the bar. The app computes the accuracy that is above chance at p < 0.05 for your number of test trials (binomial, Combrisson & Jerbi 2015) and says whether the headline clears it: 40 two-class test trials need 0.65, not 0.50. For balanced classes it also reports the Wolpaw information transfer rate in bits per trial, with its assumptions stated.

What each design warns of

On the Execute page

When a run finishes, its design sits on the run bar beside the run id — Validation: Evaluation Protocol, across sessions — so the tiles below are never read without it.

Save knowledge of this run

Reading the results tables

Every results table in the app follows one rule: a row is an evaluation, a column is a metric. What was tested is kept as separate facts and never folded into a metric's name. Averaging (macro, weighted) is a band over the metrics it groups.

A run without an Evaluation Protocol has no pair, so it shows the Evaluation column only.

An Evaluation filter above the long tables shows one evaluation at a time. n always names what it counts — Subjects, Folds, or n train / n test. For a fine-tuned row, n train is the rows of the fine-tuning itself. Bold “best” compares only rows with the same pair, evaluation and training condition: a cross-session score is never best against a within-session one. A summary of a run with more than six pairs (every subject in turn against every other) pools them into one row per evaluation and names roles instead of parts: source and target. The design's headline metric leads the metric columns, its header in the accent colour over a rule. Right-click any table to copy it or save it as CSV.

The tiles above the tables carry the headline metric only: the metric is the tile's title and the evaluation its sub-line. A protocol run shows at most four — Within-session, the headline evaluation (Cross-session, or Cross-session · DANN / Fine-tuned), the protocol's own Fine-tuned (…) row when present, and Δ Within − Cross; Unadapted is in the tables. The Results page uses the same tiles.

Reading the train → test matrix

The session on the left is the one the model trained on; the session below is the one it was tested on. The outlined diagonal is the same-session test, on trials of that session the model never trained on. The key under the figure says this on every copy:

Fine-tuning for the other session

With the plain BCILattice Trainer a model trained on indoor is tested on outdoor as trained. To also measure how far a little labelled data from the new session takes it, set fine_tune on the Evaluation Protocol (default none):

A network is trained further from its trained weights. A model that cannot train further (LDA, SVM and the other classical models start from zero on every fit) is trained again from zero instead: on its whole training session plus the slice for target only, on the two slices for source + target. The run log, the methods text and the saved model's name say which happened.

The two fine-tuned rows have the same training condition: one model, tested on each session. The first and third rows are the same trials, so their difference is what fine-tuning cost the model on its own session; the Contrasts table tests it subject by subject. With target only the training condition reads indoor + outdoor 20% and the third row is not reported. That row needs within_check set to hold-out: without held-back trials there is nothing honest to test on, and the run says so. With both directions, the same four rows follow for outdoor → indoor.

Each fine-tuned model is kept for export as CNN · trained on indoor, fine-tuned on 20% of outdoor (or retrained with 20% of outdoor for a model that cannot train further). Each fine_tune choice, and each size, is its own validation design, so a run that fine-tunes is never ranked against one that did not, or one that fine-tuned differently. The DANN and Fine-Tuning trainers use the same columns: the training condition reads indoor + outdoor (no labels) for DANN and indoor + part of outdoor for the Fine-Tuning Trainer, with the row as trained (indoor) beside it.

Until October 2026 this setting was a tick box called adapt with one size, adapt_size. A graph saved then opens as target only with the same size, and its rows read Fine-tuned where they read Adapted.

On the Results page

An experiment that holds runs validated differently shows one design at a time — the latest run's by default — with a switch listing every design and its run count. The tiles, the reading notes, the matrix and the charts all describe the runs of that design only.

Under the design line the page has tabs. The first, Results, is the design's runs taken together. Each run of the design then has a tab of its own, holding that run's Per Pipeline, Detailed, Per Subject, Per Class, Statistics, Graphs and Training Curves — the same tabs Execute shows while the run is on screen. With one run that tab is called Run Results; with several, each is named by its run and when it ran. Running the experiment again adds a tab; it does not add a second Results tab, because the Results tab already averages every run of the design.

• Results table: pipeline × evaluation for the design shown, mean ± SD across participants — each participant averaged over the design's runs first. The best per evaluation is bold. Under it, the same Contrasts table as on Execute.
• Reporting checks: each check is a value and a result — the chance bar at your n (binomial, p < 0.05), the class ratio (unbalanced above 1.5), the smallest test (⚠ under 30) and the subjects (⚠ under 3: the drop is described, not tested). The design's caveat — a row-level split, DANN having seen the test data — is the tooltip of the Validation line.
• Pair scores such as indoor -> outdoor accuracy are never tiles — they are the train → test matrix under the table, each cell the mean over the design's runs.
• Charts come from the latest run of the design shown, the run-level picture first, in the order that design is read. They are at the foot of the Results tab; a run with twenty participants stores some four hundred pictures, and each is drawn when you pick it from the list.
• A run's tab is filled the first time you open it, and stays filled while the experiment is on screen. Its Statistics tab tests that run's own differences; nothing in it is pooled with another run.
• Opening an experiment shows a progress bar that names the part being built — scores, reporting checks, results tables, train → test matrix, performance charts — and the page appears complete.
• The study overview shows counts, not scores — Experiments, Runs, Subjects, Validation designs — and then each experiment × pipeline × evaluation for its latest design. There is no study-wide “best”: a best is only meaningful within one design, and it is bold inside that table.

In the Experiment Hub

Every page that lists or compares runs reads each run's evaluations — never one number pooled over its pipelines and evaluations. A score always says which metric, which evaluation and which pipeline it is.

Run History

The run list shows, per run, its Validation, how many Pipelines and Subjects, and its headline as three cells — Metric, Evaluation and Score (the best pipeline's mean ± SD across subjects; the tooltip names the pipeline). Run ID is an optional column and the name's tooltip.

Paired Statistics

Two tables and no notes. Scores is the Comparison table of the selected runs. Contrasts pairs each run with the first (the reference) on one metric and one correction, per pipeline + evaluation + method, paired on subject (× train → test pair for a protocol). Method is what the trainer did before the test (DANN, Fine-tuned …); this table pools every pair, so it has no Condition column:

A contrast that cannot be made is a row of — with the reason in its tooltip — runs validated differently, a pipeline one run lacks, fewer than two shared subjects. A refused cell is never blank, because a blank significance cell reads as “not significant”.

Leaderboard

One board per question — dataset × headline metric × validation design — picked from a Board list. An adapted or DANN design is its own design, so it gets its own board. A row is run × pipeline, the benchmark board's grain: a run with three pipelines is three entries and is never pooled.

• Ranked on the board's headline metric (first, emphasised); the reporting standard's companions follow it.
• Δ Within (protocol boards) is each subject's Within − Cross, averaged: the cost of transfer.
• An exploratory or leakage row keeps its scores but gets — for Rank and Δ best and sorts after the ranked rows — only a publication-grade run may be ranked or become the best.
• Two values, both shown. When the headline has a second value, the metric becomes two columns under its name, and Rank and Δ best split the same way: each column is ranked on its own values, so an adapted score is only compared with other adapted scores. Which pair depends on the run: Before / After adaptation for DANN and fine-tuning (cross-part; the before side is the unadapted score when the run measured it), Before / After fine-tuning for a model's own part after source + target fine-tuning, and Just learned / After the sequence for class-, domain-incremental and continual learning (the difference is forgetting). A side the run did not measure is —, never estimated. The website's boards show the same columns and ranks.

The Experiments list shows each experiment's latest design as Validation · Metric · Evaluation · Score (the Results page's mean ± SD) · Before → after · Runs — no colour pill, and never a best of runs, which could quote a run validated another way. Run History has the same Before → after column for each run's best pipeline, and Results Validation adds a two-value table (level · n · before · after · Δ) beside the comparison when the picked evaluation is one side of a pair — Within-session included.

Exporting the models

A protocol trains one model per training session — each in turn trains its own model — so a subject with two sessions has two models. Each is kept under its own name, by what it trained on:

Adapted and fine-tuned runs name the pair too (trained on indoor, adapted to outdoor). The model block's own object is never exported for a protocol run: each session trained a copy of it, so the original never learned anything.

Runs recorded before this

Old runs are classified once from what they already stored — their split policy and their Evaluation Protocol report — and nothing is re-run. Their design line ends with (read from the stored split). A run that stored neither is shown as Validation: not recorded; it is never guessed.

The methods text

The exported methods paragraph states the design and the reporting choices: