Why a second tab

The Results page can tell you that one pipeline scored highest. It cannot tell you the difference is real. “Highest mean” is not a finding — with twelve subjects and three pipelines, a two-point accuracy gap is what you would expect from noise alone about as often as not, and a table of best scores gives a reader no way to know which they are looking at.

Results Validation closes that gap without leaving the app. Pick a metric, pick the axis to compare across — session, experiment, pipeline, workflow, subject, run, or any combination — and it runs the test the design actually calls for, checks that test’s assumptions, reports an effect size with an interval, corrects the pairwise family, and writes the whole thing out in a form you can paste into a methods section.

Results vs Results Validation

What it reads

Every score your study has already stored. Each completed training run keeps a score per unit per pipeline per metric, and the tab turns all of them into one long table where every row carries every axis it belongs to:

value    metric    experiment       run     workflow   pipeline   subject   session
0.7318   accuracy  Motor Imagery    Run 1   EEGNet     EEGNet     sub-01    Session 1
0.6741   accuracy  Motor Imagery    Run 1   CSP+LDA    CSP+LDA    sub-01    Session 1
0.7204   accuracy  Motor Imagery    Run 1   EEGNet     EEGNet     sub-01    Session 2
   ⋮

Any column can become the grouping; any other column can become the observation unit. That is the whole idea — one table, and the design is whichever slice of it you ask for.

Setting Up a Comparison

Six controls in the left panel. The preview line under them updates as you change any of them and tells you what you are about to run before you run it.

1. Metric

Lists only what this study actually measured, most-measured first. Metric names are canonicalised before they are grouped, so F1 Score from an ML Suite metric block and f1_macro from a typed backend are one metric rather than two half-empty ones. If a canonical metric turns out to pool more than one stored name, the result says so in its caveats — they are treated as one measurement, and if they are not one, that is yours to decide.

Direction is handled for you. Loss, log loss, MSE, RMSE and MAE are lower-is-better, so the written interpretation names the right winner without you inverting anything.

2. Compare by

Tick one factor, or several. Several ticked means their combination becomes the grouping: Session × Pipeline over two sessions and three pipelines gives six levels (Session 1 · EEGNet, Session 1 · CSP+LDA, and so on).

A factor that takes only one value across the stored results is greyed out with its level count shown — there is nothing to compare it against, and that is visible before you click rather than as a refusal after. Factors that cut the data identically are labelled as such: in an ML Suite run a job is one workflow on one subject, so Pipeline and Workflow are usually the same partition under two names, and the picker says so instead of letting you build a two-factor model on a distinction that is not there.

3. Observation unit

This is the control that decides whether your n is real, and it is the one most worth understanding.

Units that are already the grouping are greyed out: a factor cannot be both, because then every unit would sit in exactly one group, nothing could be paired, and every group would hold a single value.

4. Levels

Every level of the current grouping, with a checkbox. Unticking one drops it from the comparison — useful for excluding a pilot session or a pipeline that never finished training. Exclusions are recorded and reported: a test over three of five pipelines is a claim about those three, and the caveat says which two were dropped, because excluding the levels that disagreed is exactly how a result stops being one.

5. Design

Auto detects it, and says why. The design is repeated measures when the same units were measured under every level — which is what makes pairing meaningful — and independent groups otherwise.

Units missing a level are excluded from the paired test and named in the caveats — so an n of 9 out of 12 is never unexplained. Asking for pairing on a design that cannot support it is refused by name rather than quietly run as something else.

6. Test, post-hoc, correction, α

Test defaults to Recommended, which resolves to a named test with a stated reason printed next to the result — “three independent groups, every one normal by Shapiro-Wilk and with homogeneous variances by Levene’s test”. Every alternative valid for the design stays selectable, and whatever you select is what runs.

Post-hoc is Recommended (Tukey HSD after a one-way ANOVA on independent groups, pairwise contrasts otherwise), or an explicit choice, or none. Correction is Benjamini-Hochberg FDR by default, with Holm and Bonferroni for confirmatory work and “no correction” for a genuinely pre-registered single contrast. α defaults to 0.05.

The Test Catalogue

Which tests are offered depends on the design and the number of levels, because those are what make a test applicable at all.

Omnibus tests

Repeated-measures ANOVA assumes sphericity and no Greenhouse-Geisser correction is applied, so a p between 0.01 and 0.05 is flagged as provisional and you are pointed at the Friedman test reported alongside it. That warning appears in the caveats every time, not only when it is convenient.

Two-level contrasts

Welch’s is the default over Student’s wherever a t-test is used: it costs almost nothing when variances are equal and is correct when they are not. Below twelve observations in the smaller sample, Recommended takes the rank-based test instead, because the normality a t-test assumes cannot be checked from that little data anyway.

The effect size follows the test rather than the other way round. Quoting Hedges’ g — which standardises by a pooled SD — next to a Mann-Whitney p describes two different models at once, so the rank test reports rank-biserial r instead.

Post-hoc families

With three or more levels, a significant omnibus says the levels differ but not which ones.

Correcting matters more than which correction you choose: eight levels give 28 pairs, and at α = 0.05 that family produces a false positive more than three quarters of the time uncorrected. Eleven levels (55 pairs) is the most this will print; past 60 pairs the family is refused rather than shown — see refusals.

If the omnibus did not reach α, the pairwise table still appears but is flagged as exploratory: reporting a significant pair under a non-significant omnibus is precisely the multiple-comparison problem the omnibus exists to control.

Two-factor models

Tick exactly two factors and a Type II two-way ANOVA is fitted alongside the main test, with both main effects and the interaction. That is the question a pair of one-way tests cannot answer: whether the effect of one factor depends on the other — whether the pipeline that wins on session 1 also wins on session 2.

A significant interaction is called out explicitly, because it means the main effects should not be reported on their own. Type II sums of squares are used because they are correct for the unbalanced designs that training results usually are. The model is fitted on the averaged table, not on the raw rows, so the same protection against pseudo-replication applies here as to the test above it.

Assumption checks

Levene is median-centred rather than mean-centred because BCI score distributions are routinely skewed, and the mean-centred form flags skew alone as unequal variance — which would push every such comparison onto Welch for a reason that is not heteroscedasticity.

Reading the Result

Verdict, effect, design, levels

Four tiles across the top:

The APA line and the reading

Under the tiles: a plain-language reading of what the test found, then the statistic formatted the way it would be written in a paper, then why this test and this design were chosen.

F(2, 22.0) = 168.74, p < .001, partial η² = 0.939

Repeated-measures ANOVA over 3 levels of pipeline, on 12 subjects: the levels
differ by more than sampling noise (p < .001 < α = 0.05). The effect is large
(partial η² = 0.939). "EEGNet" has the best mean accuracy (0.7311) and
"CSP+LDA" the worst (0.6755), a spread of 0.0556.

The APA line is there because the number a reader needs is not the one a table is good at showing, and assembling F(2, 22.0) = 168.74, p < .001 by hand from six cells is where transcription errors come from.

The distribution plot

A box plot per level, ordered to match the descriptives table underneath it. When the design is paired, each unit is drawn as a faint line across every level, behind the boxes.

The figure renders off the UI thread and exports to PNG, SVG or PDF through the standard export button.

Cross-checks

Every result runs its sibling test too — the rank-based counterpart of a parametric test, or vice versa — and reports it as a labelled cross-check with whether it reaches the same conclusion.

What this result does not say

The last section of every result, and the one worth reading first. It is generated from what actually happened, not from a template. It can include:

• How many stored values were averaged into how many observations, and why testing them unaveraged would inflate n.
• Which units were excluded from a paired test for not having every level.
• Which levels you excluded by hand.
• That the canonical metric pooled more than one stored metric name.
• That the omnibus was not significant, so the pairwise table is exploratory.
• That the repeated-measures ANOVA assumes sphericity with no correction applied.
• That Levene rejected equal variances while Fisher’s ANOVA was the test run.
• That the design cannot reach α at all — see below.

Effect Sizes

Every test returns one, because “p = 0.03” answers whether a difference is distinguishable from zero and never how big it is. A 0.4-point accuracy gain across 200 subjects is both highly significant and useless.

Two-level contrasts also carry a percentile-bootstrap confidence interval on the effect size itself. Where too few resamples are usable to make that interval meaningful, it is reported as absent with the reason rather than as a number describing the resampling.

What It Refuses to Do

A NaN in a results table reads as “not significant” to every reader who has seen one, so the tab refuses by name instead of producing one. Each refusal says what is missing and what to change.

Exporting for a Paper

Two buttons under the reading.

The distribution plot exports separately through the figure export button beneath it.

Worked Examples

Which pipeline actually wins?

Metric            Accuracy
Compare by        Pipeline
Observation unit  Subject
Design            Auto  →  repeated (the same 12 subjects ran all 3 pipelines)
Test              Recommended  →  Repeated-measures ANOVA
Post-hoc          Recommended  →  pairwise contrasts, FDR corrected

The paired design is the point here. Twelve subjects, three pipelines, and the between-subject spread is larger than the pipeline effect — an unpaired test would very likely miss a real difference that pairing finds comfortably.

Is performance stable across sessions?

Metric            Accuracy
Compare by        Session
Observation unit  Subject
Design            Auto  →  repeated
Test              Recommended  →  Paired t-test or Wilcoxon by cohort size

A non-significant result here is the common and useful one — but read it as “this study cannot distinguish the sessions”, not as “the sessions are equivalent”.

Does the winning pipeline depend on the session?

Metric            Accuracy
Compare by        Session  +  Pipeline        (6 crossed levels)
Observation unit  Subject
Design            Auto  →  repeated
→ Repeated-measures ANOVA over the 6 levels, plus a two-way model with
  Session, Pipeline, and the Session × Pipeline interaction.

The interaction term is the answer. A non-significant interaction means the pipeline ranking holds across sessions; a significant one means it does not, and the main effects should not be quoted on their own.

Did the v2 experiment improve on v1?

Metric            Accuracy
Compare by        Experiment
Observation unit  Subject × Session
Design            Auto  →  repeated if both experiments ran the same recordings

Does the preprocessing workflow matter?

Metric            Accuracy
Compare by        Workflow
Observation unit  Subject
→ If the picker says Workflow is the same split as Pipeline, the two are one
  partition under two names in this study and comparing them is comparing
  pipelines. Group by Pipeline instead.

Troubleshooting