BCINexus · Reading studies

Understanding metrics

Every number a study can report, what it measures, and which way is good. Accuracy going up is good; an error going up is bad; a few numbers are only good when they sit at chance.

Three ways a number can be good

Before reading any score, check which kind it is. Every metric on this page carries one of these tags, and so do the study tables in BCILattice.

Tag	Means	Examples
Higher is better	The bigger the number, the better the model.	Accuracy, kappa, ROC AUC, R², ITR
Lower is better	The number is an error or a cost; zero is perfect.	MAE, RMSE, log loss, false activations, p
Read against a reference	Neither direction is good. Compare it with a stated reference, usually chance.	Domain discriminator accuracy, teacher accuracy
Context, not a score	Describes the data or the test, not the model’s quality.	n, SD, df, decisions per minute

Accuracy, two balanced classesHigher is better

At or below chanceAbove chance, weakUsable

70% is the threshold the BCI literature commonly uses for a decoder someone could communicate with. It is a convention, not a law; what counts is how far above chance a result sits and how much it varies across participants.

The four ranges

Metrics only share a chart axis when they share a range, which is why a study’s charts are split into panels.

Range	Axis label	Examples
0 – 1	Score (0–1)	Accuracy, F1, precision, recall, ROC AUC
−1 – 1	Coefficient (−1–1)	Cohen's kappa, MCC, Pearson r, R² (which can go lower)
0 – ∞	Loss / error	Log loss, MAE, MSE, RMSE, training loss
1, 2, 3 …	Count	Median rank, effective rank

Classification

Used when the model picks a class: left hand vs right hand, target vs non-target. A study reports one of three sets. Minimal is accuracy alone. Standard, the default, adds balanced accuracy, F1, precision, recall and Cohen’s kappa. Full adds the rest of the table below.

Metric	Better	Range	Chance / good	What it tells you
Accuracy	Higher	0 – 1	Chance = 1 / number of classes	Share of trials predicted correctly. Overstates a model when one class is much more common than the others.
Balanced accuracy	Higher	0 – 1	Chance = 1 / number of classes, whatever the class balance	Accuracy computed per class, then averaged. Use it instead of accuracy when classes are unbalanced.
F1 (macro)	Higher	0 – 1	1 = perfect	Balance of precision and recall, each class counted equally. Drops sharply when the model ignores a class.
F1 (micro)	Higher	0 – 1	1 = perfect	F1 pooled over all trials. For ordinary one-label classification it equals accuracy.
F1 (weighted)	Higher	0 – 1	1 = perfect	F1 per class, weighted by how many trials each class has.
Precision (macro / weighted)	Higher	0 – 1	1 = no false alarms	Of the trials the model called class X, how many really were X.
Recall (macro / weighted)	Higher	0 – 1	1 = nothing missed	Of the trials that really were class X, how many the model found. Also called sensitivity.
Cohen's kappa	Higher	−1 – 1	0 = chance · above 0.6 substantial	Agreement with the true labels beyond what chance would give. Corrects for unbalanced classes.
Matthews correlation (MCC)	Higher	−1 – 1	0 = chance · 1 = perfect	Correlation between true and predicted labels using every cell of the confusion matrix. Hard to fool with unbalanced classes.
ROC AUC	Higher	0 – 1	0.5 = chance · above 0.8 excellent	How well the model ranks the right class above the wrong ones across every threshold. Needs class probabilities.
Log loss	Lower	0 – ∞	Guessing evenly scores ln(classes): 0.69 for two	Penalises confident wrong answers. Low means the probabilities are both right and well calibrated. Needs class probabilities.

Chance accuracy: what guessing scores

With balanced classes, a model that guesses scores 1 / k. An accuracy only means something once you know which of these bars it has to clear.

Cohen's kappaHigher is better

Worse than chanceSlightFairModerateSubstantialAlmost perfect

Bands from Landis & Koch (1977). Matthews correlation reads on the same −1 to 1 range, with 0 at chance.

ROC AUCHigher is better

Worse than chancePoorAcceptableExcellentOutstanding

Bands from Hosmer & Lemeshow, Applied Logistic Regression.

The pictures beside the numbers

A classification run also shows a confusion matrix, a per-class report, a ROC curve and a precision–recall curve. They are not single numbers, but they explain the numbers.

Confusion matrix (sketch)

Predicted: LeftPredicted: RightTrue: Left

Correct

Left called Left

Mistake

Left called Right

True: Right

Mistake

Right called Left

Correct

Right called Right

A good model puts its counts on the diagonal. Each row is one true class, so a row whose count sits mostly off the diagonal is a class the model cannot recognise, even when the overall accuracy looks fine.

ROC curve (sketch)

Curve: every threshold the model could use, from strict to lenient.

Shaded area: the AUC. 1.0 is perfect separation.

Dashed diagonal: guessing, AUC 0.5. A curve bowing towards the top-left beats it.

The precision–recall curve reads the same way, with “better” towards the top-right. Its chance line is flat at the share of the positive class, so it is the more honest picture when one class is rare.

Regression

Used when the model predicts a quantity rather than a class: a reaction time, a workload score, a joint angle. Error metrics are in the target’s own units, so they can only be compared between studies that predict the same thing on the same scale.

Metric	Better	Range	Chance / good	What it tells you
R²	Higher	−∞ – 1	0 = predicting the mean · 1 = perfect	Share of the target’s variance the model explains. Negative means worse than always predicting the average.
Explained variance	Higher	−∞ – 1	1 = perfect	Like R², but ignores a constant offset in the predictions.
Pearson r	Higher	−1 – 1	0 = no relationship	Whether predictions rise and fall with the truth. Can be high even when every prediction is off by the same amount.
MAE	Lower	0 – ∞	0 = perfect	Average size of the error, in the target’s own units.
MSE	Lower	0 – ∞	0 = perfect	Average squared error. Punishes a few large errors much more than many small ones.
RMSE	Lower	0 – ∞	0 = perfect	Square root of MSE, back in the target’s units.
Median absolute error	Lower	0 – ∞	0 = perfect	The typical error, unaffected by a few outliers.
MAPE	Lower	0 – ∞	0.10 = off by 10% on average	Average error as a share of the true value. Rows whose true value is zero are left out.

R²Higher is better

Worse than predicting the meanExplains some varianceExplains most variance

R² has no lower limit; the bar is cut at −1 for display.

MAE, MSE, RMSE, median errorLower is better

Small errorLarger errorAs large as just predicting the mean, or worse

A schematic, not a scale: there is no universal “good” error. Judge it against what predicting the average would score, and against the size of the quantity being predicted.

Multi-label

Used when one trial can carry several labels at once. A study shows subset accuracy, Hamming loss and macro F1 together on purpose: they disagree, and any one of them alone can be made to look good.

Metric	Better	Range	Chance / good	What it tells you
Subset accuracy	Higher	0 – 1	1 = every label right on every row	The harshest reading: a row counts only if all of its labels are exactly right.
Hamming loss	Lower	0 – 1	0 = no wrong labels	The most forgiving reading: the share of individual label slots that are wrong.
F1 (macro / micro / weighted)	Higher	0 – 1	1 = perfect	As for classification, computed per label. Macro notices a label the model never predicts.
F1 (samples)	Higher	0 – 1	1 = perfect	F1 computed per row, then averaged over rows.
Jaccard (samples)	Higher	0 – 1	1 = predicted set equals true set	Overlap of the predicted and true label sets, per row.
Precision / Recall	Higher	0 – 1	1 = perfect	Macro-averaged over labels.
Per-label F1	Higher	0 – 1 each	1 = perfect	One F1 for each label, so a weak label is visible.
Label ranking average precision	Higher	0 – 1	1 = true labels always ranked first	Whether the true labels get the highest scores. Needs label scores.
Coverage error	Lower	1 – labels	Best = the average number of true labels	How far down the score ranking you go before every true label is covered.
Label ranking loss	Lower	0 – 1	0 = perfect ordering	Share of label pairs ordered the wrong way round.
ROC AUC	Higher	0 – 1	0.5 = chance	Macro-averaged over labels. Needs label scores.

Self-supervised (representation) learning

A self-supervised run has no labels, so it has no accuracy. Its numbers answer two other questions: did the encoder learn anything, and did it collapse to a single point?

Metric	Better	Range	Chance / good	What it tells you
Contrastive loss	Lower	0 – ∞	Compare across epochs, not across studies	The pretext task’s own objective.
View match (top-1)	Reference	0 – 1	Must beat view match chance	How often an augmented view is matched back to its own pair. At chance the encoder learned nothing.
View match chance	Reference	0 – 1	The floor, not a result	What guessing would score on the view-match task.
View match median rank	Lower	1 – ∞	1 = the pair always found first	Where the true pair typically lands in the ranking.
Masked reconstruction MSE	Lower	0 – ∞	Must beat its mean baseline	Error when filling in masked parts of the signal, measured on the masked parts only.
Masked reconstruction baseline	Reference	0 – ∞	The reference, not a result	What predicting each feature’s average would score.
Masked reconstruction gain	Higher	signed	Above 0 = beats the baseline	Baseline error minus model error.
Embedding spread	Reference	0 – ∞	Near 0 = collapsed	How spread out the learned features are. Past a healthy level, larger is not better.
Effective rank	Higher	1 – dims	1 = collapsed	How many directions the embedding actually uses.

Training-method numbers

Some training methods report extra numbers that say whether the method itself worked. They sit next to the ordinary scores, which are the same across methods so runs stay comparable.

Domain adaptation (DANN)

Metric	Better	Range	Chance / good	What it tells you
Target accuracy	Higher	0 – 1	The headline number	Accuracy on the new domain (e.g. a new participant), whose labels were never trained on.
Source accuracy	Higher	0 – 1	—	Accuracy on the domain the model was trained on.
Domain discriminator accuracy	Reference	0 – 1	Chance is the goal	High means the two domains are still easy to tell apart. Very low usually means the discriminator never trained, which looks like success but is not.
Domain probe (inputs / features)	Reference	0 – 1	Read against chance	A fresh classifier trying to tell the domains apart, before and after the network. Two separate facts, not a before/after pair.

Domain discriminator accuracy, two domainsRead against a reference

Suspiciously low: discriminator may not have trainedNear chance: domains alignedDomains still separable

The width of the green band is illustrative. The point is the shape: the target is the middle, not either end.

Knowledge distillation

Metric	Better	Range	Chance / good	What it tells you
Student accuracy	Higher	0 – 1	The result	Accuracy of the smaller model being trained.
Teacher accuracy	Reference	0 – 1	The ceiling, not a result	Accuracy of the model being copied.
Teacher agreement	Higher	0 – 1	1 = always agrees	How often the student predicts what the teacher predicts.
Distillation KL	Lower	0 – ∞	0 = identical outputs	Distance between the student’s and teacher’s probabilities.

Continual learning

Metric	Better	Range	Chance / good	What it tells you
Average accuracy (all tasks)	Higher	0 – 1	—	Mean accuracy over every task after training on the last one.
Backward transfer	Higher	−1 – 1	0 = no forgetting · negative = forgetting	How learning later tasks changed earlier ones.
Forward transfer	Higher	−1 – 1	Above 0 = earlier tasks helped	How earlier tasks helped with later ones.
First / final task accuracy	Higher	0 – 1	—	Accuracy on the first and last task in the sequence.
Task drift (MMD)	Reference	0 – ∞	Near 0 = tasks barely differ	How different consecutive tasks are. Near zero means there was little to forget.

Multi-task

Metric	Better	Range	Chance / good	What it tells you
Mean task accuracy	Higher	0 – 1	—	Accuracy averaged over tasks.
Worst task accuracy	Higher	0 – 1	—	The weakest task, shown next to the mean so one strong task cannot hide a weak one.

Real-time (pseudo-online) evaluation

Offline scores are computed on cut-out trials, where the decoder is told when each trial starts. A pseudo-online evaluation slides a window over the continuous recording instead, as a live session would. Its numbers are not comparable to offline accuracy, and studies label them separately.

Metric	Better	Range	Chance / good	What it tells you
Detection rate	Higher	0 – 1	1 = every event caught	Share of real events the decoder answered correctly within the tolerance window.
False activations / min	Lower	0 – ∞	0 = never fires by mistake	Decisions that match no event. The number that decides whether a BCI is usable; no offline score shows it.
Decision latency (ms)	Lower	0 – ∞	Lower = faster response	Median time from an event to the decoder’s correct answer.
ITR (bits / min)	Higher	0 – ∞	0 = no better than chance	Information transfer rate: accuracy, number of choices and speed in one number. The fair way to compare BCIs with different class counts.
Decision accuracy	Higher	0 – 1	Chance = 1 / selectable classes	Share of the decoder’s decisions that were right.
Decisions / min	Context	0 – ∞	—	How often the decoder made a decision. Context for ITR.

Information per decision (Wolpaw ITR)

Each curve is zero up to chance (50% for two classes, 25% for four) and rises steeply near 100%. Multiply by decisions per minute to get bits per minute. This is why a 4-class decoder at 70% can carry more information than a 2-class decoder at 85%.

A high accuracy is not enough on its own

A decoder that is 92% accurate offline but fires eleven times a minute at rest is not usable. Read detection rate and false activations per minute together.

Statistics symbols

Group comparisons and validation tables use the short symbols a paper uses. This is the same key the app prints under its tables.

Symbol	Better	Meaning
t	Context	t statistic: the difference divided by its standard error; t(df) shows the degrees of freedom.
F	Context	F statistic (ANOVA): variation between conditions over variation within them; F(df1, df2) shows the degrees of freedom.
H	Context	Kruskal–Wallis H: a rank-based test of whether several groups differ.
χ²	Context	Chi-squared statistic (here from the Friedman test on ranks).
U	Context	Mann–Whitney U: a rank-based test of whether two groups differ.
W	Context	W statistic of the named test (Wilcoxon signed-rank, Shapiro–Wilk or Mauchly).
z	Context	z-score: how many standard deviations a value lies from the mean.
df	Context	Degrees of freedom: how many independent values the test rests on.
p	Lower	Probability of a difference at least this large if there were no real effect. Smaller is stronger evidence; below α (usually 0.05) counts as significant.
p corrected	Lower	p adjusted for running many tests at once (Holm, Bonferroni or FDR), so chance hits are not counted as findings.
α	Context	Significance threshold: a p below it counts as significant.
dz	Higher	Cohen's dz, the paired effect size: the mean difference divided by the standard deviation of the differences (about 0.2 small, 0.5 medium, 0.8 large).
d	Higher	Cohen's d: the difference between two means in standard-deviation units.
g	Higher	Hedges' g: Cohen's d corrected for small samples.
r	Higher	Correlation (or, as rank-biserial r, an effect size), from −1 to 1.
η²	Higher	Eta squared: the share of the variance explained by the factor (partial η² leaves the other factors out).
ε²	Higher	Epsilon squared: the share of the variance explained, for the rank-based Kruskal–Wallis test.
Kendall's W	Higher	Agreement of the participants' rankings, from 0 (none) to 1 (complete).
ε	Context	Greenhouse–Geisser epsilon: how far the data depart from sphericity; p is corrected by it.
ICC(3,1)	Higher	Intraclass correlation, test–retest reliability from 0 to 1 (above 0.75 is usually read as good).
κ	Higher	Cohen's kappa: agreement beyond what chance would give.
n	Context	Number of independent units the test counts (participants, sessions).
SD	Context	Standard deviation: how spread out the values are.
SEM	Context	Standard error of the mean (SD / √n): how precisely the mean is estimated.
CI	Context	Confidence interval: the range of plausible values for the true mean (a 95% CI misses it in 5% of studies).

p valueLower is better

Strong evidenceSignificant at α = 0.05Not significant

Axis cut at 0.2 for display; p runs to 1. A small p says an effect is unlikely to be chance, not that it is large. Read it with the effect size.

Effect size: Cohen's d, dz, Hedges' gHigher is better

NegligibleSmallMediumLarge

Bands from Cohen (1988). The size, not the sign, is what these bands read; a negative value means the difference runs the other way.

ICC(3,1), test–retest reliabilityHigher is better

PoorModerateGoodExcellent

Bands from Koo & Li (2016).

Training loss

loss, train_loss and val_loss are lower-is-better, but they measure whatever objective the model was trained with. Compare them across epochs of one model, never between models or studies. A validation loss that climbs while training loss keeps falling is the usual sign of overfitting.

Reading a results table

Mean ± SD. A score is usually averaged over participants or folds; SD is how much it varied between them. A 75% ± 15% result means some participants were near chance.
Train → test. A condition such as Session 1 → Session 2 means the model was trained on the first and scored on the second. Transfer across sessions or participants is usually lower than within one.
Prefixes. train_ is the score on the training data and is never a result. within_ is the same-part check, baseline_ the model before adaptation, adapted_ after it, and transfer_ the unseen part. Each is the same metric read on a different slice of data.
Before and after. When a study adapts or fine-tunes a model, it shows both values side by side. The gain between them is the finding.
Skipped, not zero. A metric the app could not compute (no probabilities, a single class in the test data) is listed as skipped with the reason, never shown as a number.

The most common way a result is overstated

Accuracy alone on unbalanced classes. If 80% of trials are one class, a model that always guesses that class scores 80%. Check balanced accuracy or kappa before trusting it.

For how studies are checked before publishing, see publish readiness and how review works.

Something missing or out of date? Email support or request a doc page. Include the page name and what you expected to find.