Community & reputation

Where things surface

How standing is earned

There is no points economy you can farm. Standing comes from things that are verifiable: what you published, what held up, and what you contributed to other people's work.

• Publishing your first study grants the researcher capability and its profile badge.
• Completed review stages are attributed to you on the studies they helped publish.
• Reproduction attempts you run — including the ones that fail — are attached to your profile.
• Citations and downstream reuse feed the impact history shown on studies and profiles.
• The leaderboard ranks contributors and reviewers on that activity.

Leaderboards and challenges

Prose claims about accuracy are hard to compare because everyone cuts their data differently. A leaderboard compares results only when they share the dataset, the way they were evaluated, and the metric.

Leaderboard

One dataset, one evaluation protocol, one metric. Each row is one model from one published study, and it links back to that model and study, so a number can always be traced.

Auto-entered

How results get onto a leaderboard. Publishing a study enters its model results by themselves: there is no submit button, and nobody ranks their own work.

Challenge

A governed task with a fixed dataset and metric, often a time window, and sometimes a hidden test set that BCINexus scores itself. Entering one is a deliberate step.

Reproduction attempt

Someone re-running a published result and recording what they got. A negative reproduction is a first-class contribution, not a complaint.

How a result gets on a leaderboard

A row is one model's score on one dataset, evaluated one way: “this model scored 0.84 ± 0.06 accuracy on this dataset, cross-subject, over 20 participants, in this study by these authors.” BCILattice writes the rows when a study is published, from the run's own per-participant scores. Nothing is typed in by hand, and every published run is included, not the best of several.

• The score is the mean ± SD across participants. Each participant's recordings are averaged first, so twenty people count as twenty observations.
• Each model gets its own row and links to its model card. A study with three models contributes three rows.
• The dataset is the BCINexus dataset the study names, or its dataset title. A dataset uploaded with a study joins the leaderboards once it has passed its own review.
• A study's rows appear once a reviewer has verified the study. Until then they show on the study page as awaiting verification.
• Only the metrics on the leaderboard list open a board (accuracy, balanced accuracy, F1, kappa, precision, recall, ROC AUC, MCC, R², MAE, RMSE, MSE, log loss). Training-set scores never do.

Results are only ranked against results evaluated the same way. Each protocol has its own leaderboard:

A row whose split was exploratory, or where participants or sessions appear on both sides of the split, is listed with that reason and is not ranked. The protocol comes from the run's cohort setting and its split policy, so it cannot be chosen after the fact.

What the author sees in BCILattice

The Publish page has a Leaderboards tab. Before publishing it lists every row the study will auto-enter: the model, the leaderboard it joins (dataset · protocol · metric), its score, and whether it will rank. If it will not rank, the tab gives the reason, and it says whether the row joins an existing board or opens a new one. Below that it names what goes on no board, and why: a metric not on the list, a training-set score, a model with no test score, or runs that were not ticked.

After the study is published, the same tab reads back where each row stands on BCINexus: Ranked #3 of 12, Listed, not ranked with the reason, or Waiting for verification. Nothing opens by itself after publishing.

Entering a challenge

When a study matches an open challenge, BCILattice says so on Publish ▸ Leaderboards and on the Experiment Hub's Leaderboard tab. A study matches when the challenge names the study's dataset (or, for a challenge without a dataset, its modality). For a challenge without a hidden test set, the study must also have recorded a result in the challenge's metric. The author decides whether to enter: press Enter… next to the challenge. The study must be published first, because the entry names the published study.

• Hidden test set: you upload predictions for the held-out examples and BCINexus computes the score. The true labels are never sent to anyone.
• No hidden test set: you pick one result the study recorded in that metric. The score cannot be typed, and the entry names the run, model and protocol it came from. A score that matches no result the study recorded is flagged for the validator.
• Every entry carries an honesty attestation, is screened for anti-cheat flags, and is validated by an admin before it is ranked.

Entries are made from BCILattice only. The website lists the challenges and their entries but has no submit form, because a form would accept a typed number.

Submitting a reproduction

• Start from the published study or model rather than reimplementing from the abstract.
• Record what you changed if you could not match the original environment exactly.
• Report the number you actually got, next to the one that was claimed.
• Attach your own reproducibility stamp so your attempt is itself reproducible.

Discussion and conduct

Public comments on a study are visible to everyone and attributed to you. Review comments are a separate, private channel — see how review works. The community guidelines apply everywhere: criticise the work, be specific, and bring evidence.

Notifications and digests

• In-app notifications cover review activity, comments and replies, authorship invitations, team events, and decisions on your work.
• Email delivery follows your notification preferences per event type — the two are driven by the same setting, so turning something off turns off both.
• A periodic digest summarises activity relevant to you without one message per event.

Related