AI-assisted: this article was drafted with AI assistance and checked against BCI Report’s published results.
Key results
Some recent model interfaces answer explicit, typed questions instead of generating free text. TypeSafe’s Jev model and Visual Jev (Yu and Yao) follow an encode-once, answer-many pattern: represent the input once, then answer several questions about it. An earlier shared-encoder article asked whether several decisions about one EEG recording could share computation; its questions were identifiers. Writing the questions in natural language is the obvious next step, and it raises a sharper question: if a model accepts a sentence, can it answer a question whose EEG label it has never been trained on?
BCI Report’s questions-in-language experiment, released on 8 October 2026, tests that directly. It replaces the identifiers with label templates or descriptions, then measures what happens when the wording changes and when the question itself is new. Here the idea is tested on EEG with frozen features and a small question-conditioned head. This article covers its primary SSVEP panel.
When a person looks at a light flickering at a steady rate, the visual cortex responds at that frequency and its harmonics: a steady-state visual evoked potential (SSVEP). A speller exploits this by giving every key its own flicker frequency, and SSVEP decoding identifies the flickering target a person was attending. BETA (Liu et al., 2020) is a public 40-target benchmark of this kind: 70 people, frequencies from 8.0 to 15.8 Hz in 0.2 Hz steps with phase offsets, recorded with 64 channels outside an electromagnetically shielded room.
The classic decoder needs no training at all. Standard CCA (Lin et al., 2006) compares a window of EEG with sine and cosine reference signals at each candidate frequency and its harmonics, and picks the frequency with the highest canonical correlation (Nakanishi et al., 2015). Filter-bank CCA (Chen et al., 2015) improves on it, still without calibration. Methods such as task-related component analysis (Nakanishi et al., 2018) are usually stronger, but they learn each person’s response to each target from labeled trials. That makes CCA an unusually strong reference for a question about frequencies a learned model never saw: it builds its references from the frequency value itself, so no frequency is ever new to it.
The panel uses BETA’s 70 people, two-second windows and participant-disjoint folds (no person appears in both training and test). In each of five declared rotations, 32 of BETA’s 40 target frequencies are available to train the head; the remaining eight are held out. Every frequency is held out in exactly one rotation, and no two held-out frequencies are adjacent on the grid.
Two EEG representations are tested separately: a plain spectrum and frozen CBraMod features. Each way of posing the question is called an arm. The language arms embed each question with frozen bge-small-en-v1.5. A small head combines the EEG features with the question vector through FiLM conditioning and produces a binary score for that question; the candidate with the highest score is the answer. The identifier arm uses a one-hot question vector instead of text, and a numeric arm uses a smooth code of the frequency; neither uses the text encoder.
Every SSVEP arm trains for 60 epochs (a declared fallback set by a fit check on training data only), with three seeds, no validation set and no early stopping. Neither encoder is fine-tuned.
The tests distinguish three abilities, and they are never merged into one score because their candidate sets and chance levels differ.
BETA, 70 people, two-second windows; every frequency is held out in exactly one of five rotations.
“Unseen” needs care. It means unseen by the EEG head. For a held-out frequency, its labeled recordings are excluded from training, scaling and batch sampling, and its frequency string is absent from training texts. The frozen text encoder has already learned language, and in the primary experiment every tokenizer token of a held-out frequency also occurs in the training texts, in other combinations. So this is a test of transferring a question representation to a new EEG label, not of a text encoder meeting entirely new words.
The wording tests are deliberately narrower than an unrestricted conversation. One of the three training templates is “The person is attending the target flickering at {label}.” Held-out sentence frames change the surrounding sentence. Three fixed alternative labels per frequency replace the training label: for 8.2 Hz they are “8.2 hertz,” “8.2 cycles per second” and “eight point two hertz.” A separate arm is trained on descriptions and tested on new ones. The complete wording lists and frequency partitions were fixed before scoring; they were written by agents from a fixed brief, not collected from users or tuned to the results.
The seen 32-way task is hard for every learned head on these features. With question numbers, the head reached 32.1% balanced accuracy on a plain spectrum and 24.2% on frozen CBraMod features, against a chance level of 3.1%. CCA reached 65.5%.
CCA works on the EEG directly, so one value serves both columns.
Replacing the question identifier with its training template already costs balanced accuracy: −5.96 percentage points (pp), 95% interval [−6.84, −5.08], with the spectrum and −3.49 pp [−4.10, −2.93] with CBraMod features. Descriptions also score lower than identifiers at both levels. None of these four comparisons meets the predeclared ±2-point margin. (A comparison is called equivalent when its whole 95% interval lies within ±2 pp; otherwise (for every comparison here) the margin is “not met,” which can still mean a small cost.)
Changing the wording adds a further cost. Every paired interval favors the training wording. Only the CBraMod alternative-label comparison is equivalent within the ±2-point margin: its cost is measurable but lies inside the tolerated margin. The other five do not meet the margin. The alternative-label result holds for these three alternatives per frequency, not for arbitrary paraphrases.
Paired differences in balanced accuracy on the seen task, with 95% intervals; the shaded band is the predeclared ±2 pp margin, and “equivalent” marks the one comparison whose whole interval lies inside it. The plain-spectrum alternative-label cost, −1.84 pp [−2.10, −1.57], just misses the margin. The two “instead of number” rows compare against the question-number head; the three rewordings compare against their own training wording. A difference can be measurable and still lie within a tolerated margin; “margin not met” is not the same as a loss of two points or more.
The eight-way unseen-frequency test asks a different question. The language head now receives a candidate frequency it never learned to detect. Call the head trained on label-template questions the template head. Three references sit beside it: a head of the same design trained on a smooth numeric code of the frequency instead of text; a baseline with no new training that averages the template head’s scores for the adjacent seen frequencies (two in the interior of the grid, one at its edges); and CCA. Together they test whether a language embedding adds useful transfer beyond numerical structure, interpolation or an analytic frequency reference.
BETA, 70 people, five rotations and three training seeds for the learned heads. Bars show participant-mean accuracy; whiskers are the published descriptive 95% intervals; the vertical line marks 12.5% chance. Blue: the language head; gray: references that use no text for the held-out frequency. “Neighbor average” is the template head’s own scores averaged over the adjacent trained frequencies. The same CCA result is repeated as a reference in both panels: CCA operates on the EEG, not on either feature representation. Intervals are not adjusted for multiple comparisons and omit dependence from overlapping cross-validation training sets.
The language head was above chance: 28.7% with the spectrum and 19.0% with CBraMod features, against 12.5%. Descriptions did about the same, 28.6% and 18.6%. But in paired comparisons every reference scored higher than the template head.
Negative values mean the template head scored lower.
The template head minus CCA is −52.21 pp [−55.13, −48.99] with the spectrum and −61.94 pp [−65.65, −58.06] with CBraMod features; neither meets the ±2-point margin. The template head also scored below the numeric code and below its own neighbor average at both levels; for these two, the protocol asks only which side of zero the interval falls, and no margin applies. The description rows are descriptive; the declared comparisons use the template head.
CCA’s advantage is specific to a task whose candidate frequencies are known. Its 80.9% here is an eight-way result and should not be compared directly with the 40-way score in the earlier BETA article.
The BETA panel passes both declared gates: the identifier head has room above chance and below ceiling, and simple per-channel features do not solve the seen task alone. Intervals come from 2,000 hierarchical paired-bootstrap draws over people, training seeds within each outer fold and, where applicable, held-out wordings; the three alternative labels are fixed. The full study has 35 primary comparisons without a multiplicity correction, and its intervals describe this fixed experiment, not other choices of model or recipe.
The panel also has a second unseen-frequency test that this article does not chart: a three-way choice between the held-out frequency and its two trained neighbors (a two-way choice at the grid edges; mean chance 34.2%), in which the trained neighbors have an advantage by design. On a plain spectrum the template head scored 19.3%, below that chance level, with no difference shown against the numeric code. On frozen CBraMod features it scored 27.0% against the numeric code’s 11.4%, both below chance, a difference of +15.61 pp [14.25, 16.93] in the language head’s favor; but those features keep the stimulus phase and neighboring targets differ in phase, so that comparison tests frequency and phase together. The plain spectrum, which carries no phase, is the clean frequency test. CCA reached 81.7% on this test.
Before any result was read, a pre-run check refit the heads on shuffled training labels and required chance-level scores. Its first two versions failed on the study’s sleep-staging panel, which is not covered here (one, then three, of 38 intervals excluded chance); every SSVEP interval included chance both times. A revised check, adopted after those failures and overriding an earlier “no further revision” clause, averaged 20 heads per configuration and passed: none of 38 intervals excluded chance, although single heads still strayed from chance in both directions (66 above and 59 below, of 760). Both failed records are retained. The sleep results and the secondary studies, including Chinese wordings and single-seed runs declared likely inconclusive, are outside the results discussed here. The full protocol and disclosures also record a planned diagnostic that was not run, a declared prediction that was not activated, and an engineering amendment that changed no question, wording, metric or margin.
Using language as a handle on classes is an active idea. CLIP (Radford et al., 2021) popularized zero-shot classification by scoring candidate text prompts, and the head here uses a similar recipe, scoring each candidate’s text and picking the highest, but with a FiLM-conditioned binary head rather than a contrastive embedding. In EEG, pretraining on EEG paired with clinical reports has supported zero-shot clinical classification (Gijsen and Ritter, 2025), and NeuroLM (Jiang et al., 2025) uses text instructions to switch among EEG tasks it was trained on.
The closest work is a preprint posted on 9 October 2026, Teaching neural decoders with language (Wen et al.). It trains EEG encoders to align with natural-language class descriptions and reports conditional unseen-frequency decoding on the 35-person SSVEP Benchmark dataset: with the same people in training and testing, its encoders chose among five held-out frequencies (20% chance) and scored numerically above filter-bank CCA with 0.6 to 1.0 s windows. The authors also report that correct pairing of descriptions and classes supports this generalization. The setups differ in ways that matter: the same people versus new people, EEG encoders trained with the language objective versus frozen features with a small head, five candidates versus eight, and short windows versus two seconds. The two results do not contradict each other; they describe different recipes.
Classical SSVEP research has its own route to untrained frequencies. Cross-stimulus transfer reuses a person’s calibration data from some frequencies to build templates for others (Wong et al., 2021; Wang et al., 2023), and SRRNet (Mai et al., 2025) regresses SSVEP templates for unseen classes from their sine–cosine reference signals. These methods describe a class by its stimulus rather than by text, and they work within one person’s calibration data and report speeds on full spellers, so their numbers are not comparable with the new-person accuracies here.
Finally, controls that remove the supposed source of information have been revealing elsewhere. Many EEG-to-text models scored about as well on noise as on EEG (Jo et al., 2026), and removing the language model from several language-model-based forecasters did not degrade forecasting performance (Tan et al., 2024). The numeric-code and neighbor-average controls play that role here.
These results suggest testing three capabilities separately: answering a trained question, recognizing a different wording of it, and answering a question whose EEG label was never trained. A model can pass one and fail another. A natural-language interface therefore needs an explicit supported-question domain, tested rewordings and a policy for unsupported questions.
Where the structure of the answer is known in advance, as with SSVEP frequencies, a method that uses that structure directly is the reference to beat. In the eight-way test a numeric code and a neighbor average both scored higher than the language head, so on that test the text embedding transferred less than numerical structure or interpolation did. This fixed recipe does not establish a ceiling for language-conditioned EEG models or validate an open-ended EEG assistant.
The unseen-frequency results can be read from the public aggregates without EEG recordings or weights:
import jsonfrom urllib.request import urlopencommit = "461ebfc1c8ad2a5e316c69dc74bbd23f48ee72d8"url = (f"https://raw.githubusercontent.com/twu3202/bci-report/{commit}/" "site/public/data/questions-in-language-update.json")with urlopen(url, timeout=30) as response: result = json.load(response)["results"]["questions-in-language"]# L0 = plain spectrum, L1 = frozen CBraMod featuresfor block in ("P-ssvep-L0", "P-ssvep-L1"): print(block) for key in ("TPL u8", "DESC u8", "NUM u8", "TPL nn8", "CCA u8"): row = result["descriptive"][block]["levels"][key] print(key, f'{100 * row["estimate"]:.1f}%')
The result page, data API and Hugging Face dataset provide the remaining comparisons and their source records.
BETA was created by Bingchuan Liu and colleagues, with Figshare version 3 recorded as upstream and the Bingchuan/BETA mirror used for the evaluation. BETA and BCI Report’s aggregate measurements are CC BY 4.0. CBraMod’s code is MIT and its weights Apache-2.0; bge-small-en-v1.5 is MIT. These results imply no endorsement by the dataset or model authors. All figures and tables were rendered from BCI Report’s published aggregates.
To cite these results: BCI Report (2026), doi:10.5281/zenodo.23123296, together with the upstream dataset.
BCI Report, which produced these evaluations, is a personal, noncommercial project; this article uses only its published aggregate results.
Can an EEG Decoder Answer a Question it Was Never Trained on? was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.