Collection The mid-training, SFT and RL data of Carbon-X, with the trained on it. Recipe: https://github.com/huggingface/carbon/tree/main/carbon-x • 4 items • Updated • 3

text stringlengths 1.36k 126k	id stringclasses 0 values	source stringclasses 1 value	metadata stringclasses 0 values
<|fim_prefix|>This 1115 bp genomic region from Arabidopsis thaliana encompasses the long non-coding RNA (lncRNA) gene AT4G07665 and its surrounding regulatory elements. The locus features a putative TATA box promoter upstream, a centrally located CpG island embedded within the lncRNA body, and multiple microRNA target ...	null	gemini_fim	null
<|fim_prefix|>This genomic region encompasses the Caenorhabditis elegans C33D3.5 protein-coding gene along with its extensive 5' and 3' flanking sequences. The locus is structurally complex, characterized by a dense cluster of upstream alternative splice sites, multiple internal promoters (TATA boxes), and a prominent ...	null	gemini_fim	null
<|fim_prefix|>This 691 bp genomic region from Escherichia coli contains the yjiX protein-coding gene, which is located on the negative strand. The sequence is notable for harboring two distinct EcoRI restriction sites, making it amenable to molecular cloning techniques. Additionally, it features two regions with elevat...	null	gemini_fim	null
<|fim_prefix|>This genomic region spans 24,802 bp on the positive strand of the Danio rerio genome and contains the pcsk1nl (proprotein convertase subtilisin/kexin type 1 inhibitor like) gene. The gene body extends from local positions 2055 to 22152. pcsk1nl functions as a serine-type endopeptidase inhibitor, playing...	null	gemini_fim	null
<|fim_prefix|>The genomic region corresponds to the Arabidopsis thaliana gene AT3G12880. This gene encodes a protein involved in fundamental cellular processes, likely regulating metabolic or signaling pathways critical for plant growth and development. The region contains multiple regulatory features, including promot...	null	gemini_fim	null
<|fim_prefix|>This 15.8 kb genomic region on human chromosome 13 encodes a complex, convergently transcribed pair of long non-coding RNAs (lncRNAs): ENSG00000289302 (on the plus strand) and LINC01058 (on the minus strand). The region is characterized by extreme regulatory density, featuring multiple broad enhancer doma...	null	gemini_fim	null
<|fim_prefix|>This genomic region in Danio rerio encodes a protein-coding gene (ENSDARG00000017947), which is a zebrafish ortholog of human PTK6. The gene product is a non-membrane spanning protein tyrosine kinase involved in signal transduction, cell differentiation, and the regulation of cell surface receptor pathway...	null	gemini_fim	null
"<|fim_prefix|>The Cmtr2 gene encodes a nucleoside-2'-O-methyltransferase involved in the 7-methylgu(...TRUNCATED)	null	gemini_fim	null
"<|fim_prefix|>This genomic region on chromosome 3 contains the RNA binding motif single stranded in(...TRUNCATED)	null	gemini_fim	null
"<|fim_prefix|>This 8889 bp genomic region in Mus musculus contains the long non-coding RNA ENSMUSG0(...TRUNCATED)	null	gemini_fim	null

Expand in Data Studio

#

	FineDNA

FineDNA is the training data of Carbon-X, the model that taught Qwen3-14B to read DNA, for all three of its training stages. Each stage is a group of configs:

stage	configs	size	starts from
Mid-training	mid_* , one per source (23)	514.30B tokens, of which the run read 302B	Qwen/Qwen3-14B
SFT	sft_mix , andsft_* for each of its 8 domains	203,923 conversations, 230.3M tokens	HuggingFaceBio/Carbon-X-Base
RL	rl_mix	98,433 prompts, 61.1M tokens	HuggingFaceBio/Carbon-X-SFT

sft_mix and rl_mix are the rows the SFT and GRPO trainers used, in the order they used them. Mid-training has no single mix: Megatron samples the mid_* configs with the weights in the mix below.

from datasets import load_dataset

ds = load_dataset("HuggingFaceBio/FineDNA", "mid_dna_mrna", split="train", streaming=True)
ds = load_dataset("HuggingFaceBio/FineDNA", "sft_mix", split="train")
ds = load_dataset("HuggingFaceBio/FineDNA", "sft_chat", split="train")
ds = load_dataset("HuggingFaceBio/FineDNA", "rl_mix", split="train")

The training recipes for every stage are in huggingface/carbon.

#

	Mid-training

The mid_* configs hold every source of the 302B-token mid-training run, one config per source: genomes and transcripts, biology text that contains DNA, English web and PDF text, and instruction data. Their files are under mid/<source>/.

Every config holds all the data Carbon-X sampled from for that source. That is not the whole upstream dataset: the DNA, for example, is GENERator and OpenGenome2 after the filtering described below. The share of training column below is the weight each config gets, and those weights are the recipe: they decide how much of each config the run read, so some configs were read several times and some only in part. Tokenize the text column with the Carbon-X tokenizer, train with these weights, and you get the same data diet as Carbon-X.

	Columns

Every mid-training config has the same columns:

column
text	the document, as it was tokenized
id	the source's document id, where it has one
source	the config name without mid_ , or for biology text the original source (pubmed ,biorxiv , …)
metadata	the source's other fields, as a JSON string

	The mix

	unique tokens	share of training in Carbon-X	tokens seen in Carbon-X
DNA	181.96B	55%	166.1B
Biology text	15.75B	15%	45.3B
English	310.22B	20%	60.4B
Instruction	6.36B	10%	30.2B
Total	514.30B	100%	302.0B

Carbon-X trained on 302B of these 514.30B tokens.

config	documents	tokens	share of training (weight)	tokens read by Carbon-X	passes
mid_dna_eukaryote_fungi	6,607,490	1.88B	1.12%	3.37B	1.79
mid_dna_eukaryote_protozoa	1,257,499	366.8M	0.19%	581.3M	1.58
mid_dna_eukaryote_plant	9,429,321	5.55B	3.08%	9.30B	1.68
mid_dna_eukaryote_invertebrate	8,716,932	13.71B	7.58%	22.90B	1.67
mid_dna_eukaryote_vertebrate_mammalian	7,169,696	18.96B	10.32%	31.16B	1.64
mid_dna_eukaryote_vertebrate_other	12,420,543	29.56B	16.21%	48.95B	1.66
mid_dna_mrna	52,702,454	19.41B	8.80%	26.58B	1.37
mid_dna_mrna_splice	56,877,102	32.93B	2.20%	6.64B	0.20
mid_dna_prokaryote	17,408,059	59.60B	5.50%	16.61B	0.28
mid_bio_p90_biorxiv	7,262	99.1M	0.24%	710.3M	7.17
mid_bio_p90_biostars	2,036	3.1M	0.01%	25.4M	8.19
mid_bio_p90_finepdfs	51,241	1.96B	4.58%	13.84B	7.07
mid_bio_p90_fineweb	8,205	79.7M	0.19%	560.2M	7.03
mid_bio_p90_pubmed	68,330	854.2M	1.99%	6.01B	7.03
mid_bio_rest	914,391	12.16B	7.00%	21.14B	1.74
mid_bio_gemini_fim	56,866	262.8M	0.50%	1.51B	5.74
mid_bio_gemini_nonfim	72,343	333.4M	0.50%	1.51B	4.53
mid_english_fineweb_edu	96,541,686	99.35B	8.00%	24.16B	0.24
mid_english_dclm	88,340,007	111.98B	8.00%	24.16B	0.22
mid_english_finepdfs_edu	17,596,723	98.89B	4.00%	12.08B	0.12
mid_instruct_nemotron_chat_v3	562,376	2.25B	3.53%	10.67B	4.75
mid_instruct_nemotron_if_v3	249,748	632.0M	0.99%	3.00B	4.75
mid_instruct_nemotron_if_v2	1,997,510	3.48B	5.47%	16.53B	4.75

Tokens is the config's size with the Carbon-X tokenizer (estimated from a 2,000-document sample per config). Tokens read by Carbon-X is share x 302B. Passes is how many times over the run read the config: below 1, it read only that fraction of it.

The full training setup we used for Carbon-X are in the Carbon-X midtraining recipe.

	What is in it

DNA. Eukaryote genomic windows from GENERator, up to 100 kbp, in six species groups. mRNA, mRNA with splice and promoter context, and prokaryote genomes (GTDB and IMG/PR) from OpenGenome2. The raw sequences are also in HuggingFaceBio/carbon-pretraining-corpus. Here they are formatted exactly as they were trained on:

• every sequence is cut to a multiple of 6 bp (the tokenizer reads 6-mers) and wrapped in <dna>…</dna> ;
• eukaryote sequences carry metadata tags half the time, chosen at random: species group and gene type, species group only, or gene type only, e.g. <mammalian_species><protein_coding_region><dna>ATG…</dna> ;
• mRNA and prokaryote sequences are uppercased, splice-junction markers (@ ) are removed, and mRNA longer than 100,000 bases is dropped.

Biology text. Web pages, papers and forum posts that contain DNA, plus synthetic annotations. The web and paper configs were built in two steps; the code is in mid/pipeline/.

1. Find documents that contain DNA:1_regex_filter.py . We ran a regex filter over FineWeb-Edu, FinePDFs, PubMed Central, bioRxiv and Biostars. A document is kept when

• it has a run of 15 or more nucleotide letters (A, T, C, G, U, N) using at least 3 different ones,
• that run is not inside curly braces, which in scraped text is usually LaTeX, and
• it mentions a molecular-biology keyword (dna, gene, primer, pcr, ...).

2. Split by DNA length:2_p90_split.py , which finds DNA withtag_dna_sequences.py . Documents whose longest DNA sequence is at least 39 bp, the 90th percentile, formmid_bio_p90_* ; everything else formsmid_bio_rest . Short runs are mostly primers and motifs, while the p90 documents carry real sequences. In our ablations, upweighting them improved DNA benchmarks.

config	what it is
mid_bio_p90_biorxiv ,mid_bio_p90_biostars ,mid_bio_p90_finepdfs ,mid_bio_p90_fineweb ,mid_bio_p90_pubmed	documents with long DNA sequences, one config per source
mid_bio_rest	the other regex hits from the same five sources, whose longest DNA sequence is under 39 bp
mid_bio_gemini_nonfim	genomic regions described by Gemini next to their sequence
mid_bio_gemini_fim	the same kind of documents in Qwen's fill-in-the-middle format: part of each document is moved to the end, and the model learns to fill it in

English. FineWeb-Edu (100BT sample), DCLM and FinePDFs-Edu, to keep the model's English.

Instruction. Nemotron Instruction-Following-Chat v3 (chat and instruction-following splits) and v2 (chat), rendered with the Qwen3 chat template, reasoning traces kept. Mid-training starts from an instruct model, and this keeps it able to follow instructions.

Biology configs also have a validation split: the documents held out of training to track loss.

#

	SFT

Supervised fine-tuning conversations for a DNA language model: 203,923 rows and 230.3M tokens, 27.4M of them in assistant turns. sft_mix holds exactly the rows the SFT trainer built from the Carbon-X SFT mixture, so training on it reproduces the mixture without any further weighting.

Each domain also has its own config, sft_<domain>, holding the sft_mix rows of that domain in the same order, e.g. load_dataset("HuggingFaceBio/FineDNA", "sft_chat").

	Columns

• messages : the conversation, as a list of{role, content} turns.
• chat_template_kwargs :{"tokenizer_kwargs": {"auto_dna_tags": <bool>}} , passed per example toapply_chat_template so the DNA tokenizer wraps sequences in<dna>...</dna> . It isfalse only for the sequence rewriting rows, whose answers are plain text. Keep it when selecting training columns.
• domain : the kind of task, one of the eight below.

	Domains

config	rows	tokens	assistant tokens	what the rows teach
sft_biology_qa	4,603	2.8M	2.2M	genetics, molecular biology and bioinformatics questions from Biology StackExchange, Biostars, real user chats and Nemotron chat data
sft_chat	8,553	2.1M	1.5M	multi-turn conversations on molecular-biology concepts, the assistant's identity, honesty about its limits, and pushing back on false claims
sft_dna_computation	44,925	19.1M	7.0M	answers computed exactly from the sequence: counting, GC content, melting temperature, translation, restriction sites, base lookup, coordinates, edits, rewriting
sft_instruction_following	5,904	3.7M	2.1M	answering in the format the user asks for, and following format constraints on biology prompts
sft_sequence_analysis	7,561	8.7M	3.6M	free-text, mostly multi-turn discussion of a pasted sequence: primer checks, annotation, coding potential, troubleshooting
sft_sequence_classification	16,396	12.8M	2.0M	multiple-choice functional genomics: promoter vs enhancer, species of origin, TF motifs, TF binding sites
sft_sequence_continuation	58,727	64.8M	0.65M	predict the next 1–32 nucleotides after a 4–8 kb eukaryotic context
sft_variant_effect_prediction	57,254	116.2M	8.2M	decide whether a single-nucleotide variant is benign or pathogenic from its sequence window, with templated reasoning

Tokens are counted with the Carbon-X tokenizer and chat template. Assistant tokens are the ones the SFT loss is computed on, and no conversation is longer than the 9,984-token training context.

#

	RL

Verifiable prompts for reinforcement learning of a DNA language model: 98,433 rows and 61.1M prompt tokens, covering 80,406 distinct prompts. Every prompt has a gold answer or constraint set that a reward function checks in code. The blend was calibrated on rollouts from Carbon-X-SFT and excludes the prompts that model saw during SFT. It is meant for one epoch. The rows of rl_mix are in the order the GRPO trainer's mixture produces (seed 0), so training on it reproduces the blend exactly.

Some prompts appear more than once on purpose, for groups that learn slowly under RL: basic_dna repeats half of each task type (10,153 second copies), bioreason repeats 30% of each subgroup and label (4,614), and dna_skills repeats every prompt (3,260).

	Columns

• prompt : the conversation to complete, as a list of{role, content} turns (2,983 rows open with a system turn).
• answer_text : the gold answer that most rewards compare against.
• task_group : which reward scores the row. Each task reward returnsNone for rows of other groups.
• task_type : the sub-task, used by thebasic_dna andseqrecqa_position rewards (null elsewhere).
• verifier_metadata : the constraints an instruction-following reply must meet (if rows only).
• gold_answer_json : the exact answer the DNA-skills verifier checks (dna_skills rows only).
• domain : the kind of task. It uses the same labels as the SFT domains. Five of the eight occur here because the blend has no chat, sequence-analysis or sequence-continuation prompts.

	Task groups

domain	task group	rows	prompt tokens	what the prompts ask
dna_computation	basic_dna	30,461	2.4M	short-sequence computations with code-checked answers: base counts, melting temperature, restriction fragments, CRISPR cut sites, translation, motif spacing
dna_computation	dna_skills	6,520	1.1M	exact operations on longer sequences: locate a base or snippet, convert coordinates, compare, edit, translate, design
dna_computation	seqrecqa_position	6,000	3.1M	localisation: the position of a base, 6-base block, variant block, 12-base span, single-base difference or snippet
variant_effect_prediction	bioreason	19,996	29.0M	is a coding-sequence or non-SNV variant pathogenic or benign
variant_effect_prediction	vepqa	7,426	14.3M	is a single-nucleotide variant pathogenic or benign, judged from its sequence window
sequence_classification	genomeqa	14,032	7.1M	multiple-choice functional genomics: promoter vs enhancer, species of origin, TF motifs and binding sites, splice sites, histone marks
biology_qa	bio_mcq	9,998	3.3M	four- and ten-option biology knowledge questions
instruction_following	if	4,000	0.65M	prompts with verifiable formatting constraints

In total, dna_computation has 42,981 rows, variant_effect_prediction 27,422, sequence_classification 14,032, biology_qa 9,998 and instruction_following 4,000.

Prompt tokens are counted with the Carbon-X tokenizer and chat template, as the GRPO trainer renders the prompts (thinking disabled). Completions are generated during training, so they are not counted.

#

	License

FineDNA is released under the Apache 2.0 license. The mid-training configs are built from third-party sources (GENERator, OpenGenome2, FineWeb-Edu, DCLM, FinePDFs-Edu, PubMed Central, bioRxiv, Biostars and Nemotron), which keep their own licenses and terms. Check the source dataset before use.

#

	Citation

FineDNA was built for Carbon-X, a joint collaboration between the research teams at Hugging Face, Zhongguancun Academy, and the Telethon Institute of Genetics and Medicine/University of Naples “Federico II”. If you find it useful, please cite it as:

@misc{benallal2026finedna,
  title        = {FineDNA},
  author       = {Ben Allal, Loubna and Tunstall, Lewis and Aubakirova, Dana and Fiusco, Maurizio and Beeching, Edward and Rasul, Kashif and von Werra, Leandro and Li, Qiuyi and di Bernardo, Diego},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/HuggingFaceBio/FineDNA}}
}

• Downloads last month
• 72

source & further reading

huggingface.co — original article Project Guardian: A Hybrid Safety System for AI Agents SCCN Unthink: semantic scene reconstruction across seven model/surface configurations When the fastest LLM reply skips the task: a proposed comparison protocol