FineDNA: Open data to train LLMs on DNA

Hugging Face released FineDNA, the full training dataset behind Carbon-X, the model that adapted Qwen3-14B to read DNA across all three of its training stages. The mid-training stage comprises 23 configs totaling 514.30B tokens, of which the run read 302B, starting from Qwen/Qwen3-14B; the SFT stage contains 203,923 conversations spanning 8 domains at 230.3M tokens. The release makes the mid-training, SFT and RL data public alongside a training recipe on GitHub, giving researchers an open path to reproduce a DNA-capable LLM.

Collection The mid-training, SFT and RL data of Carbon-X, with the trained on it. Recipe: https://github.com/huggingface/carbon/tree/main/carbon-x • 4 items • Updated • 3 https://huggingface.co/collections/HuggingFaceBio/finedna | text stringlengths 1.36k 126k | id stringclasses 0 values | source stringclasses 1 value | metadata stringclasses 0 values | |---|---|---|---| | <\|fim prefix\| This 1115 bp genomic region from Arabidopsis thaliana encompasses the long non-coding RNA lncRNA gene AT4G07665 and its surrounding regulatory elements. The locus features a putative TATA box promoter upstream, a centrally located CpG island embedded within the lncRNA body, and multiple microRNA target ... | null | gemini fim | null | | <\|fim prefix\| This genomic region encompasses the Caenorhabditis elegans C33D3.5 protein-coding gene along with its extensive 5' and 3' flanking sequences. The locus is structurally complex, characterized by a dense cluster of upstream alternative splice sites, multiple internal promoters TATA boxes , and a prominent ... | null | gemini fim | null | | <\|fim prefix\| This 691 bp genomic region from Escherichia coli contains the yjiX protein-coding gene, which is located on the negative strand. The sequence is notable for harboring two distinct EcoRI restriction sites, making it amenable to molecular cloning techniques. Additionally, it features two regions with elevat... | null | gemini fim | null | | <\|fim prefix\| This genomic region spans 24,802 bp on the positive strand of the Danio rerio genome and contains the pcsk1nl proprotein convertase subtilisin/kexin type 1 inhibitor like gene. The gene body extends from local positions 2055 to 22152. pcsk1nl functions as a serine-type endopeptidase inhibitor, playing... | null | gemini fim | null | | <\|fim prefix\| The genomic region corresponds to the Arabidopsis thaliana gene AT3G12880. This gene encodes a protein involved in fundamental cellular processes, likely regulating metabolic or signaling pathways critical for plant growth and development. The region contains multiple regulatory features, including promot... | null | gemini fim | null | | <\|fim prefix\| This 15.8 kb genomic region on human chromosome 13 encodes a complex, convergently transcribed pair of long non-coding RNAs lncRNAs : ENSG00000289302 on the plus strand and LINC01058 on the minus strand . The region is characterized by extreme regulatory density, featuring multiple broad enhancer doma... | null | gemini fim | null | | <\|fim prefix\| This genomic region in Danio rerio encodes a protein-coding gene ENSDARG00000017947 , which is a zebrafish ortholog of human PTK6. The gene product is a non-membrane spanning protein tyrosine kinase involved in signal transduction, cell differentiation, and the regulation of cell surface receptor pathway... | null | gemini fim | null | | "<\|fim prefix\| The Cmtr2 gene encodes a nucleoside-2'-O-methyltransferase involved in the 7-methylgu ...TRUNCATED | null | gemini fim | null | | "<\|fim prefix\| This genomic region on chromosome 3 contains the RNA binding motif single stranded in ...TRUNCATED | null | gemini fim | null | | "<\|fim prefix\| This 8889 bp genomic region in Mus musculus contains the long non-coding RNA ENSMUSG0 ...TRUNCATED | null | gemini fim | null | Expand in Data Studio https://huggingface.co/datasets/HuggingFaceBio/FineDNA/viewer/mid bio gemini fim/train FineDNA FineDNA is the training data of Carbon-X https://huggingface.co/HuggingFaceBio/Carbon-X , the model that taught Qwen3-14B to read DNA, for all three of its training stages. Each stage is a group of configs: | stage | configs | size | starts from | |---|---|---|---| | Mid-training mid-training | mid , one per source 23 | 514.30B tokens, of which the run read 302B | Qwen/Qwen3-14B https://huggingface.co/Qwen/Qwen3-14B | | SFT sft | sft mix , and sft for each of its 8 domains | 203,923 conversations, 230.3M tokens | HuggingFaceBio/Carbon-X-Base https://huggingface.co/HuggingFaceBio/Carbon-X-Base | | RL rl | rl mix | 98,433 prompts, 61.1M tokens | HuggingFaceBio/Carbon-X-SFT https://huggingface.co/HuggingFaceBio/Carbon-X-SFT | sft mix and rl mix are the rows the SFT and GRPO trainers used, in the order they used them. Mid-training has no single mix: Megatron samples the mid configs with the weights in the mix the-mix below. python from datasets import load dataset one mid-training source; the mid-training configs total about 950 GB, so stream them ds = load dataset "HuggingFaceBio/FineDNA", "mid dna mrna", split="train", streaming=True the SFT mixture, or one of its domains ds = load dataset "HuggingFaceBio/FineDNA", "sft mix", split="train" ds = load dataset "HuggingFaceBio/FineDNA", "sft chat", split="train" the RL blend ds = load dataset "HuggingFaceBio/FineDNA", "rl mix", split="train" The training recipes for every stage are in huggingface/carbon https://github.com/huggingface/carbon/tree/main/carbon-x . Mid-training The mid configs hold every source of the 302B-token mid-training run, one config per source: genomes and transcripts, biology text that contains DNA, English web and PDF text, and instruction data. Their files are under mid/<source / . Every config holds all the data Carbon-X sampled from for that source. That is not the whole upstream dataset: the DNA, for example, is GENERator and OpenGenome2 after the filtering described below. The share of training column below is the weight each config gets, and those weights are the recipe: they decide how much of each config the run read, so some configs were read several times and some only in part. Tokenize the text column with the Carbon-X tokenizer, train with these weights, and you get the same data diet as Carbon-X. Columns Every mid-training config has the same columns: | column | | |---|---| | text | the document, as it was tokenized | | id | the source's document id, where it has one | | source | the config name without mid , or for biology text the original source pubmed , biorxiv , … | | metadata | the source's other fields, as a JSON string | The mix | | unique tokens | share of training in Carbon-X | tokens seen in Carbon-X | |---|---|---|---| | DNA | 181.96B | 55% | 166.1B | | Biology text | 15.75B | 15% | 45.3B | | English | 310.22B | 20% | 60.4B | | Instruction | 6.36B | 10% | 30.2B | | Total | 514.30B | 100% | 302.0B | Carbon-X trained on 302B of these 514.30B tokens. | config | documents | tokens | share of training weight | tokens read by Carbon-X | passes | |---|---|---|---|---|---| | mid dna eukaryote fungi | 6,607,490 | 1.88B | 1.12% | 3.37B | 1.79 | | mid dna eukaryote protozoa | 1,257,499 | 366.8M | 0.19% | 581.3M | 1.58 | | mid dna eukaryote plant | 9,429,321 | 5.55B | 3.08% | 9.30B | 1.68 | | mid dna eukaryote invertebrate | 8,716,932 | 13.71B | 7.58% | 22.90B | 1.67 | | mid dna eukaryote vertebrate mammalian | 7,169,696 | 18.96B | 10.32% | 31.16B | 1.64 | | mid dna eukaryote vertebrate other | 12,420,543 | 29.56B | 16.21% | 48.95B | 1.66 | | mid dna mrna | 52,702,454 | 19.41B | 8.80% | 26.58B | 1.37 | | mid dna mrna splice | 56,877,102 | 32.93B | 2.20% | 6.64B | 0.20 | | mid dna prokaryote | 17,408,059 | 59.60B | 5.50% | 16.61B | 0.28 | | mid bio p90 biorxiv | 7,262 | 99.1M | 0.24% | 710.3M | 7.17 | | mid bio p90 biostars | 2,036 | 3.1M | 0.01% | 25.4M | 8.19 | | mid bio p90 finepdfs | 51,241 | 1.96B | 4.58% | 13.84B | 7.07 | | mid bio p90 fineweb | 8,205 | 79.7M | 0.19% | 560.2M | 7.03 | | mid bio p90 pubmed | 68,330 | 854.2M | 1.99% | 6.01B | 7.03 | | mid bio rest | 914,391 | 12.16B | 7.00% | 21.14B | 1.74 | | mid bio gemini fim | 56,866 | 262.8M | 0.50% | 1.51B | 5.74 | | mid bio gemini nonfim | 72,343 | 333.4M | 0.50% | 1.51B | 4.53 | | mid english fineweb edu | 96,541,686 | 99.35B | 8.00% | 24.16B | 0.24 | | mid english dclm | 88,340,007 | 111.98B | 8.00% | 24.16B | 0.22 | | mid english finepdfs edu | 17,596,723 | 98.89B | 4.00% | 12.08B | 0.12 | | mid instruct nemotron chat v3 | 562,376 | 2.25B | 3.53% | 10.67B | 4.75 | | mid instruct nemotron if v3 | 249,748 | 632.0M | 0.99% | 3.00B | 4.75 | | mid instruct nemotron if v2 | 1,997,510 | 3.48B | 5.47% | 16.53B | 4.75 | Tokens is the config's size with the Carbon-X tokenizer estimated from a 2,000-document sample per config . Tokens read by Carbon-X is share x 302B. Passes is how many times over the run read the config: below 1, it read only that fraction of it. The full training setup we used for Carbon-X are in the Carbon-X midtraining recipe https://github.com/huggingface/carbon/tree/main/carbon-x/training/midtraining . What is in it DNA. Eukaryote genomic windows from GENERator https://huggingface.co/datasets/GenerTeam/pretrain data eukaryote , up to 100 kbp, in six species groups. mRNA, mRNA with splice and promoter context, and prokaryote genomes GTDB and IMG/PR from OpenGenome2 https://huggingface.co/datasets/arcinstitute/opengenome2 . The raw sequences are also in HuggingFaceBio/carbon-pretraining-corpus https://huggingface.co/datasets/HuggingFaceBio/carbon-pretraining-corpus . Here they are formatted exactly as they were trained on: - every sequence is cut to a multiple of 6 bp the tokenizer reads 6-mers and wrapped in <dna …</dna ; - eukaryote sequences carry metadata tags half the time, chosen at random: species group and gene type, species group only, or gene type only, e.g. <mammalian species <protein coding region <dna ATG…</dna ; - mRNA and prokaryote sequences are uppercased, splice-junction markers @ are removed, and mRNA longer than 100,000 bases is dropped. Biology text. Web pages, papers and forum posts that contain DNA, plus synthetic annotations. The web and paper configs were built in two steps; the code is in mid/pipeline/ https://huggingface.co/datasets/HuggingFaceBio/FineDNA/tree/main/mid/pipeline . 1. Find documents that contain DNA: 1 regex filter.py https://huggingface.co/datasets/HuggingFaceBio/FineDNA/blob/main/mid/pipeline/1 regex filter.py . We ran a regex filter over FineWeb-Edu, FinePDFs, PubMed Central, bioRxiv and Biostars. A document is kept when - it has a run of 15 or more nucleotide letters A, T, C, G, U, N using at least 3 different ones, - that run is not inside curly braces, which in scraped text is usually LaTeX, and - it mentions a molecular-biology keyword dna, gene, primer, pcr, ... . 2. Split by DNA length: 2 p90 split.py https://huggingface.co/datasets/HuggingFaceBio/FineDNA/blob/main/mid/pipeline/2 p90 split.py , which finds DNA with tag dna sequences.py https://huggingface.co/datasets/HuggingFaceBio/FineDNA/blob/main/mid/pipeline/tag dna sequences.py . Documents whose longest DNA sequence is at least 39 bp, the 90th percentile, form mid bio p90 ; everything else forms mid bio rest . Short runs are mostly primers and motifs, while the p90 documents carry real sequences. In our ablations, upweighting them improved DNA benchmarks. | config | what it is | |---|---| | mid bio p90 biorxiv , mid bio p90 biostars , mid bio p90 finepdfs , mid bio p90 fineweb , mid bio p90 pubmed | documents with long DNA sequences, one config per source | | mid bio rest | the other regex hits from the same five sources, whose longest DNA sequence is under 39 bp | | mid bio gemini nonfim | genomic regions described by Gemini next to their sequence | | mid bio gemini fim | the same kind of documents in Qwen's fill-in-the-middle format: part of each document is moved to the end, and the model learns to fill it in | English. FineWeb-Edu https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu 100BT sample , DCLM https://huggingface.co/datasets/HuggingFaceFW/dclm 100BT-shuffled and FinePDFs-Edu https://huggingface.co/datasets/HuggingFaceFW/finepdfs edu 100BT-shuffled , to keep the model's English. Instruction. Nemotron Instruction-Following-Chat v3 https://huggingface.co/datasets/nvidia/Nemotron-SFT-Instruction-Following-Chat-v3 chat and instruction-following splits and v2 https://huggingface.co/datasets/nvidia/Nemotron-SFT-Instruction-Following-Chat-v2 chat , rendered with the Qwen3 chat template, reasoning traces kept. Mid-training starts from an instruct model, and this keeps it able to follow instructions. Biology configs also have a validation split: the documents held out of training to track loss. SFT Supervised fine-tuning conversations for a DNA language model: 203,923 rows and 230.3M tokens, 27.4M of them in assistant turns. sft mix holds exactly the rows the SFT trainer built from the Carbon-X SFT mixture, so training on it reproduces the mixture without any further weighting. Each domain also has its own config, sft <domain , holding the sft mix rows of that domain in the same order, e.g. load dataset "HuggingFaceBio/FineDNA", "sft chat" . Columns - messages : the conversation, as a list of {role, content} turns. - chat template kwargs : {"tokenizer kwargs": {"auto dna tags": <bool }} , passed per example to apply chat template so the DNA tokenizer wraps sequences in <dna ...</dna . It is false only for the sequence rewriting rows, whose answers are plain text. Keep it when selecting training columns. - domain : the kind of task, one of the eight below. Domains | config | rows | tokens | assistant tokens | what the rows teach | |---|---|---|---|---| | sft biology qa | 4,603 | 2.8M | 2.2M | genetics, molecular biology and bioinformatics questions from Biology StackExchange, Biostars, real user chats and Nemotron chat data | | sft chat | 8,553 | 2.1M | 1.5M | multi-turn conversations on molecular-biology concepts, the assistant's identity, honesty about its limits, and pushing back on false claims | | sft dna computation | 44,925 | 19.1M | 7.0M | answers computed exactly from the sequence: counting, GC content, melting temperature, translation, restriction sites, base lookup, coordinates, edits, rewriting | | sft instruction following | 5,904 | 3.7M | 2.1M | answering in the format the user asks for, and following format constraints on biology prompts | | sft sequence analysis | 7,561 | 8.7M | 3.6M | free-text, mostly multi-turn discussion of a pasted sequence: primer checks, annotation, coding potential, troubleshooting | | sft sequence classification | 16,396 | 12.8M | 2.0M | multiple-choice functional genomics: promoter vs enhancer, species of origin, TF motifs, TF binding sites | | sft sequence continuation | 58,727 | 64.8M | 0.65M | predict the next 1–32 nucleotides after a 4–8 kb eukaryotic context | | sft variant effect prediction | 57,254 | 116.2M | 8.2M | decide whether a single-nucleotide variant is benign or pathogenic from its sequence window, with templated reasoning | Tokens are counted with the Carbon-X tokenizer and chat template. Assistant tokens are the ones the SFT loss is computed on, and no conversation is longer than the 9,984-token training context. RL Verifiable prompts for reinforcement learning of a DNA language model: 98,433 rows and 61.1M prompt tokens, covering 80,406 distinct prompts. Every prompt has a gold answer or constraint set that a reward function checks in code. The blend was calibrated on rollouts from Carbon-X-SFT and excludes the prompts that model saw during SFT. It is meant for one epoch. The rows of rl mix are in the order the GRPO trainer's mixture loader produces seed 0 , so training on it reproduces the blend exactly. Some prompts appear more than once on purpose, for groups that learn slowly under RL: basic dna repeats half of each task type 10,153 second copies , bioreason repeats 30% of each subgroup and label 4,614 , and dna skills repeats every prompt 3,260 . Columns - prompt : the conversation to complete, as a list of {role, content} turns 2,983 rows open with a system turn . - answer text : the gold answer that most rewards compare against. - task group : which reward scores the row. Each task reward returns None for rows of other groups. - task type : the sub-task, used by the basic dna and seqrecqa position rewards null elsewhere . - verifier metadata : the constraints an instruction-following reply must meet if rows only . - gold answer json : the exact answer the DNA-skills verifier checks dna skills rows only . - domain : the kind of task. It uses the same labels as the SFT domains. Five of the eight occur here because the blend has no chat, sequence-analysis or sequence-continuation prompts. Task groups | domain | task group | rows | prompt tokens | what the prompts ask | |---|---|---|---|---| | dna computation | basic dna | 30,461 | 2.4M | short-sequence computations with code-checked answers: base counts, melting temperature, restriction fragments, CRISPR cut sites, translation, motif spacing | | dna computation | dna skills | 6,520 | 1.1M | exact operations on longer sequences: locate a base or snippet, convert coordinates, compare, edit, translate, design | | dna computation | seqrecqa position | 6,000 | 3.1M | localisation: the position of a base, 6-base block, variant block, 12-base span, single-base difference or snippet | | variant effect prediction | bioreason | 19,996 | 29.0M | is a coding-sequence or non-SNV variant pathogenic or benign | | variant effect prediction | vepqa | 7,426 | 14.3M | is a single-nucleotide variant pathogenic or benign, judged from its sequence window | | sequence classification | genomeqa | 14,032 | 7.1M | multiple-choice functional genomics: promoter vs enhancer, species of origin, TF motifs and binding sites, splice sites, histone marks | | biology qa | bio mcq | 9,998 | 3.3M | four- and ten-option biology knowledge questions | | instruction following | if | 4,000 | 0.65M | prompts with verifiable formatting constraints | In total, dna computation has 42,981 rows, variant effect prediction 27,422, sequence classification 14,032, biology qa 9,998 and instruction following 4,000. Prompt tokens are counted with the Carbon-X tokenizer and chat template, as the GRPO trainer renders the prompts thinking disabled . Completions are generated during training, so they are not counted. License FineDNA is released under the Apache 2.0 license. The mid-training configs are built from third-party sources GENERator, OpenGenome2, FineWeb-Edu, DCLM, FinePDFs-Edu, PubMed Central, bioRxiv, Biostars and Nemotron , which keep their own licenses and terms. Check the source dataset before use. Citation FineDNA was built for Carbon-X, a joint collaboration between the research teams at Hugging Face, Zhongguancun Academy, and the Telethon Institute of Genetics and Medicine/University of Naples “Federico II”. If you find it useful, please cite it as: @misc{benallal2026finedna, title = {FineDNA}, author = {Ben Allal, Loubna and Tunstall, Lewis and Aubakirova, Dana and Fiusco, Maurizio and Beeching, Edward and Rasul, Kashif and von Werra, Leandro and Li, Qiuyi and di Bernardo, Diego}, year = {2026}, publisher = {Hugging Face}, howpublished = {\url{https://huggingface.co/datasets/HuggingFaceBio/FineDNA}} } - Downloads last month - 72