{"slug": "finedna-open-data-to-train-llms-on-dna", "title": "FineDNA: Open data to train LLMs on DNA", "summary": "Hugging Face released FineDNA, the full training dataset behind Carbon-X, the model that adapted Qwen3-14B to read DNA across all three of its training stages. The mid-training stage comprises 23 configs totaling 514.30B tokens, of which the run read 302B, starting from Qwen/Qwen3-14B; the SFT stage contains 203,923 conversations spanning 8 domains at 230.3M tokens. The release makes the mid-training, SFT and RL data public alongside a training recipe on GitHub, giving researchers an open path to reproduce a DNA-capable LLM.", "body_md": "[Collection The mid-training, SFT and RL data of Carbon-X, with the trained on it. Recipe: https://github.com/huggingface/carbon/tree/main/carbon-x • 4 items • Updated  •  3](https://huggingface.co/collections/HuggingFaceBio/finedna)\n\n| text  stringlengths 1.36k 126k | id  stringclasses 0 \t\t\t\tvalues | source  stringclasses 1 \t\t\t\tvalue | metadata  stringclasses 0 \t\t\t\tvalues | \n|---|---|---|---|\n|  <\\|fim_prefix\\|>This 1115 bp genomic region from Arabidopsis thaliana encompasses the long non-coding RNA (lncRNA) gene AT4G07665 and its surrounding regulatory elements. The locus features a putative TATA box promoter upstream, a centrally located CpG island embedded within the lncRNA body, and multiple microRNA target ... | null |  gemini_fim | null | \n|  <\\|fim_prefix\\|>This genomic region encompasses the Caenorhabditis elegans C33D3.5 protein-coding gene along with its extensive 5' and 3' flanking sequences. The locus is structurally complex, characterized by a dense cluster of upstream alternative splice sites, multiple internal promoters (TATA boxes), and a prominent ... | null |  gemini_fim | null | \n|  <\\|fim_prefix\\|>This 691 bp genomic region from Escherichia coli contains the yjiX protein-coding gene, which is located on the negative strand. The sequence is notable for harboring two distinct EcoRI restriction sites, making it amenable to molecular cloning techniques. Additionally, it features two regions with elevat... | null |  gemini_fim | null | \n|  <\\|fim_prefix\\|>This genomic region spans 24,802 bp on the positive strand of the Danio rerio genome and contains the pcsk1nl (proprotein convertase subtilisin/kexin type 1 inhibitor like) gene. The gene body extends from local positions 2055 to 22152.   pcsk1nl functions as a serine-type endopeptidase inhibitor, playing... | null |  gemini_fim | null | \n|  <\\|fim_prefix\\|>The genomic region corresponds to the Arabidopsis thaliana gene AT3G12880. This gene encodes a protein involved in fundamental cellular processes, likely regulating metabolic or signaling pathways critical for plant growth and development. The region contains multiple regulatory features, including promot... | null |  gemini_fim | null | \n|  <\\|fim_prefix\\|>This 15.8 kb genomic region on human chromosome 13 encodes a complex, convergently transcribed pair of long non-coding RNAs (lncRNAs): ENSG00000289302 (on the plus strand) and LINC01058 (on the minus strand). The region is characterized by extreme regulatory density, featuring multiple broad enhancer doma... | null |  gemini_fim | null | \n|  <\\|fim_prefix\\|>This genomic region in Danio rerio encodes a protein-coding gene (ENSDARG00000017947), which is a zebrafish ortholog of human PTK6. The gene product is a non-membrane spanning protein tyrosine kinase involved in signal transduction, cell differentiation, and the regulation of cell surface receptor pathway... | null |  gemini_fim | null | \n|  \"<\\|fim_prefix\\|>The Cmtr2 gene encodes a nucleoside-2'-O-methyltransferase involved in the 7-methylgu(...TRUNCATED) | null |  gemini_fim | null | \n|  \"<\\|fim_prefix\\|>This genomic region on chromosome 3 contains the RNA binding motif single stranded in(...TRUNCATED) | null |  gemini_fim | null | \n|  \"<\\|fim_prefix\\|>This 8889 bp genomic region in Mus musculus contains the long non-coding RNA ENSMUSG0(...TRUNCATED) | null |  gemini_fim | null | \n\n[Expand in Data Studio](https://huggingface.co/datasets/HuggingFaceBio/FineDNA/viewer/mid_bio_gemini_fim/train)\n\n# \n\t\tFineDNA\n\t\n\nFineDNA is the training data of [Carbon-X](https://huggingface.co/HuggingFaceBio/Carbon-X), the model that taught\nQwen3-14B to read DNA, for all three of its training stages. Each stage is a group of configs:\n\n| stage | configs | size | starts from | \n|---|---|---|---|\n| [Mid-training](#mid-training) | `mid_*` , one per source (23) | 514.30B tokens, of which the run read 302B | [`Qwen/Qwen3-14B`](https://huggingface.co/Qwen/Qwen3-14B) | \n| [SFT](#sft) | `sft_mix` , and`sft_*` for each of its 8 domains | 203,923 conversations, 230.3M tokens | [`HuggingFaceBio/Carbon-X-Base`](https://huggingface.co/HuggingFaceBio/Carbon-X-Base) | \n| [RL](#rl) | `rl_mix` | 98,433 prompts, 61.1M tokens | [`HuggingFaceBio/Carbon-X-SFT`](https://huggingface.co/HuggingFaceBio/Carbon-X-SFT) | \n\n`sft_mix` and `rl_mix` are the rows the SFT and GRPO trainers used, in the order they used them. Mid-training has no\nsingle mix: Megatron samples the `mid_*` configs with the weights in [the mix](#the-mix) below.\n\n``` python\nfrom datasets import load_dataset\n\n# one mid-training source; the mid-training configs total about 950 GB, so stream them\nds = load_dataset(\"HuggingFaceBio/FineDNA\", \"mid_dna_mrna\", split=\"train\", streaming=True)\n# the SFT mixture, or one of its domains\nds = load_dataset(\"HuggingFaceBio/FineDNA\", \"sft_mix\", split=\"train\")\nds = load_dataset(\"HuggingFaceBio/FineDNA\", \"sft_chat\", split=\"train\")\n# the RL blend\nds = load_dataset(\"HuggingFaceBio/FineDNA\", \"rl_mix\", split=\"train\")\n```\n\nThe training recipes for every stage are in [`huggingface/carbon`](https://github.com/huggingface/carbon/tree/main/carbon-x).\n\n## \n\t\tMid-training\n\t\n\nThe `mid_*` configs hold every source of the 302B-token mid-training run, one config per source: genomes and\ntranscripts, biology text that contains DNA, English web and PDF text, and instruction data. Their files are under\n`mid/<source>/`.\n\nEvery config holds all the data Carbon-X sampled from for that source. That is not the whole upstream dataset: the\nDNA, for example, is GENERator and OpenGenome2 after the filtering described below. The **share of training** column\nbelow is the weight each config gets, and those weights are the recipe: they decide how much of each config the run\nread, so some configs were read several times and some only in part. Tokenize the `text` column with the Carbon-X\ntokenizer, train with these weights, and you get the same data diet as Carbon-X.\n\n### \n\t\tColumns\n\t\n\nEvery mid-training config has the same columns:\n\n| column |  | \n|---|---|\n| `text` | the document, as it was tokenized | \n| `id` | the source's document id, where it has one | \n| `source` | the config name without `mid_` , or for biology text the original source (`pubmed` ,`biorxiv` , …) | \n| `metadata` | the source's other fields, as a JSON string | \n\n### \n\t\tThe mix\n\t\n\n|  | unique tokens | share of training in Carbon-X | tokens seen in Carbon-X | \n|---|---|---|---|\n| DNA | 181.96B | 55% | 166.1B | \n| Biology text | 15.75B | 15% | 45.3B | \n| English | 310.22B | 20% | 60.4B | \n| Instruction | 6.36B | 10% | 30.2B | \n| Total | 514.30B | 100% | 302.0B | \n\nCarbon-X trained on 302B of these 514.30B tokens.\n\n| config | documents | tokens | share of training (weight) | tokens read by Carbon-X | passes | \n|---|---|---|---|---|---|\n| `mid_dna_eukaryote_fungi` | 6,607,490 | 1.88B | 1.12% | 3.37B | 1.79 | \n| `mid_dna_eukaryote_protozoa` | 1,257,499 | 366.8M | 0.19% | 581.3M | 1.58 | \n| `mid_dna_eukaryote_plant` | 9,429,321 | 5.55B | 3.08% | 9.30B | 1.68 | \n| `mid_dna_eukaryote_invertebrate` | 8,716,932 | 13.71B | 7.58% | 22.90B | 1.67 | \n| `mid_dna_eukaryote_vertebrate_mammalian` | 7,169,696 | 18.96B | 10.32% | 31.16B | 1.64 | \n| `mid_dna_eukaryote_vertebrate_other` | 12,420,543 | 29.56B | 16.21% | 48.95B | 1.66 | \n| `mid_dna_mrna` | 52,702,454 | 19.41B | 8.80% | 26.58B | 1.37 | \n| `mid_dna_mrna_splice` | 56,877,102 | 32.93B | 2.20% | 6.64B | 0.20 | \n| `mid_dna_prokaryote` | 17,408,059 | 59.60B | 5.50% | 16.61B | 0.28 | \n| `mid_bio_p90_biorxiv` | 7,262 | 99.1M | 0.24% | 710.3M | 7.17 | \n| `mid_bio_p90_biostars` | 2,036 | 3.1M | 0.01% | 25.4M | 8.19 | \n| `mid_bio_p90_finepdfs` | 51,241 | 1.96B | 4.58% | 13.84B | 7.07 | \n| `mid_bio_p90_fineweb` | 8,205 | 79.7M | 0.19% | 560.2M | 7.03 | \n| `mid_bio_p90_pubmed` | 68,330 | 854.2M | 1.99% | 6.01B | 7.03 | \n| `mid_bio_rest` | 914,391 | 12.16B | 7.00% | 21.14B | 1.74 | \n| `mid_bio_gemini_fim` | 56,866 | 262.8M | 0.50% | 1.51B | 5.74 | \n| `mid_bio_gemini_nonfim` | 72,343 | 333.4M | 0.50% | 1.51B | 4.53 | \n| `mid_english_fineweb_edu` | 96,541,686 | 99.35B | 8.00% | 24.16B | 0.24 | \n| `mid_english_dclm` | 88,340,007 | 111.98B | 8.00% | 24.16B | 0.22 | \n| `mid_english_finepdfs_edu` | 17,596,723 | 98.89B | 4.00% | 12.08B | 0.12 | \n| `mid_instruct_nemotron_chat_v3` | 562,376 | 2.25B | 3.53% | 10.67B | 4.75 | \n| `mid_instruct_nemotron_if_v3` | 249,748 | 632.0M | 0.99% | 3.00B | 4.75 | \n| `mid_instruct_nemotron_if_v2` | 1,997,510 | 3.48B | 5.47% | 16.53B | 4.75 | \n\n*Tokens* is the config's size with the Carbon-X tokenizer (estimated from a 2,000-document sample per config).\n*Tokens read by Carbon-X* is share x 302B. *Passes* is how many times over the run read the config: below 1, it read\nonly that fraction of it.\n\nThe full training setup we used for Carbon-X are in the\n[Carbon-X midtraining recipe](https://github.com/huggingface/carbon/tree/main/carbon-x/training/midtraining).\n\n### \n\t\tWhat is in it\n\t\n\n**DNA.** Eukaryote genomic windows from [GENERator](https://huggingface.co/datasets/GenerTeam/pretrain_data_eukaryote),\nup to 100 kbp, in six species groups. mRNA, mRNA with splice and promoter context, and prokaryote genomes (GTDB and\nIMG/PR) from [OpenGenome2](https://huggingface.co/datasets/arcinstitute/opengenome2). The raw sequences are also in\n[`HuggingFaceBio/carbon-pretraining-corpus`](https://huggingface.co/datasets/HuggingFaceBio/carbon-pretraining-corpus).\nHere they are formatted exactly as they were trained on:\n\n- every sequence is cut to a multiple of 6 bp (the tokenizer reads 6-mers) and wrapped in `<dna>…</dna>` ;\n- eukaryote sequences carry metadata tags half the time, chosen at random: species group and gene type, species group\nonly, or gene type only, e.g. `<mammalian_species><protein_coding_region><dna>ATG…</dna>` ;\n- mRNA and prokaryote sequences are uppercased, splice-junction markers (`@` ) are removed, and mRNA longer than\n100,000 bases is dropped.\n\n**Biology text.** Web pages, papers and forum posts that contain DNA, plus synthetic annotations. The web and paper\nconfigs were built in two steps; the code is in\n[`mid/pipeline/`](https://huggingface.co/datasets/HuggingFaceBio/FineDNA/tree/main/mid/pipeline).\n\n1. **Find documents that contain DNA:**[`1_regex_filter.py`](https://huggingface.co/datasets/HuggingFaceBio/FineDNA/blob/main/mid/pipeline/1_regex_filter.py) .\nWe ran a regex filter over FineWeb-Edu, FinePDFs, PubMed Central, bioRxiv and Biostars. A document is kept when\n  - it has a run of 15 or more nucleotide letters (A, T, C, G, U, N) using at least 3 different ones,\n  - that run is not inside curly braces, which in scraped text is usually LaTeX, and\n  - it mentions a molecular-biology keyword (dna, gene, primer, pcr, ...).\n2. **Split by DNA length:**[`2_p90_split.py`](https://huggingface.co/datasets/HuggingFaceBio/FineDNA/blob/main/mid/pipeline/2_p90_split.py) ,\nwhich finds DNA with[`tag_dna_sequences.py`](https://huggingface.co/datasets/HuggingFaceBio/FineDNA/blob/main/mid/pipeline/tag_dna_sequences.py) .\nDocuments whose longest DNA sequence is at least 39 bp, the 90th percentile, form`mid_bio_p90_*` ; everything else\nforms`mid_bio_rest` . Short runs are mostly primers and motifs, while the p90 documents carry real sequences. In our\nablations, upweighting them improved DNA benchmarks.\n\n| config | what it is | \n|---|---|\n| `mid_bio_p90_biorxiv` ,`mid_bio_p90_biostars` ,`mid_bio_p90_finepdfs` ,`mid_bio_p90_fineweb` ,`mid_bio_p90_pubmed` | documents with long DNA sequences, one config per source | \n| `mid_bio_rest` | the other regex hits from the same five sources, whose longest DNA sequence is under 39 bp | \n| `mid_bio_gemini_nonfim` | genomic regions described by Gemini next to their sequence | \n| `mid_bio_gemini_fim` | the same kind of documents in Qwen's fill-in-the-middle format: part of each document is moved to the end, and the model learns to fill it in | \n\n**English.** [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) (100BT sample),\n[DCLM](https://huggingface.co/datasets/HuggingFaceFW/dclm_100BT-shuffled) and\n[FinePDFs-Edu](https://huggingface.co/datasets/HuggingFaceFW/finepdfs_edu_100BT-shuffled), to keep the model's English.\n\n**Instruction.** [Nemotron Instruction-Following-Chat v3](https://huggingface.co/datasets/nvidia/Nemotron-SFT-Instruction-Following-Chat-v3)\n(chat and instruction-following splits) and\n[v2](https://huggingface.co/datasets/nvidia/Nemotron-SFT-Instruction-Following-Chat-v2) (chat), rendered with the\nQwen3 chat template, reasoning traces kept. Mid-training starts from an instruct model, and this keeps it able to\nfollow instructions.\n\nBiology configs also have a `validation` split: the documents held out of training to track loss.\n\n## \n\t\tSFT\n\t\n\nSupervised fine-tuning conversations for a DNA language model: 203,923 rows and 230.3M tokens, 27.4M of them in\nassistant turns. `sft_mix` holds exactly the rows the SFT trainer built from the Carbon-X SFT mixture, so training on\nit reproduces the mixture without any further weighting.\n\nEach domain also has its own config, `sft_<domain>`, holding the `sft_mix` rows of that domain in the same order, e.g.\n`load_dataset(\"HuggingFaceBio/FineDNA\", \"sft_chat\")`.\n\n### \n\t\tColumns\n\t\n\n- `messages` : the conversation, as a list of`{role, content}` turns.\n- `chat_template_kwargs` :`{\"tokenizer_kwargs\": {\"auto_dna_tags\": <bool>}}` , passed per example to`apply_chat_template` so the DNA tokenizer wraps sequences in`<dna>...</dna>` . It is`false` only for the sequence\nrewriting rows, whose answers are plain text. Keep it when selecting training columns.\n- `domain` : the kind of task, one of the eight below.\n\n### \n\t\tDomains\n\t\n\n| config | rows | tokens | assistant tokens | what the rows teach | \n|---|---|---|---|---|\n| `sft_biology_qa` | 4,603 | 2.8M | 2.2M | genetics, molecular biology and bioinformatics questions from Biology StackExchange, Biostars, real user chats and Nemotron chat data | \n| `sft_chat` | 8,553 | 2.1M | 1.5M | multi-turn conversations on molecular-biology concepts, the assistant's identity, honesty about its limits, and pushing back on false claims | \n| `sft_dna_computation` | 44,925 | 19.1M | 7.0M | answers computed exactly from the sequence: counting, GC content, melting temperature, translation, restriction sites, base lookup, coordinates, edits, rewriting | \n| `sft_instruction_following` | 5,904 | 3.7M | 2.1M | answering in the format the user asks for, and following format constraints on biology prompts | \n| `sft_sequence_analysis` | 7,561 | 8.7M | 3.6M | free-text, mostly multi-turn discussion of a pasted sequence: primer checks, annotation, coding potential, troubleshooting | \n| `sft_sequence_classification` | 16,396 | 12.8M | 2.0M | multiple-choice functional genomics: promoter vs enhancer, species of origin, TF motifs, TF binding sites | \n| `sft_sequence_continuation` | 58,727 | 64.8M | 0.65M | predict the next 1–32 nucleotides after a 4–8 kb eukaryotic context | \n| `sft_variant_effect_prediction` | 57,254 | 116.2M | 8.2M | decide whether a single-nucleotide variant is benign or pathogenic from its sequence window, with templated reasoning | \n\nTokens are counted with the Carbon-X tokenizer and chat template. Assistant tokens are the ones the SFT loss is computed on, and no conversation is longer than the 9,984-token training context.\n\n## \n\t\tRL\n\t\n\nVerifiable prompts for reinforcement learning of a DNA language model: 98,433 rows and 61.1M prompt tokens, covering\n80,406 distinct prompts. Every prompt has a gold answer or constraint set that a reward function checks in code. The\nblend was calibrated on rollouts from Carbon-X-SFT and excludes the prompts that model saw during SFT. It is meant for\none epoch. The rows of `rl_mix` are in the order the GRPO trainer's mixture loader produces (seed 0), so training on it\nreproduces the blend exactly.\n\nSome prompts appear more than once on purpose, for groups that learn slowly under RL: `basic_dna` repeats half of each\ntask type (10,153 second copies), `bioreason` repeats 30% of each subgroup and label (4,614), and `dna_skills` repeats\nevery prompt (3,260).\n\n### \n\t\tColumns\n\t\n\n- `prompt` : the conversation to complete, as a list of`{role, content}` turns (2,983 rows open with a system turn).\n- `answer_text` : the gold answer that most rewards compare against.\n- `task_group` : which reward scores the row. Each task reward returns`None` for rows of other groups.\n- `task_type` : the sub-task, used by the`basic_dna` and`seqrecqa_position` rewards (null elsewhere).\n- `verifier_metadata` : the constraints an instruction-following reply must meet (`if` rows only).\n- `gold_answer_json` : the exact answer the DNA-skills verifier checks (`dna_skills` rows only).\n- `domain` : the kind of task. It uses the same labels as the SFT domains. Five of the eight occur here because the\nblend has no chat, sequence-analysis or sequence-continuation prompts.\n\n### \n\t\tTask groups\n\t\n\n| domain | task group | rows | prompt tokens | what the prompts ask | \n|---|---|---|---|---|\n| `dna_computation` | `basic_dna` | 30,461 | 2.4M | short-sequence computations with code-checked answers: base counts, melting temperature, restriction fragments, CRISPR cut sites, translation, motif spacing | \n| `dna_computation` | `dna_skills` | 6,520 | 1.1M | exact operations on longer sequences: locate a base or snippet, convert coordinates, compare, edit, translate, design | \n| `dna_computation` | `seqrecqa_position` | 6,000 | 3.1M | localisation: the position of a base, 6-base block, variant block, 12-base span, single-base difference or snippet | \n| `variant_effect_prediction` | `bioreason` | 19,996 | 29.0M | is a coding-sequence or non-SNV variant pathogenic or benign | \n| `variant_effect_prediction` | `vepqa` | 7,426 | 14.3M | is a single-nucleotide variant pathogenic or benign, judged from its sequence window | \n| `sequence_classification` | `genomeqa` | 14,032 | 7.1M | multiple-choice functional genomics: promoter vs enhancer, species of origin, TF motifs and binding sites, splice sites, histone marks | \n| `biology_qa` | `bio_mcq` | 9,998 | 3.3M | four- and ten-option biology knowledge questions | \n| `instruction_following` | `if` | 4,000 | 0.65M | prompts with verifiable formatting constraints | \n\nIn total, `dna_computation` has 42,981 rows, `variant_effect_prediction` 27,422, `sequence_classification` 14,032,\n`biology_qa` 9,998 and `instruction_following` 4,000.\n\nPrompt tokens are counted with the Carbon-X tokenizer and chat template, as the GRPO trainer renders the prompts (thinking disabled). Completions are generated during training, so they are not counted.\n\n## \n\t\tLicense\n\t\n\nFineDNA is released under the Apache 2.0 license. The mid-training configs are built from third-party sources (GENERator, OpenGenome2, FineWeb-Edu, DCLM, FinePDFs-Edu, PubMed Central, bioRxiv, Biostars and Nemotron), which keep their own licenses and terms. Check the source dataset before use.\n\n## \n\t\tCitation\n\t\n\nFineDNA was built for Carbon-X, a joint collaboration between the research teams at Hugging Face, Zhongguancun Academy, and the Telethon Institute of Genetics and Medicine/University of Naples “Federico II”. If you find it useful, please cite it as:\n\n```\n@misc{benallal2026finedna,\n  title        = {FineDNA},\n  author       = {Ben Allal, Loubna and Tunstall, Lewis and Aubakirova, Dana and Fiusco, Maurizio and Beeching, Edward and Rasul, Kashif and von Werra, Leandro and Li, Qiuyi and di Bernardo, Diego},\n  year         = {2026},\n  publisher    = {Hugging Face},\n  howpublished = {\\url{https://huggingface.co/datasets/HuggingFaceBio/FineDNA}}\n}\n```\n\n- Downloads last month\n- 72", "url": "https://wpnews.pro/news/finedna-open-data-to-train-llms-on-dna", "canonical_source": "https://huggingface.co/datasets/HuggingFaceBio/FineDNA", "published_at": "2026-11-01 07:13:15+00:00", "updated_at": "2026-11-01 07:49:50.387934+00:00", "lang": "en", "topics": ["ai-research", "large-language-models", "machine-learning", "artificial-intelligence"], "entities": ["Hugging Face", "FineDNA", "Carbon-X", "Qwen3-14B", "HuggingFaceBio", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/finedna-open-data-to-train-llms-on-dna", "markdown": "https://wpnews.pro/news/finedna-open-data-to-train-llms-on-dna.md", "text": "https://wpnews.pro/news/finedna-open-data-to-train-llms-on-dna.txt", "jsonld": "https://wpnews.pro/news/finedna-open-data-to-train-llms-on-dna.jsonld"}}