Show HN: Anime Text Datasets for LLMs Training Muhammad Muqeet released a text-only anime dataset for fine-tuning and evaluating large language models, spanning 35,766 unique anime works, 283,477 characters, 535,642 staff credits, 276,763 episodes, 317,907 question/answer pairs, 235,395 instruction-tuning examples and 206,906 RAG chunks, split 32,189 train / 1,788 validation / 1,789 test. The release ships on Hugging Face as a full dataset, a free 30% LLM-training preview and a CC0 subset, with SHA-256 checksums and a `python -m src.cli validate` command that runs 21 checks covering uniqueness, referential integrity, split hygiene, answer grounding and license coverage. Each record carries provenance fields and one of four redistribution tiers, and the documentation tables are generated from `reports/summary.json` by `scripts/sync_docs.py` rather than edited by hand. Text-only anime dataset built for fine-tuning large language models LLM , instruction tuning, retrieval-augmented generation RAG , question answering QA , entity linking, recommendation systems, and evaluation benchmarks. The headline unit is unique anime works — one work expands into many records: characters, staff, episodes, QA pairs, RAG chunks, and relationship edges. Developer & maintainer: Muhammad Muqeet — contact@animearray.site mailto:contact@animearray.site · muqeet@animearray.site mailto:muqeet@animearray.site · muqeet@anitroves.com mailto:muqeet@anitroves.com Live: dataset site https://dataset.animearray.site/ · Hugging Face Space https://huggingface.co/spaces/muqeet-1138/anime-text-dataset · full release https://huggingface.co/datasets/muqeet-1138/anime-text-dataset · free 30% for LLM training https://huggingface.co/datasets/muqeet-1138/anime-text-dataset-preview · CC0 subset https://huggingface.co/datasets/muqeet-1138/anime-text-dataset-open | Metric | Value | |---|---| | Unique anime works with at least one title | 35,766 | | Characters | 283,477 | | Staff credits | 535,642 | | Episodes | 276,763 | | Alternative titles | 135,425 | | Relationship edges | 39,226 | | Franchise clusters | 23,980 | | Descriptions summaries + synopses | 63,396 | | Question/answer pairs | 317,907 | | Instruction-tuning examples | 235,395 | | RAG chunks | 206,906 | | Split: train / val / test | 32,189 / 1,788 / 1,789 | Coverage of the anime table share of rows carrying a value : | Field | Coverage | |---|---| | Year | 95% | | Episode count | 94% | | Per-episode records | 60% | | Studios | 63% | | Genres | 77% | | Characters | 50% | | Voice actors | 33% | | Staff credits | 65% | | Relationships | 50% | Every export ships with SHA-256 checksums, and python -m src.cli validate runs 21 checks covering uniqueness, referential integrity, enum conformance, value ranges, split hygiene, answer grounding, corpus diversity, license coverage and export schema. The tables above are generated from reports/summary.json by python scripts/sync docs.py ; they are never edited by hand. The site pages are written in docs src/ and rendered into docs/ by the same script, which keeps the {{n:…}} count tokens in the source so later runs can refresh them. A structured, factual, text-only corpus about anime: the works themselves, their casts, the people who made them, the studios, the genres and themes, the episode lists, the narrative relations between works — and, most importantly for NLP, hundreds of thousands of grounded rows of natural language about anime . Each anime work includes: - Identity and titles — canonical, English, Japanese, and up to dozens of alternative titles per work. - Discovery metadata — format, status, episode count, duration, air dates, season bucket, year. - Text content — a generated record summary built from the work's own structured fields, plus the upstream synopsis where one was published, carrying prose source and prose license see LICENSE AUDIT.md . - Scores and popularity — mean score normalised to a single 0–100 scale, popularity, favourites. - Taxonomy — genre and theme labels, source material, age rating. - Cast and crew — characters with roles and voice actors, staff with per-record credit roles. - Episodes — per-episode numbers, titles, air dates and lengths. - Relations — prequel/sequel/side-story/spin-off edges plus franchise clusters. - Training-ready text — QA pairs, instruction examples and RAG chunks, each grounded in fields of the same record. - Provenance and a license tier on every record — source, remote ID, link method, confidence, and one of four redistribution tiers. | Collection | Rows per anime | What it teaches | |---|---|---| | qa.jsonl | 6–14 | Direct question answering over one work: metadata, cast, crew, episodes, relations, franchise, multi-hop, comparison, synopsis | | instructions.jsonl | 0–12 | Task formats: extraction, summarisation, JSON repair, title normalisation, entity resolution, abstention | | rag chunks.jsonl | 4–10 | Retrieval units written as prose, each with source refs and a kind label | | descriptions.jsonl | 2 | record summary generated from structured fields, plus source synopsis with its source | QA pairs are emitted only when every value in the answer exists in the database for that work. Each question type has several phrasings chosen deterministically per work, so the corpus is not one repeated template. Roughly one pair in twenty is an explicit abstention "No episode count is recorded for this work in this dataset" so a model can learn to decline instead of inventing. Instruction examples are derivable from their own input field by construction: the source facts are rendered into the input, then the task asks for a transformation of exactly those facts. This is checked at validation time by re-reading the input and confirming each output value appears in it. RAG chunks are prose, not JSON dumps, and are labelled by family record summary , source synopsis , title aliases , characters , voice cast , staff , episodes , relations . source synopsis chunks are upstream text and carry prose source + prose license . - No images, posters, audio, subtitles, or lyrics. - No full scripts or long copyrighted passages. - No human-written question/answer pairs: the QA and instruction corpora are template-generated from structured fields. They are supervision for extraction and grounding, not a source of new facts. - Not uniformly licensed: retained synopses, biographies, episode summaries and tag definitions are third-party prose, excluded from the project's CC-BY-4.0 grant. See LICENSE AUDIT.md . These are measured, not estimated — the exact figures are regenerated into reports/final report.md https://github.com/muqeet-1138/anime-text-dataset/blob/main/reports/final report.md on every run. - Episode lists are the weakest field. Kitsu is the only reachable source that publishes per-episode records. Shikimori's episode endpoint is unavailable, AniList publishes counts but not episode lists, and MyAnimeList/Jikan is network-blocked from the collection environment. Works without a Kitsu link therefore have an episode count but no episode list . - Depth is uneven by design. Characters, staff and episodes exist only where an upstream source published them. The coverage table above gives the exact rate per field instead of an average that hides the spread. - The Anime News Network encyclopedia contributes zero rows. It is reachable only page-by-page, so a sample was collected and retained in raw storage without being linked into the exports. - The 50,000-title target is not met. The union of the reachable catalogues is exhausted at the current count; MAL would raise it and is blocked. - Franchise clusters use strong relations only SEQUEL , PREQUEL , PARENT STORY , FULL STORY , SIDE STORY , SPIN OFF , ALTERNATIVE SETTING , ALTERNATIVE VERSION , SUMMARY . Sharing a source manga or a cameo does not merge two works into one cluster. | Release | Works | Rows | Who can use it | How to get it | |---|---|---|---|---| | Full release | 35,766 | 2,436,445 | research and education | request access https://huggingface.co/datasets/muqeet-1138/anime-text-dataset — gated, manual approval | | 30% preview | 10,729 | 713,238 | anyone with a Hugging Face account | get free access https://huggingface.co/datasets/muqeet-1138/anime-text-dataset-preview — gated, automatic approval | | CC0 subset | 969 | 8,654 | anyone, no account | direct download https://huggingface.co/datasets/muqeet-1138/anime-text-dataset-open — public | | Code, schemas, docs | — | — | anyone | GitHub https://github.com/muqeet-1138/anime-text-dataset | - No purchase is required for any release. The 30% preview is free and approved instantly. The full release is approved manually for non-commercial research or education. - Gating is access control, not a licence. The full release and the preview contain third-party metadata and retained upstream prose. Approval does not grant redistribution rights from AniList, Kitsu, Shikimori or Anime News Network. - Commercial use needs permission from the relevant upstream source, except for the cc0 only rows 969 works . - Redistribution is permitted only for the public CC0 subset. Do not mirror or republish the full release or the preview. muqeet-1138/anime-text-dataset-preview https://huggingface.co/datasets/muqeet-1138/anime-text-dataset-preview is a free, automatically approved 30% sample of the full release. It exists so people can evaluate the data and train models without buying anything or requesting the full dataset. | Property | Value | |---|---| | Works | 10,729 of 35,766 30.0% | | Rows | 713,238 | | Sampling | stratified by license tier, split , seed 42 | | Child rows | every row of each sampled work is kept | | Tiers | 291 cc0 only / 3,398 permission required / 7,040 unresolved | | Splits | 9,656 train / 536 val / 537 test | | Formats | JSONL, Parquet, CSV no SQLite or dedup log | | Access | free, gated, automatic approval | | Training | allowed for non-commercial research or education | The sample is representative, not cherry-picked: every licence tier and split is preserved, so a model trained on it sees the same obligation mix as the full release. It is not a public-domain release and not a licence from the source owners. Only 291 of its works are cc0 only ; the rest carry upstream obligations. python from datasets import load dataset qa = load dataset "muqeet-1138/anime-text-dataset-preview", "qa", split="train" anime = load dataset "muqeet-1138/anime-text-dataset-preview", "anime", split="train" Nine ready-to-use gateways into the same data: | | Method | What you get | How | |---|---|---|---| | 1 | Clone the repository | code, schemas, docs, pipeline | git clone https://github.com/muqeet-1138/anime-text-dataset.git | | 2 | Download the full data | the complete exports/ tree, 2.2 GB | request access on the gated Hugging Face dataset https://huggingface.co/datasets/muqeet-1138/anime-text-dataset | | 3 | Free 30% preview | 10,729 works, 713,238 rows, instant approval | get free access https://huggingface.co/datasets/muqeet-1138/anime-text-dataset-preview | | 4 | Hugging Face datasets loader | a sharded HF Dataset | load dataset "muqeet-1138/anime-text-dataset-preview", "qa", split="train" | | 5 | Stream JSONL in Python | train-ready rows without loading everything | loop over open "exports/qa.jsonl", encoding="utf-8" | | 6 | SQLite database | joins and referential queries | sqlite3 exports/anime dataset.sqlite or Python stdlib | | 7 | Parquet + DuckDB / pandas | fast analytics | duckdb -c "select ..." or pd.read parquet | | 8 | CSV | spreadsheet-friendly flat view | open exports/anime.csv | | 9 | Training recipes | fine-tuned models | follow docs/TRAINING.md + examples/ scripts | Each gateway is covered with runnable code in docs/COMMANDS.md https://github.com/muqeet-1138/anime-text-dataset/blob/main/docs/COMMANDS.md terminals and docs/TRAINING.md https://github.com/muqeet-1138/anime-text-dataset/blob/main/docs/TRAINING.md model training . Primary format is JSONL UTF-8, one logical record per line, deterministic field order, stable IDs . Derived formats are regenerated from the same canonical data. | Format | File | Notes | |---|---|---| | JSONL | exports/ .jsonl | canonical, 19 tables | | SQLite | exports/anime dataset.sqlite | all tables, split and license tier pre-joined, indexed | | Parquet | exports/ .parquet | 10 tables, for analytics | | CSV | exports/anime.csv | flat anime table | | License subsets | exports/license tier .csv | pre-filtered anime id lists per tier | | Checksums | exports/checksums.json + per-file .sha256 | verify integrity | Every per-anime table carries split and license tier as columns, so a training script can filter without joining anything. JSON arrays are real arrays in JSONL and Parquet, not encoded strings. Hosting note : GitHub holds the code, schemas and documentation 0.7 MB . The 2.2 GB generated dataset is hosted on Hugging Face , not in Git and not as GitHub Release assets. Three releases share one schema: the full mixed-rights release gated, manual approval , a free 30% preview gated, automatic approval, 10,729 works , and the public CC0 subset ungated, 969 works . Every release publishes SHA-256 checksums. Contact: contact@animearray.site mailto:contact@animearray.site . Verified checksum verification any shell : php PowerShell Get-ChildItem exports\ .jsonl | ForEach-Object { $h = Get-FileHash $ .FullName -Algorithm SHA256 .Hash.ToLower if $h -eq Get-Content "$ $ .FullName .sha256" .Split ' ' 0 { "OK $ $ .Name " } else { "BAD $ $ .Name " } } Two ways to browse before training: - Interactive explorer site — docs/explore.html https://github.com/muqeet-1138/anime-text-dataset/blob/main/docs/explore.html renders the live statistical profile: coverage rates, year/type/status distributions, top genres, themes and studios, numeric quartiles for every length/score field, and the full export manifest. - Static profile — reports/data profile.json https://github.com/muqeet-1138/anime-text-dataset/blob/main/reports/data profile.json is the machine-readable version, regenerated any time with python scripts/data profile.py . - EDA notebook — examples/notebooks/EDA.ipynb https://github.com/muqeet-1138/anime-text-dataset/blob/main/examples/notebooks/EDA.ipynb walks the same numbers in Jupyter with charts. - Publish to Hugging Face Hub — python scripts/upload hf.py --repo muqeet-1138/anime-text-dataset --gated manual uploads the full release to a gated dataset repo. python scripts/build preview subset.py then python scripts/upload hf.py --repo muqeet-1138/anime-text-dataset-preview --source exports preview --gated auto publishes the free 30% preview. python scripts/build open subset.py then python scripts/upload hf.py --repo muqeet-1138/anime-text-dataset-open --source exports open publishes the public CC0 subset. Run with --dry-run first; token via HF TOKEN or the login saved by hf auth login . | Source | Contributed | Terms | Tier assigned | |---|---|---|---| | AniList | discovery, scores, characters, staff, relations | public GraphQL API; terms allow API use, not redistribution of content | permission required | | Kitsu | discovery, episode lists | public JSON:API; terms grant no redistribution of API data | permission required | | Shikimori | discovery, cast, crew, relations | no license or terms for API data could be located | unresolved | | Wikidata | supplementary titles | CC0-1.0 | cc0 only | | ANN encyclopedia | 789 sample pages, retained in raw storage | copyrighted | not linked into exports | | MyAnimeList / Jikan | none | — | unreachable, excluded | Each anime row is assigned exactly one tier: the strictest implied by the sources that contributed to it unresolved permission required cc0 only . The tier names describe the obligation you take on, not the copyright status of the underlying anime, which stays with its rights holders. - cc0 only — no obligation; attribution is courtesy. - permission required — AniList and Kitsu terms do not grant redistribution; obtain written permission before publishing derived records. - unresolved — no permission basis identified; do not redistribute. Machine-readable: exports/license tiers.jsonl per anime , exports/license manifest.json definitions, counts, gaps, disclaimer , exports/sources.jsonl per source , exports/provenance.jsonl per record . Human-readable: docs/LICENSE NOTES.md https://github.com/muqeet-1138/anime-text-dataset/blob/main/docs/LICENSE NOTES.md , NOTICE https://github.com/muqeet-1138/anime-text-dataset/blob/main/NOTICE , ACKNOWLEDGMENTS.md https://github.com/muqeet-1138/anime-text-dataset/blob/main/ACKNOWLEDGMENTS.md . The repository code, pipeline and project-generated text are released under CC-BY-4.0 https://github.com/muqeet-1138/anime-text-dataset/blob/main/LICENSE . That license covers the code and the text this project generates; it cannot and does not relicense upstream source content. Upstream prose is retained, attributed, and excluded from that grant. Plot synopses, character biographies, episode summaries and tag definitions are third-party expression. Every prose row carries prose source and prose license ; per-source owners and terms are in exports/attribution.json https://github.com/muqeet-1138/anime-text-dataset/blob/main/exports/attribution.json and the prose attribution block of exports/license manifest.json . Attribution is not permission. Every artifact and its rights status is listed in LICENSE AUDIT.md https://github.com/muqeet-1138/anime-text-dataset/blob/main/LICENSE AUDIT.md . | Object | Export file | Content | |---|---|---| | Anime | anime.jsonl | canonical record per unique work, with nested titles/genres/themes/studios/sources and attributed synopsis | | Characters | characters.jsonl | name, native name, localized name, role, voice actors, attributed biography | | Staff | staff.jsonl | person, credit roles, source references | | Studios | studios.jsonl | studio names + identifiers | | Genres / themes | genres.jsonl , themes.jsonl | controlled vocabulary; theme definitions attributed | | Episodes | episodes.jsonl | number, title, air date, runtime, attributed summary | | Relationships | relationships.jsonl | typed edges between works | | Alternative titles | alt titles.jsonl | language, kind, and the sources that supplied it | | Descriptions | descriptions.jsonl | record summary plus attributed source synopsis | | QA pairs | qa.jsonl | question, answer, type, template, grounded fields, prose source | | Instructions | instructions.jsonl | instruction, input, output, type, purpose | | RAG chunks | rag chunks.jsonl | prose chunk, kind , source refs , prose source | | Entity links | entity links.jsonl | internal → external IDs, method, confidence | | Franchises | franchises.jsonl | anime → franchise cluster id | | Provenance | provenance.jsonl + sources.jsonl + attribution.json | per-record, per-source, and per-prose-owner registry | | License tiers | license tiers.jsonl | per-anime redistribution tier and reason | | Field | Type | Meaning | |---|---|---| | anime id | int | stable internal ID | | type | enum | TV , MOVIE , OVA , ONA , SPECIAL , TV SPECIAL , SHORT , TV SHORT , ONA SHORT , MUSIC , PV , CM | | status | enum | FINISHED , RELEASING , NOT YET RELEASED , CANCELLED , HIATUS | | episodes , duration min | int | episode count and per-episode minutes | | aired from , aired to | date | air window YYYY-MM-DD | | season , year | enum/int | WINTER / SPRING / SUMMER / FALL and start year | | title canonical , title english , title japanese | string | titles | | mean score , popularity , favourites | number | score on a single normalised 0–100 scale | | source material , age rating | enum | e.g. MANGA , PG-13 | | quality tier | string | A / B / C | | confidence | string | high/medium/low | | synopsis , synopsis source | string | upstream synopsis and its registry id; prose source / prose license are added on export | | alt titles , genres , themes , studios , sources , franchise id , license tier , split | list/string | enriched record | Full field-level reference: docs/DATA DICTIONARY.md https://github.com/muqeet-1138/anime-text-dataset/blob/main/docs/DATA DICTIONARY.md and docs/SCHEMAS.md https://github.com/muqeet-1138/anime-text-dataset/blob/main/docs/SCHEMAS.md . - 90 / 5 / 5 train / val / test , random seed 42 . - Split per anime , never per record — every character, QA pair, chunk, and relationship of a work lands in the same bucket, so no leakage across train/test. - Every exported record carries its split so you can filter without re-walking the tables. python import json train qa = q for q in map json.loads, open "exports/qa.jsonl", encoding="utf-8" if q "split" == "train" val qa = q for q in map json.loads, open "exports/qa.jsonl", encoding="utf-8" if q "split" == "val" test qa = q for q in map json.loads, open "exports/qa.jsonl", encoding="utf-8" if q "split" == "test" Works identically from Windows CMD , PowerShell , and Git Bash / WSL . Replace python with python3 where your system uses it. git clone