{"slug": "a-structurally-chunked-pre-embedded-sqlite-corpus-of-the-eu-ai-act", "title": "A structurally chunked, pre-embedded SQLite corpus of the EU AI Act", "summary": "Researchers released a structurally chunked, pre-embedded SQLite corpus of the EU AI Act, containing 933 chunks with BGE-M3 dense embeddings for retrieval-augmented generation. The corpus is designed for legal AI research and engineering, not as authoritative legal interpretation.", "body_md": "Paper • 2603.09435 • Published\n\nThe dataset viewer is not available because its heuristics could not detect any [supported data files](https://huggingface.co/docs/hub/datasets-adding#file-formats). You can try [uploading](https://huggingface.co/docs/hub/datasets-adding) some data\nfiles, or [configuring](https://huggingface.co/docs/hub/datasets-data-files-configuration) the data\nfiles location manually.\n\n# EU AI Act — structural RAG corpus\n\nA single-file, pre-embedded SQLite corpus of the **EU AI Act** — Regulation\n(EU) 2024/1689 — chunked on **legal structure only**: one chunk per article\nparagraph, per recital, per annex point, per Article 3 definition. Chapter,\nsection, and article metadata live in columns, not smeared into the text.\n**The artifact is the corpus, not a pipeline** — a downloadable database\nfile you query locally, not a hosted service and not an authoritative\nlegal interpretation.\n\n## Release\n\n| Release | v1.0.0 (prepared 2026-07-16) |\n| Primary artifact | `aiact_openrag.db` (SQLite, embeddings included) |\n| Schema version | 2 (recorded in the `meta` table) |\n| Chunks | 933 |\n| Source text | EUR-Lex consolidated version, CELEX 02024R1689-20240712 |\n| Source commit | `f94a363` |\n| Label audit commit | `03e7aff` (\n|\n\nLicensing is mixed — no single licence applies indiscriminately to every file.The EU legal text is reused under Commission Decision 2011/833/EU and isnot relicensedby this dataset; the original corpus structure, metadata, embeddings and documentation are CC BY 4.0; adapted benchmark data is CC BY 4.0; software associated with the source repository is Apache-2.0. See[LICENSE]for the path-level division ([LICENSES.md]is the same text under the repository's canonical filename) and[NOTICE]for the attribution notice.\n\nThis is not legal advice.This corpus is a retrieval artifact for research and engineering. The chunking, tier labels, and metadata are operator-derived with documented rules — they arenot authoritative interpretations. In particular,`risk_tier`\n\nis a labelling convention derived from the Act's structure (see`docs/derivation.md`\n\n), not a legal determination. Consult the Official Journal text and qualified counsel for compliance decisions.\n\n## Contents\n\n**933 chunks**: 180 recitals, 522 article-paragraph chunks, 68 Article 3 definitions, 163 annex points** BGE-M3 dense embeddings**(1024-dim float32, L2-normalized) for every chunk`risk_tier`\n\n/`obligation_on`\n\nlabels derived**only where the text is unambiguous**— every rule cites its provision in; NULL rather than guess`docs/derivation.md`\n\n- ELI deep links to the exact provision on EUR-Lex\n- Staged application dates per Article 113\n\nSource: EUR-Lex consolidated version **02024R1689-20240712** (English). The\nEnglish consolidated text is character-identical to the OJ text\n(`32024R1689`\n\n); article numbering is identical in both, and both CELEX ids\nare recorded. The build verifies, mechanically and fatally: every article\n1–113 appears exactly once; every text node of the normative text is consumed\nby exactly one chunk and survives verbatim into the output (zero silent\ndrops); no empty chunks; Annex III point counts match an independent\nextraction.\n\n## Schema — table `EmbeddingContent`\n\n| column | type | notes |\n|---|---|---|\n`chunk_id` |\nINTEGER PRIMARY KEY | |\n`celex_id` |\nTEXT | `02024R1689-20240712` |\n`content_type` |\nTEXT | `recital` | `article` | `annex` | `definition` |\n`chapter` |\nTEXT | e.g. `CHAPTER III — HIGH-RISK AI SYSTEMS` |\n`section` |\nTEXT | e.g. `SECTION 2 — Requirements for high-risk AI systems` |\n`article_num` |\nTEXT | `6` , `6(2)` , `3(34)` , `recital 44` , `Annex III.4(a)` |\n`heading` |\nTEXT | e.g. `Article 6 — Classification rules for high-risk AI systems` |\n`heading_level` |\nINTEGER | 1 recital, 3 article/annex intro, 4 paragraph/point, 5 sub-point |\n`chunk_text` |\nTEXT | the provision text, nothing else |\n`risk_tier` |\nTEXT | direct textual classification only: `prohibited` (Art 5(1)) | `high` (Arts 6(1)/6(2) + Annex III, rebuttable per Art 6(3)) | NULL. `limited` /`minimal` are reserved, never assigned — operator-derived, see disclaimer |\n`regime_bucket` |\nTEXT | regime association: `prohibited-practices` | `high-risk` | `transparency` | `gpai` | `voluntary-codes` | NULL |\n`obligation_on` |\nTEXT | chunk-level operative subject: `provider` | `deployer` | `importer` | `distributor` | NULL |\n`eli_url` |\nTEXT | deep link to the exact provision |\n`date_in_force` |\nTEXT | EARLIEST date the provision applies, per Art 113 staging (Annex I: NULL) — an application date, not the Regulation's entry-into-force date (1 August 2024) |\n`embedding` |\nBLOB | 1024 × float32, raw bytes, L2-normalized (BGE-M3) |\n`continued` |\nINTEGER | 0 normally; 1,2,… for parts of a paragraph split at ~1000 tokens |\n\nA `meta`\n\ntable records the source URL, both CELEX ids, embedding model,\nsource-text licence notice, `schema_version = 2`\n\n, and a `*_semantics`\n\nnote\nfor each of the four derived columns (`risk_tier_semantics`\n\n,\n`regime_bucket_semantics`\n\n, `obligation_on_semantics`\n\n,\n`date_in_force_semantics`\n\n) stating exactly what each column does and does\nnot claim.\n\nThe four derived columns answer four different questions:\n\n`risk_tier`\n\n—*does this chunk's own operative text classify?*Narrow and deliberately sparse: only direct textual classifications (\"shall be prohibited\", \"shall be considered to be high-risk\").`regime_bucket`\n\n—*which regulatory regime does this provision belong to?*Structural association (chapter/annex membership and the Commission's conventional tier names), independent of whether the text classifies.`obligation_on`\n\n—*which enum role is the chunk's operative obligated subject?*Judged per chunk from its own sentences; NULL for authority-, Commission-, or AI-Office-bound and descriptive chunks.`date_in_force`\n\n—*earliest application date*per Article 113; not the entry-into-force date, and NULL where no single date is unambiguous.\n\nValue counts (933 chunks):\n\n| column | population |\n|---|---|\n`risk_tier` |\n`prohibited` 2 · `high` 35 · NULL 896 |\n`regime_bucket` |\n`prohibited-practices` 10 · `high-risk` 332 · `transparency` 7 · `gpai` 51 · `voluntary-codes` 4 · NULL 529 |\n`obligation_on` |\n`provider` 56 · `deployer` 17 · `importer` 7 · `distributor` 6 · NULL 847 |\n\n## Decoding the embeddings\n\n``` python\nimport sqlite3, numpy as np\n\ncon = sqlite3.connect(\"aiact_openrag.db\")\ncon.row_factory = sqlite3.Row\nrow = con.execute(\"SELECT * FROM EmbeddingContent WHERE article_num = '5(1)'\").fetchone()\nvec = np.frombuffer(row[\"embedding\"], dtype=np.float32)   # shape (1024,)\n```\n\n## Getting started — semantic search in ~15 lines\n\nEncoding new queries needs the embedding model (`pip install sentence-transformers`\n\n); everything else is the standard library plus NumPy.\n\n``` python\nimport sqlite3, numpy as np\nfrom sentence_transformers import SentenceTransformer\n\ncon = sqlite3.connect(\"aiact_openrag.db\")\nrows = con.execute(\"SELECT chunk_id, article_num, heading, chunk_text, embedding \"\n                   \"FROM EmbeddingContent WHERE embedding IS NOT NULL\").fetchall()\nmat = np.vstack([np.frombuffer(r[4], dtype=np.float32) for r in rows])\n\nmodel = SentenceTransformer(\"BAAI/bge-m3\")\nq = model.encode([\"is emotion recognition at work prohibited?\"],\n                 normalize_embeddings=True)[0]\nfor i in np.argsort(-(mat @ q))[:5]:\n    r = rows[i]\n    print(f\"{r[1]:<16} {r[2]}\\n   {r[3][:150]}…\\n\")\n```\n\nFilter by metadata instead of (or as well as) similarity — that is the point of structural chunking:\n\n```\n-- what the text itself classifies (direct, defensible per provision)\nSELECT article_num, chunk_text FROM EmbeddingContent\nWHERE risk_tier = 'prohibited';\n\n-- everything belonging to a regime (the broader, structural grouping)\nSELECT article_num, heading FROM EmbeddingContent\nWHERE regime_bucket = 'high-risk' AND content_type = 'article';\n```\n\n## Evaluation\n\nThe corpus was evaluated against the AI Act Evaluation Benchmark\n(Davvetas et al.) on retrieval, QA and risk-classification tasks; full\nnumbers, methodology and provenance are in\n[eval/RESULTS.md](/datasets/faitholopade/aiact-openrag/blob/main/eval/RESULTS.md).\n\nOne finding deserves a plain-language caveat here. A three-family\nmodel panel (mistral-nemo, qwen2.5:14b, claude-fable-5) rating all 339\nbenchmark scenarios showed inter-model agreement collapsing exactly on the\ntwo tiers where classification F1 is lowest: per-tier Fleiss κ was 0.238\n(limited) and 0.449 (minimal) versus 0.784 (prohibited) and 0.738\n(high-risk). This is **consistent with ambiguity or noise at the\nlimited/minimal label boundary**, and it means limited/minimal per-tier\nscores on this benchmark should not be read as pure retrieval-quality\nsignals — but it is an **exploratory finding with substantial caveats**,\nnot proof: the raters are themselves LLMs, so shared model difficulty on a\ngenuinely hard boundary cannot be separated from label noise, and the\nbenchmark's labels are LLM-generation targets under human-authored tier\nprompts rather than independently validated classifications.\n\n## Known limitations\n\nPlain language, so nobody trips on these later:\n\n**Frozen in time.** This is the consolidated version of the Act as of its consolidation date (CELEX`02024R1689-20240712`\n\n), as fetched from EUR-Lex at build time (July 2026). If the Act is amended after that, this corpus does not know. Check EUR-Lex for anything load-bearing.**Harmonised standards are not in here — on purpose.** The CEN/CENELEC standards that operationalise the Act are copyrighted and cannot be redistributed. Commission guidance without an explicit reuse notice is also excluded. A RAG system built on this corpus alone cannot answer \"which standard do I follow?\"The Regulation's operative text never classifies a system into those tiers; they exist only in the Commission's four-tier presentation. An empty result for`risk_tier`\n\nvalues`limited`\n\nand`minimal`\n\nnever occur — on purpose.`risk_tier = 'minimal'`\n\nis correct behaviour, not missing data. The conventional associations are preserved in`regime_bucket`\n\n(`transparency`\n\n≈ \"limited risk\", Art 50;`voluntary-codes`\n\n≈ \"minimal risk\", Art 95).Only 37 of 933 chunks carry it (2 prohibited, 35 high) because almost none of the Act's text directly classifies anything — most of it is regime machinery. Use`risk_tier`\n\nis deliberately sparse.`regime_bucket`\n\n(404 non-NULL) for \"provisions of the high-risk/prohibited/GPAI regimes\".**All derived columns are rule-derived, not court-decided.** Every rule and the provision it rests on is written out in, and the audit that produced the current design in`docs/derivation.md`\n\n. Where the text is ambiguous the value is NULL. Read the rules before trusting the labels.`docs/label-audit.md`\n\n## Attribution\n\n**Source text**: © European Union, 1998–2026, via[EUR-Lex](https://eur-lex.europa.eu/eli/reg/2024/1689/oj). Reuse permitted under[Commission Decision 2011/833/EU](https://eur-lex.europa.eu/eli/dec/2011/833/oj)on the reuse of Commission documents; reproduction with acknowledgement of the source. Only the versions of EU law published in the Official Journal of the European Union are authentic.**Evaluation benchmark**: scenarios and QA pairs from the[AI Act Evaluation Benchmark](https://github.com/davidath/ai-act-evaluation-benchmark)(CC-BY-4.0 data, Apache-2.0 code).\n\n## Citation\n\nIf you evaluate against the benchmark used in this corpus's evaluation, cite:\n\n```\n@article{Davvetas2026,\n  title   = {AI Act Evaluation Benchmark: An Open, Transparent, and\n             Reproducible Evaluation Dataset for NLP and RAG Systems},\n  author  = {Athanasios Davvetas and Michael Papademas and Xenia Ziouvelou\n             and Vangelis Karkaletsis},\n  journal = {arXiv preprint arXiv:2603.09435},\n  year    = {2026}\n}\n```\n\n- Downloads last month\n- -", "url": "https://wpnews.pro/news/a-structurally-chunked-pre-embedded-sqlite-corpus-of-the-eu-ai-act", "canonical_source": "https://huggingface.co/datasets/faitholopade/aiact-openrag", "published_at": "2026-07-17 08:09:39+00:00", "updated_at": "2026-07-17 08:21:18.123329+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-policy", "ai-research", "ai-tools", "large-language-models"], "entities": ["EU AI Act", "EUR-Lex", "BGE-M3", "SQLite", "CC BY 4.0", "Apache-2.0"], "alternates": {"html": "https://wpnews.pro/news/a-structurally-chunked-pre-embedded-sqlite-corpus-of-the-eu-ai-act", "markdown": "https://wpnews.pro/news/a-structurally-chunked-pre-embedded-sqlite-corpus-of-the-eu-ai-act.md", "text": "https://wpnews.pro/news/a-structurally-chunked-pre-embedded-sqlite-corpus-of-the-eu-ai-act.txt", "jsonld": "https://wpnews.pro/news/a-structurally-chunked-pre-embedded-sqlite-corpus-of-the-eu-ai-act.jsonld"}}