{"slug": "datamimic-don-t-let-your-coding-agent-invent-its-own-test-world", "title": "Datamimic – don't let your coding agent invent its own test world", "summary": "DATAMIMIC Community Edition, an MIT-licensed Python-native synthetic data engine from rapiddweller, is now MCP-ready and ships an AGENTS.md contract that directs coding agents to preserve intent as model.dm.json and stop only on verified=true. The CE release generates deterministic synthetic datasets and performs PII-aware pseudonymization across PostgreSQL, MySQL, Oracle, MS SQL, SQLite, MongoDB, CSV, JSON, XML, XLSX, DbUnit, and fixed-width formats, while the DATAMIMIC Enterprise Platform adds a probability-scored PII scanner, multi-system execution across Oracle, MongoDB, and Kafka, and EDIFACT, SWIFT MT, HL7 v2.x, and HL7 FHIR message templates. The company says the platform is deployed in regulated EU banking environments, with reference customers available under NDA.", "body_md": "**This repository contains the DATAMIMIC Community Edition (CE).** MIT-licensed, Python-native, MCP-ready.\n\nCE is fully usable standalone for deterministic synthetic data generation and PII-aware pseudonymization. The Enterprise Platform adds governed workflows, PII scanning, role-based access, audit logging, scheduling, multi-system execution, and the full operational layer that regulated enterprises require.\n\n👉 **Enterprise Platform:** [datamimic.io](https://datamimic.io)  |  📘 **Docs:** [docs.datamimic.io](https://docs.datamimic.io)  |  📅 **Book a strategy call:** [datamimic.io/contact](https://datamimic.io/contact)\n\n🤖 **AI agent?** Start at [`AGENTS.md`](https://github.com/rapiddweller/datamimic/blob/development/AGENTS.md) and use the project CLI: preserve new intent as `model.dm.json`, submit an early best attempt via `datamimic scaffold ... --format json`, repair from the structured issues, declare an expectation per stated requirement, and stop on `verified=true`. Existing raw XML uses lint plus bounded dry-run.\n\n**DATAMIMIC CE is the open-source deterministic data engine at the core of the DATAMIMIC Enterprise Platform.** It is usable standalone for synthetic data generation and PII-aware pseudonymization in any local, CI, or agent-driven workflow.\n\nThe Enterprise Platform adds the governed workflows, scanners, dashboards, and execution layer that regulated enterprises require for production-scale test-data operations.\n\n**Available in CE (this repo):**\n\n- **Generate** fully synthetic, deterministic datasets — model-driven, no source data required\n- **Pseudonymize** staging/QA exports — deterministic (seeded) or privacy-maximized (non-seeded) field transformation; PII fields identified and modeled manually in the XML pipeline\n- **Execute** single-system pipelines against PostgreSQL · MySQL · Oracle · MS SQL · SQLite · MongoDB · CSV · JSON · XML · XLSX · DbUnit · fixed-width (`.fcw` )\n- **Model behavior** — weighted state machines, composite multi-field references, control flow (`<while>` ,`<assert>` ), and a scriptable memstore for staged aggregation\n- **Emit provenance** — append-only execution logs and per-output content hash for audit re-execution\n- **Guide agents** — machine-readable capabilities, progressive reference queries, and one canonical CLI scaffold transaction; an optional MCP adapter exposes the same authoring service\n\n**The Enterprise Platform adds:**\n\n- **PII scanner** — probability-scored field detection with configurable thresholds via DataWorkbench\n- **Multi-system execution** — Oracle / MongoDB / Kafka in coordinated workflows with referential integrity\n- **Industry message templates** — EDIFACT / SWIFT MT / HL7 v2.x / HL7 FHIR generated as deterministic test/training artefacts\n- **Governance layer** — role-based dashboards, audit trails, approval flows, reusable enterprise templates, scheduler\n- **Performance core** — Rust fastpath, ML/auto-regressive engine for complex distributions, keyset and manifest building, optimised distributed execution\n- **On-premise / air-gapped deployment** — podman-compose or Helm, with consulting-led rollout\n\nDeployed in regulated EU banking environments for deterministic test data across Oracle, MongoDB, and Kafka pipelines. Reference customers available under NDA — see also [datamimic.io case studies](https://datamimic.io).\n\nThe CLI is the baseline agent contract. Install CE with `pip install datamimic-ce`;\ninside this checkout, use `.venv/bin/datamimic` so a stale global installation cannot\nchange the available schema or commands.\n\n| Need | CLI tool | Contract | \n|---|---|---|\n| Discover the live structural surface | `datamimic capabilities` | Compact machine-readable JSON index by default; `--full` for the complete manifest,`--section <name>` for one section. | \n| Learn the Intent Model progressively | `datamimic reference authoring` , then`datamimic reference authoring --category <category> --kind <kind>` | Start with the query catalogue, then load only the typed fragment needed. | \n| Author a new model | Preserve `model.dm.json` ; run`datamimic scaffold model.dm.json --format json` | One compile/lint/bounded-run/acceptance transaction per changed attempt. Stop on `verified=true` ; generated XML is runtime output. | \n| Work with existing raw XML | `datamimic lint model.xml --format json` , then`datamimic dry-run model.xml --format json` | Fix diagnostics, inspect bounded samples for intent, then use `datamimic run model.xml` only when real execution is requested. | \n| Find a DSL detail | `datamimic reference overview` , then a narrow`reference` topic/name | Query the live model and rule registries instead of guessing elements, generators, scope, distributions, or rules. | \n\n`capabilities`, authoring-reference projections, and the commands shown with\n`--format json` return machine-readable JSON. On a failed scaffold attempt, change\n`model.dm.json` using its structured validation issues, typed repair, or rule\ndiagnostics before retrying. A typed `max_count` remediation instead changes only\nthe bounded scaffold parameter to at least its reported minimum. Never repeat an\nidentical failed call. A successful scaffold result is terminal for authoring, so\ndo not lint or dry-run its generated XML again. Exact source fragments are\ndiscoverable through queries such as `--category source --kind memstore`.\n\nWhen the calling environment already exposes DATAMIMIC MCP tools, they map to the same\ncanonical contracts and implementations: `reference` → `datamimic_reference`,\n`scaffold` → `datamimic_scaffold`, `lint` → `datamimic_check`, and `dry-run` →\n`datamimic_run`. Install the adapter with `pip install \"datamimic-ce[mcp]\"`;\nregistration details belong in the [`MCP quickstart`](https://github.com/rapiddweller/datamimic/blob/development/docs/mcp_quickstart.md),\nnot in the authoring workflow. The adapter intentionally exposes only the four\ncanonical reference, scaffold, check, and bounded-run operations; domain generation\nremains a Python/CLI capability rather than a parallel MCP authoring path.\n\n**Author and verify a new model**\n\n```\nCreate the dataset I describe with DATAMIMIC.\n\nRead AGENTS.md first. In a repository checkout use `.venv/bin/datamimic`;\notherwise use the current `datamimic` CLI. Preserve my intent as\n`model.dm.json`; do not hand-write XML.\n\nStart from the minimal valid document shape in AGENTS.md (\"Authoring a new\nmodel\"). Two rules prevent most rejections: the top level allows ONLY\nversion, seed, products, expectations; product-level \"kind\"\n(generated/source/time_series) is a different vocabulary from field-level\n\"kind\" (increment, values, weighted, int_range, decimal_range, pattern,\nconstant, script). Range fields take minimum/maximum, never min/max.\n\nSubmit EARLY: run `datamimic scaffold model.dm.json --format json` with your\nbest attempt after at most one discovery call. Repair from the structured\nissues (path/code/message/allowed_fields) and diagnostics (fix_hint) — they\nteach the schema faster than more discovery. Never resubmit an unchanged\ndocument. If a remediation requests a larger max_count, retry scaffold with\nat least that value without changing the intent.\n\nDeclare an expectation for every requirement I state (counts as exact_count\nwith a \"count\" field, uniqueness, allowed values, ranges, foreign keys) —\nverified=true certifies only what you declared. Stop on verified=true; do\nnot lint or dry-run the generated XML. If I request real execution, save the\nreturned XML as a generated artifact and run that descriptor. Return the\nmodel.dm.json path and concise verification evidence.\n```\n\n**Relational hierarchy with referential integrity (fully supported — no XML\nneeded)**\n\n```\nSeed a relational dataset with referential integrity: 4 customers, each with\nexactly 2 orders.\n\nCustomers get an incrementing unique id and a region from\n{north, south, east, west}. Each order carries the REAL parent customer id\nas a foreign key and an amount between 10.0 and 500.0.\n\nFollow AGENTS.md's \"Authoring a new model\" and its structural recipes:\norders nest inside the customer product's \"children\" array; the FK field is\n{\"kind\": \"script\", \"script\": \"parent.id\"} with a foreign_key role — a\nrandomly generated FK passes schema validation but fails per-parent-count\nacceptance. Declare expectations for the customer count, customer id\nuniqueness, exactly 2 orders per customer (per_parent_count), the\norders->customers foreign key, and the amount range. Stop on verified=true\nand show the acceptance evidence.\n```\n\nRaw XML remains supported for existing descriptors (lint → dry-run → run;\nsee AGENTS.md). For new models it is a last resort: only when a scaffold\nissue explicitly classifies the requirement as `unsupported_intent` should\nan agent hand-author XML, preserving that evidence.\n\nCE and EE are **not the same engine with a feature flag**. They share the DSL and determinism contract, but EE is an independently optimised execution engine built for enterprise-scale throughput and operational control.\n\n| Capability | Community Edition (CE) | Enterprise Platform (EE) | \n|---|---|---|\n| Deterministic data generation | ✅ | ✅ | \n| Deterministic seeding in the DSL | ✅ entities + standalone literal `<key generator>` (4.0.0) | ✅ same, plus sandboxed script expressions and stdlib `random` calls | \n| **Pseudonymization — seeded***(GDPR Art. 4(5); supports Art. 25 / Art. 32)* | ✅ manual model | ✅ automated via DataWorkbench | \n| **Pseudonymization — non-seeded (privacy-maximized)** | ✅ manual model | ✅ automated via DataWorkbench | \n| Python API + XML pipelines | ✅ | ✅ | \n| Domain models: Finance, Healthcare, Demographics | ✅ | ✅ | \n| Time-series generation ( `<generate start/end/interval>` , ISO 8601, prefix-stable) | ✅ | ✅ | \n| MCP server for AI agent integration | ✅ | ✅ | \n| CLI + local execution | ✅ | ✅ | \n| **Scale** | millions of records via Python multiprocessing (and optional Ray) | **designed for billion-record workloads** — Rust fastpath, optimised multi-process execution, and keyset/manifest building on top of the shared Ray distribution layer | \n| **PII scanner** | ❌ | ✅ probability-scored field detection, configurable threshold, DataWorkbench integration | \n| **Runtime configuration profiles** | ❌ | ✅ Performance · Balanced · Flexibility | \n| **Memory management** | standard | optimised for high-volume batch and streaming | \n| **Logging granularity** | flat execution log | configurable: minimal · standard · deep nested tracing | \n| **Nested structure evaluation** | basic | deep nested generation with extended condition + ruleset evaluation | \n| **Importer / exporter logging** | ❌ | per-stage logging for importers and exporters | \n| **Error handling** | standard exceptions | structured error catalog with recovery strategies | \n| **Rust fastpath** | ❌ | performance-critical paths in Rust | \n| **Keyset and manifest building** | ❌ | reads live DB schemas to build coordinated multi-table generation plans | \n| **ML / auto-regressive engine** | ❌ | combine statistical models with conditions, rulesets, validators for complex distributions | \n\n| Capability | EE | \n|---|---|\n| Multi-user collaboration | ✅ | \n| Role-based access control (RBAC) | ✅ | \n| Audit logs + provenance dashboards | ✅ | \n| **PII scanner — probability scoring, threshold-based field flagging** | ✅ | \n| **DataWorkbench — visual field mapping and pseudonymization model builder** | ✅ | \n| Reusable enterprise template library | ✅ | \n| Scheduled execution + task runner | ✅ | \n| CI/CD pipeline integration (Tosca, Jenkins, GitLab) | ✅ | \n| Multi-system execution: Oracle, MongoDB, Kafka | ✅ | \n| **Template engine: schema-aware editors for EDIFACT, SWIFT MT, HL7 v2.x, and HL7 FHIR — customer-uploadable specs, further industry formats built per engagement on the same framework** | ✅ | \n| Audit-evidence artefacts for GDPR Art. 30 records, PCI DSS 4.0 Req. 6.5.5 (test data) reviews, and — for US Covered Entities / Business Associates — HIPAA §164.312 evidence packs | ✅ | \n| On-premise deployment + air-gapped environments | ✅ | \n| LSP-powered IDE tooling for DSL authoring | ✅ | \n\n👉 [Explore the Enterprise Platform](https://datamimic.io)  |  [Book a platform demo](https://datamimic.io/contact)\n\nThe EE core supports three runtime configuration profiles, selectable per execution context:\n\n| Profile | Optimises for | Typical use case | \n|---|---|---|\n| **Performance** | Maximum throughput via Rust fastpath, optimised multi-process execution, and Ray-based distribution | Bulk generation at billion-record volumes to PostgreSQL, Oracle, Kafka | \n| **Balanced** | Throughput + full audit logging | Standard enterprise pipeline runs with compliance requirements | \n| **Flexibility** | Deep nested evaluation, extended condition and ruleset processing | Complex domain models with ML engine combinations, multi-level referential structures | \n\nLogging depth is independently configurable per profile — from minimal (throughput-optimised) to full nested tracing across importers, exporters, and generation stages.\n\nThe EE template engine generates industry-standard financial messages from DATAMIMIC models. The workbench parses uploaded message samples, auto-detects the message type, and validates edits against the registered spec version in real time.\n\n- **Spec-aware form editing** — segments and elements rendered as structured forms with mandatory/optional indicators, per-field value suggestions, and inline custom-extension support\n- **Strict validation** against baked spec versions, with segment- and element-level error reporting\n- **Advisory mode** when a spec is unregistered or in draft — editing stays enabled, validation continues as guidance\n- **Round-trip** between the structured form view and the authoritative template text — no fidelity loss\n- **Download / adjust / upload your own spec** — customers can extend or override the baked spec catalogue without waiting for a release\n- **Live structure tree + preview** for every edit\n- **File auto-detection** — upload an existing message, the editor identifies the type and loads the matching spec\n\n| Format | Coverage | \n|---|---|\n| **UN/EDIFACT** | Schema-aware form editor; spec versions and subsets per engagement | \n| **SWIFT MT** | Schema-aware form editor; categories and SR versions per engagement | \n| **HL7 v2.x** | Schema-aware form editor; versions per engagement | \n| **HL7 FHIR** | Schema-aware form editor for FHIR resources (Patient, Observation, Encounter, …); profiles per engagement | \n| **Further industry formats** (ISO 20022 / MX, vertical dialects) | Built into the editor catalogue per customer engagement, on the same framework | \n\nCustomers can extend the spec catalogue between releases by downloading, adjusting, and uploading their own spec files directly.\n\nGenerated messages are deterministic and traceable to their source model, and syntactically valid against the registered spec. They are intended for **test and training environments only** — they are not network-validated and must not be transmitted on production SWIFTNet or EDI networks. See the [SWIFT CSP note](#supported-systems) below.\n\n| Role | What DATAMIMIC solves | \n|---|---|\n| **QA / Test Manager** | Eliminate manual test data requests. Self-service, governed, always ready. | \n| **Business Analyst** | Define data requirements in business-readable models — no scripting needed. | \n| **Platform / DevOps Engineer** | Integrate deterministic test data generation into CI/CD and scheduled pipelines. | \n| **Compliance / Audit** | Full audit trail for every generation run. Regulator-ready logs, no production data exposure. | \n| **Enterprise Architect** | One governed standard across Oracle, MongoDB, Kafka, flat files, and custom systems. | \n\nDevelopers and data engineers who need deterministic synthetic data generation or PII-aware pseudonymization in local environments, CI pipelines, or agent-driven workflows. PII field identification is manual — the EE DataWorkbench automates this step.\n\nMost test data tools produce random output. That breaks regression tests, audit trails, and cross-team reproducibility.\n\n**DATAMIMIC's determinism contract (CE):**\n\n- **Same engine version + same model + same seed = byte-identical output** , every run, every machine. Holds at three layers: the`generate_domain` facade, every domain service called directly, and every literal generator that accepts an`rng=` argument. Verified per-service on every CI run via[`tests_ce/architecture/test_service_replay_determinism.py`](https://github.com/rapiddweller/datamimic/blob/development/tests_ce/architecture/test_service_replay_determinism.py) .\n- **DSL-level seeding:**`<setup rngSeed=\"N\">` makes the whole model deterministic — every seed-less`<variable entity=\"…\">` derives a reproducible child RNG from it, and`<variable rngSeed=\"…\">` overrides it for that block (no seed anywhere → wall-clock random). Verified by[`tests_ce/integration_tests/test_determinism_seed_scenarios`](https://github.com/rapiddweller/datamimic/tree/development/tests_ce/integration_tests/test_determinism_seed_scenarios) . As of 4.0.0 the same seed also reaches standalone literal generators (`<key generator=\"…\">` ), typed/pattern keys,`DateTimeGenerator` , and cross-page`unique` picks — machine-independently.\n- **Source reads:**`distribution=\"ordered\"` reads a data source in stable file order;`distribution=\"random\"` shuffles but replays identically when`<setup rngSeed>` is set (without a seed the shuffle is non-deterministic by design, for privacy-maximized one-time deliveries). Deterministic shuffling across distributed / multi-process execution is EE.\n- **Provenance hash on every facade output** = re-executable lineage. Same input → same`determinism_proof.content_hash` , always.\n- **UUIDv5 entity identifiers** = stable across runs and machines.\n- **Single wall-clock SPOT** (`now_utc_naive()` ); raw`datetime.now()` is forbidden in production code and the clock-drift architecture gate fails CI on any reintroduction.\n- **RNG/clock runtime SPOTs** in`datamimic_ce/domains/domain_core/runtime/` :`spawn_rng` (reproducible child-RNG derivation),`now_utc_naive` , and`resolve_clock` . The same contract vocabulary the Enterprise Platform enforces end-to-end.\n\n**The Enterprise Platform (EE) goes further:** beyond the CE contract, EE makes the whole execution environment deterministic — a configurable/frozen wall-clock (not just CE's fixed anchor), and deterministic `SAFE_GLOBALS` plus the Python `random` functions, so sandboxed script expressions and any stdlib `random` call replay identically as well.\n\n``` python\nfrom datamimic_ce.domains.facade import generate_domain\n\nrequest = {\n    \"domain\": \"person\",\n    \"version\": \"v1\",\n    \"count\": 1,\n    \"seed\": \"regression-suite-42\",       # identical seed → identical output\n    \"locale\": \"en_US\",\n    \"clock\": \"2025-01-01T00:00:00Z\"      # fixed clock = stable time context\n}\n\nresponse = generate_domain(request)\n# response[\"determinism_proof\"][\"content_hash\"] is stable across runs.\n```\n\nDirect service use is equally deterministic when given a seeded RNG:\n\n``` python\nimport random\nfrom datamimic_ce.domains.finance.services import CreditCardService\n\n# Same seeded Random → byte-identical CreditCard across runs.\ncard_a = CreditCardService(rng=random.Random(42)).generate()\ncard_b = CreditCardService(rng=random.Random(42)).generate()\nassert card_a.bic == card_b.bic and card_a.card_number == card_b.card_number\n```\n\n| Scope | CE | Enterprise Platform | \n|---|---|---|\n| **Facade** (`generate_domain` registered domains) | ✅ byte-identical, CI-gated | ✅ byte-identical | \n| **Domain services** (direct use with seeded`rng=...` ) | ✅ byte-identical, CI-gated | ✅ byte-identical | \n| **Literal generators** (with seeded`rng=...` ) | ✅ byte-identical | ✅ byte-identical | \n| **RNG / clock runtime SPOTs** | ✅ `spawn_rng` ,`now_utc_naive` ,`resolve_clock` | ✅ same contract, enforced end-to-end | \n| **Architecture gates in CI** | ✅ facade replay + service replay (every service) + clock drift | ✅ 5+ gates (RNG ownership, clock drift, DSL eval, seeded-mode propagation, dataset SPOT) | \n| **Custom XML pipelines** (seeded via`<setup rngSeed>` ) | ✅ byte-identical, machine-independent (single-process) | ✅ byte-identical, distributed | \n| **Multi-system coordinated execution** (Oracle + MongoDB + Kafka in one run) | — | ✅ byte-identical end-to-end | \n| **Seeded vs unseeded pseudonymization** (deterministic clock anchor vs CSPRNG live-clock) | — | ✅ | \n| **Threat-led / TLPT-grade audit evidence** (full contract enforcement, per-stage execution logging) | — | ✅ | \n\nCE delivers contract-enforced determinism for the synthetic-data generation surface (facade, services, generators) and, as of 4.0.0, for seeded XML descriptors — byte-identical across machines, executed single-process. The Enterprise Platform extends the same contract to distributed and multi-system execution with referential integrity and the seeded/unseeded pseudonymization modes, and adds the five drift-gates that lock the contract end-to-end for regulated deployments.\n\n|  | Faker / Random generators | DATAMIMIC CE | DATAMIMIC EE | \n|---|---|---|---|\n| Reproducible output | ❌ | ✅ | ✅ | \n| Domain-aware relationships | ❌ | ✅ | ✅ | \n| Business logic constraints | ❌ | ✅ | ✅ | \n| Per-output provenance hash | ❌ | ✅ | ✅ | \n| Source data pseudonymization | ❌ | ✅ manual | ✅ automated | \n| PII field detection | ❌ | ❌ | ✅ probability-scored | \n| Enterprise governance layer | ❌ | ❌ | ✅ | \n| Multi-system execution | ❌ | ❌ | ✅ | \n| Role-based workflows | ❌ | ❌ | ✅ | \n| Designed for regulated-industry deployment (governance, audit, RBAC) | ❌ | ❌ | ✅ | \n\n``` python\n# Faker — broken relationships\nfrom faker import Faker\nfake = Faker()\npatient_age = fake.random_int(1, 99)\nconditions  = [fake.word()]\n# \"25-year-old with Alzheimer's\" — meaningless for any real test\n\n# DATAMIMIC — domain-aware, deterministic with a seed\nimport random\nfrom datamimic_ce.domains.healthcare.services import PatientService\npatient = PatientService(rng=random.Random(42)).generate()\nprint(f\"{patient.full_name}, {patient.age}, {patient.conditions}\")\n# Age-appropriate, domain-consistent — and identical every run with a fixed seed\npip install datamimic-ce\npython\nimport random\nfrom datamimic_ce.domains.healthcare.services import PatientService\n\npatient = PatientService(rng=random.Random(42)).generate()\nprint(patient.full_name, patient.age, patient.conditions)\n# Age-appropriate conditions, demographically realistic; deterministic with a seed\npython\nimport random\nfrom datamimic_ce.domains.finance.services import BankAccountService\n\naccount = BankAccountService(rng=random.Random(42)).generate()\nprint(account.account_number, account.balance)\n# Balance-consistent, locale-correct; reproducible with a seed\n```\n\nDATAMIMIC supports two pseudonymization modes with different privacy postures:\n\n| Mode | How | Legal classification | Use case | \n|---|---|---|---|\n| **Seeded** (`rngSeed` set) | Deterministic, reproducible | Pseudonymization (GDPR Art. 4(5)) | Regression testing, stable CI/CD pipelines | \n| **Non-seeded** (no`rngSeed` ) | Non-deterministic, no reversible mapping at field level | Privacy-maximized transformation | One-time data delivery, higher privacy posture | \n\n**Note on GDPR anonymization:** Full anonymization status under GDPR depends on complete field coverage across all quasi-identifiers and a re-identification risk assessment on the complete record — not on individual field transformation alone. DATAMIMIC does not make anonymization claims on behalf of the customer. Non-seeded mode maximizes privacy at the transformation level; the customer is responsible for assessing re-identification risk across the full dataset.\n\nIn CE, PII fields are identified and modeled manually in the XML pipeline:\n\n```\n<setup defaultSeparator=\",\">\n  <generate name=\"customers\" source=\"customer_export.csv\" target=\"CSV\" distribution=\"ordered\">\n    <!-- distribution=\"ordered\" reads the source in a stable order — required so the\n         Nth source row maps to the same seeded synthetic value on every run. The\n         default (\"random\") shuffles non-deterministically and would break it.\n         rngSeed on the <variable> makes the synthetic values reproducible; drop\n         rngSeed for the privacy-maximized (non-deterministic) mode. -->\n    <variable name=\"p\"   entity=\"Person\"      dataset=\"DE\" rngSeed=\"42\" />\n    <variable name=\"acc\" entity=\"BankAccount\" dataset=\"DE\" rngSeed=\"42\" />\n\n    <key name=\"first_name\" script=\"p.given_name\" />\n    <key name=\"last_name\"  script=\"p.family_name\" />\n    <key name=\"email\"      script=\"p.email\" />\n    <key name=\"iban\"       script=\"acc.iban\" />\n    <key name=\"birth_date\" script=\"p.birthdate\" />\n  </generate>\n</setup>\n```\n\nBuilt-in converters can additionally transform a key's value — e.g. irreversibly hash the original instead of replacing it, or partially mask it:\n\n```\n<key name=\"email\" script=\"p.email\" converter=\"Hash('sha256','hex')\" />\n<key name=\"iban\"  script=\"acc.iban\" converter=\"MiddleMask(8, 4)\" />\n```\n\nAvailable converters (13): `Mask`, `MiddleMask(start, end)`, `CutLength(n)`, `Substring(start, end)`, `JavaHash`, `RemoveNoneOrEmptyElement`,\n`Hash(type, format[, salt])`, `DateFormat(fmt)`, `Append`, `UpperCase`,\n`LowerCase`, `Date2Timestamp`, `Timestamp2Date`.\n\n```\ndatamimic run ./pseudonymize-customers/datamimic.xml\n```\n\n`source` is a controlled export or staging input — never a live production connection.\n\nWith `rngSeed` set: same source record → same pseudonymized output on every run. Stable for regression testing.\n\nWithout `rngSeed`: non-deterministic output — no reversible mapping exists at the field level. Stronger privacy posture for one-time delivery scenarios.\n\n**In the Enterprise Platform (EE):** the DataWorkbench PII scanner automatically scans source schemas, assigns probability scores to each field, and flags candidates above a configurable threshold. Flagged fields are wired into the pseudonymization model automatically — no manual field mapping required.\n\n```\n<setup>\n  <generate name=\"patients\" count=\"1000\" target=\"CSV\">\n    <variable name=\"patient\" entity=\"Patient\" dataset=\"US\" ageMin=\"60\" ageMax=\"80\" rngSeed=\"42\" />\n    <key name=\"full_name\"   script=\"patient.full_name\" />\n    <key name=\"age\"         script=\"patient.age\" />\n    <array name=\"conditions\" script=\"patient.conditions\" />\n  </generate>\n</setup>\ndatamimic run ./patient-scenario/datamimic.xml\n```\n\nAny `<generate>` becomes a time-series loop when given strict ISO 8601 `start`/` end`/` interval` attributes. Per iteration the script context exposes a `ts` namespace:\n\n| Variable | Type | Meaning | \n|---|---|---|\n| `ts.now` | `datetime` | Current tick | \n| `ts.step` | `int` | Position within one series ( `0..N-1` ) | \n| `ts.series` | `int` | Which series this row belongs to ( `0..count-1` ) | \n\nOutput column names — including whether to even emit a timestamp or series-id column — are entirely the user's choice via `<key>`. The primitive is domain-agnostic; the same DSL covers IoT readings, financial ticks, log streams, smart meters, anything time-indexed.\n\n```\n<setup>\n  <!-- Stock ticks: three symbols, 5-min interval, 30-min window (writes ticks.csv) -->\n  <generate name=\"ticks\" count=\"3\"\n            start=\"2026-01-01T09:30:00+00:00\"\n            end=\"2026-01-01T10:00:00+00:00\"\n            interval=\"PT5M\"\n            target=\"CSV\">\n    <key name=\"timestamp\" script=\"ts.now.isoformat()\"/>\n    <key name=\"symbol\"    script=\"['AAPL','MSFT','GOOG'][ts.series]\"/>\n    <key name=\"price\"     script=\"100 + ts.step * 0.25\"/>\n  </generate>\n\n  <!-- Sensor with diurnal seasonality, single series (count defaults to 1; writes readings.csv) -->\n  <generate name=\"readings\"\n            start=\"2026-01-01T00:00:00+00:00\"\n            end=\"2026-01-08T00:00:00+00:00\"\n            interval=\"PT1H\"\n            target=\"CSV\">\n    <key name=\"timestamp\" script=\"ts.now.isoformat()\"/>\n    <key name=\"value\"     script=\"20 - 10 * math.cos(ts.now.hour * math.pi / 12)\"/>\n  </generate>\n</setup>\n```\n\nGuarantees:\n\n- **Prefix-stable by construction** — the first N ticks of series 0 are byte-identical regardless of total window length, because each row's`ts.now` is a pure function of`start + interval * step` .\n- **Loop order is contiguous per series** — series 0's full sequence, then series 1's, etc. Makes downstream grouping trivial.\n- **Strict ISO 8601** —`start` /`end` via`datetime.fromisoformat` (Z-suffix supported);`interval` via the[`isodate`](https://pypi.org/project/isodate/) library (` PT1H` ,`PT15M` ,`PT5S` ,`P1D` ,`P1W` ,`P1DT12H` , fractional seconds for sub-second precision). Resolution:`PT0.001S` = 1 ms,`PT0.000001S` = 1 µs (Python`datetime.timedelta` microsecond floor; sub-µs intervals and constant-length-undefined units like months/years are rejected with a clear error).\n- **`count` is orthogonal** , not overloaded — it means \"outer-loop iterations of this`<generate>` \" in**all** modes (same as nested`<generate count=…>` ). In time-series mode each outer iteration is one series of N ticks, so total rows =`count × ticks_per_series` . Default`count=\"1\"` keeps single-series fixtures terse.\n- **Naming caveat** — a`<key name=\"ts\">` output column would shadow the namespace (`current_product` overrides`current_variables` in script scope), and a`<variable name=\"ts\">` is rejected at parse time. Use a different name, e.g.`timestamp` for the column.\n\nComposes with the existing `<variable>` mechanism for multi-source merges (e.g. join each tick with a sensor-metadata CSV via `<variable source=\"meta.csv\" cyclic=\"True\">` inside the same `<generate>`), with `<key condition=\"...\">` filtering, and with `<nestedKey>` sub-scopes — the `ts` namespace is visible everywhere a `<key script>` runs. See `tests_ce/integration_tests/test_timeseries/` for committed DSL fixtures + proofs (including pagination invariance).\n\nMost teams adopt CE for one of three reasons. EE is not required for any of them.\n\n**1. Reproducible test data for CI/CD pipelines.** Pin a seed against the `generate_domain` facade — or hand a seeded `random.Random` to any domain service — and you get byte-identical output across runs and machines. Both layers are gated on every CI run by [`tests_ce/architecture/`](https://github.com/rapiddweller/datamimic/tree/development/tests_ce/architecture/). Regression tests stop being flaky because the input data is stable across runs.\n\n``` python\nfrom datamimic_ce.domains.facade import generate_domain\n\nresponse = generate_domain({\n    \"domain\": \"person\", \"version\": \"v1\", \"count\": 1,\n    \"seed\": \"ci-pipeline-42\", \"locale\": \"en_US\",\n    \"clock\": \"2026-01-01T00:00:00Z\",\n})\n# Same engine version + same model + same seed → same output, every machine, every run.\n```\n\n**2. Deterministic data backend for AI agents and LLM tooling.** The CLI and Python API are the baseline surfaces for seeded, verifiable generation. The optional MCP adapter (`pip install \"datamimic-ce[mcp]\"`) exposes the canonical reference, scaffold, check, and bounded-run authoring operations. Generated domain-facade outputs include a `determinism_proof.content_hash`, so Python/CLI callers can re-execute and verify the data later — useful for agent regression tests and any workflow where the data an agent saw must be reconstructable.\n\n**3. Pseudonymization of staging and QA exports.** Manual model in CE (XML pipeline), no scanner license required. Seeded mode for stable regression test data; non-seeded mode for one-time deliveries with maximized privacy posture. See the [Pseudonymization section above](#pseudonymization--ce-manual-model).\n\nDATAMIMIC produces evidence and reproducible artifacts that support compliance work. It does not replace your DPO, your CISO, or your auditor. The following are pointers for where DATAMIMIC outputs commonly slot into established programs:\n\nBoth editions produce reproducible artefacts. CE covers single-system fixtures and provenance evidence; multi-system audit evidence with role-based dashboards is EE.\n\n| Regulation / standard | Where DATAMIMIC contributes | \n|---|---|\n| **DORA (Reg. 2022/2554)** — Art. 24 (testing of ICT tools, systems and processes; non-TLPT scope) | Reproducible test datasets for non-TLPT resilience tests; deterministic data fixtures for ICT testing programmes | \n| **ISO/IEC 27701:2019** — A.7.2.8 (records related to processing PII) and A.7.4.5 (PII minimisation) | Synthetic data in lieu of PII in non-production environments; documented model definitions as supporting evidence | \n| **HIPAA Security Rule** — §164.312 technical safeguards*(US Covered Entities / Business Associates only)* | Synthetic Patient/MedicalDevice/MedicalProcedure data for dev and test environments without ePHI exposure | \n| **GDPR** — Art. 4(5) pseudonymization definition; Art. 25 privacy by design; Art. 32 security of processing | Seeded pseudonymization with deterministic mapping; non-seeded mode for stronger privacy posture | \n| **PCI DSS 4.0** — Req. 6.5.5 (live PANs prohibited in test/development) | Synthetic PAN generation for test environments; deterministic tokenisation reproducible across runs | \n\nThese pointers do not constitute legal advice or a compliance attestation. Consult your DPO, CISO, or qualified counsel for formal compliance determinations. Full anonymization status under GDPR depends on re-identification risk across the complete dataset — see the [pseudonymization disclaimer above](#pseudonymization--ce-manual-model).\n\nCE and EE share the DATAMIMIC DSL and the determinism contract. The execution layer is separate: CE is a Python execution engine using multiprocessing (with optional Ray for distribution); EE is an independently-optimised execution engine with a Rust fastpath, ML/auto-regressive generation, keyset and manifest building from live schemas, and optimised distributed execution at billion-record scale.\n\n```\n╔══════════════════════════════════════════════════════════════════╗\n║              DATAMIMIC ENTERPRISE PLATFORM (EE)                  ║\n║                                                                  ║\n║  ┌──────────────────────────────────────────────────────────┐    ║\n║  │  PLATFORM LAYER                                          │    ║\n║  │  UI · RBAC · Governance · Audit Dashboards               │    ║\n║  │  DataWorkbench · PII Scanner · Pseudonymization Builder  │    ║\n║  │  Scheduler · Task Runner · CI/CD · Template Engine       │    ║\n║  └──────────────────────────────────────────────────────────┘    ║\n║                                                                  ║\n║  ┌──────────────────────────────────────────────────────────┐    ║\n║  │  EE CORE  (separately maintained, more advanced than CE) │    ║\n║  │                                                          │    ║\n║  │  Rust fastpath for performance-critical paths            │    ║\n║  │  ML / auto-regressive engine for complex distributions   │    ║\n║  │  Keyset and manifest building from live DB schemas       │    ║\n║  │  Optimised distributed execution at billion-record scale │    ║\n║  │  Runtime profiles: Performance · Balanced · Flexibility  │    ║\n║  │  Deep nested evaluation · Conditions · Rulesets          │    ║\n║  │  Structured error catalog · Per-stage execution logging  │    ║\n║  └──────────────────────────────────────────────────────────┘    ║\n╚══════════════════════════════════════════════════════════════════╝\n\n╔══════════════════════════════════════════════════════════════════╗\n║              DATAMIMIC COMMUNITY EDITION (CE)  — this repo       ║\n║                                                                  ║\n║  Determinism Kit · Domain Services · Schema Validators           ║\n║  Synthetic Generation · Pseudonymization (manual model)          ║\n║  Python API · XML Pipelines · CLI · MCP Server                   ║\n╚══════════════════════════════════════════════════════════════════╝\n\n         ↓              ↓              ↓              ↓\n    PostgreSQL       Oracle         MongoDB      CSV / JSON / XML\n```\n\nEE adds Kafka, EDIFACT, SWIFT MT, HL7 v2.x, and HL7 FHIR as additional targets — see [Supported systems](#supported-systems) below. Both editions share the DATAMIMIC DSL and determinism contract.\n\n| System | CE | EE | Notes | \n|---|---|---|---|\n| PostgreSQL | ✅ | ✅ | EE adds schema introspection and referential integrity | \n| MySQL | ✅ | ✅ |  | \n| Oracle | ✅ | ✅ | EE production-validated in regulated banking environments | \n| MS SQL Server | ✅ | ✅ |  | \n| SQLite | ✅ | ✅ | Lightweight CI/CD fixtures | \n| MongoDB | ✅ | ✅ | EE adds nested document generation | \n| CSV / JSON / XML | ✅ | ✅ | Flat file pipelines | \n| XLSX | ✅ | ✅ | Spreadsheet read + write (first row = header) | \n| DbUnit XML | ✅ | ✅ | `.dbunit.xml` dataset read + write | \n| Fixed-width ( `.fcw` ) | ✅ | ✅ | Self-describing column files, read + write | \n| Apache Kafka | — | ✅ | Real-time streaming, payment scenarios | \n| HL7 v2.x | — | ✅ | Test/training output via template engine | \n| HL7 FHIR | — | ✅ | Test/training output via template engine | \n| EDIFACT / SWIFT MT | — | ✅ | Test/training output only; does not satisfy SWIFT CSCF v2025 secure-zone controls (1.1 environment protection, 1.4 internet restriction). Generated messages must not be transmitted from a CSP-attested secure zone. | \n\n| Domain | Services available | \n|---|---|\n| **Healthcare** | Patient, Doctor, Hospital, MedicalDevice, MedicalProcedure | \n| **Finance** | Bank, BankAccount, CreditCard, Transaction | \n| **Insurance** | InsuranceCompany, InsuranceProduct, InsurancePolicy, InsuranceCoverage | \n| **E-commerce** | Order, Product | \n| **Public sector** | AdministrationOffice, EducationalInstitution, PoliceOfficer | \n| **Demographics** | Person (DE / US / VN locale packs), Address, City, Country | \n| **Common** | Company | \n\nAll services are versioned and seeded; each generation emits a provenance hash suitable as evidence in audit reviews. Domain services can be used directly via constructor injection, or driven through the higher-level `generate_domain({...})` facade for seed/locale/clock/count parameterisation (currently supports `person`, `address`, `patient`, `doctor` at `v1`).\n\n```\n# Discover the live structural surface as JSON\ndatamimic capabilities\n\n# Enumerate typed authoring queries, then request only the needed fragment\ndatamimic reference authoring\ndatamimic reference authoring --category field --kind weighted\ndatamimic reference overview\n\n# Compile and fully verify the canonical intent artifact; stop on verified=true\ndatamimic scaffold model.dm.json --format json\n\n# Lint a descriptor: schema, semantics, best practices — every finding carries\n# a rule id (DMxxx) and a fix hint. Exit codes 0/1/2.\ndatamimic lint my-scenario/datamimic.xml\ndatamimic lint my-scenario/datamimic.xml --format json   # diagnostics v1, CI-friendly\n\n# Safely execute bounded counts with neutralized targets and sample rows\ndatamimic dry-run my-scenario/datamimic.xml --format json\n\n# Run a verified scenario for real\ndatamimic run my-scenario/datamimic.xml\n\n# Initialize a new project\ndatamimic init my-scenario\n\n# Demos\ndatamimic demo list\ndatamimic demo create demo-healthcare\ndatamimic demo create --all --target ./my_demos\n\n# System and version info\ndatamimic info\ndatamimic version\n```\n\n| Resource | Link | \n|---|---|\n| Full documentation | [docs.datamimic.io](https://docs.datamimic.io) | \n| MCP quickstart | [docs/mcp_quickstart.md](https://github.com/rapiddweller/datamimic/blob/development/docs/mcp_quickstart.md) | \n| Developer guide | [docs/developer_guide.md](https://github.com/rapiddweller/datamimic/blob/development/docs/developer_guide.md) | \n| Enterprise platform | [datamimic.io](https://datamimic.io) | \n| GitHub Discussions | [Discussions](https://github.com/rapiddweller/datamimic/discussions) | \n| Issue tracker | [Issues](https://github.com/rapiddweller/datamimic/issues) | \n| Email support | [support@rapiddweller.com](mailto:support@rapiddweller.com) | \n\nSee [CONTRIBUTING.md](https://github.com/rapiddweller/datamimic/blob/development/CONTRIBUTING.md). CE is MIT licensed and community contributions are welcome.\n\nThe CE engine is the foundation. If you are building integrations, domain extensions, or MCP tooling on top of DATAMIMIC, we want to hear from you.\n\nMIT — see [LICENSE](https://github.com/rapiddweller/datamimic/blob/development/LICENSE).\n\nThe DATAMIMIC Enterprise Platform (EE) is a commercial product. [Contact us](https://datamimic.io/contact) for licensing.\n\n**DATAMIMIC — Deterministic, governed test data for regulated enterprises.**", "url": "https://wpnews.pro/news/datamimic-don-t-let-your-coding-agent-invent-its-own-test-world", "canonical_source": "https://github.com/rapiddweller/datamimic", "published_at": "2026-09-16 04:58:36+00:00", "updated_at": "2026-09-16 05:07:52.845168+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-tools", "ai-products", "mlops"], "entities": ["DATAMIMIC", "DATAMIMIC Community Edition", "DATAMIMIC Enterprise Platform", "rapiddweller", "DataWorkbench", "PostgreSQL", "MongoDB", "Oracle"], "alternates": {"html": "https://wpnews.pro/news/datamimic-don-t-let-your-coding-agent-invent-its-own-test-world", "markdown": "https://wpnews.pro/news/datamimic-don-t-let-your-coding-agent-invent-its-own-test-world.md", "text": "https://wpnews.pro/news/datamimic-don-t-let-your-coding-agent-invent-its-own-test-world.txt", "jsonld": "https://wpnews.pro/news/datamimic-don-t-let-your-coding-agent-invent-its-own-test-world.jsonld"}}