{"slug": "1m-multi-lingual-synthetic-corporate-emails-to-stress-test-your-rag", "title": "1M multi-lingual synthetic corporate emails to stress-test your RAG", "summary": "The Hugging Face dataset viewer for the Sinabis Group's \"1M synthetic emails\" dataset fails with an ArrowInvalid JSON parse error on row 0, preventing the 1 million multi-lingual synthetic corporate emails from being previewed in the browser. The traceback shows pandas' read_json raising \"ValueError: Expected object or value\" before pyarrow's read_json fails, so the dataset cannot be loaded for RAG stress-testing through the viewer.", "body_md": "[Duplicate](https://huggingface.co/datasets/sinabis-group/sinabis-1m-synthetic-emails?duplicate=true)\n\n## The dataset viewer is not available for this split.\n\n```\nError code: FeaturesError\nException: ArrowInvalid\nMessage: JSON parse error: Invalid value. in row 0\nTraceback: Traceback (most recent call last):\n File \"/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py\", line 324, in _generate_tables\n df = pandas_read_json(f)\n File \"/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py\", line 38, in pandas_read_json\n return pd.read_json(path_or_buf, **kwargs)\n ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.14/site-packages/pandas/io/json/_json.py\", line 815, in read_json\n return json_reader.read()\n ~~~~~~~~~~~~~~~~^^\n File \"/usr/local/lib/python3.14/site-packages/pandas/io/json/_json.py\", line 1014, in read\n obj = self._get_object_parser(self.data)\n File \"/usr/local/lib/python3.14/site-packages/pandas/io/json/_json.py\", line 1040, in _get_object_parser\n obj = FrameParser(json, **kwargs).parse()\n File \"/usr/local/lib/python3.14/site-packages/pandas/io/json/_json.py\", line 1176, in parse\n self._parse()\n ~~~~~~~~~~~^^\n File \"/usr/local/lib/python3.14/site-packages/pandas/io/json/_json.py\", line 1392, in _parse\n ujson_loads(json, precise_float=self.precise_float), dtype=None\n ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n ValueError: Expected object or value\n \n During handling of the above exception, another exception occurred:\n \n Traceback (most recent call last):\n File \"/src/services/worker/src/worker/job_runners/split/first_rows.py\", line 244, in compute_first_rows_from_streaming_response\n iterable_dataset = iterable_dataset._resolve_features()\n File \"/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py\", line 4523, in _resolve_features\n features = _infer_features_from_batch(self.with_format(None)._head())\n ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^\n File \"/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py\", line 2768, in _head\n return next(iter(self.iter(batch_size=n)))\n File \"/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py\", line 2972, in iter\n for key, pa_table in ex_iterable.iter_arrow():\n ~~~~~~~~~~~~~~~~~~~~~~^^\n File \"/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py\", line 2483, in _iter_arrow\n yield from self.ex_iterable._iter_arrow()\n File \"/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py\", line 547, in _iter_arrow\n for key, pa_table in iterator:\n ^^^^^^^^\n File \"/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py\", line 430, in _iter_arrow\n for key, pa_table in self.generate_tables_fn(**gen_kwags):\n ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py\", line 327, in _generate_tables\n raise e\n File \"/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py\", line 290, in _generate_tables\n pa_table = paj.read_json(\n io.BytesIO(batch), read_options=paj.ReadOptions(block_size=block_size)\n )\n File \"pyarrow/_json.pyx\", line 364, in pyarrow._json.read_json\n File \"pyarrow/error.pxi\", line 155, in pyarrow.lib.pyarrow_internal_check_status\n File \"pyarrow/error.pxi\", line 92, in pyarrow.lib.check_status\n raise convert_status(status)\n pyarrow.lib.ArrowInvalid: JSON parse error: Invalid value. in row 0\n```\n\nNeed help to make the dataset viewer work? Make sure to review [how to configure the dataset viewer](https://huggingface.co/docs/hub/datasets-data-files-configuration), and [open a discussion](https://huggingface.co/datasets/sinabis-group/sinabis-1m-synthetic-emails/discussions/new?title=Dataset+Viewer+issue%3A+FeaturesError&description=The+dataset+viewer+is+not+working.%0A%0AError+details%3A%0A%0A%60%60%60%0AError+code%3A+++FeaturesError%0AException%3A++++ArrowInvalid%0AMessage%3A++++++JSON+parse+error%3A+Invalid+value.+in+row+0%0ATraceback%3A++++Traceback+%28most+recent+call+last%29%3A%0A++++++++++++++++File+%22%2Fusr%2Flocal%2Flib%2Fpython3.14%2Fsite-packages%2Fdatasets%2Fpackaged_modules%2Fjson%2Fjson.py%22%2C+line+324%2C+in+_generate_tables%0A++++++++++++++++++df+%3D+pandas_read_json%28f%29%0A++++++++++++++++File+%22%2Fusr%2Flocal%2Flib%2Fpython3.14%2Fsite-packages%2Fdatasets%2Fpackaged_modules%2Fjson%2Fjson.py%22%2C+line+38%2C+in+pandas_read_json%0A++++++++++++++++++return+pd.read_json%28path_or_buf%2C+**kwargs%29%0A+++++++++++++++++++++++++%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%0A++++++++++++++++File+%22%2Fusr%2Flocal%2Flib%2Fpython3.14%2Fsite-packages%2Fpandas%2Fio%2Fjson%2F_json.py%22%2C+line+815%2C+in+read_json%0A++++++++++++++++++return+json_reader.read%28%29%0A+++++++++++++++++++++++++%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%5E%5E%0A++++++++++++++++File+%22%2Fusr%2Flocal%2Flib%2Fpython3.14%2Fsite-packages%2Fpandas%2Fio%2Fjson%2F_json.py%22%2C+line+1014%2C+in+read%0A++++++++++++++++++obj+%3D+self._get_object_parser%28self.data%29%0A++++++++++++++++File+%22%2Fusr%2Flocal%2Flib%2Fpython3.14%2Fsite-packages%2Fpandas%2Fio%2Fjson%2F_json.py%22%2C+line+1040%2C+in+_get_object_parser%0A++++++++++++++++++obj+%3D+FrameParser%28json%2C+**kwargs%29.parse%28%29%0A++++++++++++++++File+%22%2Fusr%2Flocal%2Flib%2Fpython3.14%2Fsite-packages%2Fpandas%2Fio%2Fjson%2F_json.py%22%2C+line+1176%2C+in+parse%0A++++++++++++++++++self._parse%28%29%0A++++++++++++++++++%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%5E%5E%0A++++++++++++++++File+%22%2Fusr%2Flocal%2Flib%2Fpython3.14%2Fsite-packages%2Fpandas%2Fio%2Fjson%2F_json.py%22%2C+line+1392%2C+in+_parse%0A++++++++++++++++++ujson_loads%28json%2C+precise_float%3Dself.precise_float%29%2C+dtype%3DNone%0A++++++++++++++++++%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%0A++++++++++++++ValueError%3A+Expected+object+or+value%0A++++++++++++++%0A++++++++++++++During+handling+of+the+above+exception%2C+another+exception+occurred%3A%0A++++++++++++++%0A++++++++++++++Traceback+%28most+recent+call+last%29%3A%0A++++++++++++++++File+%22%2Fsrc%2Fservices%2Fworker%2Fsrc%2Fworker%2Fjob_runners%2Fsplit%2Ffirst_rows.py%22%2C+line+244%2C+in+compute_first_rows_from_streaming_response%0A++++++++++++++++++iterable_dataset+%3D+iterable_dataset._resolve_features%28%29%0A++++++++++++++++File+%22%2Fusr%2Flocal%2Flib%2Fpython3.14%2Fsite-packages%2Fdatasets%2Fiterable_dataset.py%22%2C+line+4523%2C+in+_resolve_features%0A++++++++++++++++++features+%3D+_infer_features_from_batch%28self.with_format%28None%29._head%28%29%29%0A++++++++++++++++++++++++++++++++++++++++++++++++++++++++%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%5E%5E%0A++++++++++++++++File+%22%2Fusr%2Flocal%2Flib%2Fpython3.14%2Fsite-packages%2Fdatasets%2Fiterable_dataset.py%22%2C+line+2768%2C+in+_head%0A++++++++++++++++++return+next%28iter%28self.iter%28batch_size%3Dn%29%29%29%0A++++++++++++++++File+%22%2Fusr%2Flocal%2Flib%2Fpython3.14%2Fsite-packages%2Fdatasets%2Fiterable_dataset.py%22%2C+line+2972%2C+in+iter%0A++++++++++++++++++for+key%2C+pa_table+in+ex_iterable.iter_arrow%28%29%3A%0A+++++++++++++++++++++++++++++++++++++++%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%5E%5E%0A++++++++++++++++File+%22%2Fusr%2Flocal%2Flib%2Fpython3.14%2Fsite-packages%2Fdatasets%2Fiterable_dataset.py%22%2C+line+2483%2C+in+_iter_arrow%0A++++++++++++++++++yield+from+self.ex_iterable._iter_arrow%28%29%0A++++++++++++++++File+%22%2Fusr%2Flocal%2Flib%2Fpython3.14%2Fsite-packages%2Fdatasets%2Fiterable_dataset.py%22%2C+line+547%2C+in+_iter_arrow%0A++++++++++++++++++for+key%2C+pa_table+in+iterator%3A%0A+++++++++++++++++++++++++++++++++++++++%5E%5E%5E%5E%5E%5E%5E%5E%0A++++++++++++++++File+%22%2Fusr%2Flocal%2Flib%2Fpython3.14%2Fsite-packages%2Fdatasets%2Fiterable_dataset.py%22%2C+line+430%2C+in+_iter_arrow%0A++++++++++++++++++for+key%2C+pa_table+in+self.generate_tables_fn%28**gen_kwags%29%3A%0A+++++++++++++++++++++++++++++++++++++++%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%7E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%5E%0A++++++++++++++++File+%22%2Fusr%2Flocal%2Flib%2Fpython3.14%2Fsite-packages%2Fdatasets%2Fpackaged_modules%2Fjson%2Fjson.py%22%2C+line+327%2C+in+_generate_tables%0A++++++++++++++++++raise+e%0A++++++++++++++++File+%22%2Fusr%2Flocal%2Flib%2Fpython3.14%2Fsite-packages%2Fdatasets%2Fpackaged_modules%2Fjson%2Fjson.py%22%2C+line+290%2C+in+_generate_tables%0A++++++++++++++++++pa_table+%3D+paj.read_json%28%0A++++++++++++++++++++++io.BytesIO%28batch%29%2C+read_options%3Dpaj.ReadOptions%28block_size%3Dblock_size%29%0A++++++++++++++++++%29%0A++++++++++++++++File+%22pyarrow%2F_json.pyx%22%2C+line+364%2C+in+pyarrow._json.read_json%0A++++++++++++++++File+%22pyarrow%2Ferror.pxi%22%2C+line+155%2C+in+pyarrow.lib.pyarrow_internal_check_status%0A++++++++++++++++File+%22pyarrow%2Ferror.pxi%22%2C+line+92%2C+in+pyarrow.lib.check_status%0A++++++++++++++++++raise+convert_status%28status%29%0A++++++++++++++pyarrow.lib.ArrowInvalid%3A+JSON+parse+error%3A+Invalid+value.+in+row+0%0A%60%60%60%0A%0A%0Acc+%40lhoestq+%40cfahlgren1.) for direct support.\n\n# \n\t\tSinabis Oil – Synthetic Email Dataset\n\t\n\nMore than **1.25 million emails** from the everyday business of a fictional\nenergy company. They span eight and a half years and four languages, and they\ninclude real attachments and calendar invitations. Every person, company and\nevent is made up. The content was written by large language models, and the\n`demo-gen` generator built the structure around it.\n\nAll figures in this file were **computed from the files themselves**\n(as of 1 Oct 2026). They are not target values from the configuration.\n\n## \n\t\tAt a glance\n\t\n\n| Emails | **1,251,102** (13.1 GB,`.eml` per RFC 5322) | \n| Business cases | **376,417** : 262,776 email threads and 113,641 single emails | \n| Meetings | **9,470** meetings with 45,483 calendar emails, plus 3,020 standalone`.ics` files | \n| Document attachments | **57,890** : 33,467 PDF · 13,485 DOCX · 10,938 XLSX | \n| Time span | January 2018 to July 2026, about 146,000 emails per year | \n| Languages | English, German, French, Italian | \n| People | 140 employees in 15 departments, 60 external contacts | \n| Organisations | 1 main company + 39 others (group companies, suppliers, authorities, customers) | \n\n## \n\t\tWhat it is about\n\t\n\n**Sinabis Oil & Gas GmbH** (Wilhelmshaven, Germany, fictional) runs drilling\noperations in the North Sea, a refinery, pipelines and tank farms. The emails\nshow the day-to-day business of such an operation. There are **29 topics**,\nincluding:\n\nNorth Sea drilling campaign · refinery turnaround · pipeline inspection · HSE incident report · IT security incident · data protection case · audit · supplier tender · CAPEX budget · shift planning · environmental permit · offshore emergency power · HR topics and more.\n\nAbout 12 % of the emails are private or informal exchanges between colleagues. Single emails are newsletters, memos, circulars, payment reminders and maintenance notices.\n\n### \n\t\tWho is writing?\n\t\n\n| Group | Count | Examples | \n|---|---|---|\n| Sinabis Oil & Gas GmbH | 140 people, 15 departments | E&P, Drilling, Refinery, HSE, Procurement, IT, Legal/Compliance | \n| Sinabis group companies | 21 | Sinabis Logistik, Sinabis Finanzdienst, Sinabis Consulting | \n| Suppliers and service providers | 11 | Nordsee Rig Services, Baltic Drilling Supplies, Friesland Inspection | \n| Authorities | 4 | Landesbergamt Nord, Hafenbehörde Wattkant, Umweltagentur Küstenland | \n| Energy customers | 3 | Ostsee Stromwerk, Nordlicht Kraftwerke, Küstenstadt Wärme | \n\nActivity is **unevenly distributed**, like in a real mailbox: about 145\ninternal senders write 99 % of the internal emails, some of them tens of\nthousands each. People only appear during their period of employment.\n\n## \n\t\tLanguage distribution\n\t\n\nThe language of each email was determined from its subject and its own text, without the quoted history, using stop-word detection. Very short texts, such as calendar replies like \"Accepted: …\" or \"Zugesagt: …\", were assigned by their subject line.\n\n| Language | Main set | Earlier set | **Total** | **Share** | \n|---|---|---|---|---|\n| English | 481,765 | 199,677 | **681,442** | **54.5 %** | \n| German | 306,392 | 32,991 | **339,383** | **27.1 %** | \n| French | 102,762 | 26,694 | **129,456** | **10.3 %** | \n| Italian | 97,069 | 3,752 | **100,821** | **8.1 %** | \n| **Total** | 987,988 | 263,114 | **1,251,102** | | \n\nEvery email is **monolingual**: subject, body, signature, disclaimer, legal\nnotice and history header are all in the same language.\n\n**Attachments by language**\n\n| | | DOCX | XLSX | \n|---|---|---|---|\n| English | 20,021 | 6,690 | 5,323 | \n| German | 7,965 | 4,084 | 3,482 | \n| French | 3,365 | 1,352 | 1,128 | \n| Italian | 2,116 | 1,359 | 1,005 | \n\n## \n\t\tModels used\n\t\n\nIn the main set, the tag at the end of the file name (`…-V1.eml`, `…-QW.eml`)\nshows which model wrote the email.\n\n| Tag | Model | Emails | Share | \n|---|---|---|---|\n| `V1` –`V6` | Qwen3.6-35B-A3B (AWQ, 4-bit) | 781,765 | 62.5 % | \n| *(none)* | DeepSeek V4 Flash (no thinking), incl. all 263,114 of the earlier set | 335,068 | 26.8 % | \n| `QW` | Qwen 3.6 | 88,786 | 7.1 % | \n| *(none)* , calendar | **no LLM** , rule-based | 45,483 | 3.6 % | \n\nThe PDF, DOCX and XLSX attachments are real files. The model only supplies the content; the generator builds the files itself.\n\n## \n\t\tDownloads\n\t\n\nThe dataset is split into archives so you only need to download what you're interested in:\n\n| Archive | Contents | \n|---|---|\n| `Sinabis_Demo_Dataset_emails.zip` | 1,251,102 `.eml` files | \n| `Sinabis_Demo_Dataset_chats.zip` | chat/messenger conversations | \n| `Sinabis_Demo_Dataset_appointments.zip` | standalone `.ics` calendar files | \n\nEach archive keeps the internal folder structure shown below. `README.md` and\n`docs/` are plain files in the repo, not zipped.\n\n## \n\t\tFolder structure\n\t\n\n```\n.\n├── README.md\n├── docs/\n│ └── generation/\n│ ├── prompts.json\n│ ├── chat_prompts.json\n│ └── company.json\n├── Sinabis_Demo_Dataset_emails.zip\n│ └── batch-0001 … batch-0031/\n├── Sinabis_Demo_Dataset_chats.zip\n│ └── batch-0001 … batch-0002/\n└── Sinabis_Demo_Dataset_appointments.zip\n```\n\nThe batch folders only split the data so that no single folder holds a million files. They carry no meaning.\n\n## \n\t\tHow everything is linked\n\t\n\n### \n\t\tEmail threads\n\t\n\n```\n--.eml\n │ │ └─ model (see above), missing for DeepSeek and calendar\n │ └────── position in the thread (01, 02, …)\n └────────────────── SHA-256 over the whole thread, same for all its emails\n```\n\n- All emails of one case share the **same file name prefix** .\n- The headers link the emails: `Message-ID` ,`In-Reply-To` and`References` each point to the previous email.\n- Every email contains the **quoted history** (`> …` ) of all earlier emails,\nso a single file already shows the whole thread.\n- 99 % of threads have **3 to 6 emails** .\n\n### \n\t\tAttachments\n\t\n\n- Attachments are **embedded in the email** (MIME, Base64), not stored as\nseparate files next to it.\n- The document is the **source the email refers to** , for example an\ninspection report or a budget sheet, not a copy of the email text. It has the\nsame language and the same date as the email.\n\n### \n\t\tMeetings\n\t\n\n- A meeting consists of **several emails with an `.ics` attachment** : the\ninvitation (`REQUEST` ) to all attendees, acceptances or declines (`REPLY` )\nto the organiser, and possibly a cancellation (`CANCEL` ).\n- These emails share the same file name prefix. 9,470 meetings produce 45,483 calendar emails this way.\n- The standalone `.ics` files in`Sinabis_Demo_Dataset_appointments.zip` are\nmeetings without an email (`appt___