# I Built a Company That Doesn't Exist to Test an AI Product > Source: > Published: 2026-08-28 02:57:13+00:00 *These articles come from lessons learned while building Eterna Clarity and the operating system I use to run it.* One of the hardest test environments I have built for an AI product is a company that does not exist. The underlying problem is less strange than that sentence: Clarity is supposed to take the kinds of files and photos people already have, process them, organize them and turn them into something usable. Testing that with real customer data before launch creates an obvious privacy problem. Testing it with a folder full of toy files creates a different problem: the product can look excellent because the test world is unrealistically easy. So I needed synthetic data, and then discovered that making synthetic files is easy. Making a synthetic business believable enough to expose real product failures is much harder. The first version of the Business corpus passed plenty of automated checks. The PDFs opened, the DOCX files parsed, the spreadsheets were valid, the first pages had visual variation, and the file counts and formats were where I expected them to be. There were invoices, operational documents, spreadsheets, exports and presentations. Then I looked at the corpus the way a customer might. Some documents were barely populated. Some spreadsheets had only a few rows. Data followed obvious algorithmic patterns. Explanatory boilerplate appeared where an actual business would have transaction detail. A file could satisfy the technical definition of “invoice” without looking like something a vendor would ever send. The test data was structurally valid and operationally ridiculous. That distinction matters for AI products because models are extremely good at exploiting regularity. If every invoice is clean, short and laid out the same way, you may be measuring how well the system handles your generator rather than how well it handles invoices. If a spreadsheet has four rows, you are not learning what happens when the model has to reason across 300. If every business document is independent, you are not testing whether the system can connect an invoice to the purchase order, packing slip, credit memo and monthly statement that belong to the same transaction. A benchmark can be perfectly reproducible and still be a weak representation of reality. The funniest failure was also one of the most useful. AI had generated a business story around an assumed company identity. Then another pass shifted the story again and started treating a vendor as though it were the business itself. The files were synthetic, but that did not mean the relationships were allowed to be arbitrary. I had already reviewed and retained the supporting image set, and those images contained evidence. Across the 58 retained Business images, Harbor Lane appeared repeatedly as the operating company. Equipment carried HLS identifiers. Opening and closing checklists named Harbor Lane Services. Insurance and service records pointed to the same entity. Northline Packaging appeared in shipping and procurement material. Cedar Table Cafe appeared as a recurring customer/project. One damaged delivery carton made the relationship almost embarrassingly clear: FROM Northline Packaging, TO Harbor Lane Services, with an order number and packing-slip number printed on the box. At that point the right response was not to generate a prettier story. It was to reconstruct the synthetic business from its own evidence. Harbor Lane Services became the company. Northline Packaging became the vendor it had always been in the image evidence. Cedar Table Cafe / JOB-1047 became a recurring client-project anchor, and other vendors and assets were kept only in roles the retained source material could support. The strange lesson was that synthetic data still needs provenance. If one generated artifact becomes the reason another generated artifact exists, the test environment can drift into a self-reinforcing fiction. You need some authority that says which parts of the synthetic world are fixed and which parts are allowed to vary. Once the business identity was grounded, the next rebuild changed the unit of design. I was no longer asking whether I could create 34 realistic-looking files; I was asking what this business would have to be doing for those 34 files to exist. That produced a much better corpus. A Northline purchase order connects to an order confirmation, packing slip, damaged-delivery evidence, invoice, credit memo and account statement. Cedar Table JOB-1047 has a quote, work order, change order, completion record and handoff material. Equipment HLS-EQ-018 appears across maintenance and service history. Monthly operating records connect to expenses, fuel, mileage, timesheets, inventory and vendor relationships. The identifiers recur deliberately because real business records do not live as 34 isolated short stories. In the final validation, JOB-1047 appeared across the corpus 193 times, HLS-EQ-018 appeared 66 times, the Northline order NP-24091 appeared 29 times and its packing slip PK-24077 appeared 16 times. Those counts are not targets by themselves; they are evidence that the test world contains relationships a processing system can either preserve or destroy. That gives me something much more valuable than asking whether the model understands one PDF. I can ask whether the whole system understands that several files belong to the same business event. The final accepted born-digital layer contains 34 files: 22 PDFs, four DOCX records, three XLSX workbooks, three CSV exports, one long operating plan and one training deck. The number of files is less interesting than the workload inside them. The PDFs total 69 pages. The three workbooks each have eight operational sheets and together contain roughly 1,550 data rows and 1,800 formulas. The CSV exports add 252 timesheet rows, 143 continuous mileage trips and 68 operational contacts. The monthly operating plan is more than 5,000 words, and the training deck is 20 substantive slides. That scale is deliberate. A product that performs well on a five-line invoice and a four-row spreadsheet may fail differently when the same job contains hundreds of rows, repeated vendors, formulas, project references, dates, exceptions and partially redundant evidence. Retrieval changes. Summarization changes. Cost changes. Context selection changes. Error propagation changes. This is one reason I do not like reducing benchmark design to a file count. A 40-page packet is not one unit of work in the same sense as a photograph. An eight-sheet inventory workbook is not equivalent to a one-page receipt. Realistic evaluation has to account for processing extent as well as source count. The process gave me a more useful way to think about synthetic test data. I now look for at least five layers of realism: structural realism, document realism, entity realism, transactional realism and workload realism. Structural realism asks whether the file actually behaves like the format it claims to be: can it be opened, parsed, rendered and processed without corruption? Document realism asks whether the invoice looks like an invoice, whether an insurance packet contains the density and schedules that type of packet normally contains, and whether a spreadsheet has formulas and operational sheet roles rather than a decorative grid. Entity realism asks whether companies, vendors, customers, assets and people remain in coherent roles. Transactional realism asks whether dates, quantities, references, amounts and statuses reconcile when several files describe the same event. Workload realism asks whether the mixture is difficult in the same ways real customer data will be difficult: long documents, short documents, exports, images, messy batches, repeated entities, historical records and edge cases. A corpus can pass the first layer and fail all four others. That was exactly what happened to mine. The answer is not to replace automation with “looks good to me.” The final rebuild has aggressive automated validation. Every file is hashed. PDF pages are rendered and checked for density. DOCX depth, tables and pagination are measured. XLSX formulas, row counts and reference errors are audited. CSV schemas and row counts are checked. The presentation is slide-boundary tested. Forbidden placeholder language is scanned. Cross-file identifiers are counted. Candidate, active and benchmark copies are hash-compared. The final v2 promotion passed with zero validation issues and zero warnings, and all 34 candidate files matched the 34 active files and 34 benchmark copies exactly. Those checks became useful only after the acceptance criteria represented the thing I actually cared about. The first corpus also had automated checks. They simply proved the wrong claim: that the files existed, opened and varied structurally. They did not prove that an experienced business operator would believe the records came from a functioning company. That is a recurring evaluation mistake — improving the measurement system without first asking whether the measurement corresponds to the real-world failure you are trying to prevent. A major reason to use synthetic test data is obvious: I can build an aggressive public-demo and benchmark corpus without putting real customer records at risk. But “synthetic” is not itself a quality standard. NIST's work on synthetic data separates privacy from utility and fidelity for a reason. Data can be safe to share and still be useless for the task you want to test. Recent work on realistic AI evaluations is moving in the same direction. OpenAI's GDPval, for example, deliberately uses work products based on real occupational tasks because academic-style benchmarks often do not represent what people actually do at work. I think the same principle applies at a smaller product level. If your product is supposed to organize a business, test it on something that behaves like a business. If it is supposed to understand messy household records, give it a household with repeated people, purchases, equipment, warranties, photos and documents that overlap. The goal is not photorealism for its own sake; it is to create the dependencies, ambiguity, scale and inconsistency that make the production problem hard. That is now the standard I care about most. A good test environment should not be designed to make the product look intelligent. It should be designed to give the product enough reality to fail honestly. That means a synthetic world needs history. It needs entities that recur, long boring documents, exceptions, files that disagree in useful ways and files that are redundant in realistic ways. It needs enough scale that shortcuts become visible, and it needs a source of truth for the parts that cannot drift. It also needs human review because some failures are obvious to a person long before they are captured by a metric. The Harbor Lane corpus is completely synthetic. No customer had to give me their invoices, insurance packet, mileage history or staff timesheets to build it. But the problems it is designed to expose are very real. That is the point. *AI disclosure: This article is based on my own Eterna build notes and experience. I used AI as a drafting and editing partner; I reviewed the final piece and stand behind the technical substance and claims.*