{"slug": "show-hn-let-formal-specs-test-ai-implementations", "title": "Show HN: Let formal specs test AI implementations", "summary": "SpecGuardian, a new open-source tool from developer mrasu, generates Go conformance tests from FizzBee formal specifications to check whether an AI-generated implementation reaches only states the spec allows. In the included cache-aside tutorial, SpecGuardian injects an I/O error immediately before cache invalidation and compares the observed state against spec-permitted states; a reordered WriteAndInvalidate that deletes the cache entry before writing to the database fails the test with \"actual state does not match any allowed state,\" leaving the database holding \"old\" while the cache entry is gone. The example runs with Go and Docker Compose via `go test -count=1 -tags=specguardian ./conformance/...`, with state observation and providers configured in `specguardian.yml`.", "body_md": "Let formal specs test AI implementations.\n\nA [FizzBee](https://fizzbee.io/) spec describes the state transitions your system allows. However, the spec does not test whether your implementation follows those transitions. SpecGuardian turns FizzBee's results into tests that check whether your implementation reaches an allowed state.\n\nSpecGuardian lets you verify that an AI-generated implementation actually conforms to your specification.\n\nIn this cache-aside example, a `WriteAndInvalidate` action stores a new value in the database, then invalidates the corresponding cache entry. The [tutorial spec](https://github.com/mrasu/spec-guardian/blob/main/examples/tutorial_cache_aside_write/docs/spec/CacheAside.fizz) models each operation as atomic and allows a crash before or between them:\n\n```\n@conformance\nrole CacheAside:\n  fair action WriteAndInvalidate:\n    # Yield to allow a crash before the first I/O.\n    atomic:\n      pass\n\n    # Write to the database first, then invalidate the cache.\n    db.write(self.key, self.write_value)\n    cache.delete_cache(self.key)\n```\n\nSpecGuardian [generates Go tests](https://github.com/mrasu/spec-guardian/blob/main/examples/tutorial_cache_aside_write/conformance/specguardian/cacheaside_conformance_generated_test.go) from FizzBee's results. They compare the observed state after the action with states allowed by the spec, including states reached when an I/O operation fails.\n\nFor example, the generated Go test injects an error into the implementation just before cache invalidation, then checks whether the resulting state is allowed by the spec.\n\nConsider an ordering bug in `WriteAndInvalidate`: cache invalidation runs before the database write.\n\n```\nfunc (ca *CacheAside) WriteAndInvalidate(ctx context.Context, input *Input) (*Output, error) {\n+  if err := ca.cache.Delete(ctx, input.Key); err != nil {\n+    return nil, err\n+  }\n+\n   if err := ca.db.Write(ctx, input.Key, input.Value); err != nil {\n     return nil, err\n   }\n\n-  if err := ca.cache.Delete(ctx, input.Key); err != nil {\n-    return nil, err\n-  }\n-\n   return &Output{Key: input.Key, Value: input.Value}, nil\n}\n```\n\nBoth orders produce the same final state on success. But when the database write fails after cache invalidation, the database still holds `old` value and the cache entry is gone. The spec writes to the database first, so stopping at any point cannot produce this combination.\n\nThe test fails with the following output (excerpt):\n\n```\nactual state does not match any allowed state:\n    I/O events:\n      1. valkey.Do\n      2. sql.BeginTx [fault injected; call not executed]\n```\n\nThe mismatch excerpts show the unchanged database value and missing cache entry:\n\n```\nFields.DB.Records[0].Value (-want +got):\n  string(\n-  \"new\",\n+  \"old\",\n  )\nFields.Cache.CachedRecords (-want +got):\n  map[string]string{\n-  \"cached\": \"old\",\n  }\n```\n\nTo try this example locally, start with the included cache-aside write tutorial. You'll need Go and Docker Compose; the FizzBee exploration results and generated tests are included.\n\nClone the repository and run the example:\n\n```\ngit clone https://github.com/mrasu/spec-guardian.git\ncd spec-guardian/examples/tutorial_cache_aside_write\ndocker compose up -d\ngo test -count=1 -tags=specguardian ./conformance/...\n```\n\nThe test passes, checking normal completion and injected I/O failures. Apply the diff above to `cache_aside.go` and rerun the `go test` command to reproduce the failure. Restore the original order to make the test pass again.\n\nHere's how SpecGuardian works:\n\n1. FizzBee explores the possible states described by the spec.\n2. SpecGuardian generates Go tests from the exploration results.\n3. Either AI or you write the setup and Action execution code, configure providers in `specguardian.yml` , and write any remaining state observation code.\n4. The tests run a function normally and with an error injected immediately before a supported I/O call, then compare the observed state with states allowed by the spec.\n\n`specguardian.yml` configures state observation and comparison. See the [configuration schema](https://github.com/mrasu/spec-guardian/blob/main/docs/specguardian.schema.yaml) for available settings and the [example configuration](https://github.com/mrasu/spec-guardian/blob/main/examples/tutorial_cache_aside_write/specguardian.yml) for an example.\n\nSpecGuardian checks whether an Action ends in a state allowed by the FizzBee spec, including after an injected I/O failure. It complements other tests:\n\n- **Different from unit tests:** It checks the resulting application state against the spec, beyond assertions about individual functions.\n- **Different from integration tests:** The expected states come from a FizzBee spec defined independently of the implementation.\n\nTo generate tests for your project, install [FizzBee](https://fizzbee.io/) and run these commands from your Go module directory:\n\n```\nfizz --output-dir path/to/output_dir path/to/YourSpec.fizz\ngo get -tool github.com/mrasu/spec-guardian\ngo tool spec-guardian init --fizzbee-output-dir path/to/output_dir\n\n# Configure observation and comparison.\n$EDITOR specguardian.yml\n\ngo generate ./conformance\n\n# Replace <role> with the generated role name and connect the tests to your application.\n$EDITOR \"conformance/specguardian/<role>_conformance_impl_test.go\"\n\ngo mod tidy\ngo test -tags=specguardian ./conformance/...\n```\n\nSee the [user guide](https://github.com/mrasu/spec-guardian/blob/main/docs/user-guide.md) for the complete setup.\n\n- [Cache-aside write tutorial](https://github.com/mrasu/spec-guardian/blob/main/examples/tutorial_cache_aside_write) is the completed project from the[user guide](https://github.com/mrasu/spec-guardian/blob/main/docs/user-guide.md) .\n- [Cache-aside](https://github.com/mrasu/spec-guardian/blob/main/examples/cache_aside) demonstrates the cache-aside pattern.\n- [Idempotent request](https://github.com/mrasu/spec-guardian/blob/main/examples/idempotent_request) demonstrates idempotent requests.\n- [Iceberg](https://github.com/mrasu/spec-guardian/blob/main/examples/iceberg) is a real-world example that tests[apache/iceberg-go](https://github.com/apache/iceberg-go) .\n\nSpecGuardian currently generates Go tests for one FizzBee Action invocation at a time. See the [guardian directory](https://github.com/mrasu/spec-guardian/blob/main/guardian) for the available I/O hooks.\n\nSpecGuardian is under development. Its public APIs or generated Go code may change.", "url": "https://wpnews.pro/news/show-hn-let-formal-specs-test-ai-implementations", "canonical_source": "https://github.com/mrasu/spec-guardian", "published_at": "2026-11-01 14:11:06+00:00", "updated_at": "2026-11-01 14:20:01.674329+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-agents", "ai-safety"], "entities": ["SpecGuardian", "FizzBee", "mrasu", "Go", "Docker Compose", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-let-formal-specs-test-ai-implementations", "markdown": "https://wpnews.pro/news/show-hn-let-formal-specs-test-ai-implementations.md", "text": "https://wpnews.pro/news/show-hn-let-formal-specs-test-ai-implementations.txt", "jsonld": "https://wpnews.pro/news/show-hn-let-formal-specs-test-ai-implementations.jsonld"}}