Ergo Framework – Erlang/OTP-style distributed systems in Go

Ergo Framework 3.3.0, an Erlang/OTP-style distributed systems toolkit for Go, adds distributed tracing, a testing framework, and an Observer that serves as a full cluster inspector with an MCP endpoint for AI agents. The Saturn module is now MIT licensed, making the entire ecosystem free for production use. The release includes breaking changes requiring applications to embed app.Application.

Ergo Framework 3.3.0 is about seeing what a running cluster is doing, and proving what it will do before it runs. A trace now follows a message across nodes, the Observer became a full cluster inspector that an AI agent can drive, and a testing framework ships with the framework itself. Saturn is MIT from this release on, so every module of the ecosystem is free for production use. Highlights Distributed tracing . A trace context propagates across Send , Call , spawn and every hop a message takes between nodes, with configurable sampling and cross-node clock-skew correction. The new Pulse application exports the collected spans over OTLP to Tempo, Jaeger or anything else that speaks it, and ships a Grafana dashboard. Testing framework . ergo.services/ergo/testing gives four layers on one fluent assertion grammar: unit runs a single actor against a mock node, stage runs real nodes over the real protocol, mock provides standalone gen. mocks for code that consumes the interfaces, and check is the assertion core they share. You assert over messages, calls, spawns, links, monitors and events, positively and negatively. Observer, and MCP for AI agents . The Observer application is now a full cluster inspector: node info with live memory graphs, the process list with mailbox depth, latency, running time and wakeups, per-process detail with supervision tree and live HandleInspect state, meta processes, applications, the network stack and its wire-type registry, events, a live log stream, and on-demand goroutine and heap profiles with no special build. The same application serves an MCP endpoint: 38 tools and 13 resource lenses over the live cluster, 26 of them read-only, with capability ceilings that can only narrow. One Observer inspects every node in the cluster , because every node already runs the built-in system application. act.Router . A behavior that picks the destination from the message instead of taking the next worker in order. It can own its routes or resolve them through the node registry, the routing table is editable while it runs, and high-priority traffic bypasses routing so an admin path stays open while the routed path is busy. EDF schema evolution . With NetworkFlags.EnableSchemaEvolution on both nodes, fields appended to the end of a registered struct stay wire-compatible: a peer that does not know them skips them, a peer missing them zero-fills. Off by default, so EDF stays strict unless you ask for it. Grid . A new application giving a distributed registry, key lifecycle monitors and per-key process groups. Every node keeps a full local copy, so a lookup is a local read; writes are serialized per key by a shard actor and replicate in the background. AP and eventually consistent. argus . A vet tool for the actor-model invariants the compiler cannot express: shared memory in a message, an unbounded wait in a callback, a goroutine reaching back into actor state, a HandleCall that can never reply, and thirty-one more. Runs as go vet -vettool=$ which argus ./..., with a baseline mode for codebases that have never seen it. Breaking changes Applications must embed app.Application . ApplicationBehavior gained lifecycle callbacks and Load lost its node gen.Node parameter: // before type MyApp struct{} func a MyApp Load node gen.Node, args ...any gen.ApplicationSpec, error { ... } func a MyApp Start mode gen.ApplicationMode {} // after type MyApp struct{ app.Application } func a MyApp Load args ...any gen.ApplicationSpec, error { ... } The embed supplies the framework plumbing and no-op defaults for the new callbacks. If you used node inside Load , switch to a.Node . If your Start was an empty stub, delete it; if it had logic, its signature is now Start ref gen.Ref, mode gen.ApplicationMode . Step-by-step in the Application documentation. Also - gen.ProcessBehavior requires ProcessKind gen.ProcessKind . The base behaviors act.Actor , act.Supervisor , act.Pool , act.Router , act.WebWorker report their own, so only hand-written behaviors are affected. - The trace build tag is now verbose : go run --tags verbose ./cmd. Also in this release - Network . Automatic fragmentation of large messages, application-level keepalive that catches the silent failures TCP keepalive cannot, a handshake deadline, and a fix for the dead loop two nodes could enter when dialing each other at the same time. - Observability without pauses. Node.Info and the heap inspector no longer stop the world; memory and GC figures come from runtime/metrics. New NodeShortInfo / RemoteNodeShortInfo / ProcessShortInfo are the cheap counterparts for polling a whole cluster, alongside per-event metrics, process lifecycle counters, per-process running time, wakeups, init time, and mailbox latency under -tags=latency . - Application teardown. An application's Terminate now runs after every one of its processes has terminated, its own Terminate included, so closing a shared resource there is safe. Detached processes are stopped with the application instead of outliving it. - Errors that keep their causes. gen.Errorf collects %w arguments so errors.Is and errors.As still find them, and the chain survives the wire when both nodes enable NetworkFlags.EnableWrappedErrors. - Radar , Health , Sentry . One application serving Kubernetes probes and a Prometheus endpoint on one port; the Health actor behind it; and a logger that forwards framework panics and errors to Sentry with stack traces from the panic site. - Metrics actor expanded well beyond node/network telemetry, with a complete Grafana cluster dashboard. - Target Manager rewritten lock-free. One lock used to serialize every link, monitor and event subscription on the node; a node now carries hundreds of thousands of them without contention - Saturn moved from Business Source License 1.1 to MIT. No node cap, no licence to buy. Documentation New articles on docs.ergo.services https://docs.ergo.services : - FAQ https://docs.ergo.services/faq - the questions that come up before the first node starts - AI Agents https://docs.ergo.services/ai-agents - the skills and agents shipped for working on an Ergo codebase - Router https://docs.ergo.services/actors/router - act.Router , its routing modes and runtime management - Distributed Tracing https://docs.ergo.services/advanced/distributed-tracing - samplers, business spans and exporters - Inspecting Actor State https://docs.ergo.services/advanced/inspecting-state - HandleInspect and reading a live process - Inspecting With Observer https://docs.ergo.services/advanced/observer - the cluster inspector, panel by panel - Inspecting With an AI Agent https://docs.ergo.services/advanced/mcp - the MCP surface, its lenses and capability ceilings - Actor Model Vet Tool https://docs.ergo.services/tools/argus - argus , its rules and how to adopt it on existing code Testing: - Overview https://docs.ergo.services/testing/overview - the four layers and which one answers which question - Check https://docs.ergo.services/testing/check - the assertion grammar shared by every layer - Mock https://docs.ergo.services/testing/mock - standalone gen. mocks for code that consumes the interfaces Extra library: - Grid https://docs.ergo.services/extra-library/applications/grid - distributed registry, key monitors and process groups - Pulse https://docs.ergo.services/extra-library/applications/pulse - OTLP exporter for the collected spans - Radar https://docs.ergo.services/extra-library/applications/radar - health probes and Prometheus metrics on one port - Health actor https://docs.ergo.services/extra-library/actors/health - Kubernetes liveness, readiness and startup probes - Sentry logger https://docs.ergo.services/extra-library/loggers/sentry - forwards panics and errors to a Sentry project Full list of changes: CHANGELOG.md