Wide Events is a Rails gem that puts the production context a coding agent needs onto one OpenTelemetry root span per request or job.
In one production search request, the root event showed 30.0 seconds total duration, 446 ms of Postgres time, and 29.4 seconds of outbound HTTP time. That was enough to focus the investigation on an external dependency. The trace then identified a POST
that took 28.9 seconds.
The trace contained 82 spans and 20,261 bytes of attribute JSON. The root event contained 40 attributes and 1,420 bytes. This is not a token benchmark, but it shows why the root event is a more compact starting point for an agent.
A coding agent starts with an unusual advantage: it can search every model, controller, job, migration, and test in a few seconds.
It also starts with a serious blind spot. The repository cannot tell it:
Those answers often exist somewhere, but “somewhere” might mean a trace waterfall, application logs, a feature-flag service, product analytics, and a database console. Pulling all of that into a context window is expensive and usually requires several joins that were never designed in advance.
A wide event changes the starting point. The app accumulates the context it learns while processing one unit of work, then attaches the completed flat map to the OpenTelemetry root span. The span is marked main=true
, so every request or job can be queried as one row.
request or job
-> Rails and domain context accumulate
-> child spans contribute dependency counts and timings
-> one flat map is flushed onto the root span
-> ClickHouse stores one queryable row
The trace still exists. Wide Events gives it an application-shaped index.
If you do not run tracing, the same map can be emitted as one JSON log line.
Wide Events requires Ruby 3.2 or newer and Rails 7.1 or newer.
gem "wide_events"
bundle install
bin/rails generate wide_events:install
For a first local event, no tracing setup is required:
WideEvent.configure do |config|
config.enabled = true
config.sink = :log
end
Hit a route and the event appears as a single JSON line. In production, the default sink writes the attributes onto the current OpenTelemetry root span. The deployment behind this article exports those spans to self-hosted ClickStack, which combines ClickHouse and HyperDX. The gem itself works with an OTLP backend you already use.
The gem supplies request and Active Job boundaries. The useful application context is deliberately yours to define.
Rails knows more about a request than generic instrumentation ever can. After authentication, it knows the actor and account. When a flag is evaluated, it knows the exact variant. After a search completes, it knows whether semantic search ran and what survived its thresholds.
That is where I add attributes:
WideEvent.set(
"user.type" => Current.person&.class&.name,
"user.account.id" => Current.account&.id
)
Opaque identifiers can still be sensitive. Whether to record them is a policy decision, not something the gem can decide. Names, email addresses, request parameters, and free text do not belong in this event by default.
For a hybrid search:
WideEvent.set(
"search.result_count" => results.size,
"search.semantic" => search.semantic?,
"search.semantic_fetched" => search.semantic_fetched,
"search.semantic_kept" => search.semantic_kept,
"search.semantic_top_cosine" => search.semantic_top_cosine
)
For a phase whose duration matters:
report = WideEvent.phase("report_upload") do
upload_report
end
That adds report_upload.duration_ms
and returns the block's result.
Handled failures get stable names:
rescue Search::Unavailable => error
WideEvent.error!(
slug: "err-search-unavailable",
exception: error,
expected: true
)
keyword_results
end
An exception that escapes the request or job is recorded automatically with error=true
and no slug. The missing slug distinguishes an escaped exception from a failure the application intentionally handled and named.
Instrumentation has a never-raise contract. Attribute writes outside an open request or job are safe no-ops, and failures inside the telemetry code are reported without being raised into application code. The block passed to phase
behaves like normal application code, including propagating its own exceptions.
Free-form attributes become a mess when nobody owns their meaning. Wide Events therefore treats its registry as executable schema rather than documentation.
search.semantic:
type: boolean
set_by: SearchController#index
pii: none
notes: False when the keyword-only fallback ran.
search.semantic_top_cosine:
type: float
set_by: Search
pii: none
The production workflow is intentionally strict:
This costs more than inventing a key in a dashboard. In return, both humans and agents get stable names, declared types, ownership, and a place to review privacy.
“One event per request” should not be confused with “one event per user action.” Asynchronous work crosses a real execution boundary and should remain visible.
I inspected one production reply-draft flow. The initiating request and the generation job were linked by an opaque generation identifier, but they were correctly emitted as separate events:
| Field | Request | Generation job |
|---|---|---|
| Duration | 230.6 ms | 3,159.7 ms |
| HTTP status | 204 | n/a |
| Postgres queries | 59 | 14 |
| Model latency | n/a | 2,995.96 ms |
| Input tokens | n/a | 3,035 |
| Output tokens | n/a | 112 |
| Cost | n/a | $0.009268 |
The job event also recorded the prompt version, model, input size, counts of thread messages, notes, goals and todos, and the output character count. It did not record the prompt or completion.
That distinction is important in an application subject to healthcare privacy requirements. Shape and outcome are useful for production analysis. Content is both risky and usually unnecessary.
The generation job's full trace contained 35 spans and 288 attributes. Its root event contained 42 attributes. The attribute JSON was 11,534 bytes across the trace and 1,384 bytes on the root event. Again, this does not replace the trace. It gives an agent a compact first read of what happened.
The gem ships two plain-Markdown skills:
bin/rails generate wide_events:skills
One skill teaches an agent how to add instrumentation, including naming, registry, testing, and privacy conventions. The other teaches the read path: start with root events, narrow the cohort, inspect individual traces, and verify a deployed change against a new build.
A first query can be small:
SELECT
Timestamp,
TraceId,
SpanAttributes['http.route.controller'] AS route,
round(Duration / 1e6, 1) AS duration_ms,
SpanAttributes['stats.postgres_query_count'] AS query_count,
SpanAttributes['stats.postgres_query_duration_ms'] AS postgres_ms,
SpanAttributes['stats.http_call_duration_ms'] AS http_ms,
SpanAttributes['error'] AS error
FROM otel_traces
WHERE ResourceAttributes['deployment.environment'] = 'production'
AND SpanAttributes['main'] = 'true'
AND Timestamp > now() - INTERVAL 15 MINUTE
ORDER BY Duration DESC
LIMIT 20
The loop is:
The last step matters. An agent that can edit code but cannot check the running result is still handing the experiment back to a person halfway through.
The skills do not magically grant production access. The agent still needs a read-only query path, appropriate credentials, and limits on what it may retrieve. In this case, ClickStack exposes ClickHouse through MCP, and the production queries explicitly exclude identifiers and free text.
I originally described unnamed exceptions as a ready-made queue of failures that nobody had handled. The real data made that claim less neat.
The query was dominated by expected framework noise, especially routing errors. The missing slug still tells you that application code did not name the path, but it does not prove every row deserves engineering attention.
A practical query needs local exclusions or an explicit expected-error policy:
SELECT
SpanAttributes['exception.type'] AS exception_type,
SpanAttributes['http.route.controller'] AS route,
count() AS occurrences
FROM otel_traces
WHERE ResourceAttributes['deployment.environment'] = 'production'
AND SpanAttributes['main'] = 'true'
AND SpanAttributes['error'] = 'true'
AND SpanAttributes['exception.slug'] = ''
AND SpanAttributes['exception.type'] != ''
AND Timestamp > now() - INTERVAL 7 DAY
AND SpanAttributes['exception.type'] NOT IN (
'ActionController::RoutingError',
'ActionController::TooManyRequests'
)
GROUP BY exception_type, route
ORDER BY occurrences DESC
LIMIT 50
That is one reason I wanted the article grounded in production rows. A tidy API design can still produce a noisy operational query.
I also checked the basic grain before using the data. In the seven-day production slice ending August 4, 2026, the table contained 1,185,563 root events and exactly 1,185,563 distinct trace IDs. There were no duplicate root rows. That does not prove every deployment will be configured correctly, but it does verify the one-root-row invariant in this deployment at meaningful volume.
Wide events are deliberately high-cardinality. You need a backend that handles that shape, plus an explicit retention and privacy policy. ClickHouse is a good fit for my deployment, but operating it is still work.
The registry adds friction. Teams that will not review attribute names or data classification will not get much value from pretending they have a schema.
Root events also cannot contain every detail. They should contain the dimensions and outcomes you expect to group, filter, compare, and hand to an agent. Traces, logs, and profiles remain better for lower-level evidence.
Finally, “compact” is workload-dependent. The two traces in this article had roughly 8x and 14x more serialized span-attribute data than their root events. Those are two measured examples, not a universal compression ratio and not a claim about exact model-token usage.
Wide Events is MIT licensed on GitHub. The log sink produces the first event without requiring an observability stack; the OpenTelemetry sink works with an OTLP backend you already operate.
I extracted the gem from the production Rails application described here. I am especially interested in feedback on three things: the registry workflow, the attribute naming conventions, and whether the generated skills describe an agent verification loop you would actually trust against production.
Disclosure: I used OpenAI Codex to run sanitized, read-only queries against production telemetry, challenge the claims, and edit this article. I reviewed the source code, query results, and every number before publication.