I run AI Change Watch, a small independent project that
It runs on Cloudflare Workers, which means that when someone tells me "your site 500'd an hour ago",
the obvious tool is useless. wrangler tail
is a live stream. It shows you what is happening now.
It cannot show you an hour ago.
There is a way to read the past, and there are four traps in it that cost me most of a day.
Workers can write their invocation logs to a queryable store, and a REST endpoint reads it back. First
the worker has to be opted in — this is the whole config:
// wrangler.jsonc
{
"observability": { "enabled": true }
}
Then you can ask for events in a time range:
curl -sX POST \
"https://api.cloudflare.com/client/v4/accounts/$ACCOUNT/workers/observability/telemetry/query" \
-H "Authorization: Bearer $CF_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"queryId": "anything",
"timeframe": { "from": 1786000000000, "to": 1786003600000 },
"limit": 500,
"view": "events",
"parameters": {
"datasets": ["cloudflare-workers"],
"filters": [
{ "id": "f1", "key": "$workers.event.response.status",
"type": "number", "operation": "eq", "value": 500 }
]
}
}'
from
and to
are epoch milliseconds, not ISO strings. A read-scoped API token is enough — the
one I already had for deploys worked unchanged.
Each event carries more than you would guess:
$workers.outcome ok | exceededCpu | canceled
$workers.cpuTimeMs / $workers.wallTimeMs
$workers.event.request.path / .search
$workers.event.request.headers['user-agent']
$workers.event.request.cf.asOrganization the ASN owner
$workers.event.response.status
$metadata.error
That is the tool. Now the traps.
I started by filtering on the field that sounds right:
{ "key": "$workers.outcome", "operation": "eq", "value": "exceededCpu" }
and found nothing, repeatedly, while the site was demonstrably returning 500s.
Because for every one of them:
$workers.outcome = "ok"
The worker ran. It produced a response. It returned it. That the response was an error page is not the
runtime's problem — the invocation succeeded. outcome
describes the worker, not the HTTP result.
So outcome
is the wrong axis for application errors. Filter on $workers.event.response.status
for
what the user saw, and read $metadata.error
for the throw. outcome
is for failures the runtime
itself noticed: CPU limit, cancellation.
This is worth internalising because it inverts the usual relationship. In most stacks "the request
failed" and "the handler failed" are the same event. At the edge they are two different fields — and
the one with the friendlier name is the one that will not tell you.
This is the one that actually cost me the day.
I asked for 5xx across a 24-hour window, got nine events, and concluded I was chasing a single bug. The
same 24 hours, walked in 4-hour slices and concatenated, returned 956 — across 92 URLs and three
unrelated causes.
I re-ran the comparison today, on 404s, to check it was not a one-off:
24h asked as one query → 26 events
same 24h in 4h slices → 266 events
The single query returned 10% of what was there. Not a rounding difference — a different
conclusion. And nothing in the response says so: no truncation flag, no "results were sampled" field.
You get a well-formed answer that happens to be mostly missing.
So the loop, not the query:
const out = [];
for (let h = 24; h > 0; h -= 4) {
const from = Date.now() - h * 3600_000;
const to = Date.now() - (h - 4) * 3600_000;
const ev = await queryEvents({ from, to, limit: 500 });
if (ev.length >= 500) console.warn(`slice ${h}h hit the limit — narrow it`);
out.push(...ev);
}
The ev.length >= 500
check matters as much as the slicing. A slice that returns exactly your limit is
truncated, and you have to narrow that slice further. Without the warning you cannot tell "500 events
happened" from "500 events fit".
exists
matches empty strings, and includes
ignores case Two smaller ones, both of which produced confidently wrong numbers before I noticed.
** operation: "exists" matches a key that is present but empty.** I wanted requests Cloudflare had
{ "key": "$workers.event.request.cf.verifiedBotCategory", "operation": "exists" }
That field is present on every request and is ""
on almost all of them, so the filter matched the
entire dataset and I briefly believed the whole site was bot traffic. Use exists
only for keys
genuinely absent on what you are excluding — sec-fetch-mode
is a real example, since non-browsers do
not send it.
** operation: "includes" is case-insensitive.** Filtering user agents for
bot
and for Bot
returnedThe events view and the dashboard's invocation count disagree, and both are right. On one day my web
worker showed 24,085 telemetry events against 8,772 invocations — roughly 2.7 events per
invocation.
So:
workersInvocationsAdaptive
in the GraphQL analytics API, not from counting eventsMixing them gives a number that is wrong by a factor you cannot see.
// Past-tense debugging: "what 500'd between 3am and 4am".
async function queryEvents({ from, to, limit = 500, filters }) {
const r = await fetch(
`https://api.cloudflare.com/client/v4/accounts/${ACCOUNT}/workers/observability/telemetry/query`,
{ method: 'POST',
headers: { Authorization: `Bearer ${TOKEN}`, 'Content-Type': 'application/json' },
body: JSON.stringify({
queryId: 'q', timeframe: { from, to }, limit, view: 'events',
parameters: { datasets: ['cloudflare-workers'], filters },
}) });
const d = await r.json();
return d?.result?.events?.events ?? [];
}
…called from the slicing loop above, with the results grouped in plain JavaScript rather than by asking
the API to group them. (view: "calculations"
with a groupBy
on a high-cardinality key returns only a
few groups, quietly — the same failure mode as trap 2: a well-formed answer that is mostly missing.)
The retention window is limited. I have reliably queried three days back and would not build a workflow
that assumes more; for anything you need to keep, pull it out and store it yourself.
One more, learned the embarrassing way: ** cf.asOrganization is the ASN owner, not the bot.** Requests
wrangler tail
is for watching. For asking, use the telemetry API — and remember that ** outcome: "ok" means the worker succeeded, not that your user did**, and that a query covering a wide window will hand
The tracker this came out of is at aichangewatch.com — it watches AI vendor docs for changes, and the 500s that started all this were a REST detail endpoint quietly falling through to its collection endpoint.