{"slug": "what-a-coding-agent-taught-me-about-a-b-test-telemetry", "title": "What a Coding Agent Taught Me About A/B Test Telemetry", "summary": "A developer used a coding agent to build three A/B/C test variants of a price-tag scanning screen in an Android app for store staff, then documented the telemetry work behind the experiment. The agent reframed the analytics design around a single \"scan session\" entity delivered via two events (session_start and session_finish), and proposed a prepared BigQuery table, scanner_ab.sessions, that turns raw Firebase exports into one row per scan session. The developer notes the agent also worked around a sandbox network policy blocking the BigQuery API by routing through Cloud Shell.", "body_md": "We had three designs for a new price-tag scanning screen in an Android app used by store staff. Claude Design produced the mockups, I showed them to colleagues and ran a vote. There was no clear winner.\n\nSo I suggested the obvious thing: if we can't decide, let the users decide. A routine Android task turned into an A/B/C test.\n\nA coding agent wrote the new screen and all three layouts. That turned out to be the easy part, because I had never run an A/B test. I knew the idea from articles and videos: split users into groups, show different variants, collect metrics, compare. Between that description and a real experiment in production there were a lot of questions I couldn't answer:\n\nInstead of reading another round of abstract examples, I went through the whole thing on the real task, with the agent. This post is about the parts you can reuse: the event model, the data layer, the traps in the Firebase to BigQuery export, and the one mistake no amount of SQL fixes.\n\nMy first idea was simple: the more behavior you want to analyze, the more events you send. I started instrumenting a separate event for everything: opening the scanner, a successful scan, an error, a \"move closer\" hint, various user actions.\n\nThe agent proposed a different model. For the experiment it mostly needed two events:\n\n```\nbarcode_scan_scanner_session_start\nbarcode_scan_scanner_session_finish\n```\n\nThe difference was in what each event carried.\n\n```\nsession_start\n  session_id, variant (A/B/C), store, device, launch source, ...\n\nsession_finish\n  session_id, result, code type, scan duration,\n  \"too far\" hints, mismatches during double confirmation,\n  focus / zoom / torch usage, decoded on first try, ...\n```\n\nThis was the most useful thing I learned in the whole project, before a single number came in. I would not have designed it this way myself.\n\nI had thought of analytics as a list of events: something happens, you send an event. The agent effectively designed an entity, a **scan session**, and the two events were just how the start and the outcome of that session got delivered. From then on, questions could be asked about a whole scan attempt instead of about individual clicks.\n\nSending events to Firebase Analytics is easy. Analyzing this experiment in the Firebase console quickly isn't:\n\n`event_params`;` session_id`;\nI had never worked with the Firebase Analytics export to BigQuery. The agent explained why the analysis needed it and then walked me through the setup: where to go, what to enable, what to click.\n\nThe learning order was reversed. Normally it's documentation, then an example, then test data, then finally a real project. Here the real task, the real events and the real users came first, and I learned exactly as much BigQuery as the agent needed to keep going.\n\nOnce the export worked, the next problem appeared. Raw Firebase events are an awkward interface even for an agent. To answer \"how many successful sessions did variant B have?\" you have to unpack `event_params`, find each start, find its finish, join them on `session_id`, handle missing finishes, and only then compute the metric. Doing that in every query means repeating a large block of SQL over and over.\n\nThe agent proposed a separate dataset, `scanner_ab`, with a prepared table `scanner_ab.sessions` in which **one row is one scan session**:\n\n```\nAndroid app\n  ↓\nFirebase Analytics\n  ↓\nBigQuery events_*          (raw export)\n  ↓\nscanner_ab.sessions        (one row = one scan session)\n  ↓\nanalysis queries\n```\n\nThis is where I understood why analytics people build prepared layers at all. Raw events stay raw. The prepared layer turns them into the business entity you actually ask questions about.\n\nThis is how the pipeline actually works, including the details I would not have known to look for.\n\n**Reaching BigQuery from a sandboxed agent.** The agent runs in a sandbox whose network policy blocks the BigQuery API. Instead of fighting that, it goes through Cloud Shell:\n\n```\ngcloud cloud-shell ssh --authorize-session --quiet \\\n  --command=\"bq query --use_legacy_sql=false '<SQL>'\"\n```\n\n`bq` sometimes exits with 0 after an internal error.\n**`events_*` also matches the intraday tables.** A wildcard `FROM events_*` includes `events_intraday_YYYYMMDD`, so on the day the daily table lands, the same events show up twice. Deduplicate by `session_id`, or filter with `_TABLE_SUFFIX NOT LIKE 'intraday%'`.\n\n**The sessions table maintains itself.** It's updated by a `MERGE` keyed on `session_id` over a sliding window of `events_*`, which also absorbs the intraday duplicates, and runs as a daily scheduled query. Analysis never touches the raw events.\n\n**The data disagreed with the documentation.** The experiment runbook said the launch parameter was `source`; in the real events it's `extra_source`. The runbook said a cancelled scan has `result = 'cancel'`; the real value is `cancel_back`. A query written from the docs returns zeros, with no error.\n\n**Every parameter arrived as a string.** Numeric fields need `SAFE_CAST` before any arithmetic or quantiles.\n\n**A metric can die silently.** `torch_used` stopped meaning anything once the torch button was removed on devices without a flash unit. The field is still in the data; it just no longer says anything about lighting.\n\nI now practically don't write these queries. I ask the agent in plain language:\n\nCompare A/B/C for the last three days.\n\nBreak the results down by business unit.\n\nAnalyze by device model.\n\nIn the analytics there's a drop. Compare it with Crashlytics.\n\nThe agent works out which data it needs, writes the SQL, runs it and returns a table. One sentence from me can turn into a long query: period filtering, session assembly, `UNNEST(event_params)`, joining start and finish, checking for incomplete sessions, grouping by store and device, quantiles, guardrail metrics, cuts by variant. I couldn't write those queries without first learning BigQuery and analytical SQL properly.\n\nThe result comes back as a table with roughly this shape (illustrative, not real numbers):\n\n| Variant | Sessions | Success | Cancel | p75 | TooFar | \n|---|---|---|---|---|---|\n| A | … | … | … | … | … | \n| B | … | … | … | … | … | \n| C | … | … | … | … | … | \n\nThen I ask the next question: \"Now split B by tablet model\", or \"Is this a widespread problem or a few specific stores?\"\n\nWhat makes this work isn't that an LLM can write SQL. LLMs have done that for a while. It's that the agent designed the data it was later going to analyze: it chose the telemetry, implemented it in the app, explained how to get raw events into BigQuery, proposed the prepared layer, and then started using that layer for its own queries.\n\nIn our experiment, the variant is assigned to a **store**, not to a person or a scan.\n\nThat means a thousand scans in one store are not a thousand independent participants. If one large store scans far more than the others, you can't pour all sessions into one table and declare a winner. The analysis has to happen at the store level first, and only then can variants be compared.\n\nOn top of that, stores differ in things the experiment doesn't control: tablet models, lighting, staff, load, local technical problems.\n\nThe agent can find these limitations and build the right cuts. But the final conclusion can't become \"the AI said B is better.\" Someone has to understand how the experiment was set up and what conclusion the data actually supports. Otherwise you get a very tidy table with the wrong meaning.\n\nAt one point the data showed a noticeable drop, and the question changed from \"which variant wins?\" to \"why did the numbers get worse?\"\n\nBecause every attempt is a session, the same layer could answer that too: split the problem by store, by business unit, by device model; check whether it's one person, one tablet or a broad change; then line up the moment of degradation with Crashlytics. The pipeline built for a product experiment became part of a technical investigation.\n\nThe session model helps here in a way separate events don't. Cancelling the scanner is an explicit result inside `session_finish`. A session that has a start and no finish at all is a different signal, and that is the one worth checking against crash reports.\n\nIf you're instrumenting your first experiment:\n\n`session_start` / `session_finish` pair with a shared `events_*` wildcard.\nThe coding agent defined the events and their attributes, implemented the instrumentation in the Android app, explained why the analysis needed BigQuery and guided me through the export setup, proposed and built `scanner_ab.sessions`, and now writes and runs the queries, checks data quality, builds the cuts and compares drops with Crashlytics.\n\nI brought the product question, asked the questions in plain language, and remain responsible for the part that doesn't come out of a query: how the experiment is set up and which conclusion the data actually allows.\n\nI didn't become a BigQuery specialist or an analyst. But I can now work on a task that previously would have required learning several new tools first. I don't work with BigQuery through SQL; I work with the agent in the language of engineering questions, and the agent works with BigQuery through SQL.", "url": "https://wpnews.pro/news/what-a-coding-agent-taught-me-about-a-b-test-telemetry", "canonical_source": "https://dev.to/hram/what-a-coding-agent-taught-me-about-ab-test-telemetry-4md1", "published_at": "2026-09-27 09:33:46+00:00", "updated_at": "2026-09-27 10:01:16.016029+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "mlops"], "entities": ["Firebase Analytics", "BigQuery", "Claude Design", "Google Cloud Shell", "Android"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/what-a-coding-agent-taught-me-about-a-b-test-telemetry", "markdown": "https://wpnews.pro/news/what-a-coding-agent-taught-me-about-a-b-test-telemetry.md", "text": "https://wpnews.pro/news/what-a-coding-agent-taught-me-about-a-b-test-telemetry.txt", "jsonld": "https://wpnews.pro/news/what-a-coding-agent-taught-me-about-a-b-test-telemetry.jsonld"}}