Can We Stop Running Recurring Analysis as a Scheduled LLM Prompt?

A developer argues that running recurring analysis as scheduled LLM prompts is flawed because the LLM's nondeterministic query generation confounds metric changes with query variations. The proposed fix is to move the LLM from runtime to build time, freezing metric definitions as versioned code for deterministic execution.

Every major assistant now ships some version of scheduled prompts. You write "every Monday at 9am, analyze last week's signup funnel and tell me what changed," pick a cadence, and it runs. For a news briefing, this is great. For analysis, I think it's a mistake — and not a small one. I want to argue that the problem isn't the scheduler. It's where the LLM sits relative to it. Nobody schedules an analysis because they want a report. They schedule it because they want to know what changed . That's the whole purpose. A one-off analysis answers "what is the state of things." A recurring analysis answers "is the state of things moving, and in which direction." The second question is the only reason the cadence exists. And detecting change requires one thing above all else: the instrument has to hold still . If the query is regenerated from a natural language prompt on every run, then a movement in your number has two possible causes: And you have no way to separate them. This is textbook confounding, and you've built it into the foundation of your monitoring. The tempting response is "modern models are good at SQL, it'll be fine." But accuracy isn't really the issue — consistency is. Even a model that writes correct SQL every single time can write differently correct SQL each run. Here is the failure mode that actually bites: -- Run 1 SELECT COUNT DISTINCT u.user id FROM users u LEFT JOIN subscriptions s ON s.user id = u.user id WHERE u.created at = '2026-08-01' AND u.created at < '2026-09-01'; -- Run 2 — same prompt, two weeks later SELECT COUNT DISTINCT u.user id FROM users u INNER JOIN subscriptions s ON s.user id = u.user id -- users with no subscription silently vanish WHERE u.created at '2026-08-01' AND u.created at <= '2026-09-01'; -- window shifted by a day Neither query is broken. Neither throws an error. Both are defensible readings of the same English sentence. The number just moves by a few percent. The usual suspects, in my experience: LEFT JOIN quietly becoming INNER JOIN = / and < / <= NOT IN vs NOT EXISTS and how each treats NULL COUNT vs COUNT DISTINCT ... on a fanned-out join UTC vs local, and whether the day boundary matches the rest of the orgThis is worse than an outright failure. A crash is loud and you fix it. This is silent, plausible, and it corrupts the time series. Six months later, somebody asks why March looks odd, and you cannot reconstruct what query produced March's number — because it no longer exists anywhere. The fix isn't to stop using LLMs. It's to move the LLM from runtime to build time . Concretely, split the work into phases: 1. Explore — interactive, LLM-heavy. This is where "analyze the signup funnel" belongs. Iterate, argue with the model, throw things away. Nondeterminism is a feature here; you want different angles. 2. Freeze — emit code. Once you know what you're measuring, have the model write it out as SQL or a script, and put it in version control. The metric definition is now an artifact with a name and a hash. 3. Review — a human reads the diff. This step looks bureaucratic and it is the single most valuable one. It converts "the metric definition changed" from an invisible accident into an explicit, attributable, reviewable event. A scheduled prompt can never give you this, because there is no diff to look at — the change happens inside a sampling distribution. 4. Run — cron, or whatever your orchestrator is. Deterministic code, deterministic schedule. 5. Interpret — LLM again, downstream. More on this below. If this sounds familiar, it's because the BI world has already had this argument and reached the same conclusion. dbt's Semantic Layer, Omni, Dremio, Cortex Analyst — the shared thesis is that metric definitions must be codified so the LLM chooses which metric rather than how to compute it. dbt make the point directly: with a semantic layer, the model can't produce correct-looking numbers that differ subtly between runs, because the logic is fixed. What strikes me is that this conversation is happening almost entirely inside data engineering, in the vocabulary of text-to-SQL and governance. Meanwhile the "schedule an AI task" conversation is happening somewhere else entirely, in productivity blogs, and the two have not met. They're the same problem. I don't want this to read as "keep LLMs out of the pipeline." There are two places they belong, and once you look closely they're the same place. Interpreting the numbers. The metric is computed by frozen code. Then you hand the resulting numbers to a model and ask for commentary: "conversion fell 12% week over week, concentrated in the paid-social cohort." That's a judgment, it's genuinely hard to express as code, and — crucially — the model is consuming numbers, not producing them. The nondeterminism stays downstream of the measurement. Understanding unstructured input. If a step in your pipeline classifies free-text support tickets by theme, or extracts entities from a PDF, no amount of SQL is going to replace it. This is exactly what language models are for. The unifying shape: in both cases the LLM's job is natural language in, fixed structure out . It's a function with a declared output contract. What it must never be is the thing that decides the contract. So the line I'd draw isn't "LLM or not." It's: The LLM may consume or produce values. It may not define how values are computed. Freezing the query removes one source of drift. It doesn't make the pipeline safe, and I think it's worth being honest that two failure modes survive — one on the data side, one on the model side. Your schema and your semantics keep moving even when your SQL doesn't. Someone adds a new value to an enum. An event gets renamed and the old name is dual-written for a month. A column's meaning quietly changes because a new signup flow populates it differently. Your frozen query keeps running, keeps returning numbers, and keeps being wrong. Note that this is the same silent-plausible-drift problem as before, just relocated. Freezing moved it, it didn't kill it. Countermeasure: assertions alongside the metric. Every scheduled run should check its own inputs and outputs: Guard the metric, not just the job. A green exit code means nothing if the underlying data changed shape. assert 8 000 <= row count <= 20 000, f"row count outside expected band: {row count}" assert null rate "plan type" < 0.02, "null rate spiked — upstream schema change?" assert set observed categories <= KNOWN CATEGORIES, f"unknown category values: {set observed categories - KNOWN CATEGORIES}" The important part is what happens when these fire. Don't treat it as a flaky job to be retried. Treat it as a signal that the metric definition needs to be revisited by a human . The architecture is: freeze, plus an automated trigger to reconsider the freeze. Countermeasure: version the metric itself. Stamp a metric version on every output row and surface it on the dashboard. When you revise a definition, you then have to make an explicit decision — backfill history under the new definition, or break the series and show the break — instead of silently splicing two incompatible time series together and squinting at the result. If an LLM inference step remains in the pipeline, it has its own drift, from two directions. The provider changes the model under you. Pin model versions explicitly. Treat an unpinned model in a scheduled pipeline the same way you'd treat pip install without a lockfile. The input distribution changes under a pinned model. Pinning is necessary but not sufficient. If your ticket classifier suddenly reports 30% more "billing" tickets, is that because customers are complaining more about billing, or because the phrasing of incoming tickets shifted in a way that pushes the classifier around? Same confound as at the start of this post, one layer down. Countermeasure: a golden set on every run. Keep a fixed, hand-labeled sample and push it through the classifier alongside the real data each time. If the golden set's outputs move, the classifier moved. If the golden set is stable and production numbers moved, the world moved. Now you can actually tell the two apart — which was the entire point of scheduling the thing in the first place. These two are the failure modes I've run into myself, which almost certainly means the list is incomplete rather than complete. I'm sure there are whole categories here I haven't hit yet — late-arriving data and how it interacts with a frozen window, DST and the days that have 23 or 25 hours in them, warehouse engine upgrades changing float or sort behavior, upstream backfills silently rewriting history under a metric you already published, whatever else is out there. If you've operated something like this in production and watched it break in a way I haven't described, please leave a comment . I'd like to turn this section into something more like a proper catalogue, and I'll learn more from your war stories than from anything else. Disagreement is very welcome too — if you think scheduled prompts are fine for analysis and I've overstated the risk, I want to hear the case. The scheduler was never the problem. Cron is fine. Airflow is fine. The problem is putting a probabilistic process where a fixed instrument needs to be. Let the LLM write the pipeline once. Don't let it be the pipeline.