LLM monitoring in production: what to watch, and how failures become tests

A practitioner-written guide to LLM monitoring in production argues that standard application performance monitoring misses LLM failures because refusals and safety stops return HTTP 200 — Anthropic's stop-reason documentation states a refusal arrives as "a normal HTTP 200 response, not an error" with stop_reason set to refusal, while Google's API reference lists more than twenty finish reasons on successful responses, including SAFETY, MAX_TOKENS, MALFORMED_FUNCTION_CALL and TOO_MANY_TOOL_CALLS. The guide, whose author discloses building the testing tool EvalShift, covers per-request cost, online eval sampling costs, rate-based alerting, OpenTelemetry's status and tool comparisons as of November 2026, and cites Anthropic's September 2025 postmortem of three infrastructure bugs, one of which misrouted 0.8% of Sonnet 4 requests. It closes on converting production failures into offline test cases so the next change is checked against them.

← all posts https://www.evalshift.dev/blog LLM monitoring in production: what to watch, and how failures become tests LLM monitoring in production: the metrics to watch, online evals and what they cost, alerting on rates, OpenTelemetry's status, and the tools compared. LLM monitoring is watching an application built on a language model while real users are on it: whether calls succeed, how long they take, what they cost, and whether the answers and actions are still the ones you tested. The first three are ordinary monitoring with new units. The fourth is the hard one, because a wrong answer comes back with the same status code as a right one. Most guides to LLM monitoring stop at the dashboard. This one covers the signals worth watching and how each is computed, what it costs to score a sample of live traffic, how to alert on a rate without paging someone over noise, where the OpenTelemetry standard stands, and the tools as their docs and pricing pages read in November 2026. It ends with the step most setups skip: turning a production failure into a test case, so the next change is checked against it. A disclosure: I build EvalShift, which is not a monitoring tool. It covers the testing half of that last step, and it has a short section near the end. The rest works with any tool. What LLM monitoring is Three terms get used for overlapping things, and the tools blur them further. | | Question it answers | When it runs | Output | |---|---|---|---| | LLM monitoring | Is something wrong right now? | Continuously, on live traffic | Metrics over time, and alerts | | LLM observability | Why did this request go wrong? | On demand, after an alert or a complaint | A trace: every model call, tool call and retrieval in one request | | Offline evaluation | Will this change make things worse? | Before release, on a fixed set of cases | A comparison of two versions | Monitoring is built on observability data: you cannot chart tool-call errors if tool calls are not recorded as spans. So one product does both, and "LLM observability platform" and "LLM monitoring tool" name the same list of vendors. Monitoring and offline evaluation feed each other. Monitoring sees real inputs and has no expected answer for them. Offline testing https://www.evalshift.dev/blog/llm-testing has expected answers and only the inputs someone thought to collect. Why ordinary APM misses LLM failures An application performance dashboard watches status codes, latency and throughput. An LLM application can fail while all three look healthy. A failed response returns 200. Anthropic's stop reason docs ↗ https://platform.claude.com/docs/en/build-with-claude/handling-stop-reasons say a refusal arrives as "a normal HTTP 200 response, not an error", with stop reason set to refusal . Google's API reference ↗ https://aistudio.google.com/api-reference/generate-content lists more than twenty finish reasons on a successful response, among them SAFETY , MAX TOKENS , MALFORMED FUNCTION CALL and TOO MANY TOOL CALLS . OpenAI's chat completion object ↗ https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create carries a refusal field on the message and a content filter finish reason. None of these moves an error-rate chart. A wrong answer has no field at all. A reply that invents a refund policy, or an agent that issues the refund without looking up the order, finishes with a normal stop reason. The only way to count these is to check the content. Cost is a property of each request. A one-line question and a long agent task hit the same endpoint and cost very different amounts. A loop in an agent shows up on the invoice before it shows up anywhere else. Behavior changes without a deploy. In September 2025 Anthropic published a postmortem ↗ https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues of three infrastructure bugs that degraded Claude's responses. One misrouted 0.8% of Sonnet 4 requests when it was introduced on August 5, and 16% in its worst hour on August 31. Anthropic wrote that "the evaluations we ran simply didn't capture the degradation users were reporting", and that it would start running quality evaluations continuously on production systems. If the provider's own checks can miss a regression in the model you call, your deploy log will not show it either. LLM monitoring metrics: what to watch Five groups. The first two are what an APM tool already does, in different units. The last three are specific to LLM applications. | Group | Signal | How to compute it | |---|---|---| | Operational | Error rate by type | Failed model calls over all model calls, split into rate limit 429 , provider error 5xx , timeout and invalid request 400 | | Operational | Latency | p50 and p95, for the whole task and for each model call | | Operational | Time to first token | For streamed replies, time from the request to the first chunk | | Operational | Rate-limit headroom | Remaining requests and tokens, read from response headers | | Cost | Tokens per task | Input, output and reasoning tokens summed over every call in the task | | Cost | Cost per successful task | Tokens times price for each token type, summed, divided by the tasks that succeeded | | Cost | Cache hit rate | Input tokens read from cache over total input tokens | | Quality | Deterministic check pass rate | Code checks on every output: it parses, matches the schema, has the required fields | | Quality | Judged quality | An LLM judge on a sample, one pass-or-fail criterion at a time | | Quality | User feedback | Thumbs, edits, retries and escalations to a human, per session | | Quality | Refusal and truncation rate | Responses by stop or finish reason | | Agent | Tool-call error rate | Tool executions that raised or returned an error over all tool executions | | Agent | Steps per task | Model calls and tool calls per task, p95 as well as the median | | Agent | Loops | Tasks that repeat a tool call with identical arguments, and tasks that hit the step cap | | Change | Model served | Responses counted by the model the response names | | Change | Provider notices | Deprecation pages and changelogs, checked on a schedule | Operational signals Split errors by type before you chart them, because the responses differ. A 429 means back off or raise a limit, a 5xx means retry or fail over, and an invalid request after a deploy means your request shape is wrong. Current models reject parameters that older ones accepted, so the same error can arrive with a model upgrade https://www.evalshift.dev/blog/llm-testing . Rate limits can be watched before they bite. Anthropic's rate limit docs ↗ https://platform.claude.com/docs/en/api/rate-limits list response headers such as anthropic-ratelimit-input-tokens-remaining and anthropic-ratelimit-output-tokens-remaining , plus retry-after when a 429 carries one. Chart the remaining share at peak hour and you see the ceiling before users do. For anything streamed to a person, time to first token is the latency users feel. Total duration grows with the length of the answer. Cost signals Measure cost per task, not per call. An agent task is several model calls plus tool results fed back in as input, and a cheaper model that takes one more round per task can cost more. The cheapest LLM API https://www.evalshift.dev/blog/cheapest-llm-api works through that arithmetic. Count cached tokens separately, or the cost chart is wrong. In Anthropic's prompt caching docs ↗ https://platform.claude.com/docs/en/build-with-claude/prompt-caching , the input tokens field holds only the tokens after the last cache breakpoint, and the total is cache read input tokens + cache creation input tokens + input tokens . Cache reads are billed at 0.1 times the base input price or less, and a five-minute cache write at 1.25 times. A prompt edit that moves the cache breakpoint can multiply input cost without changing the request count, and the cache hit rate is the chart that shows it. Quality signals Run code checks on every response. They cost nothing and never disagree with themselves: the output parses, it validates against the schema, the order ID from the input appears in the reply. Refusals and truncations come free with the response, so count responses by stop reason too. User feedback is the signal closest to the truth. Do not rely on thumbs alone: add the implicit signals your product already has, such as the user rephrasing the same question, editing the draft before sending it, or asking for a human. For everything else, such as groundedness and whether the question was answered, the option is an LLM judge on a sample. That has its own section below. Agent signals An agent adds failures that sit between the model call and the reply. A tool execution fails, and the model may recover or may tell the user it worked. A task loops on the same search until it hits the cap. Chart the tool-call error rate per tool, steps per task at p95, and the count of tasks that hit your step limit. All three need tool calls recorded as their own spans, which is the first thing to check in a tracing library. Agent evaluation metrics https://www.evalshift.dev/blog/agent-evaluation-metrics has the definitions, and evaluating agent tool calls https://www.evalshift.dev/blog/evaluating-agent-tool-calls covers why the reply alone hides these failures. Change detection Record the model that answered, not only the model you asked for. Each provider's response names it: OpenAI's model field is "the model used for the chat completion", and Gemini's modelVersion is "the model version used to generate the response". OpenAI also returns a system fingerprint , which "represents the backend configuration that the model runs with". This matters because the two can differ. Google's model docs ↗ https://aistudio.google.com/docs/models say a latest alias points to the newest release and is swapped with two weeks' notice by email, and its changelog ↗ https://aistudio.google.com/docs/changelog records gemini-pro-latest and gemini-flash-latest switching to the Gemini 3 previews on January 21, 2026. The same changelog says that from December 1, 2026, requests to gemini-2.5-pro will automatically route to gemini-3.8-flash , and requests to gemini-2.5-flash to gemini-3.5-flash-lite . Those requests will keep returning 200. So chart responses by the model named in the response, and alert when a value appears that was not there yesterday. Then put the provider's retirement dates in a calendar. The deprecation schedule https://www.evalshift.dev/blog/llm-model-deprecation-schedule lists them for OpenAI, Anthropic and Google. Online evals: scoring a sample of live traffic An online eval is a check that runs on production traces after they are logged. Every platform in the table below has the same three controls: a filter for which traces to score, a sampling rate, and a choice between a code evaluator and an LLM judge. They run in the background. Braintrust's docs ↗ https://docs.braintrust.dev/observe/score-online say evaluations run "asynchronously in the background without adding latency to your application". Sampling is the cost control. Arize's online eval guide ↗ https://arize.com/docs/ax/evaluate/online-evals suggests 100% for low-volume or critical applications, 10 to 50% for high volume and 1 to 5% for very high volume, starting at 10 to 20% until you trust the evaluator. LangSmith's example ↗ https://docs.langchain.com/langsmith/online-evaluations is a rate of 0.1, and Datadog's is 10%. What a judge on production traffic costs There are two bills: the judge model's tokens, and whatever the platform charges for holding the result. For the tokens, Langfuse's LLM-as-a-judge docs ↗ https://langfuse.com/docs/evaluation/evaluation-methods/llm-as-a-judge estimate $0.01 to $0.10 per assessment, depending on the judge model and the size of the input. Take 100,000 traces a month, score 10% of them with two judges, and you run 20,000 assessments: $200 to $2,000 a month at that estimate. Score every trace and it is $2,000 to $20,000. The platform side, as of November 2026: - + Langfuse counts every score as a billable unit, alongside traces and observations. - + LangSmith has no per-score charge, but a new online evaluator upgrades the traces it scores ↗ https://docs.langchain.com/langsmith/usage-and-billing to 180-day retention unless you opt out, at $2.50 per 1,000 on top of the base trace price. - + Braintrust counts scores: 50,000 are included on Pro, then $1.50 per 1,000. - + Datadog bills on LLM spans and lists evaluations as unlimited on both plans. LangSmith also lets you cap an evaluator's LLM spend per week, after which it pauses until the limit resets. If your platform has that setting, use it before the first rule goes live. Three rules for a judge in production Give it one criterion with a pass or fail answer. Check its verdicts against 30 to 50 traces you labeled by hand before you alert on it. And filter before you sample: a judge for refund conversations should see only refund conversations. When to trust an LLM judge https://www.evalshift.dev/blog/when-to-trust-an-llm-judge has the audit procedure, and LLM-as-a-judge https://www.evalshift.dev/blog/llm-as-a-judge covers the known biases. Alert on a rate with its uncertainty The alerting built into these tools is a threshold on an average over a time window. LangSmith's alerts ↗ https://docs.langchain.com/langsmith/alerts cover run count, cost, errors, feedback score and latency. Braintrust's examples ↗ https://docs.braintrust.dev/deploy/monitor include "Score drops below 0.8". That works for latency and error counts, where an hour holds thousands of data points. It breaks on sampled quality scores, where the window may hold fifty. Say the judged pass rate has been 92% and the alert fires below 90%. Today's sample has 100 scored traces and 87 passed. The 95% interval around 87 out of 100 runs from 79.0% to 92.2%, and the baseline sits inside it. Nothing has been shown to change. With 1,000 scored traces and 870 passes the interval is 84.8% to 88.9%, the baseline is outside it, and the drop is real. Three settings follow from that: - + Size the window to the sample. Choose the window so it holds a few hundred scored traces, even if that makes it a day and not an hour. A quality alert that needs a day is still faster than a support ticket. - + Compare with a baseline, not a fixed number. Alert when this window's interval excludes last week's rate. A fixed threshold fires the day traffic shifts toward harder questions. - + Keep hard failures on counts. Schema failures, 5xx errors, tool exceptions and an unknown model in the response need no statistics. Alert on the first few. How many eval cases do you need https://www.evalshift.dev/blog/how-many-eval-cases-do-you-need has the same arithmetic for an offline suite. OpenTelemetry for LLM monitoring: where the standard stands OpenTelemetry has semantic conventions for generative AI: agreed names for the spans, attributes and metrics an LLM call produces, so that any instrumentation library can feed any backend. They are what lets you switch monitoring vendors without re-instrumenting. They are also not finished. As read on November 13, 2026: - +The conventions moved out of the main specification. The pages on opentelemetry.io ↗ https://opentelemetry.io/docs/specs/semconv/gen-ai/ now say they are no longer maintained there and point to a dedicated repository, semantic-conventions-genai ↗ https://github.com/open-telemetry/semantic-conventions-genai , created in May 2026. - +Every page in that repository is marked Development , not Stable. The repository has no tagged release, and its README lists the schema URL as "TODO". - +The unreleased changelog holds 20 entries marked breaking. One replaces the gen ai.client.token.usage histogram with separate input and output token metrics. Another renames gen ai.usage.cache creation.input tokens to gen ai.usage.cache write.input tokens . What the conventions define maps closely onto the table above: | You want to chart | Convention | |---|---| | Model asked for and model served | gen ai.request.model , gen ai.response.model | | Tokens, cached and reasoning tokens | gen ai.usage.input tokens , gen ai.usage.output tokens , gen ai.usage.cache read.input tokens , gen ai.usage.reasoning.output tokens | | Stop reasons | gen ai.response.finish reasons | | Latency and time to first token | gen ai.client.inference.duration , gen ai.client.inference.time to first chunk | | Tool executions and their errors | An execute tool span with gen ai.tool.name and error.type | | Steps per agent run | gen ai.invoke agent.inference calls , gen ai.invoke agent.tool calls | Prompts and completions are off by default. The span conventions ↗ https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md say instrumentations should not capture instructions, inputs or outputs unless the user opts in. That is the right default, and it means a judge has nothing to read until you turn content capture on and decide where the content lives. Backends already accept the format. Datadog says ↗ https://www.datadoghq.com/blog/llm-otel-semantic-convention/ it natively supports the GenAI conventions from version 1.37 up. Langfuse receives OTLP ↗ https://langfuse.com/integrations/native/opentelemetry and maps gen ai. attributes onto its own data model. Phoenix ↗ https://arize.com/docs/phoenix is built on OpenTelemetry with its own OpenInference instrumentation. My advice: instrument through OpenTelemetry, because it is the only portable option, pin the instrumentation library's version, and expect renames until the conventions reach Stable. LLM monitoring tools Checked against each vendor's docs and pricing page on November 13, 2026. Prices are list prices for the cloud product. | Tool | License and hosting | Online evals | Alerts | Free tier | First paid tier | |---|---|---|---|---|---| | Langfuse ↗ https://langfuse.com/pricing | MIT except ee folders; self-host free | LLM judge on observations, with filters and sampling | Monitors on cost, latency and scores, to Slack, webhook or GitHub Actions | 50k units a month, 30 days of data | Core: $29 a month, 100k units, then $8 per 100k | | LangSmith ↗ https://www.langchain.com/pricing | Proprietary; self-host on Enterprise | LLM judge on runs or threads, filter and sampling rate | Run count, cost, errors, feedback score, latency, to Slack, PagerDuty or webhook | 1 seat, 5k traces a month | Plus: $39 per seat, 10k traces, then $5 per 1,000 | | Braintrust ↗ https://www.braintrust.dev/pricing | Proprietary; BYOC on Enterprise | Scoring rules on spans, traces or groups, with sampling | Email, Slack or webhook | 1 GB, 10k scores, 14 days | Pro: $249 a month, 5 GB, 50k scores, 30 days | | Arize Phoenix and AX ↗ https://arize.com/pricing/ | Phoenix: Elastic License 2.0, self-host free. AX: commercial | On AX: LLM judge and code, continuous, with sampling | On AX: monitors to email, Slack, PagerDuty, OpsGenie, Teams or webhook | AX Free: 25k spans, 15 days | AX Pro: $50 a month, 50k spans, 30 days | | Opik ↗ https://www.comet.com/site/pricing/ | Apache-2.0; self-host free | LLM judge and code on traces, threads or spans, with sampling | On errors and scores, to Slack, PagerDuty or webhook | 25k spans a month, 60 days | Pro: $19 a month, 100k spans, then $5 per 100k | | Datadog Agent Observability ↗ https://www.datadoghq.com/pricing/?product=llm-observability | Proprietary SaaS | Managed, custom LLM judge and code evaluators, with sampling | Datadog monitors | 40k LLM spans a month, 15 days | Pro: $160 a month billed annually, 100k LLM spans, then $3.50 per 10k | | Helicone ↗ https://www.helicone.ai/pricing | Apache-2.0; gateway and logging | Scores you submit | Error rate and cost, to Slack or email | 10,000 requests, 7 days | Pro: $79 a month plus usage | Notes on reading it: - + The meters differ. Langfuse counts every trace, observation and score. LangSmith counts traces. Braintrust counts gigabytes and scores. Datadog counts LLM spans only, not tool, workflow or retrieval spans. An agent that makes ten model calls per request is one LangSmith trace, ten Datadog LLM spans and eleven or more Langfuse units. LangSmith alternatives https://www.evalshift.dev/blog/langsmith-alternatives and Braintrust vs LangSmith https://www.evalshift.dev/blog/braintrust-vs-langsmith work through examples. - + Retention is short. 14 days on LangSmith's base traces and Braintrust's free plan, 15 on Datadog and AX Free, 30 on Braintrust Pro. A failure you mean to turn into a test has to be exported before then. - + Datadog renamed the product. Its docs and pricing page call it Agent Observability, and the URLs still say LLM Observability. - + Helicone has changed hands twice. Mintlify acquired it ↗ https://www.mintlify.com/blog/mintlify-acquires-helicone in March 2026 and said it would continue in maintenance mode. As of mid-November 2026 its site carries a banner saying it is now part of Experiential Labs, with details to come. Check where that stands before you build on it. How to choose: if traces must stay on your infrastructure, Langfuse, Phoenix or Opik. If you are on LangChain or LangGraph, LangSmith. If you already pay for Datadog, start there. If experiments are the center of your workflow, Braintrust. The best LLM evaluation tools https://www.evalshift.dev/blog/best-llm-evaluation-tools compares the same platforms on their evaluation features. From alert to test case Monitoring tells you after the fact: a user saw the wrong answer before the judge scored it. What monitoring can do is make sure each failure happens in production once. That takes five steps, and the monitoring tool does none of them for you. 1. Open the trace, not the chart. Find one failing request and read all of it: the input, the tool calls, the tool results, the reply. 2. Export it before retention deletes it. Copy the input, the conversation so far and the tool results into a file in your repository, with personal data removed. 3. Write the expected behavior by hand. This is the step that gets skipped. The recorded run is what the agent did, and it did the wrong thing. The test case needs what it should have done: the right tool, the right arguments, the fact the reply must contain. 4. Tag it with the failure. refund-without-lookup , wrong-language , loop . The tag is how you know later why the case exists. 5. Run the suite before the fix ships, and on every change after. The fix has to pass the new case without failing the old ones. From then on a prompt edit, a model upgrade or a provider's forced migration is checked against that failure before release. LLM regression testing in CI https://www.evalshift.dev/blog/llm-regression-testing-in-ci shows the pull-request version, and eval-driven development https://www.evalshift.dev/blog/eval-driven-development is the same loop treated as the way you build. The loop runs the other way too. When a retirement date appears or an alias is about to move, you have a suite made of your own worst days to run the new model against before the provider switches it for you. Build a golden suite from production traffic https://www.evalshift.dev/blog/build-a-golden-suite-from-production-traffic covers collecting the ordinary cases that sit next to the failures. Where EvalShift fits Disclosure again: I'm the founder. EvalShift https://www.evalshift.dev/docs does not do production monitoring. It has no dashboards, no alerts and no online evals, so run one of the tools above next to it. What it covers is steps 2 to 5. The capture SDK https://www.evalshift.dev/docs/sdk records agent runs in production: the input, each model call with its model ID, token counts and latency, and each tool call with its result. It writes them as files to local disk or to a bucket you own on S3, Google Cloud Storage or Azure, and nothing is sent to EvalShift. Redaction runs in process before a capture is written, and EVALSHIFT SAMPLE RATE https://www.evalshift.dev/docs/sdk-config records a fraction of runs. One command turns a recorded run into a case in the golden suite: evalshift capture promote cap 3f9a2c --suite support --tag refund-without-lookup Two details matter for failures. promote refuses a capture whose trace contains an error event unless you pass --allow-errored . And the expected tool calls it writes are marked provenance: captured , which the docs https://www.evalshift.dev/docs/golden-suite describe as transcribed from the model's own recorded call, where nobody has checked it is right. For a failure it is wrong by definition, so you edit the case file to the correct calls and mark them reviewed . That is step 3. After that, evalshift compare runs the current model and a candidate over the suite and returns a verdict from the migration policy https://www.evalshift.dev/docs/migration-policy you wrote, and the GitHub Action https://www.evalshift.dev/docs/action does the same on a pull request. With captures.max age set, evalshift doctor warns when the newest case in a suite is older than that, or when cases were recorded on a model other than the one you run now. FAQ What is LLM monitoring? LLM monitoring is watching an application built on a language model while real users are on it: whether model calls succeed, how long they take, what they cost, and whether the answers and tool calls are still correct. The last part needs checks on the content, because a wrong answer returns the same HTTP 200 as a right one. What is the difference between LLM monitoring and LLM observability? Monitoring answers whether something is wrong right now, with metrics over time and alerts. Observability answers why one request went wrong, with a trace of every model call, tool call and retrieval in it. Monitoring is built on observability data, so one product usually does both. What metrics should you monitor for an LLM application? Error rate by type, latency and time to first token; tokens and cost per task and the cache hit rate; the pass rate of code checks, a judged quality score on a sample, user feedback and the refusal rate; for agents, tool-call errors, steps per task and loops; and the model named in each response. How do you monitor LLM output quality in production? Run code checks on every response, count responses by stop reason, collect user feedback, and run an LLM judge on a filtered sample of traces with one pass-or-fail criterion per judge. Validate the judge against 30 to 50 hand-labeled traces before alerting on it. How much does it cost to run LLM judges on production traffic? Langfuse's docs estimate $0.01 to $0.10 per assessment, depending on the judge model and input size. Scoring 10% of 100,000 monthly traces with two judges is 20,000 assessments, or $200 to $2,000 a month in judge tokens, before any per-score or retention charge from the platform. What are the best LLM monitoring tools? As of November 2026: Langfuse, Arize Phoenix or Opik if traces must stay on your own infrastructure, LangSmith for teams on LangChain or LangGraph, Braintrust for experiment-centered teams, and Datadog Agent Observability for teams already on Datadog. All of them trace requests, and all but self-hosted Phoenix add online evals with sampling and alerts; for Phoenix those are on Arize AX. Is OpenTelemetry ready for LLM monitoring? It is usable and not yet stable. As of November 13, 2026, the GenAI semantic conventions live in their own repository, every page is marked Development, there is no tagged release, and the unreleased changelog holds 20 entries marked breaking. Instrument through it, pin the library version and expect attribute renames. How do you detect that a model changed in production? Record the model named in each response, not only the one you requested, and alert when a new value appears. Gemini returns it in the modelVersion field and OpenAI in the model field. Add the provider's deprecation dates to a calendar, because a rerouted request still returns 200. Keep reading - + LLM testing https://www.evalshift.dev/blog/llm-testing : the types of tests that run before release, and what to rerun when the model changes. - + Build a golden suite from production traffic https://www.evalshift.dev/blog/build-a-golden-suite-from-production-traffic : recording real runs and promoting them to test cases. - + When to trust an LLM judge https://www.evalshift.dev/blog/when-to-trust-an-llm-judge : auditing a judge before it scores live traffic. - + LLM model deprecation schedule https://www.evalshift.dev/blog/llm-model-deprecation-schedule : the retirement dates to put next to your alerts.