{"slug": "scarab-diagnostic-field-test-033-prometheus-remote-write-label-order-boundary", "title": "Scarab Diagnostic Field Test #033 - Prometheus Remote-Write Label Order Boundary", "summary": "Scarab Systems conducted a field test on Prometheus remote-write label ordering, identifying a boundary mismatch where invalid input could pass through a conversion path before label sorting was enforced. The fix adds explicit label-order validation at the remote-write request edge, ensuring invalid data is rejected before internal conversion. The patch was merged in pull request #18978.", "body_md": "Target: [prometheus/prometheus](https://github.com/prometheus/prometheus)\n\nIssue: [prometheus/prometheus#11505](https://github.com/prometheus/prometheus/issues/11505)\n\nPull request: [prometheus/prometheus#18978](https://github.com/prometheus/prometheus/pull/18978)\n\nField Lab record: [Prometheus #11505](https://github.com/scarab-systems/scarab-field-lab/tree/main/field-tests/prometheus-prometheus-11505)\n\nThis field test was about a small but important remote-write boundary in Prometheus:\n\nlabels are required to be sorted lexicographically, but incoming remote-write labels could pass through a conversion path before that requirement was enforced.\n\nThat sounds narrow because it is narrow.\n\nBut this is exactly the kind of case Scarab is meant to make visible.\n\nNot \"the remote-write subsystem is broken.\"\n\nNot \"rewrite ingestion.\"\n\nNot \"let an agent go hunting.\"\n\nThe question was smaller:\n\nwhere does repo truth say invalid input should stop, and where was the actual boundary allowing that truth to blur?\n\nThe public Field Lab record for this case is here:\n\n[https://github.com/scarab-systems/scarab-field-lab/tree/main/field-tests/prometheus-prometheus-11505](https://github.com/scarab-systems/scarab-field-lab/tree/main/field-tests/prometheus-prometheus-11505)\n\nThe record includes the public issue, the public pull request, the repair scope, validation summary, and non-claims.\n\nIt contains public links, status, validation, and claim boundaries only. It does not contain SDS source code or non-public product material.\n\nThat distinction matters.\n\nThe public artifact is the field record, not SDS itself.\n\nSDS surfaced a boundary mismatch around remote-write label ordering.\n\nThe public issue already identified the expected rule:\n\nremote-write labels are supposed to be sorted lexicographically.\n\nThe meaningful diagnostic result was not just \"sort labels.\"\n\nIn fact, the repair does not sort the incoming data.\n\nThe result was that invalid remote-write input should be rejected before it can be made to look acceptable by a later conversion path.\n\nThat is the boundary.\n\nThe input contract belongs at the edge where the remote-write request is being interpreted.\n\nIf invalid input crosses that edge and then gets normalized, the system can lose the ability to distinguish valid caller behavior from repaired caller behavior.\n\nThe failure shape was subtle:\n\nThe concern is not that sorting is technically impossible.\n\nThe concern is that sorting at the wrong layer changes what the receiver is saying to the caller.\n\nIf a remote-write sender sends invalid data, the receiver should be able to reject that data as invalid input.\n\nIf the receiver silently normalizes it, the caller gets a different contract:\n\n\"send whatever order you want; the receiver will fix it.\"\n\nThat was not the boundary described by the issue.\n\nThe boundary in this field test was:\n\nremote-write request validity must be checked before incoming label data is converted into Prometheus' internal label representation.\n\nThat is a code boundary, but it is also a contract boundary.\n\nOn one side is the external remote-write request.\n\nOn the other side is Prometheus' internal representation.\n\nThe repair belongs at the crossing point.\n\nOnce the request has crossed into internal representation, a validation failure can become harder to prove because the data may already have been reshaped.\n\nThat is why the patch is intentionally placed before append behavior, and before the conversion path can erase the original ordering problem.\n\nThe patch adds explicit label-order validation for both supported remote-write paths covered by the repair:\n\nThe pull request is here:\n\n[https://github.com/prometheus/prometheus/pull/18978](https://github.com/prometheus/prometheus/pull/18978)\n\nThe changed files are public:\n\n`storage/remote/write_handler.go`\n\n`storage/remote/write_handler_test.go`\n\nThe patch does not redesign remote-write ingestion.\n\nIt does not change remote-read.\n\nIt does not introduce a new validation framework.\n\nIt keeps the repair at the specific boundary identified by the field test.\n\nThis is one of the recurring lessons from Scarab field testing:\n\nwhen the boundary is clear, the repair should get smaller, not larger.\n\nA code agent can produce a large patch very quickly.\n\nThat is not the hard part anymore.\n\nThe hard part is deciding what the patch is allowed to claim.\n\nFor this case, the claim is not:\n\n\"remote-write validation is now perfect.\"\n\nThe claim is:\n\n\"incoming remote-write labels that are out of lexicographic order are checked before they can be normalized into the internal label representation.\"\n\nThat is a much narrower statement.\n\nIt is also a more reviewable one.\n\nThe practical bug is about label ordering.\n\nThe larger field-test value is about method.\n\nModern code agents are very good at moving through a repository and producing edits. That can be useful, but it can also make patches feel suspicious when the intent is unclear.\n\nScarab's position is different:\n\nfirst find the truth boundary, then patch only that boundary.\n\nIn this case, the repo truth was that labels must be sorted.\n\nThe boundary failure was that remote-write input could cross into a representation where the original invalid ordering was no longer visible.\n\nThe patch follows from that.\n\nNot because the agent guessed a fix.\n\nBecause the public issue, the code surface, and the validation boundary pointed to a narrow repair.\n\nThat is the shift I am testing in public:\n\ncode-agent work should not begin with \"what can we generate?\"\n\nIt should begin with \"where did the repo stop enforcing what it already says is true?\"\n\nThe repair was validated in a Linux arm64 container with the Prometheus test workflow used for the public PR.\n\nValidation recorded in the Field Lab:\n\n`make test`\n\npassed`make lint`\n\npassed`go test ./storage/remote -count=1`\n\npassedAt the time this draft was prepared on June 19, 2026, the public pull request was open and ready for review.\n\nThe public status is:\n\nThat status matters too.\n\nThis field report does not claim upstream acceptance.\n\nIt claims a public diagnostic record, a narrow repair, and a submitted PR.\n\nMaintainers decide whether the patch belongs upstream.\n\nResult:\n\ndiagnostic proof and repair submitted.\n\nThe field test produced:\n\nThe patch is intentionally boring.\n\nThat is the point.\n\nThe interesting thing is not that a code change exists.\n\nThe interesting thing is that the repair boundary is explainable without exposing proprietary diagnostic logic or asking maintainers to trust a black box.\n\nThis field test supports a narrow public claim:\n\nSDS identified a remote-write label-order boundary in Prometheus where invalid input needed to be checked before conversion could normalize it, and a human-submitted repair was prepared for that boundary.\n\nIt does not claim:\n\nThe Field Lab exists to keep those claims separate.\n\nThis field report was prepared with AI-assisted editing from public field-test notes, public issue and PR records, and the public Field Lab record. The diagnostic claim, repair boundary, and final wording were human reviewed.\n\nScarab Diagnostic Suite is proprietary. The Field Lab publishes public case records, issue links, validation summaries, and claim boundaries only.\n\nSDS finds evidence. People make claims. Maintainers decide.", "url": "https://wpnews.pro/news/scarab-diagnostic-field-test-033-prometheus-remote-write-label-order-boundary", "canonical_source": "https://dev.to/scarab-systems/scarab-diagnostic-field-test-033-prometheus-remote-write-label-order-boundary-44lk", "published_at": "2026-06-19 23:46:09+00:00", "updated_at": "2026-06-20 00:07:48.555780+00:00", "lang": "en", "topics": ["developer-tools", "ai-infrastructure"], "entities": ["Scarab Systems", "Prometheus", "prometheus/prometheus", "Scarab Field Lab"], "alternates": {"html": "https://wpnews.pro/news/scarab-diagnostic-field-test-033-prometheus-remote-write-label-order-boundary", "markdown": "https://wpnews.pro/news/scarab-diagnostic-field-test-033-prometheus-remote-write-label-order-boundary.md", "text": "https://wpnews.pro/news/scarab-diagnostic-field-test-033-prometheus-remote-write-label-order-boundary.txt", "jsonld": "https://wpnews.pro/news/scarab-diagnostic-field-test-033-prometheus-remote-write-label-order-boundary.jsonld"}}