Parallel Constrained Decoding: accuracy + probability calibration eval (30 labeled cases, 2 models, temperature sweep)

A developer published a companion evaluation of the Parallel Constrained Decoding engine shipped with the Qwen-2.5-1B-RLCD model and its Hugging Face Space, measuring decision accuracy and probability calibration against plain autoregressive JSON generation. The eval uses 30 hand-written incident-routing cases with five labeled fields each, run across Qwen2.5-1.5B-Instruct-4bit at temperature 1.0 and Qwen2.5-7B-Instruct-4bit at temperature 2.0 on Apple Silicon via MLX. The author notes the release itself does not publish accuracy or calibration numbers.

Companion eval for the "Parallel Constrained Decoding" engine shipped with harshatheg/Qwen-2.5-1B-RLCD / HF Space drinkmoonshine/parallel-constrained-decoding. Measures the thing the release does not publish: decision accuracy and probability calibration, side by side with plain autoregressive JSON generation. Setup pip install mlx mlx-lm Apple Silicon git clone https://huggingface.co/spaces/drinkmoonshine/parallel-constrained-decoding cp eval cases.json run eval.py parallel-constrained-decoding/ cd parallel-constrained-decoding python run eval.py --model mlx-community/Qwen2.5-1.5B-Instruct-4bit --temp 1.0 --tag 1.5B T1.0 python run eval.py --model mlx-community/Qwen2.5-7B-Instruct-4bit --temp 2.0 --tag 7B T2.0 Eval set 30 hand-written incident-routing cases eval cases.json , 5 fields each priority / requires escalation / department / customer impact / sla breach risk , gold labels derived from the written rubric in the same file. The rubric is also written into the field descriptions, because the parallel engine's prefill catalog only keeps the first line of each description. This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters "context": "Incident: db-primary-01 CPU pinned at 100% for the last 40 minutes. Checkout requests are failing with 503 for roughly 40% of all sessions. Acme Global Tier 1 enterprise reports their storefront is down. No data loss observed so far.", "gold": { "priority": "P0 CRITICAL", "requires escalation": "true", "department": "INFRASTRUCTURE", "customer impact": "WIDESPREAD", "sla breach risk": "true" } }, { "id": "c02", "kind": "clean", "context": "A single user on the Free plan says the CSV export button produces an empty file. Everything else in their workspace works and they can export via the API as a workaround. No other reports of this.", "gold": { "priority": "P2 NORMAL", "requires escalation": "false", "department": "PRODUCT SUPPORT", "customer impact": "LIMITED", "sla breach risk": "false" } }, { "id": "c03", "kind": "clean", "context": "Security alert: an engineer's laptop shows an unexpected OAuth grant to an unknown third-party app, and the session token has been used from an unrecognised IP address in another country. The employee has since been locked out.", "gold": { "priority": "P1 HIGH", "requires escalation": "true", "department": "SECURITY", "customer impact": "NONE", "sla breach risk": "false" } }, { "id": "c04", "kind": "clean", "context": "A customer writes in asking how to configure single sign-on with their Okta tenant. They are not blocked, just evaluating the feature. No error messages, no outage.", "gold": { "priority": "P3 LOW", "requires escalation": "false", "department": "PRODUCT SUPPORT", "customer impact": "NONE", "sla breach risk": "false" } }, { "id": "c05", "kind": "clean", "context": "Our payment provider is returning hard declines for every credit card transaction since 14:20 UTC, affecting all customers in the EU region including 12 enterprise accounts. Invoice generation is also stalled.", "gold": { "priority": "P0 CRITICAL", "requires escalation": "true", "department": "BILLING", "customer impact": "WIDESPREAD", "sla breach risk": "true" } }, { "id": "c06", "kind": "clean", "context": "The marketing site footer shows the wrong copyright year. Nobody is blocked; this was noticed during a design review. It is an internal cosmetic fix.", "gold": { "priority": "P3 LOW", "requires escalation": "false", "department": "PRODUCT SUPPORT", "customer impact": "NONE", "sla breach risk": "false" } }, { "id": "c07", "kind": "clean", "context": "Alert: api-gateway p99 latency jumped from 180ms to 2.4s after the 09:15 deploy. About 15% of all requests are timing out. Two enterprise customers have opened tickets; internal dashboards are still responding.", "gold": { "priority": "P1 HIGH", "requires escalation": "true", "department": "INFRASTRUCTURE", "customer impact": "WIDESPREAD", "sla breach risk": "true" } }, { "id": "c08", "kind": "clean", "context": "A customer's account was charged twice for the September subscription. They are requesting a refund for the duplicate charge. Their service is unaffected and the amount is 49 USD.", "gold": { "priority": "P2 NORMAL", "requires escalation": "false", "department": "BILLING", "customer impact": "LIMITED", "sla breach risk": "false" } }, { "id": "c09", "kind": "clean", "context": "Detection: a public S3 bucket belonging to our analytics team was found to contain internal build logs for three weeks. No customer data was in it. The bucket has been made private, and we need a forensic review of access logs.", "gold": { "priority": "P1 HIGH", "requires escalation": "true", "department": "SECURITY", "customer impact": "NONE", "sla breach risk": "false" } }, { "id": "c10", "kind": "clean", "context": "Storage cluster reports 92% capacity and growing 3% per day. Writes still succeed, no customer impact yet, but we will hit a hard limit in about three days if nothing changes.", "gold": { "priority": "P2 NORMAL", "requires escalation": "false", "department": "INFRASTRUCTURE", "customer impact": "NONE", "sla breach risk": "false" } }, { "id": "c11", "kind": "clean", "context": "The TLS certificate for our public API expired at 00:05 UTC. Every SDK client is now rejecting connections, so all API traffic from all customers is failing. Automated renewal did not fire.", "gold": { "priority": "P0 CRITICAL", "requires escalation": "true", "department": "SECURITY", "customer impact": "WIDESPREAD", "sla breach risk": "true" } }, { "id": "c12", "kind": "clean", "context": "A user asks whether the mobile app supports offline mode for field technicians, and how to enable it. No incident, no outage, purely a capability question.", "gold": { "priority": "P3 LOW", "requires escalation": "false", "department": "PRODUCT SUPPORT", "customer impact": "NONE", "sla breach risk": "false" } }, { "id": "c13", "kind": "clean", "context": "Billing anomaly: 1,800 invoices were generated with the wrong tax rate 7% instead of 21% across all EU customers during last night's batch run. The invoices have already been emailed and one enterprise customer has flagged it to their finance team.", "gold": { "priority": "P1 HIGH", "requires escalation": "true", "department": "BILLING", "customer impact": "WIDESPREAD", "sla breach risk": "true" } }, { "id": "c14", "kind": "clean", "context": "A pentest report identifies an authentication bypass in the password reset flow: an attacker can reset any account by supplying a crafted request. The finding is marked confirmed and exploitable today.", "gold": { "priority": "P0 CRITICAL", "requires escalation": "true", "department": "SECURITY", "customer impact": "WIDESPREAD", "sla breach risk": "true" } }, { "id": "c15", "kind": "tricky", "context": "The billing dashboard is showing an alert about an expired TLS certificate on the payments gateway. Payments themselves are still processing normally through a fallback path. No customer has reported anything.", "gold": { "priority": "P2 NORMAL", "requires escalation": "true", "department": "SECURITY", "customer impact": "NONE", "sla breach risk": "false" } }, { "id": "c16", "kind": "tricky", "context": "From the internal Slack: 'just a small note, the auth service has been returning 500s to everyone since the deploy, no rush'. Login is fully unavailable for all users of the product.", "gold": { "priority": "P0 CRITICAL", "requires escalation": "true", "department": "INFRASTRUCTURE", "customer impact": "WIDESPREAD", "sla breach risk": "true" } }, { "id": "c17", "kind": "tricky", "context": "Ticket body truncated logs follow :\n2026-09-16T02:11:04Z INFO worker-3 batch ok cost=0.31\n2026-09-16T02:11:19Z INFO worker-3 retry=2 queue=billing\n2026-09-16T02:12:00Z ERROR database connection refused: too many clients\n2026-09-16T02:12:01Z WARN billing: 2400 outstanding invoices unprocessed\n2026-09-16T02:12:44Z INFO worker-3 shutting down\nCustomer-facing note: users cannot view their invoices at all, and this affects every account we serve.", "gold": { "priority": "P0 CRITICAL", "requires escalation": "true", "department": "INFRASTRUCTURE", "customer impact": "WIDESPREAD", "sla breach risk": "true" } }, { "id": "c18", "kind": "tricky", "context": "A customer on the Team plan says that after the last release the 'share' button sometimes takes two clicks. They are on macOS Safari and can still share documents normally with a second click. A second customer mentions the same thing.", "gold": { "priority": "P2 NORMAL", "requires escalation": "false", "department": "PRODUCT SUPPORT", "customer impact": "LIMITED", "sla breach risk": "false" } }, { "id": "c19", "kind": "tricky", "context": "Security team reports credential stuffing against our login endpoint: 2.3 million attempts in 20 minutes from a botnet, and roughly 600 customer accounts were accessed successfully. Login remains up for everyone else.", "gold": { "priority": "P0 CRITICAL", "requires escalation": "true", "department": "SECURITY", "customer impact": "WIDESPREAD", "sla breach risk": "true" } }, { "id": "c20", "kind": "tricky", "context": "Our own staging environment is down. No customers are affected because staging is internal-only, and the production cluster is healthy. The team wants it fixed before tomorrow's demo. This is the fourth time this week.", "gold": { "priority": "P2 NORMAL", "requires escalation": "false", "department": "INFRASTRUCTURE", "customer impact": "NONE", "sla breach risk": "false" } }, { "id": "c21", "kind": "clean", "context": "A prospective customer emails asking whether we offer a SOC 2 Type II report and a data processing agreement before they sign. No incident; this is a pre-sales compliance question.", "gold": { "priority": "P3 LOW", "requires escalation": "false", "department": "PRODUCT SUPPORT", "customer impact": "NONE", "sla breach risk": "false" } }, { "id": "c22", "kind": "clean", "context": "One enterprise customer's dedicated cluster is returning 502s on every request since 11:40. Their 400 seats are fully blocked. Other customers are unaffected because clusters are isolated.", "gold": { "priority": "P1 HIGH", "requires escalation": "true", "department": "INFRASTRUCTURE", "customer impact": "LIMITED", "sla breach risk": "true" } }, { "id": "c23", "kind": "clean", "context": "Finance flags that about 300 customers' September invoices were never emailed because the PDF renderer silently failed. The charges went through; only the documents are missing. Affected customers are all on the Enterprise plan.", "gold": { "priority": "P1 HIGH", "requires escalation": "true", "department": "BILLING", "customer impact": "WIDESPREAD", "sla breach risk": "true" } }, { "id": "c24", "kind": "tricky", "context": "A customer reports that our docs page for the REST API returns a 404 for the /v2/charges endpoint. They figured out the new path from the changelog. Nobody else has reported it; engineering says the redirect was dropped in a refactor.", "gold": { "priority": "P2 NORMAL", "requires escalation": "false", "department": "PRODUCT SUPPORT", "customer impact": "LIMITED", "sla breach risk": "false" } }, { "id": "c25", "kind": "clean", "context": "Ransomware note found on a file server used by the internal finance team. The server is isolated from production. Attackers encrypted 2TB of internal spreadsheets and demand payment. No customer systems are reachable from that network segment.", "gold": { "priority": "P1 HIGH", "requires escalation": "true", "department": "SECURITY", "customer impact": "NONE", "sla breach risk": "false" } }, { "id": "c26", "kind": "clean", "context": "Capacity alert on the search cluster: shard rebalancing is 40 minutes behind schedule. Queries are still served within SLA and no customer has complained. The team wants to rebalance during the next maintenance window.", "gold": { "priority": "P3 LOW", "requires escalation": "false", "department": "INFRASTRUCTURE", "customer impact": "NONE", "sla breach risk": "false" } }, { "id": "c27", "kind": "tricky", "context": "Support inbox: 'We updated our card and now the subscription page shows a spinning loader forever. It has been like this for three days and we cannot access anything in the product.' Two more customers on the same plan report the identical symptom.", "gold": { "priority": "P1 HIGH", "requires escalation": "true", "department": "BILLING", "customer impact": "LIMITED", "sla breach risk": "false" } }, { "id": "c28", "kind": "clean", "context": "An employee lost a personal phone that had the corporate email client and was not enrolled in MDM. The device holds cached email and nothing else. The account password has been rotated already.", "gold": { "priority": "P1 HIGH", "requires escalation": "true", "department": "SECURITY", "customer impact": "NONE", "sla breach risk": "false" } }, { "id": "c29", "kind": "clean", "context": "A customer with 3 seats accidentally deleted a project and wants it restored from last night's backup. They have been blocked for two hours. They are not on an enterprise contract.", "gold": { "priority": "P2 NORMAL", "requires escalation": "false", "department": "PRODUCT SUPPORT", "customer impact": "LIMITED", "sla breach risk": "false" } }, { "id": "c30", "kind": "clean", "context": "Global outage: the primary region is unreachable and the failover region never took over because its credentials expired. All customers worldwide are seeing errors on every page. Recovery is not yet started.", This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters "description": "Urgency tier: P0 CRITICAL = core service fully unavailable or data loss; P1 HIGH = major function severely degraded or a customer completely blocked; P2 NORMAL = one or a few users affected with a workaround; P3 LOW = cosmetic, documentation or pre-sales questions.", }, "requires escalation": { "type": "boolean", "description": "true if priority is P0 CRITICAL or P1 HIGH, or if a security event occurred unauthorized access, leaked credentials, expired certificate, malware, exploitable vulnerability ; false otherwise.", "description": "Owning team by the incident subject: BILLING = invoices, payments, subscriptions, charges; INFRASTRUCTURE = servers, databases, latency, capacity, deploys; SECURITY = unauthorized access, credentials, certificates, malware, vulnerabilities; PRODUCT SUPPORT = how-to, configuration or feature questions.", }, "customer impact": { "type": "enum", "choices": "NONE", "LIMITED", "WIDESPREAD" , "description": "NONE = internal only or no customer affected; LIMITED = one customer or a small subset under 10% of users; WIDESPREAD = all users, more than 10% of users, or several paying customers blocked at once.", }, "sla breach risk": { "type": "boolean", "description": "true if an enterprise customer is affected and priority is P0 CRITICAL or P1 HIGH; false otherwise.", }, } FIELDS = list SCHEMA def norm v : if isinstance v, bool : return "true" if v else "false" return str v .strip def pct x : return f"{100 x:.1f}%" def argval flag, default : return sys.argv sys.argv.index flag + 1 if flag in sys.argv else default print f"样本 {len cases } 条（clean {sum 1 for r in recs if r 'kind' == 'clean' } / tricky {sum 1 for r in recs if r 'kind' == 'tricky' }），5 字段 → 共 {len cases 5} 次字段决策"