{"slug": "turning-database-grading-into-rl-rewards-post-training-agents-with-thinkingbox", "title": "Turning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox", "summary": "Microsoft and Hugging Face reported that 50 steps of GRPO post-training on Qwen3.8-27B, using executable database-state checks from ThinkingBox-format tasks as rewards, raised average success on ThinkingBox-Bench from 51.70% to 60.78% and lifted tasks passing all 20 attempts from 38 (7.50%) to 107 (21.10%), while pass@20 fell from 453 tasks (89.35%) to 446 (87.97%). The training pool was selected using what the teams call offline policy-relative difficulty, estimating each task's success probability once with the base policy before training and holding the pool fixed. ThinkingBox-Training is now available on GitHub, and the benchmark comprises 507 tasks that check an agent's final database state rather than its claims of completion.", "body_md": "[Viewer • Updated  •  513 •  326  •  16](https://huggingface.co/datasets/microsoft/ThinkingBox-Bench)  \n\n# \n\t\tTurning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox\n\t\n\n [Enterprise Article](https://huggingface.co/blog)\n\n This is the follow up to our first blog post, [The Agent Said It Was Done. The Database Disagreed.](https://huggingface.co/blog/microsoft/thinkingbox) ThinkingBox-Training is now available on [GitHub](https://github.com/microsoft/thinkingbox-training).\n\n*We built ThinkingBox to evaluate an agent's work by comparing the final database state with the expected outcome. We then used the same kind of executable signal as a reward on training tasks outside ThinkingBox-Bench. For Qwen3.8-27B, average success rose from 51.70% to 60.78%, while tasks completed correctly on all twenty attempts rose from 38 to 107 with 50 GRPO steps.*\n\n*Figure 1. ThinkingBox turns executable checks of terminal database state into rewards for post-training agents on tasks outside ThinkingBox-Bench.*\n\n#### \n\t\tThis is a joint blog by Microsoft and Hugging Face, special thanks to Mirco Milletari and Xiang Jiang from Microsoft, Sergio Paniego from Hugging Face and our former intern Zhuochun Li (University of Pittsburgh) for co-authoring/reviewing efforts.\n\t\n\nWe built ThinkingBox to catch agents that say they are done when they are not.\n\nEach of its 507 tasks in ThinkingBox-Bench (our benchmark dataset) hands an agent a real backend and a set of tools, lets it work, then checks what really changed in the database. [In the first post](https://huggingface.co/blog/microsoft/thinkingbox), we caught a model closing a customer's ticket as **resolved** while the carrier exception behind it was still open. Nine competent-looking tool calls, one **wrong** database row.\n\nThen we noticed what we had built. For most tasks, an executable check can return 1 or 0 directly from the final database state. That is not just a grader. **This can be a reward function.**<sup>+</sup> \n\nSo we constructed a training pool from ThinkingBox-format tasks outside the benchmark and used their executable outcomes as rewards. The catch was choosing **which** tasks are suitable for agent training: most tasks are either already solved or hopeless for the model in front of you, and both teach it nothing.\n\nDifficulty is policy-relative: a task is hard for the current model only until that model learns it.\n\nWe call this **offline policy-relative difficulty**. We estimated each task's success probability once using the base policy before training and kept the selected training pool fixed throughout the run.\n\n### \n\t\tWhat happened after training\n\t\n\nWe used offline policy-relative difficulty to build the training pool, then fine-tuned Qwen3.8-27B with GRPO for 50 steps. To recap, we evaluate each of the 507 benchmark tasks 20 times. pass@1 measures average success across all 10,140 attempts. pass@20 measures the share of tasks solved at least once. Observed 20/20 measures the share of tasks that pass every recorded attempt.\n\n*Table 1. Qwen3.8-27B results before and after 50 steps of GRPO on the offline-selected training pool.*\n\n| Metric | Base |  | RL fine-tuned | Change | \n|---|---|---|---|---|\n| **Observed 20/20** | 38 (7.50%) | → | **107 (21.10%)** | **+69 tasks** | \n| pass@1 (successful attempts) | 5,242 (51.70%) | → | **6,163 (60.78%)** | **+9.08 pt** | \n| pass@20 (tasks solved at least once) | 453 (89.35%) | → | **446 (87.97%)** | **−7 tasks** | \n\nTraining sharply **increased** repeatable success, with a small decline in at-least-once coverage. The rest of the post explains how we selected practice tasks, built the reward loop, and measured that tradeoff.\n\n**What does that improvement look like on one task?** A traveler requests a room downgrade, but company policy requires manager approval first. Before training, the agent changed the booking and issued refunds without approval in 15 of 20 attempts. After RL fine-tuning, it preserved the booking and payment state in all 20.\n\n*Figure 2. One corporate hotel-booking task, 20 attempts per model. The base Qwen3.8-27B model changed the booking without required manager approval in 15 attempts. The RL fine-tuned model preserved booking and payment state in all 20; all expected records matched in 19.*\n\n## \n\t\tContents\n\t\n\n*In a hurry? The headline result is above. The implementation is in [Run the training stack](#run-the-training-stack).*\n\n## \n\t\tWhat the benchmark found, in sixty seconds\n\t\n\n[ThinkingBox-Bench](https://huggingface.co/datasets/microsoft/ThinkingBox-Bench) grades agents on the database records they leave behind. Each of its 507 tasks starts a real backend, hands the agent a set of MCP tools and a simulated LLM user with private context, and at the end runs executable checks against the terminal database state. Checks are against the backend record that needs to be correct, not the specific tool calls. This incentivizes the model to discover possibly new solution paths rather than “rediscover” the human-backed solution, that might be suboptimal albeit correct.\n\n[In our first post](https://huggingface.co/blog/microsoft/thinkingbox) we ran eighteen models through that benchmark twenty times each and reported a gap that headline accuracy hides. Kimi-K3 solved 93.89% of tasks *at least once* but completed only 68 of 507 tasks (13.41%) on **all twenty recorded attempts**. Claude Opus 5 inverted it: narrower coverage, far more consistency.\n\nAnd bigger or better model did not close it. Claude Opus 5.5 scored 0.66 percentage points higher than Claude Opus 5 on pass@1, yet passed **exactly the same number of tasks, 241**, on all twenty attempts. That gain in headline accuracy bought zero additional dependability.\n\nIf you are choosing a model for work that touches production database records, pass@1 should not be read alone. Observed 20/20 shows which tasks succeed on every attempt, which is closer to what production reliability requires.\n\nThe executable terminal-state verdict was also the training reward. Before unpacking the training loop, here is what changed on the sealed benchmark.\n\n## \n\t\tResults: the dependability column moved\n\t\n\nAfter training Qwen3.8-27B, we evaluated the selected step-50 checkpoint on the 507-task benchmark it had never trained on. The 507 tasks represent an out-of-distribution test set: their domains, databases, and most MCP tools were not encountered during RL training.\n\n*Table 2. ThinkingBox-Bench pass@1 (%) by domain, for selected models from [our first blog's Table 2](https://huggingface.co/blog/microsoft/thinkingbox) and our fine-tuned model for different domains and overall. Observed 20/20 numbers are over the entire benchmark with 507 tasks. Each model is evaluated on all 507 tasks for 20 repeated trials. Bold marks the column leader; underline marks the runner-up.*\n\n| Model | Retail | Auto insurance | Travel | Neobank | Consulting | Overall | Observed 20/20 | \n|---|---|---|---|---|---|---|---|\n| Claude Opus 5.5 | __80.97__ | **68.40** | 54.28 | **71.25** | __61.58__ | **67.16** | **241 (47.53%)** | \n| Claude Opus 5 | 80.71 | __65.80__ | 49.95 | __70.62__ | **66.19** | __66.50__ | **241 (47.53%)** | \n| GPT-5.4 | 76.33 | 62.65 | **68.12** | 65.34 | 54.60 | 65.36 | 128 (25.25%) | \n| GPT-5.6 Sol | 67.65 | 65.30 | 60.34 | 59.09 | 57.52 | 61.91 | 82 (16.17%) | \n| **Qwen3.8-27B RLFT** | 67.91 | 56.45 | __62.69__ | 62.40 | 54.51 | 60.78 | 107 (21.10%) | \n| Claude Sonnet 4.6 | 72.35 | 54.40 | 58.94 | 56.39 | 54.31 | 59.19 | 102 (20.12%) | \n| GPT-6 Astra | 71.73 | 46.55 | 55.87 | 60.87 | 56.83 | 58.31 | __231 (45.56%)__ | \n| Kimi-K3 | **82.24** | 50.80 | 61.83 | 41.35 | 51.63 | 57.37 | 68 (13.41%) | \n| Qwen3.8-27B | 64.03 | 47.85 | 53.41 | 47.88 | 45.69 | 51.70 | 38 (7.50%) | \n\n**The headline result is the observed-20/20 row in Table 1.** A **27B model** that initially completed 38 (7.5%) of ThinkingBox-Bench on every attempt now completes 107 (21.1%) on every attempt, almost thrice. For comparison, observed 20/20 for Kimi-K3 is 68, GPT-5.4 with 128, GPT-6 Astra has 231, and Claude Opus 5.5 leads with 241. \n\n**Average success and repeatability produce different orderings.** In Table 2, we notice that GPT-5.6 Sol scored slightly higher on pass@1 than the RL fine-tuned Qwen3.8-27B (61.91% versus 60.78%) but passed fewer tasks on all 20 recorded attempts: 82 versus 107. Claude Sonnet 4.6 scored lower on pass@1, at 59.19%, yet passed nearly the same number of tasks every time, at 102. Neither metric alone captures the full picture.\n\nThe observed pattern is consistent with sharpening: repeatable success rose while at-least-once coverage fell slightly. This is in-line with [prior work](https://arxiv.org/abs/2504.13837). One explanation is that probability distribution moved toward trajectories that work repeatedly while some low-probability paths disappeared. \n\nOverall, the model did not become broader. It became more repeatable. The next question is where those gains landed.\n\n## \n\t\tWhere the gains landed\n\t\n\n*Figure 3. Domain-level base scores, RLFT scores, and percentage-point changes. Green change cells show gains and red cells show declines; blue shading indicates the score level within each metric.*\n\nPass@1 and observed 20/20 rose in all five domains. Pass@20 was mixed: it increased in neobank and consulting, but declined in retail, travel, and auto insurance. Auto insurance had the largest breadth loss (−7.0 points), while retail had the largest observed-20/20 gain (+24.5 points).\n\n## \n\t\tWhat dependability costs\n\t\n\nThe [first post](https://huggingface.co/blog/microsoft/thinkingbox) priced consistency using simple math: divide the **estimated cost of the full 20-run campaign** by the number of tasks the model passed every single time.\n\n**Cost per dependable task = estimated cost of the 507 × 20 campaign ÷ tasks passing 20/20**\n\nApplying the same cost math to the fine-tuned checkpoint's measured token usage:\n\n*Table 3. Estimated evaluation cost and dependability for base and RL fine-tuned Qwen3.8-27B.  Cost per successful attempt divides total estimated campaign cost (507 x 20) by the number of passing attempts.*\n\n| Measure | Base |  | RL fine-tuned | Change | \n|---|---|---|---|---|\n| Campaign cost, 20 runs | $925.80 | → | **$855.58** | **−7.6%** | \n| Cost per successful attempt | $0.177 | → | **$0.139** | **−21.4%** | \n| Tasks passing 20/20 | 38 | → | **107** | **2.8× as many** | \n| **Cost per dependable task** | $24.36 | → | **$8.00** | **−67.2%** | \n\nAt $8.00 per dependable task, the fine-tuned model came close to GPT-5.4 ($6.80), GPT-6 Astra ($7.45), and Claude Opus 5.5 ($7.80) in the companion comparison.\n\n#### \n\t\tThe single-attempt view\n\t\n\nFigure 4 prices cost per *successful attempt* against pass@1.\n\n*Figure 4. The golden arrow traces Qwen3.8-27B base pareto cost position to the RLFT (diamond): pass@1 rises from 51.70% to 60.78% while estimated cost per successful attempt falls from $0.177 to $0.139. Ringed circles mark the Pareto cost frontier.*\n\nThe move places the fine-tuned checkpoint close to GPT-5.6 Sol, although GPT-5.6 Sol remains cheaper and slightly higher on pass@1. Four models priced in [our paper](https://arxiv.org/abs/2608.19741) fall outside the plotted range.\n\nThe dollar values are a comparative serving index, not invoice or cloud bills. Serving the fine-tuned checkpoint would be the same as the base model. The calculation includes model inference cost, and excludes training run and policy-probe cost for curating the training dataset.\n\nThese results tell us what moved. The next sections explain the training loop and why selecting the right tasks matters the most.\n\n## \n\t\tThe training loop\n\t\n\n*Figure 5. The six stages, now in context: candidate tasks feed in, the first three run once offline, the last three repeat for every group in the training run. The trained checkpoint is evaluated against ThinkingBox-Bench.*\n\nSelection runs once, offline, against the base policy model, i.e in this case Qwen3.8 27B; training then uses that fixed pool that has been curated. The dashed amber arrow in Figure 5 marks the path we did not take: re-probing difficulty during the training run.\n\nEvery rollout is a complete episode, not a text completion. It gets a fresh backend, an isolated MCP session, and GPT-5 Chat as the simulated user. Most rewards come from deterministic terminal-state checks; a minority of narrow rubric requirements can invoke the configured judge (GPT-5 Chat).\n\n**Training and evaluation must interpret Qwen output identically.** That includes reasoning extraction, tool calls, argument typing, rendering, and isolated task state. Otherwise the reward measures a different agent from the one evaluated. The shared interpretation code is in [`trainer/core/tb_interpret.py`](https://github.com/microsoft/thinkingbox-training/blob/8c04db18da362357dc8a2d4ab187f44a8d63e20a/trainer/core/tb_interpret.py).\n\nFor the reference run, we ran full-parameter RL fine-tuned Qwen3.8-27B on 187 tasks using token-mean GRPO. Each update generated 18 groups of 8 rollouts, or 144 episodes, across 24 H100s. Training ran synchronously with parameter and optimizer offload; AdamW used a `1e-6` learning rate and zero weight decay. We ran the training for 50 steps.\n\nZero-variance groups were replaced [DAPO-style](https://arxiv.org/abs/2503.14476) during training. System errors were handled separately: a backend timeout is not an incorrect agent outcome, so affected groups were refilled rather than scored as failures.\n\nThat explains how retained tasks were trained. The remaining question is which tasks entered the training pool.\n\n## \n\t\tOffline policy-relative difficulty\n\t\n\nBefore diving deep into the selection of training tasks, a disclaimer:\n\n**No ThinkingBox-Bench task was included in the RL training pool.** The benchmark was reserved for evaluation.\n\nThinkingBox-Bench is out-of-distribution relative to the RL training pool: its domains, databases, policies, and most tools were not encountered during RL training. The two sets still share the ThinkingBox framework, simulated-user machinery, executable terminal-state grading, a few common tools, and a similar task style. These shared elements limit the scope of the transfer claim.\n\nWhy selection matters comes down to one arithmetic fact about GRPO. If all eight rollouts fail, the reward vector is `[0, 0, 0, 0, 0, 0, 0, 0]`. If all eight succeed, it is `[1, 1, 1, 1, 1, 1, 1, 1]`. In either case, centering within the group produces no task-reward advantage because no trajectory did better than another.<sup>++</sup>\n\nA mixed group such as `[0, 0, 0, 1, 1, 1, 1, 1]` supplies the contrast GRPO can learn from. That distinction is expensive for a stateful agent: every rollout provisions an isolated backend, runs a real tool session, and calls a separate model to play the user.\n\nThe question that shaped this work was therefore not only *which model or optimizer?* It was: **which candidate tasks should enter the fixed training pool?**\n\nWe used an offline pass-rate filter on the posterior to select the training pool. It estimates which tasks are likely to produce contrasting terminal rewards under the base policy.\n\nSo we probe. Each candidate task is run repeatedly against the **base policy** model. The plain-language rule is simple: tasks the model always fails are unlikely to teach it yet; tasks it always passes are already learned; **tasks it solves sometimes are where outcomes can provide contrast.**\n\n### \n\t\tEstimating difficulty from limited probes\n\t\n\nEach candidate task is run *n* times, producing *k* successes. We model its latent success probability with Jeffreys posterior probability:\n\n**success probability | k successes in n probes  ~  Beta(k + ½, n − k + ½)**\n\n*Figure 6. Offline policy-relative difficulty is estimated from repeated base-policy probes and used to retain tasks likely to produce contrasting rewards.*\n\nWe partition the unit interval into five buckets with boundaries at 0.08, 0.40, 0.60, and 0.90, chosen as hyperparameters for these runs. Each task is assigned to the bucket containing the greatest posterior mass:\n\n- **Rarely solved [0, 0.08):** drop\n- **Difficult [0.08, 0.40):** keep\n- **Intermediate [0.40, 0.60):** keep\n- **Frequently solved [0.60, 0.90):** keep for full-parameter RLFT\n- **Nearly always solved [0.90, 1]:** drop\n\nFurther details are in [Appendix C.4 of our ThinkingBox paper](https://arxiv.org/pdf/2608.19741).\n\n### \n\t\tOne task, end to end\n\t\n\n*Figure 7. Zooming in, one task through all six stages: GPT-5.4 screening, base-model probes, difficulty classification, a rollout group, terminal-state grading, and a GRPO update.*\n\nFigure 7 follows one candidate task that can potentially enter the training pool. For the Qwen3.8 27B full-parameter pool, GPT-5.4 first served as a solvability screen. The base policy then succeeded on few of the probes e.g 3 of 8, producing a Beta(3.5, 5.5) posterior. Its largest mass, 54.0%, fell in the Difficult band.\n\nWhen sampled for training, the task produced the mixed reward group `[0, 0, 0, 0, 1, 1, 1, 1]`. Successful trajectories therefore received above-mean task advantage and failed trajectories below-mean advantage, giving GRPO contrast to learn from. A different policy model could place the same task in a different difficulty band.\n\nThe final Qwen3.8-27B curated training pool contained 187 tasks.\n\n## \n\t\tWhat earns a reward\n\t\n\nNothing about the trajectory. The reward is the task's own executable verdict.\n\nWhen an episode ends, the sandbox extracts what actually changed in the backend and runs the task's deterministic checks against the terminal state. Any trajectory that produces the correct end state passes. Wrong, missing, or extra side effects fail. The reward is 1 or 0.\n\nIn the benchmark, the database grader can be thought of as the “examiner”: it caught agents that reported success and left the wrong record. In training, the same kind of check can be seen as the “teacher” on a different, disjoint set of tasks.\n\n**Two reward paths, one binary verdict**\n\n**Executable check (the large majority).** When the required outcome is represented in terminal state or side effects, a deterministic test accepts or rejects it; no LLM judge call is needed.\n**Judge-assisted check (a minority).** When a narrow requirement cannot be determined cleanly from database state, the configured LLM judge answers a binary rubric question used by the executable test.\n\n## \n\t\tRun the training stack\n\t\n\nWe release the data-selection, rollout, reward, and Verl integration used to train Qwen3.8-27B in [microsoft/thinkingbox-training](https://github.com/microsoft/thinkingbox-training). The repository README is the entry point. The detailed guides are:\n\n- [`docs/data-preparation.md`](https://github.com/microsoft/thinkingbox-training/blob/8c04db18da362357dc8a2d4ab187f44a8d63e20a/docs/data-preparation.md) for task hydration, policy probes, benchmark exclusion, and train/validation splitting;\n- [`docs/reference-q38-run.md`](https://github.com/microsoft/thinkingbox-training/blob/8c04db18da362357dc8a2d4ab187f44a8d63e20a/docs/reference-q38-run.md) for the validated 24-H100 Qwen3.8 configuration;\n- [`docs/evaluation.md`](https://github.com/microsoft/thinkingbox-training/blob/8c04db18da362357dc8a2d4ab187f44a8d63e20a/docs/evaluation.md) for merging a checkpoint and running ThinkingBox-Bench 507 x 20.\n\n### \n\t\tBefore you start\n\t\n\nTested on Linux and WSL with Python 3.12 and [uv](https://docs.astral.sh/uv/). You need NVIDIA GPUs with a compatible CUDA/NCCL stack, a local Qwen3.8-27B checkpoint, and OpenAI-compatible endpoints for the simulated user and judge. The validated reference configuration uses 3 nodes x 8 H100 GPUs; smaller custom configurations depend on model size, context length, and rollout concurrency.\n\nThe public sample task list exercises the interface. It is not the private 187-task training pool used for the reported run. Supply an authorized training source that remains disjoint from the public benchmark.\n\n### \n\t\tInstall\n\t\n\nKeep the three repositories side by side:\n\n```\nmkdir -p ~/thinkingbox-workspace\ncd ~/thinkingbox-workspace\n\ngit clone https://github.com/microsoft/thinkingbox.git\ngit clone https://github.com/microsoft/thinkingbox-data.git\ngit clone --branch main https://github.com/microsoft/thinkingbox-training\n\ngit -C thinkingbox checkout 892964e7226044e5463ad188da119df838f4bec1\ngit -C thinkingbox-data checkout thinkingbox-bench-v1.0\n```\n\n### \n\t\tCreate the validated environment\n\t\n\nThe released stack targets Linux, Python 3.12, Verl 0.9.0, and the pinned package lock:\n\n```\ncd ~/thinkingbox-workspace/thinkingbox-training\n\nuv venv --python 3.12\nsource .venv/bin/activate\n\nuv sync --frozen --extra fast --extra tracking --extra eval-proxy\nuv pip install -e ../thinkingbox\nuv pip install --config-settings editable-mode=compat \\\n  -e ../thinkingbox-data/servers/thinkingbox_tools\nuv pip install --config-settings editable-mode=compat \\\n  -e ../thinkingbox-data/servers/tb_business_ops_servers_202606\n\npython scripts/prepare_verl.py --dest .deps/verl --install\npython scripts/verify_verl_install.py\nuv pip check\n```\n\nThe final two commands should report a verified patched Verl checkout and no broken requirements. Do not run another automatic package sync afterward: it can replace the patched source installation with the vanilla Verl wheel.\n\n### \n\t\tPrepare the training data\n\t\n\nStart with the publicly provided samples, or add your own authorized dataset in ThinkingBox format. Then run the repository's preparation script:\n\n```\ncd ~/thinkingbox-workspace/thinkingbox-training\nsource .venv/bin/activate\npython data/prepare_data.py --help\n```\n\nThe script aggregates policy probes, applies offline policy-relative difficulty filters and benchmark exclusions, and writes the selected `train.yaml` and `test.yaml` files. [`docs/data-preparation.md`](https://github.com/microsoft/thinkingbox-training/blob/8c04db18da362357dc8a2d4ab187f44a8d63e20a/docs/data-preparation.md) documents the input fields, loader contract, and complete command.\n\n### \n\t\tConfigure and inspect the launch\n\t\n\nFollow [`docs/reference-q38-run.md`](https://github.com/microsoft/thinkingbox-training/blob/8c04db18da362357dc8a2d4ab187f44a8d63e20a/docs/reference-q38-run.md) to start Typesense and the MCP servers, configure the simulated user and judge, set the model, task-list, checkpoint, and topology variables, and prepare the Ray cluster. Keep credentials in runtime secret injection rather than committed files or shell history.\n\nBefore allocating a training job, use the guide's dry-run command:\n\n```\nDRY_RUN=true ./scripts/run_train.sh \\\n  trainer.total_training_steps=50 \\\n  trainer.save_freq=5\n```\n\nIt must exit zero without starting Ray or allocating GPUs. Read the emitted `python -m verl.trainer.main_ppo` command and every resolved override before launching the run.\n\n### \n\t\tTrain\n\t\n\nThe launcher configures training; it does not provision hosts. On the reference three-node topology, export the same environment on every node, mount `CHECKPOINT_DIR` at the same absolute path, then start Ray:\n\n```\n# head node\nROLE=head NODE_IP=\"$HEAD_IP\" GPUS_PER_NODE=8 \\\n  ./scripts/start_ray.sh\n\n# each worker node\nROLE=worker NODE_IP=\"$WORKER_IP\" \\\n  HEAD_ADDRESS=\"$HEAD_IP:6379\" GPUS_PER_NODE=8 \\\n  ./scripts/start_ray.sh\n```\n\nAfter `ray status` reports all 24 GPUs, launch once from the head node:\n\n```\nexport RAY_ADDRESS=auto\nexport GPUS=0,1,2,3,4,5,6,7\n\n./scripts/run_train.sh \\\n  trainer.project_name=thinkingbox-training \\\n  trainer.experiment_name=q38-full-rlft \\\n  trainer.total_training_steps=50 \\\n  trainer.save_freq=5\n```\n\n### \n\t\tEvaluate the checkpoint through OpenEnv\n\t\n\nThe [first ThinkingBox\npost](https://huggingface.co/blog/microsoft/thinkingbox#run-it-yourself)\ncovers Typesense, the MCP Session Proxy, and OpenEnv setup. Merge the trained\ncheckpoint following\n[`docs/evaluation.md`](https://github.com/microsoft/thinkingbox-training/blob/8c04db18da362357dc8a2d4ab187f44a8d63e20a/docs/evaluation.md),\nthen serve it with vLLM.\n\nFrom the OpenEnv repository root, set `MERGED_MODEL` to the merged model\ndirectory and `THINKINGBOX_DATA` to the pinned benchmark checkout. The\nhistorical checkpoint evaluation used this eight-A100 serving configuration:\n\n```\nvllm serve \"$MERGED_MODEL\" \\\n  --served-model-name q38-rlft \\\n  --host 127.0.0.1 \\\n  --port 8001 \\\n  --tensor-parallel-size 8 \\\n  --dtype bfloat16 \\\n  --max-model-len 262144 \\\n  --gpu-memory-utilization 0.85 \\\n  --max-num-seqs 16 \\\n  --max-num-batched-tokens 8192 \\\n  --language-model-only \\\n  --enable-chunked-prefill \\\n  --enable-prefix-caching \\\n  --reasoning-parser qwen3 \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser qwen3_xml\n```\n\nIn a separate terminal, start the OpenEnv compatibility proxy for the Qwen chat template from the ThinkingBox Training checkout. It listens on port 8014 and forwards to vLLM on 8001:\n\n```\ncd ~/thinkingbox-workspace/thinkingbox-training\nsource .venv/bin/activate\nbash scripts/launch_sysmerge.sh\n```\n\nKeep it running during evaluation. From another terminal, check readiness:\n\n```\ncurl --fail http://127.0.0.1:8014/readyz\n```\n\nUpdate the agent entry in the existing `thinkingbox.yaml`; retain its MCP, simulated-user, and judge configuration:\n\n```\norchestrator:\n  agent_model:\n    type: aoai\n    endpoint_url: http://127.0.0.1:8014/v1/chat/completions\n    deployment: q38-rlft\n    credential:\n      type: api-key\n      api_key: <your_api_key>\n    is_reasoning: true\n    temperature: 1.0\n    top_k: 20\n    top_p: 0.95\n    max_completion_tokens: 8192\n    timeout: 1800\n```\n\nStart OpenEnv as in the first post, with `OPENENV_TB_CONFIG` pointing to this\n`thinkingbox.yaml` and `OPENENV_TB_DATASET` to `$THINKINGBOX_DATA`. Once\n`/ready` succeeds, run the full protocol of 507 tasks × 20 attempts:\n\n```\nuv run --project envs/thinkingbox_env --frozen \\\n  thinkingbox-eval \\\n  --config \"$PWD/thinkingbox.yaml\" \\\n  --base-url http://127.0.0.1:8000 \\\n  --dataset \"$THINKINGBOX_DATA\" \\\n  --output q38-rlft-full.jsonl \\\n  --errors-output q38-rlft-full-errors.jsonl \\\n  --repeat 20 \\\n  --coverage-profile full \\\n  --aggregate-output q38-rlft-full-aggregate.json \\\n  --message-timeout 1800\n```\n\nThe `full` profile validates all 10,140 expected task/repetition keys before\naggregation.\n\n## \n\t\tWhere this goes next\n\t\n\nThe useful part of this work is not one Qwen3.8-27B checkpoint. It is the training loop and training data curation that turns executable outcomes into rewards.\n\nIf you are training or evaluating an agent that touches real database records:\n\n1. **Probe before you train.** Take your own agent task set, run the base policy against it*n* times, and look at the band histogram. In our experience teams often have not inspected this distribution, and it is usually more lopsided than expected.\n2. **Check your parity.** Replay the same recorded model output through the training and evaluation parsers, and verify that both extract identical tool names and arguments. Otherwise, training may reward behavior that evaluation interprets differently.\n3. **Report both coverage and repeatability.** State how many tasks pass at least once in*k* attempts and how many pass all*k* . A model that occasionally finds a solution is not necessarily one that repeats it consistently.\n\nFurther details can be found in the following links:\n\n- **Environment:**[`envs/thinkingbox_env`](https://github.com/huggingface/OpenEnv/tree/main/envs/thinkingbox_env)\n- **OpenEnv:**[ThinkingBox environment documentation](https://huggingface.co/docs/openenv/environments/thinkingbox)\n- **Framework:**[microsoft/thinkingbox](https://github.com/microsoft/thinkingbox) ·[tutorial](https://github.com/microsoft/thinkingbox/blob/main/docs/tutorial.md)\n- **Benchmark:**[microsoft/thinkingbox-data](https://github.com/microsoft/thinkingbox-data) ·[v1.0 release](https://github.com/microsoft/thinkingbox-data/releases/tag/thinkingbox-bench-v1.0)\n- **Dataset viewer:**[microsoft/ThinkingBox-Bench](https://huggingface.co/datasets/microsoft/ThinkingBox-Bench)\n- **Paper:**[arXiv:2608.19741](https://arxiv.org/abs/2608.19741) or[HF](https://huggingface.co/papers/2608.19741)\n- **RL training:**[microsoft/thinkingbox-training](https://github.com/microsoft/thinkingbox-training)\n- **Blog:**[The Agent Said It Was Done. The Database Disagreed.](https://huggingface.co/blog/microsoft/thinkingbox)\n\n### \n\t\tCite the paper\n\t\n\n```\n@article{li2026thinkingbox,\n  title   = {One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows},\n  author  = {Li, Zhuochun and Ko, Youngmin and Keramati, Ali and Kundu, Tuhin and\n             Tsai, Liang-Chun and Ferri, Nicola and Milletari, Mirco and Liu, Jiaxiang and\n             Lopez Pelaez, Susana Palmaz and Wang, Yuepeng and Smolyakov, Vadim and\n             Jiang, Xiang and Olafsson, Kjartan and Guy, Tommy},\n  journal = {arXiv preprint arXiv:2608.19741},\n  year    = {2026},\n  url     = {https://arxiv.org/abs/2608.19741}\n}\n```\n\nThinkingBox code is MIT-licensed; the benchmark data is [CDLA-Permissive-2.0](https://cdla.dev/permissive-2-0/).\n\nDisclaimer: every task in the public benchmark is a synthetic reconstruction. The workflows and policies are modeled on real enterprise agent patterns; the customers are not real.\n\n*ThinkingBox, ThinkingBox-Bench and ThinkingBox-Training is built by the Microsoft Copilot Studio team in partnership with Toloka with collaborators from the University of Pittsburgh, Northwestern University, Columbia University and UC Irvine who interned at Microsoft. Questions are welcome in the comments section below or on [github](https://github.com/microsoft/thinkingbox/discussions) or by email at [firstnamelastname@microsoft.com](mailto:firstnamelastname@microsoft.com).*\n\n<sup>+</sup> *A minority of tasks also include narrow binary rubric requirements for outcomes that the database cannot capture cleanly; those checks can invoke the configured LLM judge.*\n\n<sup>++</sup> *That zero-gradient problem has a standard batch-time fix: oversample, discard the degenerate groups, refill. The full-parameter run did exactly that, similar to [DAPO](https://arxiv.org/abs/2503.14476) with replacement of zero-variance groups. But refilling still pays full price for every wasted rollout before throwing it away.*", "url": "https://wpnews.pro/news/turning-database-grading-into-rl-rewards-post-training-agents-with-thinkingbox", "canonical_source": "https://huggingface.co/blog/microsoft/thinkingbox-training", "published_at": "2026-10-22 22:24:34+00:00", "updated_at": "2026-10-22 22:42:41.316679+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "machine-learning", "large-language-models", "ai-tools"], "entities": ["Microsoft", "Hugging Face", "ThinkingBox", "ThinkingBox-Bench", "ThinkingBox-Training", "Qwen3.8-27B", "GRPO", "Mirco Milletari"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/turning-database-grading-into-rl-rewards-post-training-agents-with-thinkingbox", "markdown": "https://wpnews.pro/news/turning-database-grading-into-rl-rewards-post-training-agents-with-thinkingbox.md", "text": "https://wpnews.pro/news/turning-database-grading-into-rl-rewards-post-training-agents-with-thinkingbox.txt", "jsonld": "https://wpnews.pro/news/turning-database-grading-into-rl-rewards-post-training-agents-with-thinkingbox.jsonld"}}