Microsoft Research released Orchard, an open-source framework for training and evaluating AI agents across software engineering, web navigation, and personal-assistant tasks, on August 3. The framework centers on Orchard Env, a Kubernetes-native environment service for reusable sandbox management. Microsoft and the accompanying arXiv paper report strong open-model results, including 73.0% on SWE-bench Verified with value-model reranking.
Microsoft Research released Orchard, an open-source framework for scalable agentic AI research, on August 3. The project combines a reusable environment layer, training recipes, data, and evaluation methods for agents that operate across software engineering, browser-based computer use, and personal-assistant workflows.
At the center of the release is Orchard Env, a lightweight Kubernetes-native service for sandbox lifecycle management. Microsoft Research describes it as a reusable layer spanning task domains, agent harnesses, and pipeline stages, including data collection, reinforcement-learning rollouts, and evaluation. The release materials state that the infrastructure can be used with deployment harnesses including Codex, OpenClaw, and ZeroClaw.
Three agent recipes
The accompanying Orchard paper presents three domain-specific implementations built on the environment layer:
- • Orchard-SWE targets software-engineering agents. The paper describes credit-assignment supervised fine-tuning, Balanced Adaptive Rollout for sparse-reward reinforcement learning, on-policy distillation, rubric-based process rewards, and historical-experience distillation for value-model reranking. - • Orchard-GUI targets vision-language agents that use graphical interfaces. According to the paper, its 4B model was trained from 0.4K distilled trajectories and 2.2K open-ended tasks. - • Orchard-Claw targets personal-assistant tasks. The paper reports training with 0.2K synthetic tasks and evaluation on Claw-Eval.
The arXiv version revised July 30 reports that Orchard-SWE, built on the Qwen3.5-35B-A3B backbone, reached 69.7% on SWE-bench Verified with rubric-based process-reward RL and 73.0% with value-model reranking. The authors characterize the latter as a state of the art among open-source methods and state that it approaches frontier systems using models more than 10 times larger.
For GUI tasks, the paper reports a 68.4% average success rate across WebVoyager, Online-Mind2Web, and DeepShop. It reports that Orchard-Claw reached 59.6% pass@3 on Claw-Eval, rising to 73.9% when paired with the ZeroClaw harness.
Versioned benchmark results matter
Microsoft's May publication page reports earlier Orchard-SWE results: 64.3% on SWE-bench Verified after supervised fine-tuning and 67.5% after supervised fine-tuning plus RL, using Qwen3-30B-A3B-Thinking. The later arXiv revision uses a different backbone and adds later-stage training and reranking results. Practitioners comparing claims should therefore treat the 67.5% and 73.0% figures as results from different versions and configurations, rather than directly interchangeable scores.
The technical contribution is less the existence of another agent harness than the attempt to make the environment layer reusable across agent domains and training stages. In comparable agent-training programs, environment fidelity, sandbox reproducibility, and evaluation repeatability can become limiting factors alongside model quality. An open implementation that separates these components could make it easier for research teams to reproduce rollouts, carry data pipelines across tasks, and test alternative training objectives.
The reported results also illustrate a broader pattern in agent research: task-specific data, reward design, and inference-time selection can materially affect outcomes even when the base model is relatively small. Benchmark users will need to assess the released environments, task contamination controls, harness settings, and evaluation protocols alongside headline scores, particularly where results depend on reranking or a stronger external harness.
Key Points #
- 1Microsoft released Orchard Env as a Kubernetes-native reusable sandbox layer, targeting agent training, RL rollouts, and evaluation across multiple domains.
- 2The revised paper reports 73.0% on SWE-bench Verified with reranking, but comparisons require attention to backbone, training stage, and inference configuration.
- 3Reusable environments can reduce replication friction in agent research, where sandbox fidelity and evaluation consistency often constrain cross-task experimentation.
Scoring Rationale #
Orchard is a substantial open research release spanning agent environments, training recipes, and benchmarked models across three task domains. Its reported SWE-bench and GUI results are notable for teams studying reproducible agent training, though practical impact depends on release completeness and independent replication.
Sources #
Primary source and supporting public references used for this report.
Practice interview problems based on real data
1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.