A pull request is green.
The specialist reviewers found nothing that should block it. The risk classifier marks the change as low risk. The human-review branch disappears, and the change continues toward production.
Every component may have done exactly what it was asked to do.
The remaining question is whether the system was right to stop asking a human.
That question jumped out at me in Gergely Orosz's diagram of OpenAI's agentic software factory.
The public workflow includes code-writing agents, CI, specialist agent reviews, risk classification, deployment agents, production monitoring, and feedback loops. The article says that areas of the codebase can opt into automatic approval for low-risk pull requests, while higher-risk changes can receive stricter review.
I have not tested OpenAI's system. I am looking at the evaluation problem the design raises.
The small diamond marked "Low-risk change?" is not merely organizing work. Its answer can determine whether human review remains in the release path.
That makes it a release decision.
A green CI run can provide strong evidence about the change under the conditions the pipeline checked.
The code built. The selected tests passed. The linters accepted it. A performance harness may have found no unacceptable regression. Specialist review agents may have found no issue within their assigned domains.
All of that matters.
None of it automatically proves that the change belonged in the low-risk route.
The classifier is making a different claim. It is saying that the available evidence, affected surface, expected consequence, and uncertainty are compatible with less human scrutiny.
That claim needs evidence of its own.
Otherwise, a team can evaluate every component around the gate while leaving the routing decision itself mostly assumed. The code is tested. The review agents are monitored. The deployment agent watches production. But the decision that removed the human is treated as a label rather than behavior that can be right or wrong.
Risk is not a permanent property attached to a file or type of change.
A change that was low risk yesterday may stop being low risk after a dependency changes, a permission boundary moves, a feature becomes widely used, or a once-correct operational rule is reversed.
The code can still pass the same local checks.
The route can still be wrong.
That is why evaluating the gate requires more than collecting examples of clean deployments. Easy successes mostly confirm that obvious low-risk changes can pass through a low-risk path. The harder cases sit near the boundary.
For example: These are challenge cases, not claims about observed OpenAI failures. Their purpose is to ask whether the classification still holds when the surrounding conditions become less convenient.
The gate should not be judged only by whether the pull request was green when it arrived.
It should be judged against the consequence of the route it selected.
Did the supposedly low-risk change preserve the required production state?
Did it trigger a rollback, incident, manual intervention, or delayed repair?
Did monitoring capture the condition that actually mattered, or only the signals the deployment agent expected to matter?
Would an informed reviewer, given the same evidence available at classification time, have selected the same route?
This does not mean every negative production event proves the classifier was wrong. Systems fail for many reasons, and hindsight can make risk look more obvious than it was. The evaluation has to preserve what was knowable when the decision was made.
But it also cannot stop at the classifier's own explanation. A confident risk label is still a claim produced by the system being evaluated.
The useful answer comes from connecting the decision to independent evidence about the resulting state.
If a team wants to defend an automatic approval later, the record needs to make the decision reconstructable. At minimum, I would want to know:
That record turns "the system marked it low risk" into something another person can examine.
It also creates the foundation for replay. When a policy, dependency, or requirement changes, the team can rerun earlier decisions and ask whether changes previously sent down the easy path would still belong there.
Without that evidence, the gate can report that it made the correct decision, but the organization cannot show why the decision was defensible.
The Nugalaxy Evaluation Cases examine this broader gap between visible success and the behavior the evidence can actually support.
An agentic software factory can make code creation, review, deployment, monitoring, and repair much faster.
That speed changes the role of the risk classifier. It is no longer sorting a small queue for convenience. It is controlling where scarce human judgment enters a high-volume system.
False positives have a cost. If the gate sends harmless changes to humans too often, the bottleneck returns and reviewers learn to distrust the alerts.
False negatives have a different cost. The system removes review precisely where a human might have challenged the assumptions shared by the surrounding agents.
Both errors matter, but they are not equally consequential in every part of the codebase. The evaluation has to reflect the real cost of the wrong route, not merely report overall classification accuracy.
This is where the harness contract matters. The system needs to capture the decision inputs, expected route, executed route, and production outcome in a form that can be compared consistently. Read more about harness engineering here.
The public diagram shows an ambitious feedback system. Code is written, checked, reviewed, classified, deployed, observed, and repaired through connected agentic loops.
The more capable that factory becomes, the less useful it is to ask only whether each pull request passed CI.
The factory is also deciding when its own evidence is sufficient to proceed without a person.
That decision should be evaluated as seriously as the code it lets through.
A green build can show that the change passed the checks.
It cannot, by itself, show that removing the human was safe.