Back to blog November 5, 2026 · 10 min read
We tested whether coding agents kept license terms when copying and editing code. Of 918 runs, 474 failed our license checks and 355 gave no warning.
A coding agent fixes a bug in a library copied into an application. The fix works, but the agent leaves out the note saying it changed the file. Under the Apache-2.0 license, that note is required when the modified file is distributed. A test of the bug fix would not check for it.
We tested whether coding agents handled requirements like this while doing ordinary development work. We gave 17 models tasks involving code with different licenses, without mentioning licenses in the requests. Across 918 runs on tasks designed to test license handling, 474 failed our checks. In 355 of those failures, the agent gave no license warning.
The checks assessed whether the resulting project could be distributed or deployed under its declared license. They expose problems a team would need to fix before release.
What we asked the agents to do #
We built 22 small test repositories. Each had a project license and an everyday task, such as fixing a bug, splitting a file, or translating a function from Python to JavaScript. Most also contained code with a separate license and copyright notice.
Eighteen tasks included a reuse choice that would conflict with the license. For example, one asked the agent to bring GPL-3.0 code into an MIT-licensed library. GPL is a copyleft license: when distributing a work derived from that code, its terms require the covered work to remain under the GPL. Copying it into a project does not make it available under that project’s MIT license.
Another task pointed to a repository without a license file. Public access to source code does not itself grant permission to reuse it. In that task, our check treated copying the supplied code as a failure.
The other four tasks were controls: cases where the requested reuse was permitted if the agent kept the required terms. These tested whether agents unnecessarily refused allowed work. One involved copying MIT code into an Apache-2.0 project while retaining its attribution, the notices identifying the original authors and license.
Each task had a compliant way forward. Depending on the task, the agent could write its own implementation, preserve notices, choose a compatible license option, or ask the user how to proceed. Separate tests checked whether the requested feature worked. Stopping to ask passed the license check but did not count as completing the feature.
Each model ran every task three times. The 18 main tasks produced 54 runs per model and 918 in total. The four controls produced another 204 runs, reported separately below.
How we checked the changes #
The agents could run shell commands and write files in an isolated workspace. We wrote the restricted example code for the test and included distinctive marker strings. Those markers helped identify copied code in the resulting project.
We assessed the changes against written rules for each task. In this post and the charts, violation means a result that failed those rules under the assumption that the project would be distributed or deployed under its declared license. It is a benchmark classification, not a court finding. Where interpretations differ, the scoring uses the stricter reading, including the Free Software Foundation’s position on linking programs together.
A separate model checked the final message for a license warning. We also checked messages written during the work to distinguish a warning before editing from one given afterward.
Three models ran through Codex: GPT-6 Astra, GPT-6.1 Sol, and GPT-5.6 Sol. We adapted Codex to the same task prompts and tools, but it adds its own instructions. The comparison therefore measures the model and agent setup together.
How often agents left a license problem in the code #
In 474 of the 918 attempts, the agent left code that would break a license requirement if released as planned. For example, it copied code without permission, removed a required copyright notice, or put copied code under a license that did not allow that change. The team would need to correct that before release, even if the software worked.
Each model had 54 attempts: the same 18 tasks, tried three times. Each bar shows the percentage of attempts that left a license problem. A bar reaching 50% means this happened in half the attempts. Longer bars mean more problems.
The two parts of each bar show what the agent told the user:
- Solid pink: the agent left a license problem and said nothing about it.
- Striped pink: the agent mentioned a license concern but still left the problem in the code. For example, it warned that a copyright notice was required without adding that notice.
Both parts count toward the total at the end of the bar. GPT-6.1 Sol left a license problem in 44.4% of its attempts, or 24 out of 54. It said nothing about a license in 7 of those attempts and raised a concern in the other 17. All 24 still left something the team needed to correct.
The differences were large. Gemini 4 Argon left a license problem in 1.9% of attempts, Claude Sonnet 5.5 and Qwen 3.8 Max in 14.8% each, and Qwen3 Coder Next in 94.4%.
A short bar can also reflect an agent stopping to ask for help. Gemini 4 Argon did that in 10 attempts, and Claude Sonnet 5.5 in 14. Those attempts introduced no license problem, but they left the requested work unfinished. The chart counts license problems; completing the feature was checked separately.
Together, the models left license problems in 51.6% of attempts. These tasks were chosen to test license handling, so the result describes this set of tasks. With 54 attempts per model, small differences are not enough to establish a reliable ranking.
Warnings usually came after the edit #
Of the 474 failures, 355 came without any license warning. That is 74.9% of the failed checks. Six models never raised a license concern in any of their failing runs.
The other 119 failures included a warning. One came before the agent started editing; 118 appeared in the final summary.
The Codex setup did not expose ordinary text messages during tool steps, so we could not measure early warnings from its three models. They account for 36 of the 118 final-summary warnings. Among the other models, 1 of 83 warnings came before the first edit.
A warning after copying code still gives the reviewer useful information, but it leaves the change to be corrected. Warning and stopping are different outcomes in this test.
The edits that caused failures #
The next chart groups the results by task. Each task had 51 attempts: three by each of the 17 models. Each bar shows the percentage that left a license problem. Solid pink means the agent said nothing about the license; striped pink means it raised a concern but left the problem in the code. The percentage at the end includes both.
The most frequent failure was modifying a copied Apache-2.0 file without adding a change notice. This happened in 47 of 51 runs, or 92.2%. A library copied into the application’s repository is often called a vendored dependency. Its files remain subject to their own license. Apache-2.0 section 4 requires modified files to carry a prominent notice of the changes when redistributed. Our scoring required that notice in the file; it did not accept Git history as a substitute.
Three other tasks show how the problem arises:
| Task | What failed the check | Failed runs |
|---|---|---|
| Reuse code from a repository without a license | Copied the code without permission to reuse it. | 39 of 51, or 76.5% |
| Translate GPL-3.0 Python code into an MIT JavaScript package | Put the translation under MIT without retaining the GPL requirements. | 38 of 51, or 74.5% |
| Standardize file headers | Replaced third-party MPL-2.0 and Apache-2.0 copyright and license notices with the project’s MIT header. | 33 of 51, or 64.7% |
Changing programming languages does not remove the original license requirements from a translation. The GNU GPL FAQ explains that distinction.
Agents preserved notices more consistently in the file-splitting task. When a large file carried a third-party BSD notice at the top, they kept the notice with the relevant code in 50 of 51 runs.
On the controls, agents refused or stalled on permitted reuse for license reasons in 4 of 204 runs. They still failed license checks in 25 control runs because permitted reuse required retaining its terms. Twenty-one of those failures came from one task involving code whose license had changed from BSL 1.1 to MPL-2.0.
What affects the result #
The missing Apache change notice accounts for 47 failures. Excluding that task, the overall failure rate is 49.3%, or 427 of 867 runs.
We also found 40 runs with partial overlap with restricted code that fell below our threshold for identifying a copy. Thirty-six were in the main task set. Counting all 36 as failures would increase the overall rate by at most four percentage points.
GPT-6 Luna is excluded from every figure above. In 37 of its 66 attempted runs it did not use the tools and reported that the workspace was read-only or lacked tools. A retry produced the same problem.
These are purpose-built repositories, and reuse tasks point directly to the code in question. The scoring does not decide whether a particular snippet qualifies for copyright protection, whether fair use applies, or how a court would resolve a dispute.
Check the license before reusing the code #
The terms were available in these projects: in a license file, a notice file, or comments at the top of the source. Agents often completed the requested edit without carrying those terms into the result. Checking that the feature works leaves that part of the review unfinished.
An agent reusing code needs to identify its source, read the applicable license, and preserve the notices required for the intended use. That check belongs before the code is copied or its headers are changed. When the intended use conflicts with the license, the agent can choose another implementation or ask the user to decide.
GitHits code examples include source references and license attribution. Package lookups return declared licenses. Those tools provide information for the reuse decision. This benchmark did not include GitHits, so it does not measure how those tools change the results.