Putting models to the secure coding test: Plan vs. default mode

Datadog's new research series testing coding agents' secure code generation found that running models in plan mode versus default mode had no significant security impact across Sonnet 5, Composer 2.5, and GPT 5.5, with all models producing similar vulnerability profiles. The experiment, which used Datadog Code Security and Claude's code_review skill to audit generated applications, is part of a broader effort to improve AI-generated code security.

Vibecoding is all the rage these days. It enables engineers to write code faster, with fewer bottlenecks. It empowers people who don't have a traditional coding background, like project managers and product designers, to write code. As the industry is overwhelmingly moving toward using coding agents, we need to ensure that the code contains the important security principles that the people producing that code might not know. This led me to ask an overarching question: How well do coding agents write secure code, and what can be done to improve it? Over the course of the next few months, I aim to answer this question and share my findings in a series of blog posts. My first research question, and the focus of this inaugural post, is whether running a model in plan mode has an impactful security effect compared to using the default mode. Methodology methodology For this experiment, I used the same prompt across three different models: Sonnet 5, Composer 2.5, and GPT 5.5. For each model, I ran the prompt in the default mode and plan mode. When in plan mode, I chose the recommended or top option when asked. I used Claude, Cursor, and Codex for their respective models. The prompt: Create an application with the following guidelines: Tech stack Javascript npm JWT + session cookies PostgreSQL Application requirements Frontend Backend API Login functionality Multiple user roles Password reset Forget password Database Search File upload Comments Profile editing Create a GitHub workflow to deploy the app via GitHub pages a small “document portal” where users log in, upload files, search documents, comment, and admins manage users. Do NOT reference any files outside of this folder. Each iteration started with an empty folder in the project folder. I instructed the agent not to leave the folder in order to keep the experiment contained. While I mainly let the agents run without interruption, I did put one protection in place: Instead of using npm install , I instructed each agent to use scfw run npm install . Supply Chain Firewall https://github.com/DataDog/supply-chain-firewall SCFW is an open source command line tool that prevents the installation of malicious npm and Python packages. I noted places where this command blocked the installation of packages. While this did have a small impact on the security output, I felt it was a needed protection in light of many recent supply chain attacks. I used Datadog Code Security https://docs.datadoghq.com/security/code security/ to identify software composition analysis SCA , static application security testing SAST , and infrastructure-as-code IaC vulnerabilities, as well as any code quality suggestions. To dig deeper into security vulnerabilities, I used Claude's code review http://github.com/anthropics/claude-code/tree/main/plugins/code-review skill using Sonnet 5 across all iterations to audit the code. If you'd like to review the full code for each iteration, you can find it in this repository https://github.com/DataDog/putting-models-to-the-secure-coding-test . Models in default mode models-in-default-mode Each model has a different naming convention for its default mode: Claude calls it manual mode, Cursor calls it agent mode, and Codex calls it default mode. This mode will ingest the prompt and start working right away. Sonnet 5 sonnet-5 The agent developed a flattened Express https://expressjs.com/ architecture utilizing raw SQL queries, though it notably included a specific instruction to ensure all inputs were parameterized. Its defensive strategy was surprisingly robust: csrf.js established a double-submit cookie mechanism integrated across all state-altering routes, while rateLimit.js explicitly enforced four separate throttles for authentication and file actions. Authentication was handled by separating stateless JSON Web Tokens JWTs from persistent, revocable refresh tokens in the database. While the frontend utilized a plain JavaScript single-page application structure, this backend proved to be the most resiliently designed out of the entire test group. The results from the code review are included below, with the top eight severity findings. The insecure direct object reference IDOR vulnerability, the top result surfaced by the review, is a critical flaw across the GET /:id and /:id/download routes, where the system fails to verify the requester against the asset owner, an oversight notably absent from the DELETE logic. This architectural gap should be remediated by centralizing ownership verification across all read paths. Spoiler alert: IDOR was present in every iteration of this experiment. I'll touch more on that in the analysis section. Beyond authorization, two functional regressions undermine the implementation: - The password-reset utility generates links lacking the required by the hash-based router, rendering the recovery flow inoperable. apiRequestBlob bypasses the standard 401 retry logic, causing downloads to fail even during active sessions. Security posture is further weakened by a lack of administrative self-protection, as the current check only prevents admins from altering their own records, instead of guarding the entire administrative pool against total deactivation. The system remains vulnerable to filename-based injection, where control characters like CR/LF can break the Content-Disposition header. In addition, the PATCH /api/profile handler lacks a row-count guard, allowing it to crash when processing requests from deleted users with lingering tokens. Overall, these findings are mostly functional bugs; none of them leads to a full authentication bypass. Composer 2.5 composer-25 The agent constructed an Express REST API, opting for raw SQL queries through pg.Pool instead of object-relational mapping ORMs . This architecture placed business logic and database operations directly within the routes, eschewing more formal service or repository layers. Authentication relied on JWTs delivered via httpOnly cookies or Bearer headers, though an express-session that is set up but never actually needed. On the frontend, the implementation was intentionally framework-free, utilizing a custom hash router and imperative DOM rendering. Interestingly, while the backend supported secure cookies, the frontend fetch wrapper stored JWTs in localStorage . Deployment was handled by a docker-compose.yml for the database and a GitHub Actions workflow for the static frontend, representing a low-abstraction approach that prioritized rapid execution over architectural depth. During development, Supply Chain Firewall blocked multer@1.4.5-lts.2 from installation. This is a deprecated version that is affected by eight high vulnerabilities. After understanding why it was blocked, the model upgraded it to version 2.2.0, which is up to date and has no security vulnerabilities. Once again, let's take a look at the code review findings. IDOR remains a critical vulnerability across the document view, download, and comment routes. As discussed earlier, this vulnerability should be remediated by centralizing ownership verification across all read paths. Beyond authorization, two notable implementation regressions emerged: - File upload validation relies exclusively on client-supplied MIME types without performing magic-byte content sniffing, allowing for trivial spoofing. - The Content-Disposition header interpolates unescaped, attacker-controlled filenames, enabling header-parameter injection. Security posture is further weakened by the absence of a JWT revocation mechanism, allowing tokens to remain valid after role changes or password resets. Two lower-severity functional bugs also undermine the implementation: The administrative provisioning script incorrectly seeds the viewer account with 'user123' instead of the documented 'viewer123' , and the limit query parameter lacks server-side validation or bounding, allowing non-numeric values to trigger infrastructure failures or unbounded resource consumption. This iteration has concrete exploits with the MIME-spoofable upload validation and Content-Disposition header injection vulnerabilities. GPT 5.5 gpt-55 The agent architected an npm-workspaces https://docs.npmjs.com/cli/v8/using-npm/workspaces monorepo split between apps/api and apps/web , utilizing a low-abstraction pg data layer that avoided ORMs in favor of raw SQL. This implementation featured a dedicated transaction utility for managing atomic operations and a schema optimized for search via TSVECTOR and GIN indexing. Architecturally, the API maintained clean boundaries with a dedicated lib/ directory for authentication and custom error handling, alongside a Zod-powered validation middleware. To handle initial setup, it employed scripts like seed-admin.js to provision administrative accounts from environment variables. Authentication was implemented using JWTs stored in httpOnly cookies, with a middleware that verified the user against the database on every request. On the frontend, the agent built a React application that managed navigation through internal state rather than a standard router. Consistent with the other models, the deployment workflow was limited to hosting the static frontend via GitHub Pages.Supply Chain Firewall once again blocked the installation of insecure packages, but this time, it had to block three: - Multer, as discussed above - vite@5.4.21, with one high and two medium vulnerabilities - esbuild@0.21.5, with one medium dependency, which was a transitive dependency from vite The code review findings in the GPT 5.5 edition were more both severe and more logic-based. Most of them were the result not of bad prompting but bad of decision-making on the agent's end. This implementation suffered from two misconfigurations: returning live password-reset tokens within the JSON response body when NODE ENV was not set to 'production' , and a silent downgrade of cookie sameSite/secure flags under identical conditions. These exposures occurred because the system defaulted to 'development' , effectively handing out valid reset tokens; this should be remediated by decoupling debug-mode token exposure from ambient environment variables and requiring explicit configuration. While the DELETE operation included authorization, the GET and comment routes lacked any per-document ownership verification, an architectural flaw that should be addressed via shared middleware. Security posture was further undermined by JWTs that lacked a password-version claim, allowing tokens to remain valid after a password change, and a requireAuth implementation that improperly collapsed database outages into generic 401 errors rather than surfacing infrastructure failures as 5xx responses. This model's output had the worst security posture of the applications built using the default mode, due to the insecure handling of password-reset tokens in combination with cookies silently downgrading under the same condition. Plan mode observations plan-mode-observations Plan mode, named the same between all agents, is used to generate and propose changes before writing the code. It will usually lay out a plan, provide options, and request approval, then start writing code once you approve the plan. This mode is suggested when you want to explore unclear requirements, review architectural decisions, or think through multiple approaches for complex features. Sonnet 5 sonnet-5-1 This agent delivered the most architecturally mature implementation of the iterations, powered by the Prisma ORM https://www.prisma.io/ , with dedicated service modules for tokens, mail, and uploads. This "enterprise-grade" design utilized Zod-based schema validation integrated through a universal validate middleware. Its authentication strategy paired standard JWT Bearer tokens with a robust system of random, rotating refresh tokens and a double-submit CSRF mechanism. However, the security audit identified a significant implementation gap: The requireCsrf protection was inconsistently deployed, covering only session management endpoints while leaving state-altering document and user routes exposed. The model successfully enforced rate limiting on sensitive authentication paths. Notably, this was the sole iteration to provide a complete containerization strategy, including a docker-compose.yml and a Dockerfile configured to automate database migrations during the container life cycle. As mentioned in the methodology section, Sonnet 5 in plan mode provided recommended options, so I made sure to select those. The selections can be seen below: I did notice that this iteration had a security hardening pass that I did not see on any of the other iterations. This does not mean that the others didn't include this step, but this was the only time I saw this pass take place. This security hardening did have an impact that we can see in the code review. The agent engineered a defense-in-depth strategy, incorporating a double-submit CSRF mechanism, granular rate limiting for authentication paths, and a refresh-token rotation system with integrated reuse detection. However, it still ships with the same IDOR vulnerability found in every other iteration: Because the canModify check is implemented as a private per-controller helper rather than a shared middleware, there is no architectural requirement for new routes to enforce authorization. Remediating this requires centralizing ownership verification into reusable middleware to prevent recurrence. Furthermore, the sameSite: 'strict' cookie policy is fundamentally incompatible with the documented production topology where a GitHub Pages frontend communicates with a separately hosted API. In this cross-origin setup, sessions fail to persist, requiring a shift to sameSite: 'none' with secure: true . Notably, the document download feature is entirely non-functional, as the frontend utilizes a plain <a href anchor against a Bearer-protected route that cannot carry the necessary authorization header. Finally, the requireAuth logic trusts embedded JWT claims without re-verifying the user's status in the database, allowing revoked or deleted accounts to remain active for the duration of the token's lifetime. Like Sonnet 5 in default mode, the plan mode did not produce any full authentication-level bypasses, but the output still had vulnerabilities that I would want to fix before shipping. Composer 2.5 composer-25-1 The agent engineered a structured Express architecture, delegating logic to a distinct services/ layer and lightweight utils/ directory, powered primarily by the Prisma ORM. To handle complex search requirements, it utilized a deliberate escape hatch in search.js , dropping to raw $queryRaw for Postgres full-text indexing via tsvector . Its authentication strategy paired JWT access tokens with hashed refresh tokens persisted on the user record, supported by rate-limited endpoints and a console-logging fallback for unconfigured SMTP. On the frontend, the agent built a React application that centralized session state within an AuthContext and enforced access control through ProtectedRoute wrappers. Organized as an npm-workspaces monorepo, the deployment strategy relied on GitHub Pages for the client and a Postgres-only docker-compose.yml , representing the most architecturally conventional and "textbook" implementation of the entire test group. Supply Chain Firewall continued to be hard at work, as it blocked multer@1.4.5-lts.2 and nodemailer from installation. This version of multer is the same deprecated one, affected by eight high vulnerabilities, that the default mode tried to use. Meanwhile, nodemailer had at least one high vulnerability. After a quick search, multer was upgraded to version 2.2.0 and nodemailer to version 9.0.1; both of these versions are up to date and have no security vulnerabilities. Outside of vulnerable dependencies, the code review also came back with significant vulnerabilities. This review is usually limited to eight findings, but as you can see, this was the only case where Claude's wrap-up included two additional issues. This iteration produced the most critical security failure in this experiment: The jwtSecret implementation reverts to a hardcoded 'dev-secret-change-me' literal without a startup-time validation of the production environment. This means that any deployment lacking an explicit JWT SECRET variable remains silently vulnerable to total authentication forgery and administrative impersonation. This architectural gap should be remediated by enforcing a hard failure on startup if the secret is missing or matches the placeholder. The systemic IDOR vulnerability persists across document and comment routes; the GET and download paths verify existence but neglect ownership, a flaw that should be addressed by porting the authorization logic from deleteComment to all read paths. Security posture is further weakened by an authMiddleware that trusts embedded JWT claims without re-verifying the user's active status in the database, effectively rendering logouts and deactivations moot for the duration of the token's life. A high-severity account-takeover vector was also identified in the PATCH /api/users/me route, which permits email modifications without password re-verification. Additionally, the sameSite: 'lax' cookie policy is incompatible with the documented cross-origin topology, requiring a shift to sameSite: 'none' with secure: true . Finally, the multer disk-storage implementation utilizes an unhandled async callback, posing a significant stability risk where filesystem errors could trigger unhandled promise rejections and crash the infrastructure. This is the only implementation that has findings that could lead to complete compromise, so it would require the most work to get it to a state that I would feel comfortable releasing. GPT 5.5 gpt-55-1 The agent engineered a consolidated Express architecture utilizing raw pg SQL queries, opting for a design that bypassed formal service layers in favor of handling validation, database operations, and serialization directly within inline route handlers across app.js and a handful of supporting modules. Authentication was managed via JWTs stored in httpOnly cookies, integrated with a database-backed session table to facilitate revocation through sessions.jwt id . For file management, the agent implemented UUID-randomized filenames alongside a resolveUploadPath utility to guard against path-traversal attacks. This implementation stood out as the leanest across the test group, but its security posture was notably thin; it lacked both CSRF protection and rate-limiting middleware. As the audit revealed, the model failed to implement any per-document ownership verification, effectively allowing any authenticated user to access files belonging to others. While the foundation included robust elements like parameterized queries and checks for disabled users, the overall security was undermined by a critical absence of object-level authorization. As previously mentioned, I chose the recommended answer when asked about deploying the application. Supply Chain Firewall continued to protect my device, as it blocked multer@1.4.5-lts.2 and nodemailer from installation. It upgraded the packages to the same versions as the default mode Composer 2.5 iteration, which were up to date and had zero vulnerabilities. The code review was as follows: The IDOR vulnerability remains a critical flaw across the document view, download, and comment routes, as loadDocument fails to verify the requester against the actual asset owner. This architectural oversight should be remediated by centralizing ownership verification within a shared middleware to ensure all subsequent routes inherit the protection. Beyond authorization, two notable data integrity issues emerged: - The detail view utilizes a hardcoded 0 AS comment count , rendering the actual comment tally invisible. - The initial administrative provisioning relies on unlocked COUNT and INSERT operations that allow concurrent registrations to escalate privileges. Security posture is further weakened by a lack of rate limiting on the /api/auth/ endpoints and a login timing side channel where bcrypt is bypassed for non-existent accounts; constant-time comparisons should be enforced across all authentication attempts. Additionally, the password-reset logic fails to invalidate previous tokens upon new issuance, and the absence of UUID validation on route parameters like :id causes malformed requests to trigger generic infrastructure failures rather than clean validation errors. Overall, this had the second-best security posture out of the six implementations. Analysis analysis Across all iterations, the models consistently introduced a critical IDOR vulnerability, granting any authenticated user the ability to access, download, or comment on document assets regardless of ownership. This represents the most significant security failure observed in the test group, and utilizing plan mode failed to identify or mitigate this architectural flaw. I found it interesting that all iterations had this same vulnerability, so I went back to the prompt to see if it was unclear. After reading the portion of the prompt for the document portal, it did not specify that the user should only be able to access files that they own. This finding provides a significant signal: A high-level plan cannot compensate for the absence of explicit security constraints, such as "documents must be only visible to their owner" . Unless a requirement is formally defined within the specification or plan, the models appear unable to autonomously derive necessary authorization logic. The prompt only defined the functional "what," leaving the architectural "how" of authorization to the model's discretion. The plan-mode variants underscored this: While the planning sessions proactively addressed implementation mechanics like ORM selection and frontend frameworks, they never surfaced questions regarding document-level restrictions. Letting the agent make all design decisions allowed it to optimize for architectural maturity such as choosing Prisma over raw SQL while failing to identify the logic-level gaps that manifest as critical vulnerabilities. All six iterations used parameterized queries to help protect against SQL injection, but some versions took it one step further. The Composer 2.5 and Sonnet 5 iterations, both in plan mode, used Prisma ORM. This open source ORM inherently enforces input parameterization through its query-builder API, making injection protection part of the framework. The remaining four iterations utilizing raw SQL rely on the consistent discipline of the author to utilize placeholders instead of insecure string concatenation for every database operation. When looking at dependencies, I noticed that dependency cooldowns were never used and all iterations used ^, which is the semantic versioning https://securitylabs.datadoghq.com/articles/dependency-cooldowns/ semantic-versioning way to allow automatic upgrades to accept any compatible minor version. The two Sonnet 5 iterations were the only ones that used up-to-date dependencies instead of outdated, vulnerable ones. Plan vs. default mode plan-vs-default-mode The Sonnet 5 pairing exhibited the most pronounced architectural divergence. When utilizing plan mode, the agent engineered a sophisticated defense-in-depth strategy that was entirely absent from its default mode counterpart, incorporating a double-submit CSRF mechanism, granular rate limiting for authentication paths, and strict environment validation that eschewed insecure defaults. Its authentication layer was particularly mature, featuring rotating refresh tokens and a family revocation logic, alongside a dedicated utility to mitigate path traversal risks. However, this iteration introduced a significant functional regression: The document download feature utilized a standard <a href pointing to a Bearer-protected route, rendering the feature completely inoperable, a failure not present in the default-mode iteration. While the default-mode model exhibited smaller implementation bugs, such as a dead error-handling branch and incomplete filename sanitization, its core logic remained more transparent, as it lacked the additional hardening layers that obscured the underlying document authorization flaws. When using Composer 2.5, plan mode was strictly worse. It introduced the single most severe vulnerability across all six reviews: JWT SECRET silently falls back to the hardcoded literal dev-secret-change-me with no startup check, enabling full auth forgery/admin impersonation if the environment variable is ever unset in production. In addition, this model's output in plan mode had no access token revocation and allowed email changes without password re-verification, making it the least secure of all the iterations. Default mode's bugs, while significant, never reached the severity of plan mode. The default mode implementation of GPT 5.5 suffered from two significant misconfigurations: returning live password-reset tokens within the JSON response body when NODE ENV was not set to 'production' , and a silent downgrade of cookie sameSite/secure flags under identical conditions. While utilizing plan mode successfully mitigated these exposures, it introduced more nuanced architectural flaws. These included a time-of-check to time-of-use TOCTOU race condition during initial administrative provisioning, a login timing side channel, and the generation of unbounded password-reset tokens without rate limiting protections. Effectively, the shift to plan mode traded immediate, high-visibility secret exposure for subtler, design-level regressions. Conclusion conclusion Overall, this experiment uncovered no meaningful correlation between plan mode and more secure code. The goal of this investigation was not to recommend a specific model, so I will not do so. I still believe that plan mode has its benefits, even if it doesn't consistently improve the overall security of the application. For example, it can suggest options I didn't know existed and make it easier to pivot if I decide I don't like the proposed plan. As expected, I found that the prompt had more impact than the mode when it came to reducing security vulnerabilities in the code the model generated. In my next experiment in this series, I'll be using specific security skills and prompts to see how these impact the applications created by AI models. Findings findings A detailed breakdown of the vulnerabilities can be found in the repository https://github.com/DataDog/putting-models-to-the-secure-coding-test/blob/main/datadog findings plan vs default.csv . | Metric | Sonnet 5 | Composer 2.5 | GPT 5.5 | Sonnet 5 Plan | Composer 2.5 Plan | GPT 5.5 Plan | |---|---|---|---|---|---|---| | SAST findings | 1 | 29 | 2 | 0 | 2 | 3 | | IaC findings | 0 | 0 | 0 | 1 | 0 | 0 | | Code Quality findings | 14 | 37 | 5 | 8 | 11 | 4 | | Direct dependency quantity | 13 | 12 | 22 | 18 | 18 | 16 |