{"slug": "i-tested-rails-baseline-by-leaving-architecture-out-of-the-prompt", "title": "I Tested Rails Baseline by Leaving Architecture Out of the Prompt", "summary": "Rails Baseline, a production-ready Rails foundation, was tested by its developer by handing the packaged ZIP to fresh coding-agent sessions without architecture instructions. The test, using Claude and Codex, showed that agents could discover and follow the repository's existing patterns for tenancy, authorization, plan limits, and entitlements, with one agent even catching and fixing a policy-scope mistake. The developer concluded that a starter designed for agents should be tested by giving incomplete product prompts to see if agents can find the right patterns.", "body_md": "Note:\n\n[Rails Baseline]is the production-ready Rails foundation I tested here. It gives a new SaaS application and the coding agents working in it clear, conventional patterns to build from.\n\nI did something a little different before shipping Rails Baseline 1.0.\n\nInstead of spending another week adding starter features I thought somebody might need, I took the actual packaged ZIP, opened it as if I had just bought it, and handed it to fresh coding-agent sessions.\n\nThen I gave them product prompts.\n\nI did not tell them to use Pundit. I did not say to scope records through the current account, use public IDs, put paid access behind Entitlements, or use Solid Queue for background work. Those were all patterns already in the repository, and I wanted to see if the agent would find them without the prompt turning into an architecture checklist.\n\nOne test used Claude. The other used Codex.\n\nBoth went pretty well.\n\nThat sentence is intentionally boring. I was not trying to figure out which model could win a Rails benchmark. I wanted to test the codebase.\n\nRails Baseline is a normal Rails application, but part of the reason I built it was for coding agents.\n\nI have spent years building SaaS apps and eventually wrote *Build A SaaS App in Ruby on Rails 8*. A lot of the hard-won stuff from doing that is not a particular controller action or gem install. It is knowing where certain decisions belong and trying not to make a different version of the same decision six months later.\n\nCoding agents make this more obvious because they can produce so much code so quickly.\n\nIf the repository has a good pattern, the agent can usually find and continue it. If the repository is silent, the agent still has to build something. At that point it either asks, guesses, or follows whatever general Rails pattern seems reasonable from training.\n\nA starter that claims to work well with agents should probably be tested by giving an agent incomplete product instructions and seeing what happens.\n\nSo that is what I did.\n\nThe first test started from the RC1 release ZIP, outside the Rails Baseline source repository. I onboarded it as a new application and gave a fresh Claude session a prompt to build a small feedback board product.\n\nThe basic product was pretty normal SaaS stuff. Accounts could create boards, anonymous visitors could leave feedback, feedback had statuses, and plan limits changed how many boards an account could have. Paid accounts also got a CSV export.\n\nI intentionally described those things as product requirements.\n\nI did not say something like:\n\n`Current.account`\n\nfor tenancy.`Entitlements.enabled?`\n\nfor paid capabilities.`Limits.value`\n\nfor board counts.`HasPublicId`\n\nfor URLs.That would prove I can write a detailed prompt. I already know I can do that.\n\nThe more interesting question was whether Claude would inspect the app and reach those conclusions itself.\n\nIt mostly did.\n\nThe board records were account-owned. Authorization followed the existing policy shape. The free and paid board counts went through the application's plan-limit boundary instead of being scattered through controllers. The paid CSV feature used the entitlement layer. Public URLs followed the existing public-ID convention.\n\nThere was also a policy-scope mistake during the build. Claude caught it and fixed it during its own pass.\n\nGood. That is much closer to what I actually want from an agent session than a first pass that looks impressive but quietly breaks account isolation.\n\nThe CSV export exposed something Rails Baseline had not really answered yet.\n\nWhere should a generated file live?\n\nIt would have been easy to react by adding Active Storage, S3 setup, an Export model, and some generic file-delivery service to the starter. That felt like exactly the kind of thing I did not want Baseline to become.\n\nA lot of products do need durable generated files. Plenty do not. A CSV that can be generated quickly and downloaded immediately may not need to be stored at all.\n\nSo the answer became documentation instead of another core subsystem. I added a generated-artifacts recipe that talks through when synchronous generation is enough and when durable storage becomes a product decision.\n\nThat gave me something else to test later.\n\nThe second test started from RC2 and used a fresh Codex session.\n\nThis prompt was more demanding. Build an account-owned endpoint monitor that checks public HTTP/HTTPS URLs on a schedule. Free accounts get three monitors checked every 15 minutes. Paid accounts get 25 monitors checked every five minutes. Keep recent check history, support a paid CSV export, and optionally expose a public status page.\n\nAgain, the prompt described the product. I did not name the Rails Baseline implementation pieces I expected it to discover.\n\nCodex found most of them.\n\nIt used account scoping and the existing Pundit context. It used persisted public IDs, Entitlements, Limits, tenant-aware jobs, and the Solid Queue recurring configuration. For the CSV export, Codex found the generated-artifacts recipe from the first dogfood and chose not to invent persistent storage for a file that could be generated on demand.\n\nThat last part made me pretty happy. The first test found a missing decision. I documented it, and a completely fresh agent used that documentation in the second test.\n\nThat is basically the behavior I wanted from Rails Baseline in the first place.\n\nAn endpoint monitor has one security problem that a feedback board does not have: the application has to make HTTP requests to URLs supplied by users.\n\nCodex correctly treated that as an SSRF problem instead of just checking whether the string started with `https://`\n\n.\n\nIts implementation resolved DNS, rejected private and other unsafe address ranges, pinned the connection to the validated address, used explicit timeouts, verified TLS, disabled environment proxies, and did not follow redirects.\n\nRails Baseline did not teach any of that.\n\nI do not really consider that a failure of the starter. Safe outbound HTTP is one of those capabilities that only some products need, and there are enough security details that \"just use `Net::HTTP`\n\n\" is not much of a recipe.\n\nStill, the test found a real boundary where an agent had to leave the existing precedent and design something new. That became one of the small recipes added before 1.0.\n\nThe scheduling side found another one.\n\nBaseline already showed recurring Solid Queue work and tenant context inside jobs, but it did not show the shape needed when each account's schedule depends on product state. Codex ended up with a minute-level dispatcher that finds monitors which are due and enqueues their actual check jobs.\n\nThat also became a short recipe instead of a new scheduling framework.\n\nThe endpoint monitor app ended up calling the model `HttpMonitor`\n\nbecause `Monitor`\n\ncollides with a Ruby constant and `EndpointMonitor`\n\ncollided with the onboarded application namespace.\n\nThe routes still used the product-facing name:\n\n```\nresources :monitors\n```\n\nAt one point, a view used Rails' polymorphic routing shortcut:\n\n```\n<%= link_to monitor.name, monitor %>\n```\n\nRails saw an `HttpMonitor`\n\ninstance and looked for `http_monitor_path`\n\n, which did not exist. The route helper was `monitor_path`\n\n.\n\nThat one made it through the agent's test suite, and I hit it manually while clicking around the app.\n\nThe fix was easy, but I liked finding this too. A coding agent following the architecture well does not mean it stops making normal implementation mistakes. It also gave Baseline one more small Rails-specific recipe: if your model class and route resource intentionally use different names, either avoid polymorphic routing there or test a navigation path that exercises it.\n\nNo framework required.\n\nThe useful result was not \"Claude passed\" or \"Codex passed.\"\n\nBoth agents were capable enough that the interesting differences came from what the repository gave them to work with.\n\nWhen Rails Baseline had a clear local pattern, the agents usually found it. Account ownership, authorization, plan access, limits, jobs, IDs, and normal Rails/Hotwire UI conventions all carried through into new product code without me naming those pieces in the prompt.\n\nWhen the app did not have a pattern, the agent had to make a product decision. Sometimes that was exactly right. Other times it exposed a missing piece of documentation that was general enough to add back to the starter.\n\nThat seems like a healthier way to build an agent-friendly codebase than trying to anticipate every future feature with another abstraction.\n\nI would rather have 20 boring, well-used patterns and a handful of good recipes than 80 generic service objects waiting for a use case.\n\nThe best part of the process was how close it felt to an actual customer using the product.\n\nI was not running tests inside the Baseline source repo and congratulating myself because they were green. The ZIP was packaged, copied somewhere else, onboarded into a new app, and then handed to an agent that had no history from building Rails Baseline.\n\nThat caught setup issues, product gaps, documentation gaps, and one wonderfully ordinary Rails routing bug.\n\nFor future Baseline releases, I think this kind of test is more valuable than adding another sample feature just so the feature list gets longer.\n\nGive a fresh agent a product request. See what it copies. See where it guesses. Fix the parts that should have been obvious from the repository.\n\nThen stop before the starter turns into the product it is supposed to help you build.\n\nIf you want to see the Rails app these tests were run against, Rails Baseline is at [railsbaseline.com](https://railsbaseline.com).", "url": "https://wpnews.pro/news/i-tested-rails-baseline-by-leaving-architecture-out-of-the-prompt", "canonical_source": "https://dev.to/rob__race/i-tested-rails-baseline-by-leaving-architecture-out-of-the-prompt-4f3p", "published_at": "2026-08-28 05:56:53+00:00", "updated_at": "2026-08-28 06:19:19.010960+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["Rails Baseline", "Claude", "Codex"], "alternates": {"html": "https://wpnews.pro/news/i-tested-rails-baseline-by-leaving-architecture-out-of-the-prompt", "markdown": "https://wpnews.pro/news/i-tested-rails-baseline-by-leaving-architecture-out-of-the-prompt.md", "text": "https://wpnews.pro/news/i-tested-rails-baseline-by-leaving-architecture-out-of-the-prompt.txt", "jsonld": "https://wpnews.pro/news/i-tested-rails-baseline-by-leaving-architecture-out-of-the-prompt.jsonld"}}