A Stranger Writes Half of My AI Matcher's Prompt

A developer behind the Upwork Scout job-alert side project described hardening its Claude Haiku-based job matcher against indirect prompt injection by constraining the model's output to a strict JSON schema with a bounded score and a 200-character reason string, returning null on any failure so jobs fall back to plain filter matching. The developer argued that prompt wording cannot reliably prevent injection, so the design instead limits what the feature can reach, keeps authority in application-level user permissions, and treats model output as untrusted input by escaping it in the alert email template just like job titles and descriptions.

Every night a cron job in Upwork Scout, my job-alert side project, asks Claude Haiku how well a freelance job fits a specific freelancer. The prompt has two halves. One half is the freelancer's own profile. The other half is the job post: title, budget, skills, client stats and up to 1,400 characters of description. I did not write that description. A stranger on Upwork did. It lands in the model's context next to my scoring rules, and the model has no reliable way to tell my instructions apart from text that merely sits near them. That is the whole AI security problem in one paragraph. We tend to argue about prompt wording; the fixes that work are boring engineering around the model, the same rules you apply to any input you did not write. Normal software keeps a clear line. Code decides what happens; data gets processed. A language model erases that line. Your instructions, the user's question, a retrieved document and an uploaded file all arrive as one stream of text, and the model acts on all of it. Two things follow. Anyone who can put text in front of the model has some influence over what it does. And the output is generated, not looked up, so you cannot fully predict it. The direct version of prompt injection is a user typing "ignore your instructions". The one that matters more is indirect: the instruction rides inside content the model reads on someone's behalf, like an email, a knowledge base document, a support ticket or a job post. No prompt wording reliably fixes this. So I stopped asking "can it be injected?" and started asking a better question. Design so the answer is small. For the Upwork Scout matcher, it is honestly small: score and a string reason , with additionalProperties: false . null on any failure so that job falls back to plain filter matching. js const parsed = JSON.parse text as { score: number; reason: string }; const score = Math.max 0, Math.min 100, Math.round Number parsed.score ; if Number.isFinite score return null; return { score, reason: String parsed.reason || "" .slice 0, 200 }; So the worst case is a job post that talks its way to a 100 with a glowing one-line reason, landing at the top of somebody's alert email. That is a real failure. A freelancer could spend Connects on a bad job because my ranking told them to. But it is a bounded failure. Nobody's data leaves, nothing gets written, and the damage is one row in one email. Compare that with a feature that can query the whole account, or one with write tools. Same injection, very different worst case. That is why the first control is not in the prompt at all: limit what the feature can reach, and keep authority in your application, decided by the signed-in user's permissions rather than by whatever the model concludes from the conversation. I wrote about the write-tools case separately in giving an AI agent write access one verb at a time https://dev.to/nabeelbaghoor/give-an-ai-agent-write-access-one-verb-at-a-time-32o6 , and about a voice-agent version of the read case in the lookup tool that just took a phone number https://dev.to/nabeelbaghoor/nobody-hacked-it-the-lookup-tool-just-took-a-phone-number-1hhh . This is the rule I wish more teams internalized. Whatever the model returns should be handled like input from an unknown user, because partly it is. A stranger's text shaped it. In Upwork Scout, that reason string goes two places: the alert email and the dashboard. The dashboard is React, so the string is rendered as text by default. The email is a hand-built HTML template, so the reason goes through the same tiny escape function as the job title and description: function esc s: string : string { return s.replace /&/g, "&amp;" .replace /</g, "&lt;" .replace / /g, "&gt;" ; } Nothing clever. The point is that model output takes the same path as every other untrusted string, not a special "it came from our AI, it is fine" path. The rendering risk is worse when your UI renders rich text. A model can be steered into producing a markdown image or link whose URL carries user data in the query string. The moment the page loads that image, the data has left your system without anyone clicking anything. If you render model output as markdown, restrict link destinations, do not auto-load external images, and escape HTML by default. The same applies downstream. If output feeds a query, a spreadsheet, an email or code that runs, validate it at that boundary. The most damaging failure in a business AI feature is rarely a weird answer. It is a correct answer built from data the user should never have seen. Three places this happens: A public AI feature is a model someone else pays for. People will try to use your support bot as a free general assistant, push huge inputs through it, or script it. Here Upwork Scout is a mixed report card, and I would rather be honest about it. The nightly matcher only runs on jobs that already passed the user's hard filters, verdicts are cached per user and job, and there is a cap of 150 AI scores per user per day. That part is in good shape, and the cost side is in cheap filters first, LLM last https://dev.to/nabeelbaghoor/cheap-filters-first-llm-last-running-an-ai-matcher-inside-a-cron-job-702 . The profile import endpoint, where a user pastes their Upwork profile and Claude structures it into fields, requires a signed-in session and truncates input to 24,000 characters, but it has no per-user rate limit yet. Writing this post is how I noticed. It is on the list. Basic controls cover most of this: per-user and per-account rate limits, input size limits, sign-in for expensive actions, a narrow scope that declines unrelated requests, and spending alerts on the provider account. The broader cost story is in model calls are metered https://dev.to/nabeelbaghoor/model-calls-are-metered-most-ai-features-are-built-like-they-are-free-1e4 . For every vendor you send data to, know what you send, whether they retain it, whether it trains anything, and where it is processed. Then look at your own logs. Prompts and responses are gold for debugging, so keep them, but they contain whatever users typed and whatever your system retrieved. They need the same access control, retention and redaction as the database the data came from. A side note from my own matcher: on failure it logs only the job ID and the error message, not the prompt, which is the kind of decision that is easy to make on day one and annoying to retrofit. Before launch, and again whenever the model, the prompt or a data source changes: AI features are not secured by clever prompts. They are secured the way the rest of your app is: access decided in your own code, untrusted input and output treated as untrusted, each component limited in what it can reach, and a clear picture of where data goes. Do that from version one and the model's unpredictability becomes a quality problem instead of an incident. The longer, less code-heavy version of this, written for founders evaluating a team building AI into their product, is on the Null Studio blog: AI feature security https://nullstud.io/blog/ai-feature-security/ . What is the worst thing an injected instruction could do in a feature you have shipped? I am curious how many of us have actually written that answer down.