Most prompt engineering articles explain techniques: Zero‑shot, Few‑shot, Chain‑of‑Thought, ReAct, Tree‑of‑Thought. Production engineers need something different: a framework that tells them which prompting strategy to choose based on the task, trade‑offs, and required context.
This article presents that framework.
Start with a production problem
Imagine you’re building a service that converts natural language into SQL.
Zero‑shot succeeds on simple queries, but starts failing once joins, aggregations, or nested filters enter the picture.
Do you add more instructions? Examples? Chain‑of‑Thought? A schema?
This isn’t a “prompting trick” problem. It’s a design decision.
In this article, you’ll get:
A simple decision tree for choosing prompt patterns.
A short 💡 Production Tip from an engineering perspective
Think of it as an engineering design framework for production prompts.
The core architecture: pattern + context → prompt → system
Before diving into patterns, anchor the big picture:
Task drives the choice of Pattern.
Pattern tells you what Context you need.
Pattern + Context form the Prompt.
The LLM output goes through Validation before hitting your Application.
This is the mental model for the rest of the article.
Engineering mental model
A production prompt is two things:
Pattern — how the model should think and structure its work Examples: Zero‑shot, Schema‑driven, Few‑shot, Chain‑of‑Thought (CoT), ReAct, Tree‑of‑Thought (ToT), self‑critique, prompt chaining.
Context — what the model needs to know
Instructions and constraints
Examples
JSON / DB / API schemas
Retrieved documents (RAG) Tool / function definitions
Logs, code, requirements
Evaluation rubrics
Stop conditions
Context engineering isn’t about attaching more information — it’s about attaching the right information. Irrelevant context increases token cost, can distract the model, and often reduces answer quality. Curate context the way you curate APIs: minimal, versioned, and purposeful.
Your job is to:
Pick the right pattern for the task and constraints.
Curate the right context package so the pattern can work reliably.
Everything below is built around this idea.
The decision tree
Start here when designing any LLM‑backed feature.
What is the task?
Classification / tagging
Extraction / structuring
Generation (text, code, SQL, configs) Reasoning / planning
Multi‑step / tool‑using (agents) Review / critique (code, design, security)
- Does it require non‑trivial reasoning?
If no → favor simpler patterns (Zero‑shot, Schema‑driven, Few‑shot).
If yes → consider CoT, ToT, ReAct.
- Do you need strict output format?
If yes → Schema‑driven / structured output.
If no → free‑form text / reasoning patterns.
- Do you need tools or external data (APIs, DB, search, logs)?
If yes → ReAct.
If no → pure‑reasoning or single‑step patterns.
Use this as your first pass. Then refine with the pattern details below.
Pattern selection matrix
These patterns aren’t competitors — they optimize for different constraints. The goal isn’t to use the “most advanced” technique, but the simplest one that reliably solves your task.
A simple way to visualize the escalation path:
Prompt architecture
No matter the pattern, most production prompts share a common structure:
You don’t always need all of these, but thinking in these blocks makes prompts easier to test, version, and reuse.
Modern models already perform well with clear instructions.
Don’t use when
SQL generation, financial reports, legal text.
Code generation needing strict syntax.
High‑stakes classification where mistakes are costly.
Trade‑offs
Prompt components
Role (optional): “You are a senior backend engineer.” Task: one clear sentence.
Constraints: length, style, “no extra text”.
Output format: “Respond with a single token: INFO / WARN / ERROR.”
Input: clearly delimited.
What to attach
Clear instruction + output format.
Domain constraints (SLOs, policies). Production example
Classifying log lines for ingestion into a telemetry pipeline where latency matters and low‑volume mistakes are acceptable.
💡 Production Tip Before moving from Zero‑shot to Few‑shot, first check whether the failures are caused by ambiguous instructions. Clearer prompts are often cheaper than adding examples.
Pattern 2: Schema‑Driven / Structured Output
Use when Downstream code expects a fixed shape: JSON, SQL, configs, API specs.
You want testable, parseable outputs.
Don’t use when
You only need free‑form text for human consumption.
You have no validation or retry logic on the backend.
Trade‑offs
Prompt components
Schema definition (JSON Schema, SQL grammar hints, or clear example).
Strict instruction: “Respond ONLY with valid JSON following this schema. No extra text.”
1–2 examples of valid outputs (if schema is non‑trivial).
Task + constraints.
Modern LLM APIs often support structured outputs or function calling. When available, use native schema validation instead of relying solely on prompt instructions.
What to attach
JSON schema / OpenAPI / DB schema.
Naming conventions or style rules.
Production example
Generating SQL from natural language with a defined DB schema and strict output format for parsing and testing.
💡 Production Tip Treat the schema as part of your API contract. Version it, test against it, and enforce it in code — not just in the prompt.
Pattern 3: Few‑Shot
Use when You need a specific format or style (JSON, code style, SQL dialect).
Moderate‑complexity tasks where examples reduce variance.
You have a small set of clean “golden” outputs.
Don’t use when
You need hundreds of examples to compensate for a weak schema or unclear instructions.
The core issue is reasoning, not format.
Trade‑offs
Prompt components
Short role + task.
2–5 representative examples:
Diverse inputs.
Exactly the output format you want.
Clear separation between examples and real input.
Output schema or style notes.
What to attach
Representative examples.
Output schema or style guide snippet.
Production example
Extracting fields from raw logs into a canonical JSON structure for ingestion into a data warehouse.
💡 Production Tip If you find yourself adding more examples, and improve your schema and instructions first. Examples amplify clarity; they don’t replace it.
Pattern 4: Chain‑of‑Thought (CoT)
Chain‑of‑Thought prompting was introduced by Wei et al. (2022) and shown to elicit multi‑step reasoning in LLMs (arXiv:2201.11903). [web:51][web:61]
You can afford extra tokens/latency for higher correctness and explainability.
Don’t use when
Simple classification, extraction, or tasks where single‑pass is already reliable.
You’re extremely latency/cost sensitive and don’t need reasoning.
Trade‑offs
Prompt components
Reasoning should be requested only when the task genuinely requires multi‑step analysis or trade‑off evaluation. For many modern LLMs, concise task and constraint descriptions are sufficient.
Evaluation criteria (optional): “Consider latency, cost, and maintainability.”
Final answer separator: “Final answer: …” so you can parse it. Optionally 1–2 examples of “reasoning + final answer”.
What to attach
Problem statement + constraints.
Architecture docs or requirements.
SLOs or business context.
Production example
Designing a retry policy for an HTTP client in a microservice. You want the model to reason over network failures, idempotency, SLOs, and tooling constraints, then output a concrete configuration.
💡 Production Tip If you don’t need to show reasoning to users, consider stripping it before returning the final answer to keep responses clean and reduce token cost in follow‑ups.
Pattern 5: ReAct (Reason + Act)
ReAct (Reasoning + Acting) was introduced by Yao et al. (2022) as a framework that interleaves reasoning traces with tool actions (arXiv:2210.03629). [web:53][web:56]
Use when
Agents that must call APIs, DBs, search, or code exec.
Tasks needing up‑to‑date data or multi‑step workflows (research, data lookup, incident triage).
Don’t use when
Summarization, translation, sentiment analysis.
Any task where tool calling only adds latency without value.
Trade‑offs
Prompt components
Role: “You are an assistant that can call these tools: …”
Tool list:
Name
Description
Input schema (JSON‑style if possible) Rules:
When to call a tool vs answer directly.
Maximum iterations / steps.
Retry behavior on tool errors.
Final answer format: “After you have enough information, output a final answer in this format…” What to attach
Tool / API specs (OpenAPI snippets, JSON schemas).
Auth / scope constraints (“You may not call admin tools”).
Example traces of good tool usage (1–2 short examples).
Production example
An incident investigation agent that queries logs, metrics, and recent deploys, then synthesizes a root‑cause hypothesis and remediation plan.
💡 Production Tip Every additional tool call increases latency and failure modes. Treat tools like network requests — only invoke them when they’re actually needed.
Pattern 6: Tree‑of‑Thought (ToT)
Tree‑of‑Thought prompting was introduced by Yao et al. (2023) as a generalization of CoT that lets models explore multiple reasoning paths and self‑evaluate (arXiv:2305.10601). [web:58][web:65]
You want a second pass to catch obvious mistakes or style issues.
Don’t use when
Simple tasks where a single pass is already reliable.
You’re extremely latency‑sensitive and quality is “good enough”.
Trade‑offs
Prompt components
First pass: task + constraints.
Second pass: “Critique this output against the requirements. List issues, then produce a revised version.”
Optional rubric: “Focus on correctness, performance, security, readability.”
What to attach
Requirements / rubric.
Coding standards or design guidelines.
Production example
Reviewing an automatically generated API contract or config file to flag versioning, auth, and idempotency issues before committing.
💡 Production Tip Use self‑critique selectively on high‑risk outputs. Blindly applying it to every call can explode cost without proportional gains.
Pattern 8: Prompt Chaining
Prompt chaining is the practice of splitting one complex prompt into multiple smaller, focused prompts.
Unlike the other techniques, prompt chaining is an orchestration pattern. Instead of changing how one prompt behaves, it changes how multiple prompts collaborate.
Use when One prompt is becoming too large or unfocused.
Validation is required between stages.
Multiple independent subtasks exist.
Don’t use when
The task is small and fits cleanly in a single call.
You’re already latency‑bound and can’t afford multiple calls.
Many real‑world systems rely on chaining simpler prompts instead of one complex prompt because it’s easier to debug, test, and evolve.
💡 Production Tip Design each step in the chain as its own mini‑API: clear input, clear output schema, and explicit failure modes.
Special case: Retrieval‑Augmented Generation (RAG) Retrieval‑Augmented Generation (RAG) isn’t a prompting pattern — it is a context engineering strategy that can be combined with several prompting patterns.
Tasks require knowledge beyond the model’s training data.
You have a document corpus (docs, tickets, runbooks, knowledge base).
Trade‑offs
Extra infra: retrieval, chunking, embedding, versioning.
Latency: retrieval + prompt.
Reliability: depends heavily on retrieval quality.
Prompt components
Role + task.
Instructions on how to use retrieved context: “Use only the provided context. If the answer isn’t in the context, say so.”
Retrieved documents or summaries as context.
Output format.
What to attach
Retrieved chunks (with source IDs). Confidence thresholds and fallback behavior.
Poor retrieval cannot be fixed by better prompting. Retrieval quality often has a larger impact on answer quality than prompt wording.
Common mistakes
These are the patterns of failure you’ll see in real systems:
❌ Using ReAct when no tools are needed.
❌ Adding 20 few‑shot examples instead of improving instructions or schema.
❌ Using Chain‑of‑Thought for simple extraction or classification.
❌ Forgetting schema validation and retry logic.
❌ Dumping entire documentation into the prompt “just in case”.
❌ Optimizing prompts before measuring baseline accuracy, latency, and cost.
If you recognize any of these in your system, you’ve found your next improvement. How to know when to change patterns
Before switching to a more complex prompting strategy, measure:
If a simpler pattern consistently meets your quality targets, resist the temptation to introduce more complexity. Observability matters. Treat prompts as observable software artifacts. Log prompt versions, context versions, latency, validation failures, and retry counts. Without telemetry, prompt optimization becomes guesswork.
Putting it together: a real production system
A customer support assistant might use:
Schema‑driven prompting to produce structured tickets,
RAG to retrieve product documentation,
ReAct to query account information, and
Prompt Chaining to validate the final response before returning it.
Real production systems often combine multiple patterns rather than relying on just one.
Prompts are code
Treat prompts like source code:
Version them
Review them
Test them
Benchmark them
Observe them
Roll them back
Closing: three rules to live by
Production prompting isn’t about mastering every technique. It’s about choosing the simplest pattern that reliably solves the task, supplying the right context, and validating the output with measurable metrics.
Start simple. Use the simplest pattern that satisfies the task and your constraints.
Add context before adding complexity. Better schemas, examples, and docs often beat jumping to CoT/ToT/ReAct.
Measure before optimizing. Track accuracy, latency, cost, and retry rates. Let data, not hype, drive your prompt architecture.
When you do this consistently, you stop “prompt engineering” in the toy sense and start designing LLM‑enabled systems that are testable, observable, and production‑grade.