Prompts feel like configuration, so it is tempting to park them in whatever managed prompt-management service your platform hands you. You get versioning, a console, and a clean API, and for a while that is the right call. The trouble shows up later, under load, when prompt retrieval sits on the hot path of every inference call and the service starts pushing back.
That is the wall I hit. The prompt-management layer was throttling requests. Because every call had to fetch its prompt before doing any real work, that throttle became a ceiling on the whole pipeline. The model was not the bottleneck. Prompt lookup was.
I moved prompt serving into DynamoDB and kept the managed service as the authoring source of truth. The throttling went away, and the design has been running well since.
Prompt-management services are built for authoring and governance, not for being read thousands of times a minute on a latency-sensitive path. They come with request quotas, and those quotas are usually shared across the account rather than scaled to one workload's read traffic. When read volume climbs, you hit the quota long before you hit any limit that has to do with the prompts themselves.
The tell is subtle at first. You start seeing throttling and too-many-requests errors on prompt retrieval, so you add retries with exponential backoff and maybe a fallback for when the service says no. That is reasonable hygiene, but once you are wrapping a plain config lookup in retry logic, the store has quietly become the throughput limit of the application. You can optimize the model, batch the work, and tune everything downstream, and none of it matters, because the request is already stalled waiting for its prompt.
A prompt lookup is about the simplest access pattern there is. You have a key, and you want the item back. That is a single-item read by primary key, which is exactly what DynamoDB is good at.
GetItem. No scans, no filtering, predictable single-digit-millisecond reads.
Caching prompts is easy. Keeping the cache honest is the part worth getting right.
The managed service stays the source of truth for prompt content. DynamoDB holds the serving copy that the hot path reads. To keep them in agreement without re-up everything on every deploy, I sync on a checksum.
The sync is write-through. When it detects a change at the source, it writes the new prompt into DynamoDB first, and readers pick it up from there afterward. The flow is simple:
This turns every sync into a cheap compare instead of a blind rewrite. Unchanged prompts cost a comparison and no write. Only a genuine edit at the source triggers a write. Because the write lands before any read needs it, a new prompt propagates cleanly: the serving copy is already updated by the time the hot path asks for it.
The nice property is that the checksum is the change signal. You do not need a separate "is this dirty" flag or a manual version bump. If the content is identical, the checksum is identical, and the write is skipped. If someone edits a prompt, the checksum moves and the new text is written on the next sync.
DynamoDB does not hand you versioning the way a prompt-management service does, but you can model it, and it is worth saying so plainly because "DynamoDB has no versioning" is a common misread.
If a single current copy per prompt is all you need, the checksum layout above is enough. If you want history, rollback, or an audit trail, add a version to the key:
CURRENT) that names the active version.
The hot path reads the current pointer by key, so serving stays a single keyed read. History stays queryable with a Query on the partition key. The checksum still decides when a new version row gets written, so you are not stacking a new version for an unchanged prompt. If concurrent syncs are possible, a conditional write (ConditionExpression on the stored checksum) keeps a stale sync from overwriting a newer one.
The tradeoff is simple: the single-copy layout is less to manage, the versioned layout buys rollback and audit. Both are key reads on the hot path, so neither reintroduces the throttling problem.
The migration itself was small, which is part of why it was the right call.
The result is that prompt retrieval stopped being a shared, rate-limited dependency and became a key read against a store that scales with the traffic hitting it.
This is not an argument against prompt-management services. They earn their place for authoring, review, and governance. The mistake is letting that same service sit on a high-volume read path it was never sized for.
A few things worth keeping in mind:
Moving prompts into DynamoDB was a small change with a disproportionate payoff. The throttling that was capping the pipeline is gone, retrieval is a clean key lookup, and the checksum sync keeps the serving copy honest against the source of truth. It has held up in production since.