A few months ago the ask landed on my desk that a lot of infra people are getting right now: "The XYZ (Non Tech) teams want to build on our LLMs. Can you give them access to Bedrock?"
Simple enough on paper. AWS Bedrock was already in our account, the models were enabled, IAM can hand out permissions. Ten minutes of work.
Except it isn't ten minutes of work, and if you treat it like it is, you'll regret it in about three weeks.
There are basically two, and neither is good once you think past the first user.
Option one is to hand out IAM credentials, or a role, that can call Bedrock. Fine for one person. Do it a few times, though, and you've got AWS keys living in .env
files, a CI variable here, someone's Postman collection there, maybe a notebook somewhere you'll never hear about. Rotating any of it means finding every place it landed. And IAM permissions tend to be broader than you meant them to be a policy that lets someone call Bedrock usually lets them reach a few other things too, because nobody scopes these tightly on day one and then day one becomes a year ago.
Option two is one shared key behind an internal endpoint, and everyone points at that. Quicker to stand up. But now every request looks the same, so you can't tell whose code is burning what. A single bad loop can drain the month's budget before lunch and you won't know which team shipped it. And you can't cut off one person without rotating the key for all of them.
Both fall apart the moment more than two or three people are involved. And the part that stung most: when finance eventually asks why the AI line jumped, you've got nothing to show them.
There's a whole category of tools for this now people call them AI gateways, or LLM gateways. LiteLLM, Portkey, Helicone, Cloudflare's AI Gateway, and a pile of others. They all sit between your apps and the model provider and give you back the two things you want: control over who can call what, and a record of what everyone spent. Same idea as an ordinary API gateway, pointed at tokens.
I spent a while looking at the hosted ones, and for most teams I'd tell you to pick one and move on. We didn't. The reason is unglamorous. We're a fintech company, and routing prompts and completions, some of which touch data I don't want leaving our network, through somebody else's SaaS just to get a usage graph was a trade I wasn't going to make. I wanted the whole thing inside our own perimeter, talking straight to Bedrock, with the logs in a database we own. So I built a small version.
The whole design comes down to one decision, and it's a blunt one. Nobody gets AWS credentials. Not developers, not services, nothing except the gateway itself.
What people get instead is a key the gateway invents a prefixed token, llmkey_9f3c…
or whatever, that AWS has never heard of. You drop it in your app, point the client at the gateway's URL instead of the Bedrock endpoint, and you're done. The only place real AWS credentials exist is the gateway process, in environment variables, and they never get copied out.
Everything else is downstream of that. Once a request shows up carrying a key you handed out, you know who's calling. And once you know who's calling, you can start refusing things.
A few things, and they're set per key.
You can pin the models a key is allowed to touch. Most internal tools don't need the top-tier model a ticket summarizer is perfectly happy on something small and fast, and there's no reason to let that key reach for a model that costs several times more, whether by accident or because somebody got curious one afternoon.
You can hand each key a token budget and count against it on every call. Nothing fancy, just a number and a running total. When a key's creeping toward its limit I can see it on the dashboard before it's a problem, and when it hits the ceiling it stops. That one thing removes the runaway-script scenario, which is the one that keeps you up at night.
And keys carry names and owners. So when you're staring at usage you're looking at "the internal analytics job" or an actual person, not a random string you have to go trace back at 11pm.
Here's what a call does. It's not exciting:
Step 5 is the reason all the reporting works later. Every call becomes a row: input tokens, output tokens, latency, model, cost. Every screen in the app is just a query over that one table.
Stripped down, the check in the middle is about this simple:
def handle_request(gateway_key, model_id, prompt):
key = keys.lookup(gateway_key)
if not key or not key.active:
raise Unauthorized("Invalid or disabled key")
if model_id not in key.allowed_models:
raise Forbidden(f"Key not permitted to use {model_id}")
if key.tokens_used >= key.token_limit:
raise QuotaExceeded("Token budget exhausted")
response = bedrock.invoke_model(model_id, prompt)
usage = response.usage # input / output token counts
cost = price(model_id, usage.input_tokens, usage.output_tokens)
log_usage(key.id, model_id, usage, cost)
keys.increment_tokens(key.id, usage.total_tokens)
return response
Cost is the part people assume will be fiddly. It isn't, as long as you don't bake the prices into the code. Bedrock charges a per-1K rate for input and a different per-1K rate for output, the rates differ by model, and they change over time. So they live in settings, and the cost function just reads them:
def price(model_id, input_tokens, output_tokens):
rate = pricing[model_id] # per-1K rates, editable in settings
return (input_tokens / 1000 * rate.input_price
+ output_tokens / 1000 * rate.output_price)
Because that runs and gets stored on every request, nobody has to reconstruct spending afterward. I can pull it apart by model, by key, over whatever window. The first time I did that and found one key sitting on most of the month's cost, it turned into a five-minute chat with the team that owned it instead of a week of guessing.
On top of that log table is the part everyone opens first: usage over time, which models are getting hammered, cost per period, the biggest keys ranked by spend. If you've seen any observability dashboard it looks like that, except the axes are tokens and money.
One thing mattered more than I expected. The people who need to look at this aren't always the people who should be able to change it. Someone on finance, or a team lead, wants to watch the numbers. They've got no reason to be creating keys or editing the price table. So there are roles: full admins who can do everything, and read-only accounts that can look and nothing else. Small feature. Saved an awkward conversation more than once.
A few things I didn't see coming, or saw coming and ignored:
None of these are hard. They're the sort of thing you only really learn by getting them a little wrong first.
For us, yeah. Not because it's clever, it isn't, it's a reverse proxy with a database bolted to it. It's worth it because of what changed day to day. Someone wants Bedrock access, I generate a key, done, no IAM ticket. No real credential has left the gateway even once. A key misbehaves, I revoke that one key and everyone else carries on.
And when AI spend comes up, which it does constantly now, I open a page instead of a spreadsheet.
The last one is the part I'd push hardest on. Every company I talk to is having some flavor of the same conversation right now: who's using these models, on what, at what cost, and whether any of it is governed. You can answer that with a policy doc that's stale the week you write it, or you can let your infrastructure answer it by itself. I know which of those stays current.
If you've just caught the "get everyone onto our LLMs" request yourself, that's the one bit of advice I'd hand you. Don't pass out keys. Stand something small up in front of Bedrock first. It's a weekend of work and it saves you from a mess that's a real pain to clean up later.