{"slug": "amazon-bedrock-tokenomics-data-lake", "title": "Amazon Bedrock tokenomics data lake", "summary": "AWS published a centralized observability data lake for Amazon Bedrock that consolidates model invocation logs from linked AWS accounts into a single Amazon Athena table, with CloudFormation templates, Lambda code and SQL queries available in the AWS Sample Bedrock Invocation Data Lake GitHub repository. The data lake forwards invocation logs through CloudWatch subscription filters and Amazon Data Firehose, capturing per call the prompt, model response, input and output token counts, model used, stop reason, account and timestamp, so platform and ML engineers can attribute usage and cost across teams. AWS said the design addresses the gap where invocation logs exist but stay locked inside individual accounts, leaving teams unable to debug spikes or spot wasteful prompts.", "body_md": "## [Artificial Intelligence](https://aws.amazon.com/blogs/machine-learning/)\n\n# Amazon Bedrock tokenomics data lake\n\nIf you need an observability data lake for your [Amazon Bedrock](https://aws.amazon.com/bedrock/) workloads, you’re not alone. When you’re running thousands of model invocations a day across multiple accounts, after a request is sent you might be wondering, what happened? You can see the aggregate bill, but you can’t see what happened inside each call.\n\nYou might start to ask questions: what prompt was sent, what the model returned, how many tokens it used, which model handled it, and whether the response was any good! If you’re a platform or machine learning (ML) engineer looking after a fleet of Amazon Bedrock workloads across teams, that’s a real problem. You can’t debug a spike, you can’t spot wasteful prompts, and you can’t tell one team’s usage from another’s.\n\nWe built something to help: a centralized observability data lake for your Amazon Bedrock invocations. In this post, I cover what native data you have and how we designed the lake. I also show how to use it to see your prompts and derive the per-invocation cost. The [AWS CloudFormation](https://docs.aws.amazon.com/cloudformation/) templates, [AWS Lambda](https://aws.amazon.com/lambda/) code, and SQL queries are all in the [AWS Sample Bedrock Invocation Data Lake GitHub repository](https://github.com/aws-samples/sample-bedrock-token-cost-allocation). This post walks through the architecture and deployment steps. The repo has everything you need to deploy it.\n\n## What Amazon Bedrock visibility do you have today?\n\nAmazon Bedrock offers [model invocation logging](https://docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html), and it’s genuinely powerful. You turn on invocation logs in the [settings in each AWS account you are using Bedrock](https://us-east-1.console.aws.amazon.com/bedrock/home?region=us-east-1#/settings). It delivers a full record of requests and responses to [Amazon Simple Storage Service (Amazon S3)](https://aws.amazon.com/pm/serv-s3/) or [Amazon CloudWatch](https://aws.amazon.com/cloudwatch/).\n\nThe data exists, but it’s locked inside individual accounts. What you really want is a single, fast, queryable view of every invocation across your whole organization. That’s what the data lake gives you.\n\n## What is the Amazon Bedrock observability data lake?\n\nA centralized [Amazon Bedrock invocation log](https://docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html) data lake pulls raw model invocation logs from your linked accounts into [Amazon CloudWatch](https://aws.amazon.com/cloudwatch/). Subscription log filters then forward those records to your central account. The central account processes the data through [Amazon Data Firehose](https://aws.amazon.com/firehose/), and exposes them through [Amazon Athena](https://aws.amazon.com/athena/) as one clean table you can query across your organization.\n\nEverything is deployed as Infrastructure as Code using two [AWS CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html) templates, so you can roll it out repeatably across every account in your organization. This can be found in the [AWS Sample Bedrock Invocation Data Lake GitHub repository](https://github.com/aws-samples/sample-bedrock-token-cost-allocation).\n\n## What does each Amazon Bedrock invocation log capture?\n\nAfter we run this solution, you will have a combined set of data across the linked account you deploy to so you can query with Athena. What you get per invocation after the logs are flattened into the processed table:\n\n- **The prompt** the user or application sent.\n- **The model response** that came back.\n- **Input and output token counts** , so you can measure the true weight of each call.\n- **The model** used, so you can understand utilization of models across teams.\n- **The stop reason** (for example,`max_tokens` ), which tells you when a response was cut off.\n- **The account and timestamp** , so you can attribute and time-slice everything.\n- **The**[AWS Identity and Access Management (IAM) role](https://aws.amazon.com/iam/?trk=40032cd2-744c-4b2c-ad0c-cf60894e0921&sc_channel=ps&ef_id=CjwKCAjws_DTBhB_EiwAXZknGfru2_JWQb1nqidRQ4APsbkaGf1W2ywetPh_unJq1IEvVAlNJfLWzRoCpg4QAvD_BwE&gads_camp=23528572748&gads_ag=198422384371&gads_ad=795811958070&gads_kw=iam&gads_matchtype=e&gads_network=g&gads_device=c&gads_geo=9198785&gad_campaignid=23528572748&gbraid=0AAAAADjHtp-pAfh6ucw0EYbg13-jUO8NM&gclid=CjwKCAjws_DTBhB_EiwAXZknGfru2_JWQb1nqidRQ4APsbkaGf1W2ywetPh_unJq1IEvVAlNJfLWzRoCpg4QAvD_BwE) , so you can see who sent the prompt.\n\nAfter every invocation is captured like this, you go from “We used X tokens on Amazon Bedrock” to “Here is exactly what each model calls contained, and how efficient it was.” That’s the visibility platform and ML teams have been missing.\n\n## How is the Amazon Bedrock data lake designed?\n\nThe data lake moves each invocation through five stages, from the moment Amazon Bedrock writes a log to the point you can query it in Athena.\n\n### 1. Amazon Bedrock writes an invocation log\n\nEvery time an application calls an Amazon Bedrock model, Amazon Bedrock can record what happened: which model, which account, which IAM identity made the call, how many input and output tokens were used, and the prompt and response text. You turn this on once per account with a single API call, pointing Amazon Bedrock at a CloudWatch log group and giving it a role that can write there.\n\n### 2. CloudWatch forwards the logs to a central account\n\nEach source account gets a subscription filter on its Amazon Bedrock log group. Think of it as a permanent forwarding rule that says “send every new event to this address.” The address is an Amazon CloudWatch Logs destination living in your central account. Source accounts don’t get permissions on the central pipeline. They hand events to the destination, and the central account’s own role takes it from there.\n\n### 3. A small Lambda tidies each record\n\nCloudWatch doesn’t hand over clean JSON. It bundles events into batches and wraps them compressed and encoded, so what arrives looks like gibberish until you unwrap it. A small Lambda function does three jobs:\n\n- Unwraps the batch back into individual log records.\n- Throws out the noise. CloudWatch sends occasional housekeeping messages and plain-English notices like “Permissions are correctly set for Amazon Bedrock logs.” These aren’t invocation data, and letting them through breaks the next stage.\n- Flattens one awkward field. The caller’s identity arrives nested one level down. The Lambda lifts it to the top so it’s straightforward to query later.\n\n### 4. Amazon Data Firehose converts to Parquet and writes to S3\n\nAmazon Data Firehose buffers the records and writes them to an Amazon S3 bucket, converting from JSON to Parquet on the way through. Firehose also files each object under a date path (`year=2026/month=08/day=27/`).\n\n### 5. Use Athena to query the data and connect it to cost data\n\nAn [AWS Glue](https://aws.amazon.com/glue/) table tells Athena how to read those files, so you can query the lake with ordinary SQL. You can see which users are using the most tokens, which models are commonly selected, and what people are prompting. If you want, you can also connect this to the [AWS Cost and Usage Report](https://aws.amazon.com/aws-cost-management/aws-cost-and-usage-reporting/) (CUR) to get the spend of user prompts by day.\n\n## What do you need before you start?\n\nA few things need to be in place before you deploy:\n\n1. [AWS Command Line Interface (AWS CLI)](https://aws.amazon.com/cli/) configured with credentials for each account.\n2. **CloudFormation**`CAPABILITY_IAM` and`CAPABILITY_NAMED_IAM` permissions.\n3. **At least one linked account** (the account running your Amazon Bedrock workloads) and**one central account** to receive the replicated logs (these can be the same account).\n4. **Python 3 with Boto3** (`pip install boto3` ), for the end-to-end test script.\n5. [Cloned repo](https://github.com/aws-samples/sample-bedrock-token-cost-allocation) with all the resources in it.\n\n## How do you deploy it?\n\n**The order matters here.** The central account stack has to exist before the linked account stacks can reference it.\n\n### Stage 1: Deploy the central data lake stack\n\nDeploy the central CloudFormation stack first, into the account that will own the pipeline. Note two values from the stack outputs before moving on: the CloudWatch Logs destination ARN (which every source account needs) and the [AWS Quick Sight](https://aws.amazon.com/quick/?trk=cb69a790-31b4-4f3f-ab38-8c131506ae2b&sc_channel=ps&ef_id=Cj0KCQjw2OnUBhC2ARIsACKyfaHOyrI2BSqC3eo7DZbbj0Ks030dwANVC-ITwsJaw_YzyJqknqgei6IaAm3WEALw_wcB:G:s&gads_camp=23528573981&gads_ag=191423962694&gads_ad=806967564154&gads_kw=amazon%20quick&gads_matchtype=e&gads_network=g&gads_device=c&gads_geo=1006993&gad_campaignid=23528573981&gbraid=0AAAAADjHtp_o1ijSErUcY81biixQPaoFW&gclid=Cj0KCQjw2OnUBhC2ARIsACKyfaHOyrI2BSqC3eo7DZbbj0Ks030dwANVC-ITwsJaw_YzyJqknqgei6IaAm3WEALw_wcB) role ARN (needed later if you deploy the dashboard).\n\n### Stage 2: Deploy the logging stack in each source account\n\nDeploy the linked account CloudFormation. Repeat this for every account running Amazon Bedrock workloads. The source stack is small and takes a single input: the destination ARN from Stage 1.\n\n### Stage 3: Turn on Amazon Bedrock model invocation logging\n\nDeploying the stack doesn’t start the logging. Amazon Bedrock logging is an account-level setting, not a CloudFormation resource. Activate it once per source account, pointing Amazon Bedrock at the log group and the logging role from Stage 2. Turn on CloudWatch delivery only.\n\n### Stage 4: Verify data is arriving\n\nRun a test invocation in a source account, wait for Amazon Data Firehose to flush its buffer ([roughly a minute, sometimes longer](https://docs.aws.amazon.com/firehose/latest/dev/basic-deliver.html#frequency)), then confirm Parquet objects have appeared under the data lake’s `data/` prefix. Finally, check your S3 bucket to see data.\n\n### Stage 5: Create the Athena view and query the data\n\nThe central stack saves the view definition as a named Athena query but doesn’t run it. Execute that saved query once to create the flattened view. Run queries in the dedicated Athena workgroup, against either the raw table or the flattened view. The view is straightforward: token counts, cache tokens, stop reason, and prompt/response previews are already flat columns.\n\n## How can you see costs of your Amazon Bedrock tokens?\n\nWhen you want a dollar figure against all of this, join the processed table to your CUR 2.0 data. The join key is the IAM ARN. You can use the following example queries in the [Bedrock Data Lake SQL](https://github.com/aws-samples/sample-bedrock-token-cost-allocation/blob/main/Bedrock%20Data%20Lake%20SQL.md) file.\n\nThat one join turns the observability lake into cost attribution too: which prompts cost the most, cost per team or account by day, and what your least efficient invocations are actually costing you. It’s a great addition, but the data lake earns its keep well before you get there.\n\n## What is tokenomics and why does this data lake matter for it?\n\nTokenomics, the practice of managing the economics of artificial intelligence (AI) token consumption, was announced as a formal FinOps domain at the 2026 FinOps Foundation conference. It’s built on the same See, Save, Run, and Plan pillars as Cloud Financial Management, but applied to tokens instead of infrastructure. The challenge is that you can’t optimize what you can’t measure. Most organizations today lack the granular data needed to answer basic tokenomics questions: which teams are consuming the most tokens, and which prompts are wasteful. With per-invocation token and cost visibility, teams can match each workload to the model that fits it. That’s exactly what this data lake gives you. With per-invocation visibility into prompts, responses, token counts, models, and IAM identities, you have the foundation to start running a tokenomics practice rather than only tracking an Amazon Bedrock bill.\n\n## Create an Amazon Bedrock invocation log dashboard\n\nIf you want to visualize your token consumption, we have an [AWS Quick Sight](https://aws.amazon.com/quicksight/) dashboard. This shows you the data from the table over time so you can share it with stakeholders in your organization. Using AWS Quick Sight, you can customize the dashboard and add organization metadata such as team information to make it your own. To deploy this, use the [cid-cmd deploy](https://github.com/aws-solutions-library-samples/cloud-intelligence-dashboards-framework/blob/main/CID-CMD.md) CLI command. Full step-by-step deployment instructions are in the [repository README](https://github.com/aws-samples/sample-bedrock-token-cost-allocation), including AWS Quick Sight role configuration and datasource setup.\n\n## What can you now do with Amazon Bedrock invocation logs?\n\nWith every invocation in one fast, queryable table, you can finally answer questions that used to be a pain:\n\n- **Debug and audit invocations –** Pull the exact prompt and response for specific calls, in an account, without digging through raw log files.\n- **Measure token efficiency –** Compare`input_tokens` against`output_tokens` and model choice across teams to find where you’re burning tokens for no reason.\n- **Spot prompt engineering opportunities –** Find requests where the response was cut off (`stop_reason = max_tokens` ), which is a good sign the prompt or config needs work.\n- **Review model families –** The data shows which model serves each pattern so teams can confirm each model matches its workload.\n- **Govern usage across teams –** Each invocation is ti", "url": "https://wpnews.pro/news/amazon-bedrock-tokenomics-data-lake", "canonical_source": "https://aws.amazon.com/blogs/machine-learning/amazon-bedrock-tokenomics-data-lake/", "published_at": "2026-11-10 16:56:36+00:00", "updated_at": "2026-11-10 17:22:52.545597+00:00", "lang": "en", "topics": ["ai-infrastructure", "mlops", "large-language-models", "ai-tools"], "entities": ["Amazon Bedrock", "Amazon Web Services", "Amazon CloudWatch", "Amazon Data Firehose", "Amazon Athena", "AWS CloudFormation", "AWS Lambda", "Amazon S3"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/amazon-bedrock-tokenomics-data-lake", "markdown": "https://wpnews.pro/news/amazon-bedrock-tokenomics-data-lake.md", "text": "https://wpnews.pro/news/amazon-bedrock-tokenomics-data-lake.txt", "jsonld": "https://wpnews.pro/news/amazon-bedrock-tokenomics-data-lake.jsonld"}}