Log Doctor: Letting AI Take the First Look at My Kubernetes Errors Using AWS Bedrock and AWS Opensearch

A developer built Log Doctor, a Kubernetes CronJob that runs every 15 minutes on EKS, pulls aggregated error logs from OpenSearch via Fluentd, sends a bounded, redacted prompt to an LLM on Amazon Bedrock, and posts a short per-service diagnosis to Slack. The design caps prompt size with an OpenSearch terms aggregation (10 services, 5 messages each, 2 samples) and strips AWS keys, bearer tokens and secrets before anything reaches the model.

Description: "Every 15 minutes, a small CronJob reads the error logs, asks an LLM on Amazon Bedrock what's going on, and posts a short diagnosis to Slack. Here's how I built it on EKS, Fluentd and OpenSearch." If you've ever been on call, you know how this goes. An alert fires: error rate on orders-api above 5% . You open the log search, filter on level: ERROR , and get back a few hundred lines. You start scrolling. Most of them are the same stack trace repeated over and over. A few are something else. One or two of those might be the real problem, but you won't know until you've read them. By the time you've worked out "okay, the database is out of connections, and the payment timeouts are probably a side effect", ten minutes have gone. That's ten minutes of reading, not fixing. What's missing is the step in between, the first pass a senior engineer does without thinking about it. Group the noise, notice which error is the loud one, form a quick theory, and decide where to look first. That first pass is mostly reading and summarizing repetitive text. That happens to be something large language models are genuinely good at. I didn't want an AI agent with access to my whole cluster. I wanted something much smaller and simpler: Every 15 minutes, look at the errors from the last 15 minutes. If there are any, post a short note to Slack, per service: what's happening, the likely cause, and what to check. If there are none, say nothing. A few rules I set for myself before writing any code: orders-api / payments-api → Fluentd → OpenSearch → Log Doctor CronJob → Amazon Bedrock → Slack Everything runs on a stack that's common in AWS shops, and it's all Terraform plus plain Kubernetes manifests: /simulate-error endpoint that fails on purpose in realistic ways: a database out of connections, a payment call timing out, and the classic 'NoneType' object is not subscriptable . app-logs-YYYY.MM.DD index. 👉 The code is here: github.com/ranjiniganeshan/log-doctor https://github.com/ranjiniganeshan/log-doctor My first thought was to fetch a few hundred log lines and give them to the model. But OpenSearch can already do the grouping, and it gives exact counts. Log Doctor runs a single aggregation: errors per service, then top messages per service, with two recent samples of each. "aggs": { "services": { "terms": {"field": cfg.service field, "size": cfg.max services, "missing": " unknown "}, "aggs": { "messages": { "terms": {"field": cfg.message field, "size": cfg.max messages per service}, "aggs": { "first seen": {"min": {"field": cfg.timestamp field}}, "last seen": {"max": {"field": cfg.timestamp field}}, "samples": { "top hits": { "size": cfg.samples per message, "sort": {cfg.timestamp field: {"order": "desc"}} , " source": {"includes": SAMPLE SOURCE FIELDS}, } }, }, } }, } }, With the defaults, that's at most 10 services, 5 messages per service and 2 samples per message. So I know the maximum size of the prompt, however bad the incident is. The Python code then trims it further: maxTokens caps the reply. There's another benefit: the model gets real counts. It's told "25 of these" instead of having to count lines itself. I spent extra time on this part. Log lines can contain whatever your users typed. If you put them in a prompt and post the answer to Slack, a stranger's text can end up in your on-call channel. So I added three protections: REDACTIONS = re.compile r"AKIA 0-9A-Z {16}" , " REDACTED AWS KEY " , re.compile r" ?i bearer\s+ a-z0-9. \- +" , "Bearer REDACTED " , re.compile r" ?i password|passwd|secret|token|api - ?key \s =: \s \S+" , r"\1= REDACTED " , re.compile r" A-Za-z0-9. %+\- +@ A-Za-z0-9.\- +\. A-Za-z {2,}" , " REDACTED EMAIL " , re.compile r"https://hooks\.slack\.com/\S+" , " REDACTED WEBHOOK " , < channel pushed to your whole team. None of these is perfect. Regex only catches the patterns you thought of, and a model can still be tricked. So the most important protection is the design itself: the model has no tools and no credentials, and all it can do is write text for Slack. If someone does trick it, the worst result is a misleading message. AWS account iam:CreateOpenIDConnectProvider . If your account is in an AWS Organization, check that no service control policy blocks it. Tools envsubst brew install gettext on macOS make and curl Slack Run these checks before you create anything. They don't cost anything, and the EC2 command is only a dry run: export AWS REGION=ap-south-1 Confirm you're in the right account aws sts get-caller-identity Check that you can launch the node type AMI=$ aws ssm get-parameter \ --name /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86 64 \ --query Parameter.Value --output text aws ec2 run-instances --dry-run --image-id "$AMI" \ --instance-type t3.medium --count 1 The EC2 command should reply with Request would have succeeded, but DryRun flag is set . 💸 This creates billable resources. Estimated cost is in the cost section. Tear it down when you're finished. Log Doctor uses Bedrock's Converse API, so it works with any Bedrock chat model. List the models you can call directly in your region: aws bedrock list-foundation-models --region ap-south-1 --by-output-modality TEXT \ --query "modelSummaries ?contains inferenceTypesSupported,'ON DEMAND' .modelId" I used openai.gpt-oss-20b-1:0 , OpenAI's open-weight model. Some models are only offered through cross-region inference profiles, with IDs that start with global. or apac. . Those can send your request to another region, so a policy that limits your account to one region can block them. Then make a real test call. A model appearing in the list doesn't prove your account can use it. export AWS REGION=ap-south-1 aws bedrock-runtime converse --model-id "openai.gpt-oss-20b-1:0" \ --messages ' {"role":"user","content": {"text":"Reply with OK"} } ' \ --inference-config '{"maxTokens":800}' Third-party models can create an AWS Marketplace subscription on first use. The CronJob's role isn't allowed to do that, which is correct for least privilege, so make this first call with your own admin identity. This model returns reasoning blocks as well as the answer, and it counts them against maxTokens . Don't set the limit too low. Log Doctor keeps only the text blocks. At api.slack.com/apps https://api.slack.com/apps , create an app, turn on Incoming Webhooks , and add one to a channel. Treat the URL like a password, because anyone who has it can post to your channel. export SLACK WEBHOOK URL='https://hooks.slack.com/services/...' It stays in your shell and goes into a Kubernetes Secret later. Nothing gets written to the repo. 📌 ADD BEFORE PUBLISHING: screenshot of the Incoming Webhooks page, with the URL hidden cp infra/terraform.tfvars.example infra/terraform.tfvars These are the settings I used: region = "ap-south-1" bedrock model id = "openai.gpt-oss-20b-1:0" node instance type = "t3.medium" cluster endpoint public access cidrs = "<your-ip /32" Then: make infra-up kubectl get nodes That creates 67 resources: a VPC, an EKS cluster with one node, a single-node OpenSearch domain, two ECR repositories and the IAM roles. For me, the whole apply took 23 minutes. OpenSearch was the slowest part at about 14 minutes, and the EKS control plane took a little over 8. kubectl get nodes NAME STATUS ROLES AGE VERSION ip-10-40-13-187.ap-south-1.compute.internal Ready <none 17m v1.36.5-eks-6e7947a How the permissions work. Neither workload has stored AWS credentials. Each one gets its own role through IRSA, and each role's trust policy is pinned to one exact service account: | Workload | Can do | |---|---| | Fluentd | POST / PUT on / bulk and /app-logs- | | Log Doctor | GET / POST on /app-logs- / search , and InvokeModel on one model | I used OpenSearch's IAM-only access policy rather than fine-grained access control because it's the simplest secure option: no master user, no role mappings, no setup step after terraform apply . The trade-off is that Fluentd writes through / bulk , which names the index in the request body, so its role could write to any index. It still can't read, search or delete. If you need per-index write control, use fine-grained access control. make build-push This builds for linux/amd64 , which matters if you're on Apple Silicon: without it, you get exec format error when the pods start on an x86 node. make slack-secret make deploy make status make deploy renders the manifests from the Terraform outputs and applies them in order: namespaces, service accounts, Fluentd, the two services, then the CronJob. Opensearch, EKS cluster is created. { "DomainNames": { "DomainName": "log-doctor", "EngineType": "OpenSearch" } } { "clusters": "log-doctor" } "nat-0c95abf4eb696bfdf" Two CronJob settings are there to handle failures: concurrencyPolicy: Forbid a slow run never overlaps the next one backoffLimit: 0 a retry could post the same summary twice If Bedrock fails, Log Doctor still posts a raw digest of top messages and counts, then exits 1. On-call gets the information without the explanation, and the failed Job stays visible. kubectl -n logging get pods NAME READY STATUS RESTARTS AGE fluentd-v9dxv 1/1 Running 0 2m2s kubectl -n apps get pods NAME READY STATUS RESTARTS AGE orders-api-5d7fc5f46f-s49kk 1/1 Running 0 2m1s payments-api-754bb84f5c-f7c2v 1/1 Running 0 2m kubectl -n log-doctor get cronjobs,jobs,pods NAME SCHEDULE TIMEZONE SUSPEND ACTIVE LAST SCHEDULE AGE cronjob.batch/log-doctor /15 Etc/UTC False 0 <none 2m Now for the fun part. Each step below shows one stage of the pipeline. 1. Cause some trouble. This sends normal traffic plus a burst of errors: 25 database failures and 4 NoneType bugs on orders-api , and 12 timeouts and 3 database failures on payments-api . make simulate-errors {"timestamp": "2026-11-05T15:50:04.398+00:00", "level": "ERROR", "service": "orders-api", "message": "Unhandled error building order confirmation", "trace id": "3dd9857419094b36a047432aba78a598", "logger": "orders-api", "error": {"type": "TypeError", "message": "'NoneType' object is not subscriptable", "stack trace": "Traceback most recent call last :\n File \"/srv/app/main.py\", line 194, in simulate error\n action \n File \"/srv/app/main.py\", line 182, in <lambda \n \"null\": \"Unhandled error building order confirmation\", lambda: build confirmation email \"c-9999\" ,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/srv/app/main.py\", line 146, in build confirmation email\n return f\"To: {customer 'email' }\" customer is None for unknown IDs - TypeError\n ~~~~~~~~^^^^^^^^^\nTypeError: 'NoneType' object is not subscriptable\n"}, "error scenario": "null"} 2. See exactly what the model will see. Wait 30 seconds for Fluentd to flush, then run this. It makes no Bedrock call and posts nothing, and it also proves the logs reached OpenSearch. make dry-run SKIP LLM=1 ERROR log summary for the last 15 minutes total errors: 44, services shown: 2 . <logs service: orders-api total errors: 29 - message: "Failed to fetch order from database" count: 25, first seen: 2026-11-05T15:50:04.846Z, last seen: 2026-11-05T15:50:04.846Z sample 1: 2026-11-05T15:50:04.846254876+00:00 pod=orders-api-5d7fc5f46f-s49kk trace id=51c260bd3fab48a793f43dec068802e9 DatabaseConnectionError: connection to server at "orders-db.internal" 10.40.3.21 , port 5432 failed: FATAL: remaining connection slots are reserved for non-replication superuser connections ... 3. Ask the model, but don't post. make dry-run Shared root cause: orders-db connection pool exhausted – “remaining connection slots are reserved for non‑replication superuser connections” is firing in both services. Orders‑api – 29 errors • What's happening: 25 DB connection errors plus 4 TypeError failures building confirmation emails. • Likely cause: The Postgres pool is saturated, and an unknown customer ID triggers a None dereference. • What to check: • run kubectl exec -it <orders-db-pod -- psql -c "SELECT FROM pg stat activity WHERE state='active';" to count active connections. • review orders-api ’s connection‑pool config max pool size , idle timeout and consider raising max connections or decreasing pool size. • audit build confirmation email to guard against missing customer records. Payments‑api – 15 errors • What's happening: 12 upstream timeouts to the payment gateway, 3 DB connection errors. • Likely cause: The gateway service is too slow or unresponsive, and the shared DB pool limit is contributing. • What to check: • check payment-gateway pod CPU/mem and logs for latency spikes kubectl logs <gateway-pod . • run curl -v http://payment-gateway.payments.svc.cluster.local:8080/v1/authorize from a pod to test connectivity. • increase payments-api ’s HTTP timeout or enable a retry/circuit‑breaker if the gateway is intermittently overloaded. 4. The real thing. make run-now Wait for a window with no errors. The job runs, finds zero errors, and posts nothing. make infra-down If you try it, I'd love to hear what it says about your logs, especially when it gets something wrong. The code is at github.com/ranjiniganeshan/log-doctor https://github.com/ranjiniganeshan/log-doctor .