# Log Doctor: Letting AI Take the First Look at My Kubernetes Errors Using AWS Bedrock and AWS Opensearch > Source: > Published: 2026-11-05 16:11:51+00:00 Description: "Every 15 minutes, a small CronJob reads the error logs, asks an LLM on Amazon Bedrock what's going on, and posts a short diagnosis to Slack. Here's how I built it on EKS, Fluentd and OpenSearch." If you've ever been on call, you know how this goes. An alert fires: *error rate on orders-api above 5%*. You open the log search, filter on `level: ERROR`, and get back a few hundred lines. You start scrolling. Most of them are the same stack trace repeated over and over. A few are something else. One or two of those might be the real problem, but you won't know until you've read them. By the time you've worked out "okay, the database is out of connections, and the payment timeouts are probably a side effect", ten minutes have gone. That's ten minutes of reading, not fixing. What's missing is the step in between, the first pass a senior engineer does without thinking about it. Group the noise, notice which error is the loud one, form a quick theory, and decide where to look first. That first pass is mostly reading and summarizing repetitive text. That happens to be something large language models are genuinely good at. I didn't want an AI agent with access to my whole cluster. I wanted something much smaller and simpler: Every 15 minutes, look at the errors from the last 15 minutes. If there are any, post a short note to Slack, per service: what's happening, the likely cause, and what to check. If there are none, say nothing. A few rules I set for myself before writing any code: ``` orders-api / payments-api → Fluentd → OpenSearch → Log Doctor (CronJob) → Amazon Bedrock → Slack ``` Everything runs on a stack that's common in AWS shops, and it's all Terraform plus plain Kubernetes manifests: `/simulate-error` endpoint that fails on purpose in realistic ways: a database out of connections, a payment call timing out, and the classic `'NoneType' object is not subscriptable`.` app-logs-YYYY.MM.DD` index. 👉 The code is here: [github.com/ranjiniganeshan/log-doctor](https://github.com/ranjiniganeshan/log-doctor) My first thought was to fetch a few hundred log lines and give them to the model. But OpenSearch can already do the grouping, and it gives exact counts. Log Doctor runs a single aggregation: errors per service, then top messages per service, with two recent samples of each. ``` "aggs": { "services": { "terms": {"field": cfg.service_field, "size": cfg.max_services, "missing": "(unknown)"}, "aggs": { "messages": { "terms": {"field": cfg.message_field, "size": cfg.max_messages_per_service}, "aggs": { "first_seen": {"min": {"field": cfg.timestamp_field}}, "last_seen": {"max": {"field": cfg.timestamp_field}}, "samples": { "top_hits": { "size": cfg.samples_per_message, "sort": [{cfg.timestamp_field: {"order": "desc"}}], "_source": {"includes": SAMPLE_SOURCE_FIELDS}, } }, }, } }, } }, ``` With the defaults, that's at most 10 services, 5 messages per service and 2 samples per message. So I know the maximum size of the prompt, however bad the incident is. The Python code then trims it further: `maxTokens` caps the reply. There's another benefit: the model gets real counts. It's told "25 of these" instead of having to count lines itself. I spent extra time on this part. Log lines can contain whatever your users typed. If you put them in a prompt and post the answer to Slack, a stranger's text can end up in your on-call channel. So I added three protections: ``` _REDACTIONS = [ (re.compile(r"AKIA[0-9A-Z]{16}"), "[REDACTED_AWS_KEY]"), (re.compile(r"(?i)bearer\s+[a-z0-9._\-]+"), "Bearer [REDACTED]"), (re.compile(r"(?i)(password|passwd|secret|token|api[_-]?key)\s*[=:]\s*\S+"), r"\1=[REDACTED]"), (re.compile(r"[A-Za-z0-9._%+\-]+@[A-Za-z0-9.\-]+\.[A-Za-z]{2,}"), "[REDACTED_EMAIL]"), (re.compile(r"https://hooks\.slack\.com/\S+"), "[REDACTED_WEBHOOK]"), ] ``` `` pushed to your whole team. None of these is perfect. Regex only catches the patterns you thought of, and a model can still be tricked. So the most important protection is the design itself: the model has no tools and no credentials, and all it can do is write text for Slack. If someone does trick it, the worst result is a misleading message. **AWS account** `iam:CreateOpenIDConnectProvider`. If your account is in an AWS Organization, check that no service control policy blocks it. **Tools** `envsubst` (`brew install gettext` on macOS)`make` and `curl` **Slack** Run these checks before you create anything. They don't cost anything, and the EC2 command is only a dry run: ``` export AWS_REGION=ap-south-1 # Confirm you're in the right account aws sts get-caller-identity # Check that you can launch the node type AMI=$(aws ssm get-parameter \ --name /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64 \ --query Parameter.Value --output text) aws ec2 run-instances --dry-run --image-id "$AMI" \ --instance-type t3.medium --count 1 ``` The EC2 command should reply with `Request would have succeeded, but DryRun flag is set`. 💸 This creates billable resources. Estimated cost is in the cost section. Tear it down when you're finished. Log Doctor uses Bedrock's Converse API, so it works with any Bedrock chat model. List the models you can call directly in your region: ``` aws bedrock list-foundation-models --region ap-south-1 --by-output-modality TEXT \ --query "modelSummaries[?contains(inferenceTypesSupported,'ON_DEMAND')].modelId" ``` I used `openai.gpt-oss-20b-1:0`, OpenAI's open-weight model. Some models are only offered through cross-region inference profiles, with IDs that start with `global.` or `apac.`. Those can send your request to another region, so a policy that limits your account to one region can block them. Then make a real test call. A model appearing in the list doesn't prove your account can use it. ``` export AWS_REGION=ap-south-1 aws bedrock-runtime converse --model-id "openai.gpt-oss-20b-1:0" \ --messages '[{"role":"user","content":[{"text":"Reply with OK"}]}]' \ --inference-config '{"maxTokens":800}' ``` Third-party models can create an AWS Marketplace subscription on first use. The CronJob's role isn't allowed to do that, which is correct for least privilege, so make this first call with your own admin identity. This model returns reasoning blocks as well as the answer, and it counts them against `maxTokens`. Don't set the limit too low. Log Doctor keeps only the text blocks. At [api.slack.com/apps](https://api.slack.com/apps), create an app, turn on **Incoming Webhooks**, and add one to a channel. Treat the URL like a password, because anyone who has it can post to your channel. ``` export SLACK_WEBHOOK_URL='https://hooks.slack.com/services/...' ``` It stays in your shell and goes into a Kubernetes Secret later. Nothing gets written to the repo. 📌 **ADD BEFORE PUBLISHING:** screenshot of the Incoming Webhooks page, with the URL hidden ``` cp infra/terraform.tfvars.example infra/terraform.tfvars ``` These are the settings I used: ``` region = "ap-south-1" bedrock_model_id = "openai.gpt-oss-20b-1:0" node_instance_type = "t3.medium" cluster_endpoint_public_access_cidrs = ["/32"] ``` Then: ``` make infra-up kubectl get nodes ``` That creates 67 resources: a VPC, an EKS cluster with one node, a single-node OpenSearch domain, two ECR repositories and the IAM roles. For me, the whole apply took 23 minutes. OpenSearch was the slowest part at about 14 minutes, and the EKS control plane took a little over 8. ``` kubectl get nodes NAME STATUS ROLES AGE VERSION ip-10-40-13-187.ap-south-1.compute.internal Ready 17m v1.36.5-eks-6e7947a ``` **How the permissions work.** Neither workload has stored AWS credentials. Each one gets its own role through IRSA, and each role's trust policy is pinned to one exact service account: | Workload | Can do | |---|---| | Fluentd | `POST` /`PUT` on`/_bulk` and`/app-logs-*` | | Log Doctor | `GET` /`POST` on`/app-logs-*/_search` , and`InvokeModel` on one model | I used OpenSearch's IAM-only access policy rather than fine-grained access control because it's the simplest secure option: no master user, no role mappings, no setup step after `terraform apply`. The trade-off is that Fluentd writes through `/_bulk`, which names the index in the request body, so its role could write to any index. It still can't read, search or delete. If you need per-index write control, use fine-grained access control. ``` make build-push ``` This builds for `linux/amd64`, which matters if you're on Apple Silicon: without it, you get `exec format error` when the pods start on an x86 node. ``` make slack-secret make deploy make status ``` `make deploy` renders the manifests from the Terraform outputs and applies them in order: namespaces, service accounts, Fluentd, the two services, then the CronJob. Opensearch, EKS cluster is created. ``` { "DomainNames": [ { "DomainName": "log-doctor", "EngineType": "OpenSearch" } ] } { "clusters": [ "log-doctor" ] } [ "nat-0c95abf4eb696bfdf" ] ``` Two CronJob settings are there to handle failures: ``` concurrencyPolicy: Forbid # a slow run never overlaps the next one backoffLimit: 0 # a retry could post the same summary twice ``` If Bedrock fails, Log Doctor still posts a raw digest of top messages and counts, then exits 1. On-call gets the information without the explanation, and the failed Job stays visible. ``` kubectl -n logging get pods NAME READY STATUS RESTARTS AGE fluentd-v9dxv 1/1 Running 0 2m2s kubectl -n apps get pods NAME READY STATUS RESTARTS AGE orders-api-5d7fc5f46f-s49kk 1/1 Running 0 2m1s payments-api-754bb84f5c-f7c2v 1/1 Running 0 2m kubectl -n log-doctor get cronjobs,jobs,pods NAME SCHEDULE TIMEZONE SUSPEND ACTIVE LAST SCHEDULE AGE cronjob.batch/log-doctor */15 * * * * Etc/UTC False 0 2m ``` Now for the fun part. Each step below shows one stage of the pipeline. **1. Cause some trouble.** This sends normal traffic plus a burst of errors: 25 database failures and 4 `NoneType` bugs on `orders-api`, and 12 timeouts and 3 database failures on `payments-api`. ``` make simulate-errors {"timestamp": "2026-11-05T15:50:04.398+00:00", "level": "ERROR", "service": "orders-api", "message": "Unhandled error building order confirmation", "trace_id": "3dd9857419094b36a047432aba78a598", "logger": "orders-api", "error": {"type": "TypeError", "message": "'NoneType' object is not subscriptable", "stack_trace": "Traceback (most recent call last):\n File \"/srv/app/main.py\", line 194, in simulate_error\n action()\n File \"/srv/app/main.py\", line 182, in \n \"null\": (\"Unhandled error building order confirmation\", lambda: build_confirmation_email(\"c-9999\")),\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/srv/app/main.py\", line 146, in build_confirmation_email\n return f\"To: {customer['email']}\" # customer is None for unknown IDs -> TypeError\n ~~~~~~~~^^^^^^^^^\nTypeError: 'NoneType' object is not subscriptable\n"}, "error_scenario": "null"} ``` **2. See exactly what the model will see.** Wait 30 seconds for Fluentd to flush, then run this. It makes no Bedrock call and posts nothing, and it also proves the logs reached OpenSearch. ``` make dry-run SKIP_LLM=1 ERROR log summary for the last 15 minutes (total errors: 44, services shown: 2). ## service: orders-api (total errors: 29) - message: "Failed to fetch order from database" count: 25, first_seen: 2026-11-05T15:50:04.846Z, last_seen: 2026-11-05T15:50:04.846Z sample 1: [2026-11-05T15:50:04.846254876+00:00] pod=orders-api-5d7fc5f46f-s49kk trace_id=51c260bd3fab48a793f43dec068802e9 DatabaseConnectionError: connection to server at "orders-db.internal" (10.40.3.21), port 5432 failed: FATAL: remaining connection slots are reserved for non-replication superuser connections ... ``` **3. Ask the model, but don't post.** ``` make dry-run *Shared root cause:* `orders-db` connection pool exhausted – “remaining connection slots are reserved for non‑replication superuser connections” is firing in both services. **Orders‑api** – 29 errors • *What's happening:* 25 DB connection errors plus 4 TypeError failures building confirmation emails. • *Likely cause:* The Postgres pool is saturated, and an unknown customer ID triggers a None dereference. • *What to check:* • run `kubectl exec -it -- psql -c "SELECT * FROM pg_stat_activity WHERE state='active';"` to count active connections. • review `orders-api`’s connection‑pool config (` max_pool_size`, `idle_timeout`) and consider raising `max_connections` or decreasing pool size. • audit `build_confirmation_email` to guard against missing customer records. **Payments‑api** – 15 errors • *What's happening:* 12 upstream timeouts to the payment gateway, 3 DB connection errors. • *Likely cause:* The gateway service is too slow or unresponsive, and the shared DB pool limit is contributing. • *What to check:* • check `payment-gateway` pod CPU/mem and logs for latency spikes (`kubectl logs `). • run `curl -v http://payment-gateway.payments.svc.cluster.local:8080/v1/authorize` from a pod to test connectivity. • increase `payments-api`’s HTTP timeout or enable a retry/circuit‑breaker if the gateway is intermittently overloaded. ``` **4. The real thing.** ``` make run-now ``` Wait for a window with no errors. The job runs, finds zero errors, and posts nothing. ``` make infra-down ``` If you try it, I'd love to hear what it says about your logs, especially when it gets something wrong. The code is at [github.com/ranjiniganeshan/log-doctor](https://github.com/ranjiniganeshan/log-doctor).