{"slug": "log-doctor-letting-ai-take-the-first-look-at-my-kubernetes-errors-using-aws-and", "title": "Log Doctor: Letting AI Take the First Look at My Kubernetes Errors Using AWS Bedrock and AWS Opensearch", "summary": "A developer built Log Doctor, a Kubernetes CronJob that runs every 15 minutes on EKS, pulls aggregated error logs from OpenSearch via Fluentd, sends a bounded, redacted prompt to an LLM on Amazon Bedrock, and posts a short per-service diagnosis to Slack. The design caps prompt size with an OpenSearch terms aggregation (10 services, 5 messages each, 2 samples) and strips AWS keys, bearer tokens and secrets before anything reaches the model.", "body_md": "Description: \"Every 15 minutes, a small CronJob reads the error logs, asks an LLM on Amazon Bedrock what's going on, and posts a short diagnosis to Slack. Here's how I built it on EKS, Fluentd and OpenSearch.\"\n\nIf you've ever been on call, you know how this goes.\n\nAn alert fires: *error rate on orders-api above 5%*. You open the log search, filter on `level: ERROR`, and get back a few hundred lines. You start scrolling. Most of them are the same stack trace repeated over and over. A few are something else. One or two of those might be the real problem, but you won't know until you've read them.\n\nBy the time you've worked out \"okay, the database is out of connections, and the payment timeouts are probably a side effect\", ten minutes have gone. That's ten minutes of reading, not fixing.\n\nWhat's missing is the step in between, the first pass a senior engineer does without thinking about it. Group the noise, notice which error is the loud one, form a quick theory, and decide where to look first.\n\nThat first pass is mostly reading and summarizing repetitive text. That happens to be something large language models are genuinely good at.\n\nI didn't want an AI agent with access to my whole cluster. I wanted something much smaller and simpler:\n\nEvery 15 minutes, look at the errors from the last 15 minutes. If there are any, post a short note to Slack, per service: what's happening, the likely cause, and what to check. If there are none, say nothing.\n\nA few rules I set for myself before writing any code:\n\n```\norders-api / payments-api → Fluentd → OpenSearch → Log Doctor (CronJob) → Amazon Bedrock → Slack\n```\n\nEverything runs on a stack that's common in AWS shops, and it's all Terraform plus plain Kubernetes manifests:\n\n`/simulate-error` endpoint that fails on purpose in realistic ways: a database out of connections, a payment call timing out, and the classic `'NoneType' object is not subscriptable`.` app-logs-YYYY.MM.DD` index.\n👉 The code is here: [github.com/ranjiniganeshan/log-doctor](https://github.com/ranjiniganeshan/log-doctor)\n\nMy first thought was to fetch a few hundred log lines and give them to the model. But OpenSearch can already do the grouping, and it gives exact counts. Log Doctor runs a single aggregation: errors per service, then top messages per service, with two recent samples of each.\n\n```\n\"aggs\": {\n    \"services\": {\n        \"terms\": {\"field\": cfg.service_field, \"size\": cfg.max_services, \"missing\": \"(unknown)\"},\n        \"aggs\": {\n            \"messages\": {\n                \"terms\": {\"field\": cfg.message_field, \"size\": cfg.max_messages_per_service},\n                \"aggs\": {\n                    \"first_seen\": {\"min\": {\"field\": cfg.timestamp_field}},\n                    \"last_seen\": {\"max\": {\"field\": cfg.timestamp_field}},\n                    \"samples\": {\n                        \"top_hits\": {\n                            \"size\": cfg.samples_per_message,\n                            \"sort\": [{cfg.timestamp_field: {\"order\": \"desc\"}}],\n                            \"_source\": {\"includes\": SAMPLE_SOURCE_FIELDS},\n                        }\n                    },\n                },\n            }\n        },\n    }\n},\n```\n\nWith the defaults, that's at most 10 services, 5 messages per service and 2 samples per message. So I know the maximum size of the prompt, however bad the incident is. The Python code then trims it further:\n\n`maxTokens` caps the reply.\nThere's another benefit: the model gets real counts. It's told \"25 of these\" instead of having to count lines itself.\n\nI spent extra time on this part. Log lines can contain whatever your users typed. If you put them in a prompt and post the answer to Slack, a stranger's text can end up in your on-call channel.\n\nSo I added three protections:\n\n```\n_REDACTIONS = [\n    (re.compile(r\"AKIA[0-9A-Z]{16}\"), \"[REDACTED_AWS_KEY]\"),\n    (re.compile(r\"(?i)bearer\\s+[a-z0-9._\\-]+\"), \"Bearer [REDACTED]\"),\n    (re.compile(r\"(?i)(password|passwd|secret|token|api[_-]?key)\\s*[=:]\\s*\\S+\"), r\"\\1=[REDACTED]\"),\n    (re.compile(r\"[A-Za-z0-9._%+\\-]+@[A-Za-z0-9.\\-]+\\.[A-Za-z]{2,}\"), \"[REDACTED_EMAIL]\"),\n    (re.compile(r\"https://hooks\\.slack\\.com/\\S+\"), \"[REDACTED_WEBHOOK]\"),\n]\n```\n\n`<!channel>` pushed to your whole team.\nNone of these is perfect. Regex only catches the patterns you thought of, and a model can still be tricked. So the most important protection is the design itself: the model has no tools and no credentials, and all it can do is write text for Slack. If someone does trick it, the worst result is a misleading message.\n\n**AWS account**\n\n`iam:CreateOpenIDConnectProvider`. If your account is in an AWS Organization, check that no service control policy blocks it.\n**Tools**\n\n`envsubst` (`brew install gettext` on macOS)`make` and `curl`\n**Slack**\n\nRun these checks before you create anything. They don't cost anything, and the EC2 command is only a dry run:\n\n```\nexport AWS_REGION=ap-south-1\n\n# Confirm you're in the right account\naws sts get-caller-identity\n\n# Check that you can launch the node type\nAMI=$(aws ssm get-parameter \\\n  --name /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64 \\\n  --query Parameter.Value --output text)\naws ec2 run-instances --dry-run --image-id \"$AMI\" \\\n  --instance-type t3.medium --count 1\n```\n\nThe EC2 command should reply with `Request would have succeeded, but DryRun flag is set`.\n\n💸 This creates billable resources. Estimated cost is in the cost section. Tear it down when you're finished.\n\nLog Doctor uses Bedrock's Converse API, so it works with any Bedrock chat model. List the models you can call directly in your region:\n\n```\naws bedrock list-foundation-models --region ap-south-1 --by-output-modality TEXT \\\n  --query \"modelSummaries[?contains(inferenceTypesSupported,'ON_DEMAND')].modelId\"\n```\n\nI used `openai.gpt-oss-20b-1:0`, OpenAI's open-weight model. Some models are only offered through cross-region inference profiles, with IDs that start with `global.` or `apac.`. Those can send your request to another region, so a policy that limits your account to one region can block them.\n\nThen make a real test call. A model appearing in the list doesn't prove your account can use it.\n\n```\nexport AWS_REGION=ap-south-1\naws bedrock-runtime converse --model-id \"openai.gpt-oss-20b-1:0\" \\\n  --messages '[{\"role\":\"user\",\"content\":[{\"text\":\"Reply with OK\"}]}]' \\\n  --inference-config '{\"maxTokens\":800}'\n```\n\nThird-party models can create an AWS Marketplace subscription on first use. The CronJob's role isn't allowed to do that, which is correct for least privilege, so make this first call with your own admin identity.\n\nThis model returns reasoning blocks as well as the answer, and it counts them against `maxTokens`. Don't set the limit too low. Log Doctor keeps only the text blocks.\n\nAt [api.slack.com/apps](https://api.slack.com/apps), create an app, turn on **Incoming Webhooks**, and add one to a channel. Treat the URL like a password, because anyone who has it can post to your channel.\n\n```\nexport SLACK_WEBHOOK_URL='https://hooks.slack.com/services/...'\n```\n\nIt stays in your shell and goes into a Kubernetes Secret later. Nothing gets written to the repo.\n\n📌 **ADD BEFORE PUBLISHING:** screenshot of the Incoming Webhooks page, with the URL hidden\n\n```\ncp infra/terraform.tfvars.example infra/terraform.tfvars\n```\n\nThese are the settings I used:\n\n```\nregion                               = \"ap-south-1\"\nbedrock_model_id                     = \"openai.gpt-oss-20b-1:0\"\nnode_instance_type                   = \"t3.medium\"\ncluster_endpoint_public_access_cidrs = [\"<your-ip>/32\"]\n```\n\nThen:\n\n```\nmake infra-up\nkubectl get nodes\n```\n\nThat creates 67 resources: a VPC, an EKS cluster with one node, a single-node OpenSearch domain, two ECR repositories and the IAM roles.\n\nFor me, the whole apply took 23 minutes. OpenSearch was the slowest part at about 14 minutes, and the EKS control plane took a little over 8.\n\n```\nkubectl get nodes \nNAME                                          STATUS   ROLES    AGE   VERSION\nip-10-40-13-187.ap-south-1.compute.internal   Ready    <none>   17m   v1.36.5-eks-6e7947a\n```\n\n**How the permissions work.** Neither workload has stored AWS credentials. Each one gets its own role through IRSA, and each role's trust policy is pinned to one exact service account:\n\n| Workload | Can do | \n|---|---|\n| Fluentd | `POST` /`PUT` on`/_bulk` and`/app-logs-*` | \n| Log Doctor | `GET` /`POST` on`/app-logs-*/_search` , and`InvokeModel` on one model | \n\nI used OpenSearch's IAM-only access policy rather than fine-grained access control because it's the simplest secure option: no master user, no role mappings, no setup step after `terraform apply`. The trade-off is that Fluentd writes through `/_bulk`, which names the index in the request body, so its role could write to any index. It still can't read, search or delete. If you need per-index write control, use fine-grained access control.\n\n```\nmake build-push\n```\n\nThis builds for `linux/amd64`, which matters if you're on Apple Silicon: without it, you get `exec format error` when the pods start on an x86 node.\n\n```\nmake slack-secret\nmake deploy\nmake status\n```\n\n`make deploy` renders the manifests from the Terraform outputs and applies them in order: namespaces, service accounts, Fluentd, the two services, then the CronJob.\n\nOpensearch, EKS cluster is created.\n\n```\n{\n    \"DomainNames\": [\n        {\n            \"DomainName\": \"log-doctor\",\n            \"EngineType\": \"OpenSearch\"\n        }\n    ]\n}\n{\n    \"clusters\": [\n        \"log-doctor\"\n    ]\n}\n[\n    \"nat-0c95abf4eb696bfdf\"\n]\n```\n\nTwo CronJob settings are there to handle failures:\n\n```\nconcurrencyPolicy: Forbid   # a slow run never overlaps the next one\nbackoffLimit: 0             # a retry could post the same summary twice\n```\n\nIf Bedrock fails, Log Doctor still posts a raw digest of top messages and counts, then exits 1. On-call gets the information without the explanation, and the failed Job stays visible.\n\n```\nkubectl -n logging get pods\nNAME            READY   STATUS    RESTARTS   AGE\nfluentd-v9dxv   1/1     Running   0          2m2s\nkubectl -n apps get pods\nNAME                            READY   STATUS    RESTARTS   AGE\norders-api-5d7fc5f46f-s49kk     1/1     Running   0          2m1s\npayments-api-754bb84f5c-f7c2v   1/1     Running   0          2m\nkubectl -n log-doctor get cronjobs,jobs,pods\nNAME                       SCHEDULE       TIMEZONE   SUSPEND   ACTIVE   LAST SCHEDULE   AGE\ncronjob.batch/log-doctor   */15 * * * *   Etc/UTC    False     0        <none>          2m\n```\n\nNow for the fun part. Each step below shows one stage of the pipeline.\n\n**1. Cause some trouble.** This sends normal traffic plus a burst of errors: 25 database failures and 4 `NoneType` bugs on `orders-api`, and 12 timeouts and 3 database failures on `payments-api`.\n\n```\nmake simulate-errors\n{\"timestamp\": \"2026-11-05T15:50:04.398+00:00\", \"level\": \"ERROR\", \"service\": \"orders-api\", \"message\": \"Unhandled error building order confirmation\", \"trace_id\": \"3dd9857419094b36a047432aba78a598\", \"logger\": \"orders-api\", \"error\": {\"type\": \"TypeError\", \"message\": \"'NoneType' object is not subscriptable\", \"stack_trace\": \"Traceback (most recent call last):\\n  File \\\"/srv/app/main.py\\\", line 194, in simulate_error\\n    action()\\n  File \\\"/srv/app/main.py\\\", line 182, in <lambda>\\n    \\\"null\\\": (\\\"Unhandled error building order confirmation\\\", lambda: build_confirmation_email(\\\"c-9999\\\")),\\n                                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\\"/srv/app/main.py\\\", line 146, in build_confirmation_email\\n    return f\\\"To: {customer['email']}\\\"  # customer is None for unknown IDs -> TypeError\\n                  ~~~~~~~~^^^^^^^^^\\nTypeError: 'NoneType' object is not subscriptable\\n\"}, \"error_scenario\": \"null\"}\n```\n\n**2. See exactly what the model will see.** Wait 30 seconds for Fluentd to flush, then run this. It makes no Bedrock call and posts nothing, and it also proves the logs reached OpenSearch.\n\n```\nmake dry-run SKIP_LLM=1\nERROR log summary for the last 15 minutes (total errors: 44, services shown: 2).\n<logs>\n## service: orders-api (total errors: 29)\n- message: \"Failed to fetch order from database\"\n  count: 25, first_seen: 2026-11-05T15:50:04.846Z, last_seen: 2026-11-05T15:50:04.846Z\n  sample 1:\n      [2026-11-05T15:50:04.846254876+00:00] pod=orders-api-5d7fc5f46f-s49kk trace_id=51c260bd3fab48a793f43dec068802e9\n      DatabaseConnectionError: connection to server at \"orders-db.internal\" (10.40.3.21), port 5432 failed: FATAL:  remaining connection slots are reserved for non-replication superuser connections\n      ...\n```\n\n**3. Ask the model, but don't post.**\n\n```\nmake dry-run\n*Shared root cause:* `orders-db` connection pool exhausted – “remaining connection slots are reserved for non‑replication superuser connections” is firing in both services.\n\n**Orders‑api** – 29 errors\n• *What's happening:* 25 DB connection errors plus 4 TypeError failures building confirmation emails.\n• *Likely cause:* The Postgres pool is saturated, and an unknown customer ID triggers a None dereference.\n• *What to check:*\n  • run `kubectl exec -it <orders-db-pod> -- psql -c \"SELECT * FROM pg_stat_activity WHERE state='active';\"` to count active connections.\n  • review `orders-api`’s connection‑pool config (` max_pool_size`, `idle_timeout`) and consider raising `max_connections` or decreasing pool size.\n  • audit `build_confirmation_email` to guard against missing customer records.\n\n**Payments‑api** – 15 errors\n• *What's happening:* 12 upstream timeouts to the payment gateway, 3 DB connection errors.\n• *Likely cause:* The gateway service is too slow or unresponsive, and the shared DB pool limit is contributing.\n• *What to check:*\n  • check `payment-gateway` pod CPU/mem and logs for latency spikes (`kubectl logs <gateway-pod>`).\n  • run `curl -v http://payment-gateway.payments.svc.cluster.local:8080/v1/authorize` from a pod to test connectivity.\n  • increase `payments-api`’s HTTP timeout or enable a retry/circuit‑breaker if the gateway is intermittently overloaded.\n```\n\n**4. The real thing.**\n\n```\nmake run-now\n```\n\nWait for a window with no errors. The job runs, finds zero errors, and posts nothing.\n\n```\nmake infra-down\n```\n\nIf you try it, I'd love to hear what it says about your logs, especially when it gets something wrong. The code is at [github.com/ranjiniganeshan/log-doctor](https://github.com/ranjiniganeshan/log-doctor).", "url": "https://wpnews.pro/news/log-doctor-letting-ai-take-the-first-look-at-my-kubernetes-errors-using-aws-and", "canonical_source": "https://dev.to/ranjiniganeshan/log-doctor-letting-ai-take-the-first-look-at-my-kubernetes-errors-using-aws-bedrock-and-aws-4kg0", "published_at": "2026-11-05 16:11:51+00:00", "updated_at": "2026-11-05 16:19:34.995910+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "mlops", "ai-infrastructure", "large-language-models"], "entities": ["Amazon Bedrock", "Amazon OpenSearch", "AWS", "Kubernetes", "Fluentd", "Slack", "Amazon EKS", "Log Doctor"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/log-doctor-letting-ai-take-the-first-look-at-my-kubernetes-errors-using-aws-and", "markdown": "https://wpnews.pro/news/log-doctor-letting-ai-take-the-first-look-at-my-kubernetes-errors-using-aws-and.md", "text": "https://wpnews.pro/news/log-doctor-letting-ai-take-the-first-look-at-my-kubernetes-errors-using-aws-and.txt", "jsonld": "https://wpnews.pro/news/log-doctor-letting-ai-take-the-first-look-at-my-kubernetes-errors-using-aws-and.jsonld"}}