{"slug": "log-parsing-with-ai-at-bronto", "title": "Log Parsing with AI at Bronto", "summary": "Bronto uses AI to automatically generate parsers for unstructured or semi-structured logs, reducing the complexity and maintenance burden of traditional regex-based methods. It details a multi-layered approach that combines high-performance Java parsers for common formats with an AI engine that analyzes log samples to create custom dissect patterns for unknown formats. The system uses AWS Bedrock to access LLMs, which generate and test these patterns before user approval, enabling real-time parsing at scale.", "body_md": "Authored by Gary Nicholls\nThis post follows on from our AWS Nova log benchmarking article, where we explored how smaller LLMs perform on log analysis tasks. That earlier post highlighted that LLMs are surprisingly good at parsing logs. While that work focused on understanding logs, this post tackles an earlier step: automatically structuring logs using AI.\nLogs are one of the oldest — and still most valuable — forms of observability. Mainframes and early Unix systems were already using logs to record system activity, with tools like syslog dating back to the early 1980s.\nEven as systems have become more distributed and complex, logs remain foundational, especially for investigating issues when things go wrong. Logs are typically written to local files before being shipped to modern observability platforms using agents like OpenTelemetry or Fluent Bit.\nThe OpenTelemetry (OTel) project is encouraging the adoption of structured JSON logs — and that's a good thing. Structured logs are easier to search, more human-readable, safer to manipulate, and more cloud-native.\nBut the reality isn't that simple. Many systems still generate unstructured or semi-structured logs where key=value pairs are embedded inside free-text messages. And even among structured formats, things vary wildly — timestamps alone appear in dozens of different formats.\nLogs reflect the unique fingerprint of each tech stack:\nWith no single standard, Bronto set out to solve the problem in an innovative way — using AI to generate parsers automatically, reducing the toil and complexity that users typically face.\nParsing logs in real time is a performance-critical operation. When ingesting millions of events per second, every millisecond counts. Regex-based parsing can be complex and hard to maintain, requiring expertise in tools like Grok or Dissect — and can become a bottleneck at scale when applied indiscriminately.\nAt Bronto, we use a multi-layered approach that separates offline detection from online parsing. Online parsing happens in real time as part of the ingestion pipeline; offline detection occurs outside the pipeline with a short delay. This hybrid approach ensures speed without sacrificing flexibility, while reducing user toil.\nWe maintain a library of high-performance Java-based parsers, optimized for the most common formats seen at high volumes across multiple customers. These are purpose-built for speed and designed to fail fast if they encounter a log that doesn't match their expected format.\nAfter applying a Java parser, we run additional lightweight processors to normalize key fields:\nFor less common but still important formats, we fall back to Dissect or Grok:\nBronto maintains a large database of both dissect and grok patterns. Due to their runtime cost, we don't attempt to apply every pattern to every event online. Instead:\nWhen we encounter unknown or proprietary formats, other tools might require users to handcraft regexes through a UI. At Bronto, we let AI do the work.\nWhen enabled, we send a sample of the dataset to an internal AI engine that analyzes the log structure and generates a custom dissect pattern. We test the pattern against a wider sample. If it matches a high percentage of events, we present the pattern and sample results to the user — they can tweak field names if desired — and once approved, the parser is saved and applied automatically to all future events in that dataset.\nSuppose your application logs look like this:\nAfter analyzing hundreds of lines, the AI generates a dissect pattern:\nThe parsed result looks like this:\nThe KVP parser then further extracts fields like sessionId\n, userId\n, and formId\nfrom the message\nvalue.\nWe use AWS Bedrock as a managed service to access LLMs (including Claude). Our infrastructure chooses the most appropriate model for each application and sends structured prompts — for example, instructing the LLM which patterns to avoid and how to handle keys like timestamps. The user doesn't have to worry about models or prompts; they just use the application.\nBedrock also provides important SaaS-grade guarantees:\nAt Bronto we believe parsing should be fast, accurate, and hands-free. Today we generate dissect patterns using AI. Soon we'll be generating Grok patterns too — bringing AI to even more complex and less structured formats.\nAs OTel continues to push for JSON-based structured logging, the hope is that log parsing becomes a less painful problem over time. But until then, automated, adaptive parsing isn't just a convenience — it's a necessity.\nBronto combines curated Java parsers, flexible Dissect/Grok matching, and AI-powered pattern generation into a unified pipeline for parsing any log format, structured or otherwise.\nIf your logs are weird or messy — we've got you.", "url": "https://wpnews.pro/news/log-parsing-with-ai-at-bronto", "canonical_source": "https://dev.to/bronto_io/log-parsing-with-ai-at-bronto-18d2", "published_at": "2026-05-21 01:06:23+00:00", "updated_at": "2026-05-21 01:32:08.152643+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "developer-tools", "cloud-computing", "data"], "entities": ["Gary Nicholls", "Bronto", "AWS", "OpenTelemetry", "Fluent Bit"], "alternates": {"html": "https://wpnews.pro/news/log-parsing-with-ai-at-bronto", "markdown": "https://wpnews.pro/news/log-parsing-with-ai-at-bronto.md", "text": "https://wpnews.pro/news/log-parsing-with-ai-at-bronto.txt", "jsonld": "https://wpnews.pro/news/log-parsing-with-ai-at-bronto.jsonld"}}