{"slug": "show-hn-intelligent-log-sanitization-and-logs-to-metrics", "title": "Show HN: Intelligent Log Sanitization and logs to metrics", "summary": "LogGuardAI, a Log4j2 plugin that automatically detects and masks sensitive data in Java logs using rule-based analysis and AI, has been released on May 27, 2026. The tool supports 20+ sensitive keyword detection, smart risk scoring, and logs-to-metrics extraction for Prometheus-style monitoring, with AI-powered sanitization via OpenAI, Anthropic Claude, or Azure OpenAI.", "body_md": "**Intelligent Log Sanitization for Java** — Automatically detect and mask sensitive data in logs using rule-based analysis and AI.\n\nLogGuardAI is a Log4j2 plugin that intercepts log events and intelligently sanitizes them to **prevent accidental exposure of sensitive data** like passwords, API keys, tokens, and personal information.\nIt also supports logs-to-metrics extraction for Log4j2, converting log lines into Prometheus-style metrics using custom regex patterns.\n\n```\n❌ Logs with secrets:\n  [LOG] User auth: userId=12345 apiKey=sk-abcdef123456 password=secret\n        ↑ Oops! Now this secret is in logs, monitoring dashboards, and backups!\n✅ Logs with LogGuardAI:\n  [LOG] User auth: userId=***** apiKey=***** password=*****\n        ↑ Secrets automatically masked, structure preserved!\n```\n\n- 🔐\n**20+ Sensitive Keyword Detection**— Automatically detects passwords, tokens, API keys, etc. - 📊\n**Smart Risk Scoring**— Multi-factor analysis (keywords, patterns, entropy) - ⚡\n**Lightning Fast**— <5ms latency, non-blocking design - 🛡️\n**Fail-Safe**— Logging never breaks, even if something fails - 📝\n**Structured Data Support**— Handles key=value, JSON, query strings\n\n- 🧠\n**AI-Powered Sanitization**— OpenAI GPT-based intelligent masking with context - 💰\n**LRU Caching**— 80-95% cache hit rate =** 80% cost savings** - 🏆\n**Enhanced Exceptions**— AI explains what went wrong - 🔍\n**Data Classification**— Automatically identify PII vs. sensitive data - 📊\n**Configurable Sampling**— Control AI costs (5%-100%)\n\n-\n📦\n\n**Batch AI Processing**— Process multiple sensitive values together (5x efficiency) -\n⚡\n\n**Async/Non-Blocking**— AI calls never block log threads (<5ms guaranteed) -\n🔄\n\n**Smart Fallbacks**— Rule-based masking if AI fails or times out -\n🐛\n\n**Spring Boot Compatibility**— Fixed Log4j2 plugin registration issues\n\n- 🤖\n**Multi-Provider AI**— OpenAI, Anthropic Claude, or Azure OpenAI - 🏗️\n**Anthropic Claude Integration**— Claude 3 models with Messages API - ☁️\n**Azure OpenAI Integration**— Custom endpoints and deployments - 🔧\n**Provider-Specific Configuration**— Azure endpoints, deployments, API versions - 🏷️\n**Enhanced Model Support**— GPT-4, Claude 3, Azure deployments - 📊\n**Configurable Batch Size**— Tune performance vs. API efficiency - 🤖\n**Multi-Provider Support**— OpenAI, Anthropic Claude, Azure OpenAI - 📈\n**Metrics Extraction**— Convert legacy unstructured logs into rich JSON metrics. - 🏗️\n**Backward Compatible**— Existing configurations work unchanged\n\n**Released:** May 27, 2026\n\n- 📊\n**Pattern-Based Metrics**— Define custom regex patterns to extract metrics directly from logs - 🎯\n**User-Defined Patterns**— Create metrics for HTTP requests, FTP transfers, database queries, etc. - 💾\n**Append-Only File**— Metrics stored in Prometheus text format with cumulative history - 📈\n**Logs-to-Metrics Conversion**— Convert Log4j2 log entries into counters and labels for monitoring and alerting - ⚙️\n**Periodic Flushing**— Configurable flush interval (default: 60 seconds) - 🔐\n**Cardinality Protection**— Configurable limits prevent unbounded metric combinations (default: 10,000) - 🔌\n**Zero Dependencies**— No external monitoring infrastructure required\n\n```\n<LogGuardLayout \n  extractMetrics=\"true\"\n  metricsFilePath=\"logs/metrics.txt\"\n  metricsFlushIntervalMs=\"60000\"\n  metricsMaxCardinality=\"10000\"\n  metricsPatterns=\"http_requests|(GET|POST|DELETE) ([/\\\\w/]+) (\\\\d{3})|http_requests_total|method,endpoint,status\"/>\n```\n\nSee [Metrics Configuration](/sanitizeai/logguardai/blob/main/docs/guides/metrics-configuration.md) for full configuration details and examples.\n\n```\n<repositories>\n    <repository>\n        <id>jitpack.io</id>\n        <url>https://jitpack.io</url>\n    </repository>\n</repositories>\n\n<dependency>\n    <groupId>com.logguardai</groupId>\n    <artifactId>logguardai</artifactId>\n    <version>0.5.0</version>  <!-- Latest: Logs to metrics -->\n</dependency>\n<Configuration>\n  <Appenders>\n    <Console name=\"Console\" target=\"SYSTEM_OUT\">\n      <LogGuardLayout/>\n    </Console>\n  </Appenders>\n\n  <Loggers>\n    <Root level=\"info\">\n      <AppenderRef ref=\"Console\"/>\n    </Root>\n  </Loggers>\n</Configuration>\n```\n\nTip: To enable pattern-based metrics, set `extractMetrics=\"true\"`\n\nand configure `metricsFilePath`\n\n, `metricsFlushIntervalMs`\n\n, `metricsMaxCardinality`\n\n, and `metricsPatterns`\n\nas shown in the v0.5 section above.\n\nLogGuardAI v0.5 makes it easy to transform Log4j2 log messages into Prometheus-compatible metrics. Use `metricsPatterns`\n\nto extract method, endpoint, status, latency, and other useful labels from structured log lines.\n\nExample: convert request logs into a counter metric for `http_requests_total`\n\nwith labels like `method`\n\n, `endpoint`\n\n, and `status`\n\n.\n\nThis is ideal for:\n\n- Log4j2 metrics extraction\n- Logs-to-metrics pipelines\n- Prometheus text metric generation from application logs\n- Regex-based log analytics without external agents\n\n``` php\n<!-- OpenAI (default) -->\n<LogGuardLayout\n    aiEnabled=\"true\"\n    aiProvider=\"openai\"\n    aiApiKey=\"${OPENAI_API_KEY}\"\n    aiModel=\"gpt-3.5-turbo\"\n    samplingRate=\"0.1\"\n    extractMetrics=\"true\"\n    batchSize=\"5\"/>\n\n<!-- Anthropic Claude -->\n<LogGuardLayout\n    aiEnabled=\"true\"\n    aiProvider=\"anthropic\"\n    aiApiKey=\"${ANTHROPIC_API_KEY}\"\n    aiModel=\"claude-3-sonnet-20240229\"\n    samplingRate=\"0.1\"\n    batchSize=\"5\"/>\n\n<!-- Azure OpenAI -->\n<LogGuardLayout\n    aiEnabled=\"true\"\n    aiProvider=\"azure-openai\"\n    aiApiKey=\"${AZURE_OPENAI_API_KEY}\"\n    aiModel=\"gpt-35-turbo\"\n    azureEndpoint=\"https://your-resource.openai.azure.com\"\n    azureDeployment=\"your-deployment-name\"\n    azureApiVersion=\"2023-12-01\"\n    samplingRate=\"0.1\"\n    batchSize=\"5\"/>\n```\n\n| Feature | v0.1 | v0.2 | v0.3 | v0.4 | v0.5 |\n|---|---|---|---|---|---|\nRule-Based Masking |\n✅ | ✅ | ✅ | ✅ | ✅ |\nAI Sanitization |\n❌ | ✅ | ✅ | ✅ | ✅ |\nLRU Caching |\n❌ | ✅ | ✅ | ✅ | ✅ |\nBatch Processing |\n❌ | ❌ | ✅ | ✅ | ✅ |\nAsync/Non-Blocking |\n❌ | ❌ | ✅ | ✅ | ✅ |\nMulti-Provider AI |\n❌ | ❌ | ❌ | ✅ | ✅ |\nMetrics Extraction |\n❌ | ❌ | ❌ | ❌ | ✅ (pattern-based) |\nLatency |\n<5ms | <5ms (rule-based) ~1500ms (API) <1ms (cached) |\n<5ms guaranteed | <5ms guaranteed | <5ms guaranteed |\nCost/10M logs |\n$0 | $95 (no cache) $19 (with cache) |\n$15-20 (with batching) | $15-20 (with batching) | $15-25 (depends on metrics config) |\nDependencies |\n0 | 0 new | 0 new | 0 new | 0 new |\nBackward Compatible |\nN/A | ✅ 100% | ✅ 100% | ✅ 100% | ✅ 100% |\n\n```\nLogger logger = LoggerFactory.getLogger(\"MyApp\");\n\n// This will be automatically sanitized by LogGuardAI:\nlogger.info(\"User login: userId=12345 token=sk-abcdef123 password=secret\");\n\n// Output:\n// User login: userId=***** token=***** password=*****\n```\n\n**Development (AI Testing)**\n\n```\n<LogGuardLayout aiEnabled=\"true\" aiProvider=\"openai\" samplingRate=\"0.05\" aiTimeoutMs=\"1000\" batchSize=\"3\"/>\n<!-- Or with Anthropic: aiProvider=\"anthropic\" aiModel=\"claude-3-haiku-20240307\" -->\n```\n\n**Production (Balanced)**\n\n```\n<LogGuardLayout aiEnabled=\"true\" aiProvider=\"openai\" samplingRate=\"0.1\" aiTimeoutMs=\"2000\" batchSize=\"5\"/>\n<!-- Or with Azure: aiProvider=\"azure-openai\" azureEndpoint=\"...\" azureDeployment=\"...\" -->\n```\n\n**High-Security (Premium)**\n\n```\n<LogGuardLayout aiEnabled=\"true\" aiProvider=\"openai\" aiModel=\"gpt-4\" samplingRate=\"1.0\" batchSize=\"10\"/>\n<!-- Or with Claude: aiProvider=\"anthropic\" aiModel=\"claude-3-sonnet-20240229\" -->\n```\n\n**Fast (No AI)**\n\n``` php\n<LogGuardLayout aiEnabled=\"false\"/>  <!-- v0.1 behavior -->\n```\n\n— 5-minute setup (with AI examples)[Quick Start Guide](/sanitizeai/logguardai/blob/main/docs/versions/v0.2/QUICK_START.md)— Deep dive into AI features (3000+ words)[AI Integration Guide](/sanitizeai/logguardai/blob/main/docs/versions/v0.2/AI_GUIDE.md)— v0.1 → v0.2 upgrade path[Migration Guide](/sanitizeai/logguardai/blob/main/docs/versions/v0.2/MIGRATION.md)\n\n— All configuration options[Configuration Reference](/sanitizeai/logguardai/blob/main/docs/guides/configuration.md)— Common issues & solutions[Troubleshooting](/sanitizeai/logguardai/blob/main/docs/guides/troubleshooting.md)— Code organization[Project Structure](/sanitizeai/logguardai/blob/main/docs/guides/project-structure.md)\n\n— How LogGuardAI works[System Overview](/sanitizeai/logguardai/blob/main/docs/architecture/overview.md)— Detailed component descriptions[Components](/sanitizeai/logguardai/blob/main/docs/architecture/components.md)— Interface reference[AIService API](/sanitizeai/logguardai/blob/main/docs/api/air-service-interface.md)\n\n— Complete version history[CHANGELOG](/sanitizeai/logguardai/blob/main/CHANGELOG.md)— v0.2 features[v0.2.0 Release Notes](/sanitizeai/logguardai/blob/main/docs/versions/v0.2/RELEASE_NOTES.md)— Previous version[v0.1.0 Docs](/sanitizeai/logguardai/blob/main/docs/versions/v0.1)\n\n— How to publish[JitPack Publishing](/sanitizeai/logguardai/blob/main/docs/guides/jitpack-publishing.md)\n\n```\n// Transaction logs with credit card & billing info\nlogger.info(\"Transaction: user_id={} amount={} card={} crypto_key={}\",\n    userId, amount, creditCard, encryptionKey);\n// Automatically masked! ✅\n// Auth logs with tokens & credentials\nlogger.info(\"Auth attempt: username={} password={} jwt_token={}\",\n    username, password, jwtToken);\n// All masked automatically! ✅\n// Headers & API keys\nlogger.info(\"Incoming request: Authorization={} X-API-Key={} User-Agent={}\",\n    bearerToken, apiKey, userAgent);\n// Headers sanitized intelligently! ✅\n```\n\nAlso: use `metricsPatterns`\n\nto extract request counts, paths and status codes for lightweight in-app metrics collection.\n\n```\nRule-Based Path:  <5ms    (fast, no external calls)\nAI Hit (cached):  <1ms    (in-memory cache)\nAI Miss (API):    ~1.5s   (to OpenAI, happens ~20% with 80% cache hit rate)\nWithout AI:              $0/month   (rule-based only)\nWith AI (no cache):      $95/month  (5% sampling)\nWith AI + caching:       $19/month  (5% sampling, 80% cache hits)\nSavings:                 80%\nRule-Based Only:        >50,000 logs/sec\nWith AI Sampling:       >45,000 logs/sec (minimal impact)\nWith Caching:           >100,000 logs/sec (cached dominate)\n```\n\n✅ **Secrets Never Exposed**\n\n- API keys, passwords, tokens, PII automatically masked\n- Fail-safe: if LogGuard fails, logging still works (never breaks!)\n- Timeout protection: no hanging requests\n\n✅ **API Key Protection**\n\n- API key stored in environment variable, never hardcoded\n- No logs sent to external services (caching keeps results locally)\n\n✅ **Non-Blocking Design**\n\n- Independent executor for AI calls\n- No thread blocking\n- Application never slows down\n\n- Java 8+\n- Maven 3.6+\n\n```\nmvn clean package\nmvn test\nls target/logguardai-0.5.0.jar\n```\n\n| Version | Release Date | Status | What's New |\n|---|---|---|---|\nv0.5.0 |\n2026-05-27 | ✅ Latest | Pattern-Based Metrics (user-defined regex metrics) |\nv0.4.0 |\n2026-05-03 | ✅ Stable | Multi-Provider AI Support |\nv0.3.0 |\n2026-05-01 | ✅ Stable | Plugin Registration Fix |\nv0.2.0 |\n2026-04-23 | ✅ Stable | AI + Caching |\nv0.1.0 |\n2026-04-22 | ✅ Stable | Rule-based masking |\n\n→ See [CHANGELOG](/sanitizeai/logguardai/blob/main/CHANGELOG.md) for complete history\n\nContributions welcome! Please:\n\n- Fork the repository\n- Create a feature branch\n- Add tests for new features\n- Submit a pull request\n\n**Questions?**[GitHub Discussions](https://github.com/sanitizeai/logguardai/discussions)** Report Issues:**[GitHub Issues](https://github.com/sanitizeai/logguardai/issues)** Full Documentation:**[docs/README.md](/sanitizeai/logguardai/blob/main/docs/README.md)\n\nMIT License - See [LICENSE](/sanitizeai/logguardai/blob/main/LICENSE) for details\n\n**Get Started**→[Quick Start Guide](/sanitizeai/logguardai/blob/main/docs/versions/v0.2/QUICK_START.md)** Enable AI**→[AI Integration Guide](/sanitizeai/logguardai/blob/main/docs/versions/v0.2/AI_GUIDE.md)** Configure**→[Configuration Reference](/sanitizeai/logguardai/blob/main/docs/guides/configuration.md)** Troubleshoot**→[Troubleshooting Guide](/sanitizeai/logguardai/blob/main/docs/guides/troubleshooting.md)\n\n**LogGuardAI: Secure your logs. Automatically.**\n\n*Built with ❤️ by sanitizeai*", "url": "https://wpnews.pro/news/show-hn-intelligent-log-sanitization-and-logs-to-metrics", "canonical_source": "https://github.com/sanitizeai/logguardai", "published_at": "2026-07-21 06:25:22+00:00", "updated_at": "2026-07-21 06:53:13.839204+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools"], "entities": ["LogGuardAI", "Log4j2", "OpenAI", "Anthropic Claude", "Azure OpenAI", "Prometheus"], "alternates": {"html": "https://wpnews.pro/news/show-hn-intelligent-log-sanitization-and-logs-to-metrics", "markdown": "https://wpnews.pro/news/show-hn-intelligent-log-sanitization-and-logs-to-metrics.md", "text": "https://wpnews.pro/news/show-hn-intelligent-log-sanitization-and-logs-to-metrics.txt", "jsonld": "https://wpnews.pro/news/show-hn-intelligent-log-sanitization-and-logs-to-metrics.jsonld"}}