{"slug": "how-i-debugged-and-fixed-memory-goroutine-leaks-in-projectdiscovery-nuclei", "title": "How I Debugged and Fixed Memory & Goroutine Leaks in ProjectDiscovery Nuclei Engine 🚀", "summary": "ProjectDiscovery Nuclei, an open-source vulnerability scanner, suffered from memory and goroutine leaks when embedded in long-running applications. A developer identified three primary leaks: an unbounded sync.Map in the HTTP-to-HTTPS port tracker, unreleased rate limiter goroutines, and persistent template caches. The fixes replaced the sync.Map with an expirable LRU cache, added proper cleanup in protocol state teardown, and introduced a Purge method for template caches, ensuring bounded memory and clean goroutine shutdown.", "body_md": "If you work in cloud security or vulnerability scanning, chances are high that you rely on **ProjectDiscovery Nuclei**—the gold standard open-source vulnerability scanner powered by YAML templates.\n\nWhile Nuclei performs exceptionally well as a standalone CLI tool, embedding it as an underlying SDK engine inside long-running microservices or continuous scanning workers introduces unique architectural challenges: **memory bloat** and **goroutine leaks** over extended execution loops.\n\nRecently, I investigated and resolved these exact engine lifecycle leaks in ** Nuclei Issue #7503** and submitted\n\nWhen embedding `NucleiEngine`\n\ninto a long-running application loop (where engines are instantiated and closed dynamically per scan target), I noticed that memory consumption climbed steadily over time, and orphaned goroutines remained active long after calling `engine.Close()`\n\n.\n\nUpon profiling the engine lifecycle in Go, I identified three primary memory leaks:\n\n`sync.Map`\n\nin HTTP-to-HTTPS Port Tracker:`HTTPToHTTPSPortTracker`\n\nstored host port mapping states in an unbounded `sync.Map`\n\n. Over thousands of target scans, this map grew infinitely without eviction.`PerHostRateLimitPool`\n\n). When an engine execution finished, worker background routines were not cleanly shut down or purged.`parsedTemplatesCache`\n\nand `compiledTemplatesCache`\n\n) retained parsed representations in memory between engine instances without an explicit cache purging mechanism during engine teardown.Instead of holding unbounded host entries in a `sync.Map`\n\n, I replaced the storage structure with an expirable LRU (Least Recently Used) cache configured with a strict capacity bound (4,096 entries) and a 24-hour TTL:\n\n```\n// Replacing unbounded sync.Map with bounded expirable LRU cache\ntype HTTPToHTTPSPortTracker struct {\n    cache *expirable.LRU[string, struct{}]\n}\n\nfunc NewHTTPToHTTPSPortTracker() *HTTPToHTTPSPortTracker {\n    return &HTTPToHTTPSPortTracker{\n        cache: expirable.NewLRU[string, struct{}](4096, nil, 24*time.Hour),\n    }\n}\n```\n\nThis guarantees that host mappings automatically expire and memory remains strictly bounded regardless of how many millions of URLs are scanned.\n\n`protocolstate.Close()`\n\nI updated the global protocol state tear-down procedure in `pkg/protocols/common/protocolstate/state.go`\n\nto release rate limiter worker routines and purge trackers upon `Close()`\n\n:\n\n```\nfunc Close(executionID string) {\n    stateLock.Lock()\n    defer stateLock.Unlock()\n\n    if state, ok := globalStateMap[executionID]; ok {\n        // Release per-host rate limiters and background goroutines\n        if state.PerHostRateLimitPool != nil {\n            state.PerHostRateLimitPool.Close()\n        }\n        // Purge HTTP to HTTPS tracker entries\n        if state.HTTPToHTTPSPortTracker != nil {\n            state.HTTPToHTTPSPortTracker.Purge()\n        }\n        delete(globalStateMap, executionID)\n    }\n}\n```\n\nFinally, I added a thread-safe `Purge()`\n\nmethod to the template parser struct and invoked interface type assertions during `NucleiEngine.Close()`\n\n:\n\n```\n// Safely purge compiled template caches on engine close\nfunc (e *NucleiEngine) closeInternal() error {\n    if e.parser != nil {\n        e.parser.Purge()\n    }\n    if purger, ok := e.executerOpts.Parser.(interface{ Purge() }); ok {\n        purger.Purge()\n    }\n    return nil\n}\n```\n\nWhen designing solutions for large open-source codebases, evaluating architectural trade-offs is essential:\n\n`Options.HTTPToHTTPSCacheSize`\n\n) would be a clean future addition.`interface{ Purge() }`\n\n) keeps the codebase decoupled and preserves backward compatibility for third-party SDK consumers using custom parsers without breaking their implementations.I validated these fixes across Nuclei unit test packages (`httpclientpool`\n\n, `protocolstate`\n\n, `templates`\n\n, and `lib`\n\n), verifying `100%`\n\nsuccess with zero memory accumulation between consecutive engine shutdowns.\n\nFixes #7503 by implementing the required leak-prevention cleanup mechanisms outlined in #7502 for long-running embedded engines.\n\n`sync.Map`\n\nin `HTTPToHTTPSPortTracker`\n\n(`pkg/protocols/http/httpclientpool/http_to_https_tracker.go`\n\n) with a size-bounded expirable LRU cache (4096 entries max, 24h TTL) and added `Purge()`\n\n.`protocolstate.Close()`\n\n(`pkg/protocols/common/protocolstate/state.go`\n\n) to release per-host rate-limit pool goroutines and purge the HTTP-to-HTTPS tracker on shutdown.`NucleiEngine.Close()`\n\n/ `closeInternal()`\n\n(`lib/sdk.go`\n\n) and `Parser`\n\n(`pkg/templates/parser.go`\n\n) to purge parsed and compiled template caches on engine close.The embedded engine can leak memory and goroutines over time during long-running usage.\n\nImplement the leak-prevention work described in #7502:\n\n`HTTPToHTTPS`\n\ntracker with an LRUWithout explicit cleanup and bounded caching, long-running embedders can accumulate memory usage and leave background goroutines running indefinitely.\n\n`HTTPToHTTPS`\n\ntracking / redirect bookkeeping`HTTPToHTTPS`\n\ntracker is size-bounded and evicts old entries.PR title: fix leaks\n\n`sync.Map`\n\nin Long-Running Apps:`sync.Map`\n\nis convenient, it lacks eviction policies. Use LRU caches with TTLs for dynamic lookup tables.`Close()`\n\n/ `Purge()`\n\nmethods to release background channels and goroutines.`if purger, ok := obj.(interface{ Purge() }); ok`\n\nenable clean resource cleanup without introducing rigid package dependencies.*Written by @Thrylox. Connect with me on GitHub!*", "url": "https://wpnews.pro/news/how-i-debugged-and-fixed-memory-goroutine-leaks-in-projectdiscovery-nuclei", "canonical_source": "https://dev.to/thrylox/how-i-debugged-and-fixed-memory-goroutine-leaks-in-projectdiscovery-nuclei-engine-4794", "published_at": "2026-06-28 01:20:58+00:00", "updated_at": "2026-06-28 02:03:45.414054+00:00", "lang": "en", "topics": ["developer-tools", "ai-infrastructure"], "entities": ["ProjectDiscovery", "Nuclei", "Go"], "alternates": {"html": "https://wpnews.pro/news/how-i-debugged-and-fixed-memory-goroutine-leaks-in-projectdiscovery-nuclei", "markdown": "https://wpnews.pro/news/how-i-debugged-and-fixed-memory-goroutine-leaks-in-projectdiscovery-nuclei.md", "text": "https://wpnews.pro/news/how-i-debugged-and-fixed-memory-goroutine-leaks-in-projectdiscovery-nuclei.txt", "jsonld": "https://wpnews.pro/news/how-i-debugged-and-fixed-memory-goroutine-leaks-in-projectdiscovery-nuclei.jsonld"}}