How I Debugged and Fixed Memory & Goroutine Leaks in ProjectDiscovery Nuclei Engine 🚀

ProjectDiscovery Nuclei, an open-source vulnerability scanner, suffered from memory and goroutine leaks when embedded in long-running applications. A developer identified three primary leaks: an unbounded sync.Map in the HTTP-to-HTTPS port tracker, unreleased rate limiter goroutines, and persistent template caches. The fixes replaced the sync.Map with an expirable LRU cache, added proper cleanup in protocol state teardown, and introduced a Purge method for template caches, ensuring bounded memory and clean goroutine shutdown.

If you work in cloud security or vulnerability scanning, chances are high that you rely on ProjectDiscovery Nuclei —the gold standard open-source vulnerability scanner powered by YAML templates. While Nuclei performs exceptionally well as a standalone CLI tool, embedding it as an underlying SDK engine inside long-running microservices or continuous scanning workers introduces unique architectural challenges: memory bloat and goroutine leaks over extended execution loops. Recently, I investigated and resolved these exact engine lifecycle leaks in Nuclei Issue 7503 and submitted When embedding NucleiEngine into a long-running application loop where engines are instantiated and closed dynamically per scan target , I noticed that memory consumption climbed steadily over time, and orphaned goroutines remained active long after calling engine.Close . Upon profiling the engine lifecycle in Go, I identified three primary memory leaks: sync.Map in HTTP-to-HTTPS Port Tracker: HTTPToHTTPSPortTracker stored host port mapping states in an unbounded sync.Map . Over thousands of target scans, this map grew infinitely without eviction. PerHostRateLimitPool . When an engine execution finished, worker background routines were not cleanly shut down or purged. parsedTemplatesCache and compiledTemplatesCache retained parsed representations in memory between engine instances without an explicit cache purging mechanism during engine teardown.Instead of holding unbounded host entries in a sync.Map , I replaced the storage structure with an expirable LRU Least Recently Used cache configured with a strict capacity bound 4,096 entries and a 24-hour TTL: // Replacing unbounded sync.Map with bounded expirable LRU cache type HTTPToHTTPSPortTracker struct { cache expirable.LRU string, struct{} } func NewHTTPToHTTPSPortTracker HTTPToHTTPSPortTracker { return &HTTPToHTTPSPortTracker{ cache: expirable.NewLRU string, struct{} 4096, nil, 24 time.Hour , } } This guarantees that host mappings automatically expire and memory remains strictly bounded regardless of how many millions of URLs are scanned. protocolstate.Close I updated the global protocol state tear-down procedure in pkg/protocols/common/protocolstate/state.go to release rate limiter worker routines and purge trackers upon Close : func Close executionID string { stateLock.Lock defer stateLock.Unlock if state, ok := globalStateMap executionID ; ok { // Release per-host rate limiters and background goroutines if state.PerHostRateLimitPool = nil { state.PerHostRateLimitPool.Close } // Purge HTTP to HTTPS tracker entries if state.HTTPToHTTPSPortTracker = nil { state.HTTPToHTTPSPortTracker.Purge } delete globalStateMap, executionID } } Finally, I added a thread-safe Purge method to the template parser struct and invoked interface type assertions during NucleiEngine.Close : // Safely purge compiled template caches on engine close func e NucleiEngine closeInternal error { if e.parser = nil { e.parser.Purge } if purger, ok := e.executerOpts.Parser. interface{ Purge } ; ok { purger.Purge } return nil } When designing solutions for large open-source codebases, evaluating architectural trade-offs is essential: Options.HTTPToHTTPSCacheSize would be a clean future addition. interface{ Purge } keeps the codebase decoupled and preserves backward compatibility for third-party SDK consumers using custom parsers without breaking their implementations.I validated these fixes across Nuclei unit test packages httpclientpool , protocolstate , templates , and lib , verifying 100% success with zero memory accumulation between consecutive engine shutdowns. Fixes 7503 by implementing the required leak-prevention cleanup mechanisms outlined in 7502 for long-running embedded engines. sync.Map in HTTPToHTTPSPortTracker pkg/protocols/http/httpclientpool/http to https tracker.go with a size-bounded expirable LRU cache 4096 entries max, 24h TTL and added Purge . protocolstate.Close pkg/protocols/common/protocolstate/state.go to release per-host rate-limit pool goroutines and purge the HTTP-to-HTTPS tracker on shutdown. NucleiEngine.Close / closeInternal lib/sdk.go and Parser pkg/templates/parser.go to purge parsed and compiled template caches on engine close.The embedded engine can leak memory and goroutines over time during long-running usage. Implement the leak-prevention work described in 7502: HTTPToHTTPS tracker with an LRUWithout explicit cleanup and bounded caching, long-running embedders can accumulate memory usage and leave background goroutines running indefinitely. HTTPToHTTPS tracking / redirect bookkeeping HTTPToHTTPS tracker is size-bounded and evicts old entries.PR title: fix leaks sync.Map in Long-Running Apps: sync.Map is convenient, it lacks eviction policies. Use LRU caches with TTLs for dynamic lookup tables. Close / Purge methods to release background channels and goroutines. if purger, ok := obj. interface{ Purge } ; ok enable clean resource cleanup without introducing rigid package dependencies. Written by @Thrylox. Connect with me on GitHub