If you work in cloud security or vulnerability scanning, chances are high that you rely on ProjectDiscovery Nuclei—the gold standard open-source vulnerability scanner powered by YAML templates.
While Nuclei performs exceptionally well as a standalone CLI tool, embedding it as an underlying SDK engine inside long-running microservices or continuous scanning workers introduces unique architectural challenges: memory bloat and goroutine leaks over extended execution loops.
Recently, I investigated and resolved these exact engine lifecycle leaks in ** Nuclei Issue #7503** and submitted
When embedding NucleiEngine
into a long-running application loop (where engines are instantiated and closed dynamically per scan target), I noticed that memory consumption climbed steadily over time, and orphaned goroutines remained active long after calling engine.Close()
.
Upon profiling the engine lifecycle in Go, I identified three primary memory leaks:
sync.Map
in HTTP-to-HTTPS Port Tracker:HTTPToHTTPSPortTracker
stored host port mapping states in an unbounded sync.Map
. Over thousands of target scans, this map grew infinitely without eviction.PerHostRateLimitPool
). When an engine execution finished, worker background routines were not cleanly shut down or purged.parsedTemplatesCache
and compiledTemplatesCache
) retained parsed representations in memory between engine instances without an explicit cache purging mechanism during engine teardown.Instead of holding unbounded host entries in a sync.Map
, I replaced the storage structure with an expirable LRU (Least Recently Used) cache configured with a strict capacity bound (4,096 entries) and a 24-hour TTL:
// Replacing unbounded sync.Map with bounded expirable LRU cache
type HTTPToHTTPSPortTracker struct {
cache *expirable.LRU[string, struct{}]
}
func NewHTTPToHTTPSPortTracker() *HTTPToHTTPSPortTracker {
return &HTTPToHTTPSPortTracker{
cache: expirable.NewLRU[string, struct{}](4096, nil, 24*time.Hour),
}
}
This guarantees that host mappings automatically expire and memory remains strictly bounded regardless of how many millions of URLs are scanned.
protocolstate.Close()
I updated the global protocol state tear-down procedure in pkg/protocols/common/protocolstate/state.go
to release rate limiter worker routines and purge trackers upon Close()
:
func Close(executionID string) {
stateLock.Lock()
defer stateLock.Unlock()
if state, ok := globalStateMap[executionID]; ok {
// Release per-host rate limiters and background goroutines
if state.PerHostRateLimitPool != nil {
state.PerHostRateLimitPool.Close()
}
// Purge HTTP to HTTPS tracker entries
if state.HTTPToHTTPSPortTracker != nil {
state.HTTPToHTTPSPortTracker.Purge()
}
delete(globalStateMap, executionID)
}
}
Finally, I added a thread-safe Purge()
method to the template parser struct and invoked interface type assertions during NucleiEngine.Close()
:
// Safely purge compiled template caches on engine close
func (e *NucleiEngine) closeInternal() error {
if e.parser != nil {
e.parser.Purge()
}
if purger, ok := e.executerOpts.Parser.(interface{ Purge() }); ok {
purger.Purge()
}
return nil
}
When designing solutions for large open-source codebases, evaluating architectural trade-offs is essential:
Options.HTTPToHTTPSCacheSize
) would be a clean future addition.interface{ Purge() }
) keeps the codebase decoupled and preserves backward compatibility for third-party SDK consumers using custom parsers without breaking their implementations.I validated these fixes across Nuclei unit test packages (httpclientpool
, protocolstate
, templates
, and lib
), verifying 100%
success with zero memory accumulation between consecutive engine shutdowns.
Fixes #7503 by implementing the required leak-prevention cleanup mechanisms outlined in #7502 for long-running embedded engines.
sync.Map
in HTTPToHTTPSPortTracker
(pkg/protocols/http/httpclientpool/http_to_https_tracker.go
) with a size-bounded expirable LRU cache (4096 entries max, 24h TTL) and added Purge()
.protocolstate.Close()
(pkg/protocols/common/protocolstate/state.go
) to release per-host rate-limit pool goroutines and purge the HTTP-to-HTTPS tracker on shutdown.NucleiEngine.Close()
/ closeInternal()
(lib/sdk.go
) and Parser
(pkg/templates/parser.go
) to purge parsed and compiled template caches on engine close.The embedded engine can leak memory and goroutines over time during long-running usage.
Implement the leak-prevention work described in #7502:
HTTPToHTTPS
tracker with an LRUWithout explicit cleanup and bounded caching, long-running embedders can accumulate memory usage and leave background goroutines running indefinitely.
HTTPToHTTPS
tracking / redirect bookkeepingHTTPToHTTPS
tracker is size-bounded and evicts old entries.PR title: fix leaks
sync.Map
in Long-Running Apps:sync.Map
is convenient, it lacks eviction policies. Use LRU caches with TTLs for dynamic lookup tables.Close()
/ Purge()
methods to release background channels and goroutines.if purger, ok := obj.(interface{ Purge() }); ok
enable clean resource cleanup without introducing rigid package dependencies.Written by @Thrylox. Connect with me on GitHub!