Prompt for model to add protection for your backend from scanners

A developer published a reusable prompt instructing AI coding assistants to implement a four-layer, fail-open backend defense against automated scanners, log exhaustion, and OTP brute-forcing. The blueprint covers verified reverse-proxy IP resolution, a Redis-backed progressive ban engine with stealth 404s and time-capped bans, an exception filter that keeps 4xx errors out of the database, and distributed OTP attempt counters with a five-failure limit. It emphasizes parameterized thresholds, fail-open behavior under Redis latency, and avoiding hardcoded secrets.

Purpose: Hand this document to any AI coding assistant or backend engineer to implement a production-grade, fail-open bot scanner defense, reverse proxy IP resolution, log exhaustion prevention, and OTP brute-force protection system in any backend service. You are an expert backend security and infrastructure architect. Your task is to implement a robust, fail-open defense system against automated scanners, web crawlers, log exhaustion attacks, and brute-force credential abuse. Follow the requirements, algorithms, and implementation blueprints defined below. Maintain fail-open availability: security controls must never degrade uptime or block legitimate users if secondary systems like Redis experience latency or outages. Do not hardcode any sensitive credentials, production ports, or environment-specific secrets. Keep all thresholds and configurations parameterized. Do not use emojis in code comments, responses, or commit messages. 1. Automated Vulnerability Scanning: Botnets and scanners Shodan, Censys, automated crawlers bombard endpoints searching for sensitive paths .env , .git , wp-login.php , phpmyadmin , backup dumps . Leaving these unmitigated wastes CPU, memory, and database bandwidth. 2. The "Self-Inflicted Proxy DoS" Trap: When running behind reverse proxies Cloudflare, Nginx, container networks , naively blocking req.ip without configuring proxy trust bans the reverse proxy itself, creating a 100% outage for all users. Reading headers like X-Forwarded-For or CF-Connecting-IP directly without verification allows attackers to spoof arbitrary IP addresses. 3. Database Log Exhaustion 404 Floods : Applications that log unhandled exceptions to a database table suffer catastrophic disk I/O and storage bloat when a scanner generates hundreds or thousands of 404 requests in seconds. 4. Carrier-Grade NAT CGNAT Collateral Damage: Mobile users share public carrier IP addresses. Permanent IP bans will inevitably block innocent users on the same cell tower or public Wi-Fi. Bans must be progressive and strictly time-capped. 5. OTP Credential Brute-Forcing: 6-digit numeric OTPs have 1,000,000 combinations. Without distributed attempt counters, bots can brute-force codes within their active validity window. You must implement four distinct defense layers: Incoming Request │ ▼ Layer 1: Reverse Proxy & Real IP Resolution - Trust only verified proxy CIDRs Cloudflare + Local/Container networks - Framework right-to-left X-Forwarded-For traversal │ ▼ Layer 2: Scanner Detection & Progressive Redis Ban Engine - Whitelist bypass for legitimate application routes /api/v1/ , etc. - Fast regex for scanner probes, dangerous extensions, and malformed URIs - Configurable strike sliding window in Redis e.g. 3 strikes in 60s - Stealth 404 responses during probing do not signal an active WAF - Progressive ban durations with 24h retention e.g. 15m - 30m - 1h max - Generic 403 Forbidden for actively banned IPs no fingerprinting - Strict fail-open design with command timeout <= 80ms │ ▼ Layer 3: Global Exception Filter Log Exhaustion Guard - Never write 4xx errors especially scanner 404s to the database - 404s logged at debug level only - Other 4xx logged at warning level - 5xx logged at error level and persisted to database/persistent storage │ ▼ Layer 4: Distributed OTP Brute-Force Protection - Redis attempt counter scoped to normalized recipient identifier - TTL matching the OTP expiration window - Hard limit of 5 failed attempts - Automatic database revocation of active OTPs upon threshold breach When generating or deploying this system, strictly enforce the following rules: 1. Parameterize Timing & Strike Thresholds: Do not hardcode strike limits e.g., 3 strikes in 60s where an attacker reading source code can calibrate scan speed to fly just under the radar. Allow overriding via environment variables. 2. Prevent Server Fingerprinting: Return generic, standard error messages e.g. standard 403 Forbidden response rather than custom signatures that identify the backend technology or internal firewall software. 3. Strict Whitelist Boundary: Never put root / , wildcard patterns, or public user-content paths into the safe route bypass list. Only include verified application route prefixes. 4. Mandatory Expiration TTL on All Cache Keys: Every key created in Redis scan:strike: , scan:ban: , scan:offenses: , auth:otp attempts: MUST have an explicit TTL. Never write unbounded keys that can lead to memory exhaustion OOM DoS . 5. No Direct Header Trust: Under no circumstances should req.headers 'x-forwarded-for' or req.headers 'cf-connecting-ip' be read directly. Trust proxy configuration must be used so the framework validates proxy hops. 6. No Secret Leaks in Exceptions: Ensure exception filters sanitize query params, request bodies, and database errors to prevent leaking database structure or environment variables in responses. Ensure req.ip resolves to the authentic remote client IP, preventing both proxy banning and header spoofing. 1. Never read req.headers 'x-forwarded-for' or req.headers 'cf-connecting-ip' directly. 2. Configure framework proxy trust using verified upstream CIDR blocks. 3. Express/Connect-based frameworks evaluate X-Forwarded-For from right to left, stripping trusted hops until the first untrusted IP is reached. Include official Cloudflare ranges and private container/bridge networks adjust if your reverse proxy infrastructure differs : - Private / Container Networks: 'loopback' , 'linklocal' , 'uniquelocal' - Cloudflare IPv4: 173.245.48.0/20 , 103.21.244.0/22 , 103.22.200.0/22 , 103.31.4.0/22 , 141.101.64.0/18 , 108.162.192.0/18 , 190.93.240.0/20 , 188.114.96.0/20 , 197.234.240.0/22 , 198.41.128.0/17 , 162.158.0.0/15 , 104.16.0.0/13 , 104.24.0.0/14 , 172.64.0.0/13 , 131.0.72.0/22 - Cloudflare IPv6: 2400:cb00::/32 , 2606:4700::/32 , 2803:f800::/32 , 2405:b500::/32 , 2405:8100::/32 , 2a06:98c0::/29 , 2c0f:f248::/32 Detect automated bot probes, increment strike counts in Redis, ban repeat offenders progressively, and return stealth responses. Before route handling, check if the client IP exists in Redis: - Redis key: scan:ban:<client ip - If key exists: immediately return HTTP 403 Forbidden with a generic message: "Forbidden" . - Extract full raw path baseUrl + path . - Attempt decodeURIComponent rawPath . - If decoding throws URIError e.g. invalid byte sequences or malformed percent-encoding like %c0%ae , treat the request immediately as a scanner probe. Do not evaluate regex or Redis strikes against legitimate application routes. If decodedPath starts with any of the configured prefixes, immediately pass to next : - '/api/v1' or application route prefix - '/api/docs' API documentation - '/health' health-check endpoints - '/favicon.ico' Evaluate non-whitelisted paths against the scanner regex: / ?:^\/|\/ ?:\.env ^/ |\.git ^/ |\.svn ^/ |\.aws ^/ |\.ssh ^/ |\.npmrc ^/ |\.netrc ^/ |\.DS Store ^/ | ^/ history|wp-admin|wp-login|wp-content|wp-includes|wp-config|phpmyadmin|pma|xmlrpc\.php|cgi-bin|actuator|eval-stdin\.php|web\.config ?:\/|$|\.|\? | ?:\.php|\.sql|\.bak|\.old|\.orig|\.save|\.swp|\.zip|\.tar|\.gz|\.tgz|\.bz2|\.rar|\.7z ?:\?. ?$/i If the pattern does not match and URI is not malformed, pass to next . When a probe is detected: 1. Increment strike counter: INCR scan:strike:<client ip 2. If counter equals 1, set window expiry: EXPIRE scan:strike:<client ip <STRIKE WINDOW SEC default: 60 seconds . 3. If strikes < threshold default: 3 : Return HTTP 404 Not Found . Never return 403 or custom messages during strikes, keeping the trap stealthy. 4. If strikes = threshold: - Increment historical offenses: INCR scan:offenses:<client ip - If offenses equals 1, set retention expiry: EXPIRE scan:offenses:<client ip 86400 24-hour memory . - Select ban duration based on offense tier: - Offense 1: 900 seconds 15 minutes - Offense 2: 1800 seconds 30 minutes - Offense 3+: 3600 seconds 1 hour maximum cap for CGNAT safety - Store ban: SET scan:ban:<client ip "1" EX <ban duration sec - Delete strike key: DEL scan:strike:<client ip - Return HTTP 404 Not Found for the probe request that triggered the ban. 5. Increment historical offenses: - Wrap all Redis operations in try/catch . - If Redis is unavailable or times out, catch the error, log a rate-limited warning cooldown of at least 30 seconds , and allow the request to proceed fail-open . - Configure Redis client with: - commandTimeout : 80ms maximum. - enableOfflineQueue : false do not buffer commands when disconnected . - maxRetriesPerRequest : 1. Prevent high-frequency scanner probes from overwhelming error logging pipelines, database tables, and disk I/O. 1. Never write 4xx errors client errors, bad inputs, missing routes, scanner 404s to the database or persistent logging storage. 2. Only write 5xx errors unhandled exceptions, database disconnects, crashes to database error tables. 3. Categorize console/stdout log levels: - 404 Not Found : Log at debug level only. In production environments where debug logs are silenced, scanner noise generates zero log volume. - Other 4xx 400, 401, 403, 422 : Log at warn level. - 5xx : Log at error level with full stack traces. 4. Ensure response bodies never expose internal database column names, raw queries, or stack traces to clients. Prevent automated brute-forcing of numeric OTP codes across clustered backend instances. 1. Track verification attempts per recipient identifier in Redis: auth:otp attempts:<normalized identifier . 2. Normalize identifier e.g. lowercase and trim email/phone . 3. On verification attempt: - Increment attempts: INCR auth:otp attempts:<identifier . - On the first attempt, set TTL equal to the OTP expiration duration e.g. 5 minutes / 300 seconds . 4. Increment attempts: 5. Enforce strict attempt threshold maximum 5 attempts : - If attempts 5 : - Delete all pending OTP records for this recipient in the database immediately hard revocation . - Remove the attempt key from Redis. - Throw a generic client exception: "Too many invalid attempts. Your verification code has expired. Please request a new code." 6. If 7. Wrap Redis logic with fail-open handling: if Redis fails, proceed to database OTP verification so users are not locked out due to cache outages. js import { NestFactory } from '@nestjs/core'; import { NestExpressApplication } from '@nestjs/platform-express'; import { AppModule } from './app.module'; const TRUSTED PROXIES = 'loopback', 'linklocal', 'uniquelocal', // Cloudflare IPv4 '173.245.48.0/20', '103.21.244.0/22', '103.22.200.0/22', '103.31.4.0/22', '141.101.64.0/18', '108.162.192.0/18', '190.93.240.0/20', '188.114.96.0/20', '197.234.240.0/22', '198.41.128.0/17', '162.158.0.0/15', '104.16.0.0/13', '104.24.0.0/14', '172.64.0.0/13', '131.0.72.0/22', // Cloudflare IPv6 '2400:cb00::/32', '2606:4700::/32', '2803:f800::/32', '2405:b500::/32', '2405:8100::/32', '2a06:98c0::/29', '2c0f:f248::/32', ; async function bootstrap { const app = await NestFactory.create<NestExpressApplication AppModule ; // Set trust proxy BEFORE any middleware reads req.ip app.set 'trust proxy', TRUSTED PROXIES ; const port = process.env.PORT || 3000; await app.listen port ; } bootstrap ; js import { Injectable, OnModuleDestroy, OnModuleInit, Logger, InternalServerErrorException } from '@nestjs/common'; import Redis from 'ioredis'; @Injectable export class RedisService implements OnModuleInit, OnModuleDestroy { private readonly logger = new Logger RedisService.name ; private isReady = false; public readonly client: Redis; constructor { const redisUrl = process.env.REDIS URL; if redisUrl { throw new InternalServerErrorException 'REDIS URL environment variable is missing' ; } this.client = new Redis redisUrl, { commandTimeout: Number process.env.REDIS COMMAND TIMEOUT || 80, enableOfflineQueue: false, maxRetriesPerRequest: 1, retryStrategy: times = Math.min times 100, 3000 , } ; this.client.on 'connect', = { this. isReady = true; this.logger.log 'Redis connected' ; } ; this.client.on 'ready', = { this. isReady = true; } ; this.client.on 'error', err: Error = { this. isReady = false; this.logger.warn Redis connection error: ${err.message} ; } ; this.client.on 'close', = { this. isReady = false; } ; } async onModuleInit { await new Promise<void resolve = { if this. isReady return resolve ; const timeout = setTimeout = { if this. isReady { this.logger.warn 'Redis not ready after 2s; running in degraded fail-open mode' ; } resolve ; }, 2000 ; this.client.once 'ready', = { clearTimeout timeout ; resolve ; } ; } ; } get isReady : boolean { return this. isReady; } async onModuleDestroy { await this.client.quit ; } } js import { Injectable, Logger, NestMiddleware } from '@nestjs/common'; import { Request, Response, NextFunction } from 'express'; import { RedisService } from '../redis/redis.service'; const SCANNER REGEX = / ?:^\/|\/ ?:\.env ^/ |\.git ^/ |\.svn ^/ |\.aws ^/ |\.ssh ^/ |\.npmrc ^/ |\.netrc ^/ |\.DS Store ^/ | ^/ history|wp-admin|wp-login|wp-content|wp-includes|wp-config|phpmyadmin|pma|xmlrpc\.php|cgi-bin|actuator|eval-stdin\.php|web\.config ?:\/|$|\.|\? | ?:\.php|\.sql|\.bak|\.old|\.orig|\.save|\.swp|\.zip|\.tar|\.gz|\.tgz|\.bz2|\.rar|\.7z ?:\?. ?$/i; const SAFE ROUTE PREFIXES = '/api/v1', '/api/docs', '/health', '/favicon.ico', ; const BAN DURATIONS SEC = 0, 900, 1800, 3600 ; const STRIKE WINDOW SEC = Number process.env.SCANNER STRIKE WINDOW SEC || 60; const MAX STRIKES = Number process.env.SCANNER MAX STRIKES || 3; @Injectable export class ScannerBlockMiddleware implements NestMiddleware { private readonly logger = new Logger ScannerBlockMiddleware.name ; private lastRedisWarnMs = 0; private readonly REDIS WARN COOLDOWN MS = 30 000; constructor private readonly redisService: RedisService {} async use req: Request, res: Response, next: NextFunction { const clientIp = req.ip; if clientIp return next ; // 1. Intercept active bans if this.redisService.isReady { try { const isBanned = await this.redisService.client.exists scan:ban:${clientIp} ; if isBanned { return res.status 403 .send 'Forbidden' ; } } catch err: any { this.warnRedisOnce err.message ; } } // 2. Decode path and check for malformed URIs const rawPath = req.baseUrl ? ${req.baseUrl}${req.path} : req.path; let decodedPath = rawPath; let isMalformed = false; try { decodedPath = decodeURIComponent rawPath ; } catch { isMalformed = true; } // 3. Whitelist bypass for legitimate API traffic if isMalformed && SAFE ROUTE PREFIXES.some p = decodedPath.startsWith p { return next ; } // 4. Scanner pattern check if isMalformed && SCANNER REGEX.test decodedPath { return next ; } // 5. Strike recording and progressive banning if this.redisService.isReady { try { await this.recordStrike clientIp, req.method, decodedPath ; } catch err: any { this.warnRedisOnce err.message ; } } // Return generic 404 to avoid revealing the honeypot return res.status 404 .end ; } private async recordStrike ip: string, method: string, path: string : Promise<void { const strikeKey = scan:strike:${ip} ; const banKey = scan:ban:${ip} ; const offenseKey = scan:offenses:${ip} ; const strikes = await this.redisService.client.incr strikeKey ; if strikes === 1 { await this.redisService.client.expire strikeKey, STRIKE WINDOW SEC ; } if strikes < MAX STRIKES { this.logger.debug Scanner probe detected from ${ip} ; return; } const offenses = await this.redisService.client.incr offenseKey ; if offenses === 1 { await this.redisService.client.expire offenseKey, 86400 ; } const banSec = BAN DURATIONS SEC Math.min offenses, BAN DURATIONS SEC.length - 1 ; await this.redisService.client.set banKey, '1', 'EX', banSec ; await this.redisService.client.del strikeKey ; this.logger.warn Temporarily blocked suspicious IP for ${banSec / 60}m ; } private warnRedisOnce message: string : void { const now = Date.now ; if now - this.lastRedisWarnMs this.REDIS WARN COOLDOWN MS { this.lastRedisWarnMs = now; this.logger.warn ScannerBlock: Redis unavailable ${message} . Failing open - checks skipped. ; } } } js import { Catch, ExceptionFilter, ExecutionContext, HttpException, HttpStatus, Logger, Optional, } from '@nestjs/common'; import { Request, Response } from 'express'; export interface PersistentErrorLogger { logError entry: any : Promise<void ; } @Catch export class GlobalExceptionFilter implements ExceptionFilter { private readonly logger = new Logger GlobalExceptionFilter.name ; constructor @Optional private errorLoggerService?: PersistentErrorLogger {} async catch exception: unknown, host: ExecutionContext { const ctx = host.switchToHttp ; const request = ctx.getRequest<Request ; const response = ctx.getResponse<Response ; let status = HttpStatus.INTERNAL SERVER ERROR; let message = 'Internal server error'; let stack: string | undefined; if exception instanceof HttpException { status = exception.getStatus ; const exceptionResponse = exception.getResponse ; message = typeof exceptionResponse === 'object' && 'message' in exceptionResponse ? exceptionResponse as any .message : exception.message; } else if exception instanceof Error { message = exception.message; stack = exception.stack; } const is4xx = status = 400 && status < 500; // Never write 4xx errors to database to prevent storage bloat if is4xx && this.errorLoggerService { try { await this.errorLoggerService.logError { message, stack, statusCode: status, endpoint: request.url, method: request.method, context: 'GlobalExceptionFilter', metadata: { path: request.path, ip: request.ip, userAgent: request.get 'user-agent' , }, } ; } catch logError { this.logger.error 'Failed to persist error log', logError ; } } // Scale logging by severity if status === HttpStatus.NOT FOUND { this.logger.debug ${request.method} ${request.url} - 404 ; } else if is4xx { this.logger.warn ${request.method} ${request.url} - ${status} ; } else { this.logger.error ${request.method} ${request.url} - ${status} , stack ; } // Sanitize 500 response so internal traces are not exposed to clients const clientMessage = status = 500 ? 'Internal server error' : message; response.status status .json { statusCode: status, message: clientMessage, timestamp: new Date .toISOString , path: request.url, } ; } } js import { Injectable, BadRequestException } from '@nestjs/common'; import { RedisService } from '../redis/redis.service'; @Injectable export class AuthService { private readonly otpValiditySeconds = 300; // 5 minutes private readonly maxOtpAttempts = 5; constructor private readonly redisService: RedisService, private readonly usersService: any, // Injected user/persistence service {} async verifyOtp otp: string, identifier: string : Promise<boolean { const cleanIdentifier = identifier.trim .toLowerCase ; const attemptKey = auth:otp attempts:${cleanIdentifier} ; if this.redisService.isReady { try { const attempts = await this.redisService.client.incr attemptKey ; if attempts === 1 { await this.redisService.client.expire attemptKey, this.otpValiditySeconds ; } if attempts this.maxOtpAttempts { // Immediately revoke active OTPs in the database const user = await this.usersService.findOneByIdentifier cleanIdentifier ; if user { await this.usersService.deleteAllRelatedOtps user.id ; } await this.redisService.client.del attemptKey ; throw new BadRequestException 'Too many invalid attempts. Your verification code has expired. Please request a new code.', ; } } catch err: any { if err instanceof BadRequestException throw err; // Fail-open: proceed with database check if Redis is unavailable } } // Proceed with validating the OTP code against database record... return true; } } If implementing in another language or backend framework, adhere to these structural equivalents: | Responsibility | Fastify Node.js | FastAPI / Starlette Python | Gin / Fiber Go | |---|---|---|---| | Proxy Trust | fastify { trustProxy: TRUSTED PROXIES } | Uvicorn --forwarded-allow-ips or Starlette ProxyHeadersMiddleware | router.SetTrustedProxies TRUSTED PROXIES | | Scanner Interceptor | addHook 'onRequest', ... | Custom ASGI middleware BaseHTTPMiddleware | Standard middleware handler c.Next | | Redis Commands | ioredis with 80ms command timeout | redis-py asyncio with socket timeout=0.08 | go-redis with DialTimeout: 80ms | | Error Handling | setErrorHandler filtering 4xx from DB | Exception handler app.add exception handler | Global recovery middleware | | Fail-Open Catch | Wrap Redis calls in try/except - fallback to next | Wrap Redis calls in try/except - proceed with request | Check err = nil - proceed without blocking | When the implementation is complete, verify behavior with these tests: 1. Proxy Trust Resolution: curl -H "X-Forwarded-For: 203.0.113.195, 172.64.0.1" http://localhost:3000/health Verify the application resolves req.ip as 203.0.113.195 client IP and not Cloudflare's IP. 2. Scanner Detection & Stealth 404: curl -i http://localhost:3000/.env curl -i http://localhost:3000/wp-login.php Verify the response is 404 Not Found empty body or standard 404, never 403 or custom scanner message . 3. Strike Escalation and Ban Activation: Send probe requests exceeding the configured threshold within the sliding window: for i in {1..3}; do curl -s -o /dev/null -w "%{http code}\n" http://localhost:3000/.env; done Subsequent request to any route: curl -i http://localhost:3000/api/v1/health Verify response is HTTP 403 Forbidden with a generic response body. 4. Whitelist Verification: curl -i http://localhost:3000/api/v1/health Verify legitimate routes under /api/v1 are never intercepted by the scanner regex. 5. Fail-Open Resilience Test: Stop the Redis instance or block its port. Execute requests to both legitimate and probed endpoints. Verify the application does not hang, throw 500 errors, or exceed 80ms latency.