{"slug": "see-which-bots-and-ai-crawlers-are-visiting-your-angular-site", "title": "See which bots and AI crawlers are visiting your Angular site", "summary": "WebDecoy published a walkthrough showing how to place its detection middleware in front of an Angular server-side rendering handler so operators can observe bot and AI crawler requests in monitor mode. The example, built on Angular 22.2 with WebDecoy's Express and Node packages at 0.18.0, runs the middleware before Angular's rendering handler and the public catalog API, using a tripwire rule for paths like /.env and a per-process rate limit of 60 requests per minute. The company notes that crawlers hitting static files or CDN cache hits never reach the Node middleware, so collection must happen at that layer instead.", "body_md": "If a crawler downloads a page without running JavaScript, an Angular service or HTTP interceptor will never see it. The request still reaches the server that delivers your site.\n\nThis walkthrough puts WebDecoy in front of Angular's server-side rendering handler. You can observe requests in monitor mode, inspect the results, and keep serving the application while you decide what needs a response.\n\nWe build WebDecoy. This tutorial was prepared with AI assistance and uses a runnable example from our public SDK repository. It covers WebDecoy request detection, not FCaptcha.\n\nThe example uses Angular SSR with an Express server:\n\n```\nCrawler or browser → Express → WebDecoy → Angular SSR\n```\n\nThe same middleware also sees requests to the application's public catalog API. It runs before both the API route and Angular's rendering handler.\n\nA separate API server cannot observe crawlers that only request HTML from another host. If your Angular app is deployed as static files, put collection at that hosting or CDN layer. Likewise, a CDN cache hit that never reaches Node is outside this middleware's coverage.\n\nUse Node.js 26.5, which was used for these checks, or another version supported by your Angular release.\n\n```\ngit clone https://github.com/WebDecoy/node.git\ncd node\n# Revision used for this walkthrough:\ngit checkout 1a6b0700d45f4437311e359d3ab1ea91927fc0bf\ncd examples/angular-monitor/app\nnpm ci --workspaces=false\nnpm run build\nnpm test\nnpm run serve\n```\n\nThe example installs the published SDK packages independently of the repository's workspaces. Open `http://127.0.0.1:4300` and select **Load public catalog**.\n\nUse the built Node server for this check. `ng serve` is not the production server whose middleware order we are testing.\n\nThe [complete source](https://github.com/WebDecoy/node/tree/1a6b0700d45f4437311e359d3ab1ea91927fc0bf/examples/angular-monitor) includes the Angular component, server, lockfile, and integration tests. The example uses Angular 22.2 and WebDecoy's Express and Node packages at 0.18.0.\n\nIn `src/server.ts`, WebDecoy runs before the call to `angularApp.handle(req)`:\n\n``` js\nimport { webdecoy } from '@webdecoy/express';\nimport { tripwire, rateLimit } from '@webdecoy/node';\n\napp.use(webdecoy({\n  apiKey: process.env['WEBDECOY_API_KEY'] || undefined,\n  mode: 'monitor',\n  honeytoken: false,\n  rules: [\n    tripwire({ paths: ['/.env', '/wp-config.php'] }),\n    rateLimit({ max: 60, window: 60 }),\n  ],\n}));\n```\n\nThis is the middleware block to integrate into your existing Express server, not a complete replacement for `server.ts`. Keep your existing routes and rendering handler.\n\n`mode: 'monitor'` records a refusal without enforcing it. The tripwire paths let us test a concrete rule without needing a real crawler. The sample does not expose a real environment file or WordPress configuration file at either path.\n\nThe rate limit is intentionally low for the demonstration. It is per process, not a shared limit across a fleet. Adjust or remove it for your workload, or use a shared store if you need one limit across replicas.\n\n`honeytoken: false` turns off automatic HTML trap injection in this example. That keeps the initial setup focused on request observation without adding DOM changes during hydration.\n\nThe server serves real static assets before the middleware and excludes its health endpoint. Other requests pass through detection. The Angular server routes use:\n\n``` js\nimport { RenderMode, ServerRoute } from '@angular/ssr';\n\nexport const serverRoutes: ServerRoute[] = [\n  { path: '**', renderMode: RenderMode.Server },\n];\n```\n\nAngular distinguishes server rendering, prerendering, and client rendering. Its [hybrid-rendering documentation](https://angular.dev/guide/ssr) explains those choices. Check how your host serves each route before assuming every page request reaches this Node process.\n\nWith the server running, send these requests from another terminal:\n\n```\ncurl -i http://127.0.0.1:4300/api/products\ncurl -i http://127.0.0.1:4300/.env\n```\n\nThe catalog should return HTTP 200 with two demo items. The tripwire request should produce a terminal record with `conclusion: \"DENY\"`, `tripwire: true`, and `wouldBlock: true`.\n\nIts HTTP response stays 404 because the application defines that response and monitor mode lets the request continue. A refusal in the decision log is not the same thing as a blocked request.\n\nThe example logs limited structured decisions. It does not log visitor IPs, full headers, request bodies, or credentials.\n\nFor cloud reporting, set your own `WEBDECOY_API_KEY` in the server environment and restart the server. Create the key in your [WebDecoy account](https://app.webdecoy.com/).\n\nKeep the key out of Angular configuration, components, and HTTP interceptors. Those can ship to the browser.\n\nThen send the reserved test request:\n\n```\ncurl -i -A 'WebDecoy-Test/1.0' http://127.0.0.1:4300/api/products\n```\n\nLook for the labeled **Test** record in Detections. This checks the reporting pipeline; it does not represent a real AI crawler visit.\n\nWithout a key, the example explicitly logs `dashboardConfigured: false` and `reportingError: true` for this test. A configured key alone is not proof of delivery either. Check the dashboard record and any reporting errors.\n\nThe production build and five integration tests passed locally. Those checks cover server-rendered HTML, catalog availability, the tripwire decision, the reserved test trigger, and rate-limit observation while responses continue.\n\nNo API key was used in those tests, so they do not verify cloud delivery or detection accuracy against real AI crawlers.\n\nStart with the requested paths, supporting signals, timing, and crawler classification. Keep a client's claimed user-agent identity separate from any verified identity. A familiar crawler name can be copied by another client.\n\nBefore deploying behind a proxy, configure trusted proxies for your actual network path. The local example trusts no forwarding headers and binds only to loopback. Incorrect proxy settings can attribute many visitors to the same address.\n\nFinally, detections are not a complete access log. Requests may be allowed locally without being reported, and caching can affect reporting frequency. Keep access logs for total request counts. Monitor mode gives you a way to investigate before changing how the site responds.", "url": "https://wpnews.pro/news/see-which-bots-and-ai-crawlers-are-visiting-your-angular-site", "canonical_source": "https://dev.to/webdecoy/see-which-bots-and-ai-crawlers-are-visiting-your-angular-site-43kg", "published_at": "2026-09-29 19:07:56+00:00", "updated_at": "2026-09-29 19:16:59.156204+00:00", "lang": "en", "topics": ["ai-crawlers", "ai-tools", "developer-tools"], "entities": ["WebDecoy", "Angular", "Express", "Node.js", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/see-which-bots-and-ai-crawlers-are-visiting-your-angular-site", "markdown": "https://wpnews.pro/news/see-which-bots-and-ai-crawlers-are-visiting-your-angular-site.md", "text": "https://wpnews.pro/news/see-which-bots-and-ai-crawlers-are-visiting-your-angular-site.txt", "jsonld": "https://wpnews.pro/news/see-which-bots-and-ai-crawlers-are-visiting-your-angular-site.jsonld"}}