# Gandalf – a self-hosted, privacy-first access guard > Source: > Published: 2026-07-08 07:51:23+00:00 ## What is Gandalf? Gandalf is NerdVPN.de's bot protection system, a self-hosted, privacy-first access guard that blocks automated bots, scrapers, crawlers, and AI agents from accessing its services. When you visit a page protected by Gandalf, he takes a brief look at your browser and decides whether you're a legitimate visitor or a suspicious automated bot. Legitimate users on supported browsers experience only a brief, unobtrusive verification step. Suspicious or anomalous requests are subject to an additional challenge or are blocked outright. *"You cannot pass. I am a servant of the Secret Fire, wielder of the flame of Anor. The dark fire will not avail you, flame of Udûn. Go back to the Shadow. You shall not pass!"* — Gandalf, The Lord of the Rings, Book II, Chapter 5: "The Bridge of Khazad-dûm" — Gandalf, The Lord of the Rings, Book II, Chapter 5: "The Bridge of Khazad-dûm" **Having trouble?** If you are on a supported browser and still can't get through, check the [troubleshooting tips](#troubleshooting)below. If you believe you have been blocked incorrectly, please [report it](#report). ## Why Bot Protection? The modern web has a bot problem and it's getting worse. According to [Cloudflare Radar](https://radar.cloudflare.com/bots) and the [CrowdSec Threat Intelligence Network](https://app.crowdsec.net/cti), nearly 45% of all HTTP traffic on the internet today is generated by automated clients rather than human users, a share that has been growing exponentially, driven by the rise of AI-powered crawlers and scrapers. For independent and FOSS services, the picture is often far worse than the internet-wide average. Many sysadmins report bot shares of up to 95% of all incoming requests. NerdVPN.de is no exception: the share of blocked bot traffic sits at around **90%**, amounting to roughly **6 million requests per day**. The majority of this traffic is harmful: it includes scrapers harvesting content without permission, AI crawlers consuming bandwidth to feed training datasets, credential-stuffing bots probing for weak passwords, and spam harvesters collecting contact information for abuse. For small and independent operators, this is not an abstract threat. Unmitigated bot traffic means real server costs, degraded performance for legitimate users, and loss of control over your own content. Beyond automated bots and scrapers, there is a growing number of bad actors who deliberately exploit publicly accessible service instances as free infrastructure for their own applications. Rather than running their own instance or using an official API, they silently piggyback on community-operated services, consuming resources, bypassing rate limits, and degrading the experience for everyone else. This kind of abuse is especially common with alternative frontends, where bad actors route their app's traffic through public instances to avoid the costs and restrictions of the original platform's API. The standard and, in my opinion, lazy answer to this problem is to outsource it, to Cloudflare, Google reCAPTCHA, or similar third-party services. But the trade-off is significant: it means routing all visitor traffic through infrastructure you don't control, handing sensitive data to US-based corporations, and accepting their terms, their outages, their business models, and their history of misusing and selling user data to the highest bidder and every government that asks them to. Gandalf exists because that trade-off is unacceptable. Bot protection should not require surrendering your users' privacy! ##### The cost of doing nothing NerdVPN.de operates several privacy-friendly alternative frontends, including Invidious (a YouTube frontend) and Redlib (a Reddit frontend). These services proxy requests to their respective upstream platforms and those platforms actively rate-limit or block instances that generate too many requests in a short period of time. Without bot protection, unchecked automated traffic would exhaust these upstream quotas within minutes (yes, really), rendering the services effectively unusable for real users. A botblocker ensures that the available capacity is reserved for humans. ##### A necessary trade-off As a consequence of this protection, certain endpoints, including APIs, RSS feeds, and other programmatic access points, are also blocked. Additionally, link preview generation in messengers and social platforms (e.g. WhatsApp, Telegram, Signal) will not work: the automated crawlers these services use to generate previews are indistinguishable from bots and are consequently intercepted as such. This is an unfortunate but unavoidable side effect: there is no reliable way to distinguish a legitimate script or crawler from an abusive bot at scale without blocking both. Programmatic access to NerdVPN.de services is against the terms of use and will be blocked, a standard browser remains the only supported client. ##### A grim future Bot detection is getting harder. Modern bots are no longer crude scripts firing raw HTTP requests; they run full browser engines, solve JavaScript challenges, mimic human interaction patterns, and in many cases have become nearly indistinguishable from real users. The gap between a bot and a human, measured purely by browser behaviour and capabilites, is narrowing to the point where it may soon vanish entirely. At the same time, a growing number of legitimate users are making themselves look more like bots than actual bots do. Privacy-conscious users increasingly harden their browser configurations: disabling JavaScript, Web Workers, Canvas API, WebGL, WebAssembly, and similar browser features in order to reduce their fingerprint and limit tracking possibilities. These are however the same signals that bot detection relies on. A visitor who has stripped their browser down to the minimum may be completely indistinguishable from an automated client. This creates a dilemma. As bots become more sophisticated, the tests required to tell them apart from humans must become harder. Harder tests, in turn, demand more from the browser and users who have intentionally limited their browser's capabilities will find themselves caught in the middle. The uncomfortable reality is that the near future will likely require visitors to permit the full execution of JavaScript, Web Workers, Canvas API, WebGL, WebAssembly (WASM), RTCPeerConnection, and dynamic imports, the same capabilities listed in the challenge requirements below, simply to prove they are human. The alternative is to accept that bot traffic becomes uncontrollable. At the same time, a newly emerging class of countermeasures, such as web DRM or ID verification, is equally inappropriate, as it violates the privacy of visitors. Moreover, such systems have a history of been circumvented by bad actors again and again without much effort, rendering them useless for actual protection and leaving them to do nothing but pointlessly harass normal, legitimate users. Users should also expect that the era of exotic, niche, or heavily patched browsers (e.g. privacy hardened forks) passing without problems is coming to an end. As the baseline for what a "normal" browser looks like becomes more precisely defined, deviating from it (intentionally or not) will increasingly trigger harder scrutiny. A modern, standard browser used on a clean residential connection will remain the most reliable path. None of this is good news for privacy. But it is the direction the web is heading and it is better to say so plainly than to pretend otherwise. *"Yet it is not our part to master all the tides of the world, but to do what is in us for the succour of those years wherein we are set, uprooting the evil in the fields that we know, so that those who live after may have clean earth to till."* — Gandalf, The Lord of the Rings, Book V, Chapter 9: "The Last Debate" — Gandalf, The Lord of the Rings, Book V, Chapter 9: "The Last Debate" ## Privacy ###### 100% Self-Hosted Gandalf runs entirely on NerdVPN.de's own servers. It uses no external services, no cloud dependencies, every request is processed locally and no data ever leaves the infrastructure. ###### No Third-Party Sharing No data is shared with, sold to, or processed by any third party. Gandalf does not integrate any external analytics, telemetry, or data brokers. ###### Sensitive Data Obfuscation Sensitive fields such as a visitor's IP address are cryptographically obfuscated before any storage and are excluded from logging. The original value is never written to disk, not even the administrator can read it back. ###### No Tracking, No Profiling Gandalf does not build profiles, track you across sessions beyond what is strictly necessary for bot detection, or use any analytics or telemetry services. *"Even the very wise cannot see all ends."* — Gandalf, The Lord of the Rings, Book I, Chapter 2: "The Shadow of the Past" — Gandalf, The Lord of the Rings, Book I, Chapter 2: "The Shadow of the Past" ## Troubleshooting / Blocked Illegitimately? If you're using a supported browser and are blocked by Gandalf or encounter other errors, start here: - Clear your browser cache and cookies. Old/stale session data is a common cause for problems. - Disable aggressive browser extensions temporarily to rule out interference. - If you're on a VPN or Tor, the exit node might be blocked on global banlists: try switching to a different exit node or disabling it. See the [VPN / Tor](#vpn)section below. - On Apple device? Apple Private Relay may interfere with Gandalf. See the [Apple Private Relay](#relay)section below. - On mobile? Make sure background data and JavaScript are not restricted by a battery saver or data saver mode. *"Who knows? Have patience. Go where you must go, and hope!"* — Gandalf, The Lord of the Rings, Book III — Gandalf, The Lord of the Rings, Book III ## Browser Compatibility ##### Browser Requirements & Challenge Dependencies If you are using a current, modern browser on a clean residential connection, Gandalf performs its verification without requiring JavaScript or any of the browser capabilities listed below. However, if your browser fails Gandalf's preliminary checks, for example due to a flagged IP address, niche browser behaviour, or excessive browser spoofing, a harder challenge may be issued. This harder challenge may depend on one or more of the following browser capabilities: - JavaScript - Web Crypto API - Web Workers - Canvas API - WebGL - WebAssembly (WASM) - RTCPeerConnection - ECMAScript: dynamic import If one of these is disabled or blocked by your browser or extensions, you will receive an error message explaining which capability is missing and how to resolve it. ##### Minimum Required Versions The following versions are the minimum required to pass Gandalf's challenges: **Niche browsers:** Palemoon, Helium, Lynx and similar niche browsers are generally compatible. See the compatibility tables below for details. ##### Compatible Browser Extensions The following list covers commonly used browser extensions and their known compatibility with Gandalf: - JShelter - must be disabled on challenges that require Web Workers - uBlock Origin - and most standard ad blockers (AdGuard, Adblock Plus) - Decentraleyes - Privacy Possum / Privacy Badger - may block Canvas API; disable if a Canvas challenge is issued - LibRedirect - NoScript - must allow scripts for NerdVPN.de on harder challenges; blocks JavaScript, Canvas, and WebGL by default - Canvas Blocker - blocks Canvas API and WebGL; disable if a Canvas or WebGL challenge is issued - Chameleon / User-Agent Switcher - browser spoofing may trigger a harder challenge; disable when accessing NerdVPN.de ##### Desktop Browsers | Browser | Passes? | Notes | |---|---|---| | Chromium / Ungoogled Chromium / Cromite | Yes | Default settings | | Firefox | Yes | Default and hardened settings | | Safari | Yes | Apple Private Relay disabled; with relay: higher challenge. | [Details below.](#vpn)Brave recently introduced built-in fingerprint spoofing, which interferes with Gandalf and may cause legitimate users to be incorrectly flagged or blocked, or have their valid sessions invalidated. ##### Mobile Browsers | Browser | Passes? | Notes | |---|---|---| | Chrome for Android | Yes | Default settings | | Firefox for Android | Yes | Default settings | | Android WebView | Yes | Default settings | | Safari on iOS | Yes | Apple Private Relay disabled; with relay: higher challenge. | Brave recently introduced built-in fingerprint spoofing, which interferes with Gandalf and may cause legitimate users to be incorrectly flagged or blocked, or have their valid sessions invalidated. **Warning:** Opera Mini injects a MITM certificate and decrypts your traffic. Not supported and not recommended.**Unsupported niche browsers:** Browsers including but not limited to Dillo and Otter are not supported due to rendering engine incompatibilities. This limitation cannot be resolved through configuration or updates to Gandalf. Please update to or switch to a supported browser to continue. **Still failing on a supported browser?** If you encounter issues while using a supported browser, the cause is likely due to local configuration changes or the usage of an IP address flagged on global banlists, for example when using a VPN or Tor exit node. See the [Network & Client Restrictions](#additional)section for details. ## Why is Gandalf Closed Source? Gandalf is proprietary and closed source, intentionally so, that's a deliberate security decision. In my opinion, Bot protection only works as long as the attacker doesn't know exactly what they're up against. The moment defensive strategies, detection heuristics, and challenge mechanics become publicly readable, they become publicly bypassable. Bots are automated; their operators will scrape, study, and adapt, often within hours of a public release. This is not a hypothetical concern. It has already happened to other well-intentioned open-source projects. [Anubis](https://anubis.techaro.lol/), for example, saw its detection logic targeted and [circumvented](https://github.com/PixelMelt/anubis-solver-rs) shortly after its source became widely indexed, a direct consequence of transparency that benefits the attacker far more than the defender. Gandalf takes the opposite approach: security through deliberate opacity. What you can't read, you can't reverse-engineer. *"He that breaks a thing to find out what it is has left the path of wisdom."* — Gandalf, The Lord of the Rings, Book II, Chapter 2: "The Council of Elrond" — Gandalf, The Lord of the Rings, Book II, Chapter 2: "The Council of Elrond" ## Is Gandalf Available for Other Sites? Gandalf is not available to third parties, neither as source code nor as a compiled binary. There are no plans to offer it as a hosted service, a downloadable release, or a licensed product. Gandalf is currently deployed exclusively on **NerdVPN.de** and will remain so for the foreseeable future. If you encounter Gandalf on another domain, it is not affiliated with this project. *"The way is shut. It was made by those who are Dead, and the Dead keep it, until the time comes. The way is shut."* — The Lord of the Rings, Book V, Chapter 3: "The Muster of Rohan" — The Lord of the Rings, Book V, Chapter 3: "The Muster of Rohan" ## Network & Client Restrictions `curl` and standard User Agents from libraries including but not limited to Dart, Go, and Python is strictly blocked. Gandalf will not make exceptions for script-based interactions or allowlist automated clients. Please use a standard browser to access NerdVPN.de services. Gandalf evaluates the reputation of every incoming IP address against multiple data sources, including [CrowdSec](https://www.crowdsec.net/) community threat intelligence feeds. CrowdSec is an open-source, collaborative security platform that aggregates real-time threat data from a global network of participants. Gandalf's integration is read-only: it periodically pulls blocklists from CrowdSec and does not send any data back. No IP addresses, request data, or any other information is shared with CrowdSec or its network. If your IP address has been flagged on one or more of these lists, for example due to prior abuse by another user sharing the same VPN exit node, datacenter subnet, or ISP, you may receive a harder challenge or be blocked outright. This is not a permanent state: IP reputations change as threat data is updated. Switching to a different IP address or network is the most reliable workaround. If you use a VPN or Tor (specifically accessing the clearnet domain rather than the onion domain), you may end up using a VPN or Tor exit node that is heavily abused by bots and other malicious actors. Consequently, these IPs are often blacklisted on global IP banlists. This is especially true for cheap or free VPN providers. With Gandalf, an attempt is made to give valid users behind those flagged IPs a fair chance. However, if you are using a VPN or Tor and are unable to pass Gandalf, the easiest solution is to change your VPN or Tor IP address. **Tor .onion domains and I2P domains** are not routed through Gandalf at all: no challenge is issued and all requests are allowed automatically. If you are connecting from a datacenter IP (such as a VPS, root server, or cloud instance), you may encounter issues because these IP ranges are frequently utilized by automated bots due to their high bandwidth and low cost. Consequently, datacenter subnets are commonly flagged on global banlists. Similar to VPNs, Gandalf tries to accommodate legitimate server traffic, but datacenter IPs generally face higher challenges. If you are unable to pass verification while using a datacenter IP, the most reliable workaround is to route your traffic through a residential network or a different ISP. **Note:** If you operate a valid domain and have full administrative control over the DNS / Reverse DNS records for your datacenter IP, please get in touch. Verified IPs can be manually added to a trust-level list. Apple Private Relay is an internet privacy service that routes user traffic through two separate relays: an ingress relay operated by Apple and an egress relay operated by content delivery networks such as Cloudflare in order to obscure IP address and destination data. Since Apple Private Relay utilizes datacenters and Cloudflare infrastructure as an exit node, all devices employing this service receive a higher Gandalf challenge due to the reliance on this shared network infrastructure. If you cannot pass Gandalf, disable the use of Apple Private Relay in your device settings. Gandalf enforces three independent rate limiting mechanisms to protect against abuse and bruteforce attacks: - **Global request limit**: a server-wide token bucket that caps the total number of requests Gandalf processes per second, regardless of source IP. If the server is under heavy load, excess requests are delayed or rejected with HTTP 429. - **Per-IP request limit**: a per-IP token bucket that limits how many requests a single IP address can make within a short burst window. Exceeding this limit results in a temporary HTTP 429 response. - **Sliding window anti-bruteforce limit**: a precise sliding window counter that tracks how many challenge attempts an IP has made over a configurable time window. This is specifically designed to prevent automated bruteforce attacks against Gandalf's challenge system. If an IP exceeds the allowed number of attempts within the window, it is banned for an escalating duration: each subsequent violation doubles the ban length (e.g. 1 min → 2 min → 4 min → …), up to a maximum of 7 days. The ban level resets after a period of clean behaviour. In line with Gandalf's privacy-first design, IP addresses are never stored in plain text for rate limiting purposes. They are cryptographically hashed before any storage, and the original value cannot be recovered, not even by the administrator. If you have been banned due to rate limiting, the challenge screen will display a countdown timer showing the remaining ban duration. Bans expire automatically; requests to lift them manually will be ignored. AI-driven browsers (e.g., Microsoft Edge AI, Perplexity Comet) and AI assistants (e.g., Copilot, Gemini, Claude) are incompatible with NerdVPN.de services. Automated AI requests are detected, trapped in a garbage-generating tarpit, and rejected. The use of AI agents to interact with this service violates the Terms of Service. Excessive unauthorized automation will result in an automatic 7-day IP ban. Requests to lift these bans will be ignored. ## Report an Issue Gandalf is wise, but not infallible. If you believe you are being blocked incorrectly, or if you encounter unexpected errors or behaviour, please report it through one of the channels below. For all inquiries, please include your **temporary session ID** from the Gandalf challenge screen (e.g. `ena@xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx` ). Open an issue with your browser, operating system, and a short description of what happened. [Open an Issue](https://git.nerdvpn.de/NerdVPN.de/gandalf-issues) Drop a message for a quick question, feedback, or if you just want to chat. [Message on Matrix](https://matrix.to/#/@weidenwiesel:nerdvpn.de) *"Even the smallest person can change the course of the future."* — Galadriel, The Lord of the Rings: The Fellowship of the Ring (2001) — Galadriel, The Lord of the Rings: The Fellowship of the Ring (2001)