kreuzcrawl began as a Rust core with bindings for ten languages. v0.3.0 ships fourteen, adds a tiered WAF-aware dispatch engine, cuts peak streaming memory from ~2.5 GB to ~20 MB, and enables SSRF defense across every outbound call path by default. It is the first release we consider API-stable.
This post covers what changed, why each decision was made, and what the harder engineering problems looked like from the inside.
| Area | v0.2.0 | v0.3.0 |
|---|---|---|
| Language bindings | 10 | 14 (+Dart, Kotlin/Android, Swift, Zig) |
| Peak streaming memory | ~2.5 GB | ~20 MB |
| SSRF protection | opt-in | on by default |
| Dispatch model | static HTTP / bypass / browser | tiered, signal-driven escalation |
| WAF fingerprints | — | 35 across 8 vendors |
| Fingerprint hot-reload | — | lock-free (ArcSwap ), 500 ms debounce |
| MCP tools | partial | 1:1 with CLI, safety-annotated |
| CLI subcommands | scrape, crawl | + batch-scrape, batch-crawl, download, citations |
| Robots / sitemap parsers | engine-internal | public modules |
| API stability | preview | stable |
v0.2.0 shipped Rust, Python, Node.js, Ruby, Go, Java, C#, PHP, Elixir, and WebAssembly.
v0.3.0 adds Dart, Kotlin/Android, Swift, and Zig — bringing the total to fourteen.
None of the per-language glue is written by hand. Every binding is generated from the Rust core by alef, our polyglot binding generator.
The Dart and Kotlin/Android packages bind through the C FFI layer (kreuzcrawl-ffi
) via dart:ffi
and JNI respectively. Swift binds through clang. Zig uses @cImport
against the same C header.
The generation pipeline also hardened in this release: the Docker publish matrix now builds each architecture natively rather than via QEMU emulation, the Dart build no longer requires the Flutter SDK for pub.dev publishes, Swift artifactbundle checksums are injected automatically, and the Elixir/PHP/Ruby releases preserve their lock files through the source-publish step.
=== "Python"
sh pip install kreuzcrawl === "Node.js"
sh npm install @xberg/kreuzcrawl === "Rust"
sh cargo add kreuzcrawl === "Go"
sh go get github.com/xberg-io/kreuzcrawl/packages/go === "Java"
xml <dependency> <groupId>io.xberg.kreuzcrawl</groupId> <artifactId>kreuzcrawl</artifactId> <version>0.3.0</version> </dependency> === "Kotlin (Android)"
groovy implementation("io.xberg.kreuzcrawl.android:kreuzcrawl-android:0.3.0") === "C#"
sh dotnet add package Kreuzcrawl === "Ruby"
sh gem install kreuzcrawl === "PHP"
sh composer require xberg-io/kreuzcrawl === "Elixir"
elixir {:kreuzcrawl, "~> 0.3"} === "Dart"
sh dart pub add kreuzcrawl === "Swift"
swift // Package.swift .package(url: "https://github.com/xberg-io/kreuzcrawl", from: "0.3.0") === "Zig"
sh zig fetch --save https://github.com/xberg-io/kreuzcrawl/archive/v0.3.0.tar.gz === "WebAssembly"
sh
npm install @xberg/kreuzcrawl-wasm
crawl_stream()
and batch_crawl_stream()
previously accumulated every page result in memory before the caller received any of them. On a large crawl — tens of thousands of pages, each carrying extracted text, metadata, links, and images — the peak working set reached approximately 2.5 GB.
The fix is a change in ownership: each page result is moved into CrawlEvent::Page
and emitted immediately. The caller receives it, processes it, and drops it. The engine never holds more than the current in-flight pages, bounded by the concurrency setting.
// The event type (unchanged externally; behavior changed internally)
pub enum CrawlEvent {
Page { result: Box<CrawlPageResult> }, // (1)
Error { url: String, error: String },
Complete { pages_crawled: usize },
}
CrawlPageResult
is boxed, moved into the variant, and dropped when the caller's loop moves past it. The engine holds no reference after the send.
from kreuzcrawl import crawl_stream
async for event in crawl_stream(engine, "https://example.com"):
if event.type == "page":
process(event) # event is dropped after this scope
Peak working set on a 10,000-page crawl with default concurrency (16): ~20 MB.
The non-streaming crawl()
is unchanged — it accumulates by contract, because callers need the complete CrawlResult
. The two code paths are kept separate. Merging them would push the accumulation pattern onto callers, which is the same problem moved one level up.
!!! tip "Choosing between crawl()
and crawl_stream()
"
Use crawl()
when you need the full result set in memory. Use crawl_stream()
for
large crawls, progress tracking, or when you process results one at a time. The memory
difference is significant at scale.
Web crawlers take URLs as input and make HTTP requests — the exact primitive an attacker needs to reach internal services. Every path that accepts a URL now validates it against an SsrfPolicy
before making the request: scrape()
, crawl()
, batch_crawl()
, sitemap fetches, robots.txt fetches, asset downloads, and link enqueue.
| Category | Ranges |
|---|---|
| Loopback | |
127.0.0.0/8 , ::1/128 |
|
| Private (RFC 1918) | |
10.0.0.0/8 , 172.16.0.0/12 , 192.168.0.0/16 |
|
| Link-local / cloud metadata | |
169.254.0.0/16 (incl. 169.254.169.254 ), fe80::/10 |
|
| Unspecified | 0.0.0.0/8 |
| Multicast | |
224.0.0.0/4 , ff00::/8 |
|
| IPv6 unique-local | fc00::/7 |
| Non-http(s) schemes | |
file:// , ftp:// , gopher:// , … |
Checking the hostname at validation time is insufficient. An attacker can register
evil.example.com
, serve a public IP at validation, then update DNS to point to
192.168.1.1
once the check passes.
The policy resolves every hostname via DNS and validates all returned IP addresses. If any resolved IP is in the deny list, the request is refused — regardless of what the others resolve to.
// From kreuzcrawl/src/net/ssrf.rs
let addresses: Vec<IpAddr> = tokio::net::lookup_host(&lookup_addr).await?
.map(|addr| addr.ip())
.collect();
for ip in &addresses {
if !is_ip_permitted(*ip, policy) {
return Err(SsrfError::DeniedByPolicy {
reason: classify_private_ip(*ip),
});
}
}
Each 30x
Location
header is re-resolved and re-validated before the next hop is taken. This closes the redirect-chain attack: a public URL that redirects to http://169.254.169.254/latest/meta-data/
is refused at the second hop. Redirect following is bounded by SsrfPolicy::max_redirects
(default: 5).
export KREUZCRAWL_ALLOW_PRIVATE_NETWORK=1
// Per-config builder — applies to a single CrawlConfig
CrawlConfig::builder()
.allow_private_networks(true)
.ssrf_allowlist_host(HostMatcher::Cidr("10.0.0.0/8".into()))
.build()
!!! warning "Wasm targets"
On wasm32
, SSRF checking is disabled — the browser's fetch API and same-origin
policy are the enforcing boundary, and tokio::net::lookup_host
is unavailable in
that context.
Before v0.3.0, the dispatch decision was static: HTTP, or bypass-vendor, or browser — chosen at config time and fixed for the duration of the crawl. This had an obvious cost problem: routing every request through a bypass provider because 5% of pages are blocked is expensive.
The new engine chains tiers and escalates based on per-attempt signals.
pub enum Tier {
Http, // plain HTTP fetch
Bypass, // vendor-managed bypass (Zyte, ScrapingBee, Bright Data, …)
Browser, // headless Chrome via Chromiumoxide
}
pub enum EscalationStrategy {
None, // HTTP only; surface all failures
BrowserOnly, // HTTP → Browser on block ← default
BypassFirst, // always use bypass (legacy behaviour)
BypassOnly, // HTTP → Bypass on block; no browser
BypassThenBrowser, // HTTP → Bypass → Browser; maximum resilience
}
All dispatch enums are #[non_exhaustive]
— new variants can be added without breaking downstream match
arms.
Detecting a WAF challenge page requires inspecting both response headers and body.
A naïve approach — one regex per fingerprint per response — scales as O(fingerprints × body_length). With 35 fingerprints that's expensive per page.
All body-pattern signals across all fingerprints are compiled into a single Aho-Corasick automaton at startup. One scan of the response body returns the set of matching pattern indices; each maps to a fingerprint via a flat Vec<usize>
.
pub struct Rules {
fingerprints: Vec<Fingerprint>,
automaton: AhoCorasick, // single automaton over all patterns
pattern_to_fp: Vec<usize>, // AC pattern index → fingerprint index
}
The body scan is capped at 100 KB (CHALLENGE_BODY_LIMIT
). WAF challenge pages are small; real content pages overwhelmingly exceed this threshold. This bounds scan cost without missing signals.
Header signals are checked first (constant time per fingerprint). If a fingerprint fires on headers alone, the body scan is skipped entirely.
Current corpus: 35 fingerprints across Cloudflare (10), DataDome (6), PerimeterX (5), Imperva (5), AWS WAF (4), F5 (2), Akamai (1), and generic corroborating patterns (2).
The fingerprint corpus is a TOML file (rules/waf_fingerprints.toml
). In Kubernetes deployments, it is managed as a ConfigMap — operators update signatures without restarting the process.
The compiled Rules
is wrapped in arc_swap::ArcSwap
. TomlClassifier::watch()
starts a filesystem watcher that atomically swaps the rule set when the file changes:
pub struct TomlClassifier {
rules: ArcSwap<Rules>,
}
impl TomlClassifier {
pub fn watch(self: &Arc<Self>, path: impl AsRef<Path>) -> Result<WatchHandle, WatchError> {
watch::start_watch(Arc::clone(self), path.as_ref())
}
}
Events are debounced 500 ms — this handles both editors that write via tmpfile+rename and the Kubernetes ConfigMap atomic projection mechanism, which produces the same sequence of filesystem events.
The engine tracks a block rate per domain using an Exponentially Weighted Moving Average. High block rates promote the starting tier: a domain that has been blocking consistently starts at Bypass
or Browser
rather than always attempting Http
first.
The DomainStatePort
trait is injectable:
#[async_trait]
pub trait DomainStatePort: Send + Sync + fmt::Debug {
async fn recommend(&self, domain: &str) -> DomainRecommendation;
async fn observe(&self, domain: &str, observation: &DomainObservation);
}
The default implementation (EwmaDomainState
) is wired in automatically.
kreuzberg-cloud replaces it with a distributed store for cross-instance domain intelligence.
use std::sync::Arc;
use kreuzcrawl::{
CrawlConfig, DispatchProfile, EscalationStrategy,
SimpleRetryPolicy, TomlClassifier,
};
let config = CrawlConfig::builder()
.dispatch(
DispatchProfile::builder()
.strategy(EscalationStrategy::BypassThenBrowser)
.retry_policy(Arc::new(SimpleRetryPolicy::new().with_max_retries(3)))
.waf_classifier(Arc::new(TomlClassifier::builtin()))
.build(),
)
.build();
The MCP server now exposes tools 1:1 with the CLI — scrape
, batch_scrape
, batch_crawl
, download
, and generate_citations
. Earlier releases had partial coverage; v0.3.0 closes the gap.
Each tool declares three safety properties from the MCP spec:
| Property | Value | Meaning |
|---|---|---|
read_only |
||
true |
||
| does not modify external state | ||
destructive |
||
false |
||
| does not delete or overwrite anything | ||
open_world |
||
true |
||
| makes network requests to caller-specified URLs |
open_world: true
is the meaningful one. MCP hosts can use it to apply additional
sandboxing or prompt for confirmation before an agent makes outbound requests. The SSRF
policy is the enforcement layer: a request to http://169.254.169.254/
returns a
SsrfPolicyViolation
error before any network activity occurs.
The server runs in two modes depending on how it is invoked:
/mcp
— used for service deployments. Enabled when the binary
is built with --features api,mcp
.
kreuzcrawl mcp
kreuzcrawl serve # exposes /mcp alongside the REST API
Four subcommands complete the CLI's 1:1 mapping with the core and MCP surfaces:
| Command | Description |
|---|---|
batch-scrape <urls…> |
|
| Scrape multiple URLs concurrently, emit structured JSON | |
batch-crawl <urls…> |
|
| Crawl from multiple seed URLs with shared concurrency budget | |
download <url> |
|
| Fetch and save assets to disk (PDF, DOCX, images, …) | |
citations <url> |
|
| Extract structured citations and references from a page | |
version |
|
| Print version and build metadata |
kreuzcrawl batch-crawl \
https://docs.example.com \
https://blog.example.com \
--depth 3 \
--format markdown
kreuzcrawl::robots
and kreuzcrawl::sitemap
are now public modules, usable without
constructing a crawl engine:
use kreuzcrawl::robots::{parse_robots_txt, is_path_allowed};
use kreuzcrawl::sitemap::{parse_sitemap_xml, parse_sitemap_index};
// Standalone robots.txt check — both functions are infallible
let rules = parse_robots_txt(robots_body, "Googlebot");
let allowed = is_path_allowed("/private/", &rules);
// Standalone sitemap parse — infallible
let urls = parse_sitemap_xml(sitemap_body);
// Sitemap index (points to child sitemaps)
let index = parse_sitemap_index(index_body);
This is useful for compliance tooling, link-graph builders, and crawl planners that need to evaluate robots.txt
access rules or enumerate URLs from a sitemap without running a full crawl.
BrowserPool
, BrowserPoolConfig
, NativeBrowserExecutor
, and
NativeBrowserExecutorConfig
are now public. Callers that run many crawls against the
same targets can construct and warm a pool once and reuse it:
use kreuzcrawl::{BrowserPool, BrowserPoolConfig, CrawlEngineBuilder};
let pool = BrowserPool::new(BrowserPoolConfig::default()); // sync, returns Arc<BrowserPool>
pool.warm().await?; // pre-open Chrome tabs up to pool capacity
let engine = CrawlEngineBuilder::new(config)
.with_browser_pool(pool)
.build()
.await?;
Without pool injection, each engine creates and tears down its own Chrome instance. With pool injection, the browser process persists across crawl jobs — useful when you are running many short crawls in a tight loop.
v0.3.0 adds two OpenTelemetry counters:
| Counter | Description |
|---|---|
crawl_waf_blocks_total |
|
| Number of times a WAF fingerprint fired, labeled by vendor | |
crawl_backend_escalations_total |
|
| Number of tier escalations, labeled by source and target tier |
These are emitted unconditionally via opentelemetry::global
— no feature gate required. Consumers that do not configure an OTel exporter incur no overhead beyond the counter increment.
The WAF subsystem also gained property-based tests, cargo-fuzz
targets covering the TOML corpus and Aho-Corasick automaton, and Criterion benchmarks measuring classification throughput at scale.
This is the first release kreuzcrawl declares stable. The commitments:
kreuzcrawl
crate public surfacekreuzcrawl-ffi
) is stable at MAJOR.MINOR. Struct layouts are frozen at MAJOR.MINOR boundaries.EscalationStrategy
, EscalationReason
, Tier
, CrawlError
, NetworkErrorKind
) are #[non_exhaustive]
. New variants are non-breaking; callers outside the crate must include wildcard arms.0.3.x
targets a Rust core at 0.3.x
.The public API surface is largely additive. Two changes require attention:
** CrawlError::WafBlocked is now a struct variant.** The previous unit variant becomes
CrawlError::WafBlocked { vendor, message }
. Match arms that destructure it need updating:
// Before
CrawlError::WafBlocked => { /* handle */ }
// After
CrawlError::WafBlocked { vendor, message } => {
eprintln!("blocked by {vendor}: {message}");
}
** SimpleRetryPolicy retry count is now exact.** The previous implementation had an off-by-one:
max_retries=3
produced 2 retries. The API also changed: new()
now takes no arguments (defaults to 3 retries); use .with_max_retries(n)
to override. Update call sites that were compensating for the off-by-one or passing a count to new()
.v0.3.0 stabilises the core surface. The areas we are actively working on:
HostMatcher
in language bindings.allowlist
field on SsrfPolicy
is currently #[alef(skip)]
— the untagged-enum FFI representation is not yet finalized. Expect it in a 0.3.x patch once the tagged-enum form is decided.ProxyProvider
trait and StaticProxyProvider
are public in this release; per-request proxy selection and rotation are landing in 0.3.x.