{"slug": "why-frisky", "title": "Why Frisky", "summary": "Frisky, a Rust-based reimplementation of the Dask parallel computing library, achieves 250,000-400,000 tasks per second with roughly 3 ยตs overhead, compared to Dask's ~2,600-3,000 tasks/s. The project is described by its creator as an art project and a fast version of Dask, emphasizing that most users do not need parallel computing and would be better off using a database, Polars, or Spark. Frisky generates detailed telemetry for AI agents to analyze, enabling an AI-driven development cycle.", "body_md": "# Why Frisky\n\nYou don't need Frisky.\n\nFrisky is a fast version of Dask and most people don't need Dask. Parallel computing is for when you're too lazy to solve a problem properly, so you bludgeon the problem with hardware. You're better off using a database, or Polars, or asking your LLM to craft you something custom. Heck, you could even use Spark.\n\nFrisky exists because I wanted to see how fast I could make a Dask-like thing today. Frisky is an art project. A very fast art project.\n\nBut Frisky is fun, and that's honestly the best reason I can find to use it, oh and that agents seem to enjoy using it. You're probably here for speed though, so let's talk about speed first.\n\n[Speed! Rust!](#speed-rust)[AI!](#ai)[New Dashboard!](#new-dashboard)[Speed Again! Fast Disk and Network!](#speed-again-fast-disk-and-network)[Graph Construction!](#graph-construction)[Lightweight!](#lightweight)[Moving Fast! Breaking Things!](#moving-fast-breaking-things)\n\n## Speed! Rust!\n\nIs it still cool to reimplement libraries in Rust? I hope so, because gosh-darn-it that's exactly what I did.\n\nFrisky is written in Rust (๐ŸŽ‰) Frisky's core is roughly 100x faster than the core of Dask. That's not to say that your code will be 100x faster (almost certainly not) but you'll no longer be able to blame task scheduling.\n\nThe Frisky scheduler can run around **250,000-400,000 tasks per second**, or\nwith an overhead of roughly **3 ยตs**.\n\nMeasured\n\nFrisky: an offline Rust harness on an Apple M4 drives the real scheduler state machine with simulated worker completions, achieving 2.5-4.5 ยตs/task. In our matching end-to-end benchmark, Dask ran at ~2,600-3,000 tasks/s.\n\n## AI!\n\nEven if Rust has lost its lustre, AI is surely in vogue (at least according to my LinkedIn feed), and agents love Frisky.\n\nThe biggest benefit to using Rust isn't speed, it's that we measure every detail of the calculation guilt-free, generating mountains of valuable telemetry and context. Frisky measures these mountains of telemetry and feeds them into downstream analytics, which in turn enable an AI development cycle at the heart of Frisky's progress.\n\nYou point your favorite AI agent to a Frisky dashboard and the `frisky`\n\nCLI and it pulls in all the context it wants:\n\n``` bash\n$ frisky --help # Agents read this\n$ frisky observe overview YOUR_DASHBOARD_URL # Then they do this\n\nOverview http://localhost:8787\n state workers 6 (0 idle) waiting 34 processing 1347 memory 7684 erred 0 queued 0\n โ†’ stuck? observe blocked\n\n perf wall-clock 14.0s workers 6 tasks 14683 spans 200000\n memory 0.5 GB / 17.2 GB (3% peak) spilled 0.0 GB unspilled 0.0 GB network recv 1.0 GB\n โ†’ per-worker memory observe workers prefix costs observe prefixes http://localhost:8787 in-flight transfers observe transfers\n\nComponents\n โ”‚0s 2 4 6 8 10 12 14sโ”‚ Total\n โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค\n compute โ”‚โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ƒโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ โ”‚ 275.6 s\n network โ”‚โ–โ–‚โ–‚โ–ƒโ–‚โ–„โ–‡โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–ƒโ–‚โ–…โ–„โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–ƒโ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–ˆโ–ˆโ–‡โ–ˆโ–ˆโ–‡โ–‡โ–‡โ–„โ–โ–โ–โ–โ–โ–‚โ–โ–โ–โ–โ–โ–‚โ–โ–โ–โ–โ–‚โ–โ–โ– โ”‚ 57.9 s\n scheduler โ”‚โ–โ–โ–โ–โ–‚โ–‚โ–‚โ–โ–โ–โ–โ–โ–โ–โ–โ–‚โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–‚โ–‚โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ– โ–โ–โ”‚ 3.0 s\n other โ”‚โ–‚โ–‚โ–‚โ–ƒโ–„โ–…โ–„โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–ƒโ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–ƒโ–ƒโ–„โ–ƒโ–ƒโ–‚โ–ƒโ–ƒโ–ƒโ–‚โ–‚โ–‚โ–‚โ–‚โ–ƒโ–ƒโ–ƒโ–‚โ–ƒโ–ƒโ–ƒโ–‚โ–‚โ–‚โ–ƒโ–ƒโ–ƒโ–‚โ–ƒโ–‚โ–‚โ–‚โ–‚โ”‚ 94.7 s\n โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜\nโ†’ zoom / full view: observe timeline http://localhost:8787 --view component\n\nCostliest span types โ€” over time\n โ”‚0s 2 4 6 8 10 12 13sโ”‚ Total\n โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค\nworker.exec.call โ”‚โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‡โ–ˆโ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ”‚ 268.6 s\ntcp.send.queue โ”‚โ–โ–โ–โ–โ–โ–โ–‚โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–‚โ–‚โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–‡โ–‡โ–ƒโ–ƒโ–†โ–†โ–…โ–…โ–‚โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ”‚ 17.2 s\ntcp.send.write_queue โ”‚โ–โ–โ–โ–โ–โ–โ–‚โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–‚โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–„โ–†โ–ƒโ–ƒโ–…โ–†โ–„โ–„โ–ƒโ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ”‚ 8.5 s\ntcp.recv.queue โ”‚โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–‚โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–†โ–†โ–โ–ƒโ–†โ–…โ–„โ–„โ–„โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ”‚ 8.5 s\ntcp.send.compress โ”‚โ–โ–โ–โ–โ–โ–‚โ–ƒโ–‚โ–โ–โ–โ–โ–โ–โ–โ–โ–‚โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–‚โ–†โ–†โ–„โ–„โ–…โ–„โ–ƒโ–ƒโ–‚โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ”‚ 7.7 s\ntcp.send.serialize โ”‚โ–โ–โ–โ–โ–โ–โ–‚โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–‚โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–ƒโ–…โ–ƒโ–‚โ–ƒโ–„โ–ƒโ–„โ–‚โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ”‚ 6.2 s\nworker.transfer.recv โ”‚โ–โ–โ–โ–โ–โ–โ–‚โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–‚โ–‚โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–„โ–„โ–‚โ–‚โ–„โ–„โ–„โ–…โ–ƒโ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ”‚ 5.5 s\nworker.deserialize โ”‚โ–โ–โ–โ–โ–‚โ–‚โ–ƒโ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–‚โ–„โ–…โ–ƒโ–‚โ–ƒโ–‚โ–‚โ–‚โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–‚โ–โ–โ–โ”‚ 4.4 s\n โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜\nMemory โ”‚โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ”‚ 2% peak\nNetwork โ”‚โ–โ–โ–โ–โ–โ–โ–โ–โ–„โ–„โ–„โ–„โ–„โ–„โ–„โ–ƒโ–ƒโ–ƒโ–ƒโ–ƒโ–ƒโ–ƒโ–ƒโ–ƒโ–ƒโ–ƒโ–ƒโ–ƒโ–ƒโ–ƒโ–…โ–…โ–…โ–…โ–…โ–…โ–…โ–…โ–…โ–…โ–‡โ–†โ–†โ–†โ–†โ–†โ–ˆโ–‡โ–‡โ–‡โ–‡โ–‡โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ”‚ 313.3 MB/s\n\n Costliest span types\nโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“\nโ”ƒ Name โ”ƒ Total โ”ƒ Per-wkr โ”ƒ Max wkr โ”ƒ\nโ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ\nโ”‚ worker.exec.call โ”‚ 268.6s โ”‚ 44.8s โ”‚ 45.7s โ”‚\nโ”‚ tcp.send.queue โ”‚ 17.2s โ”‚ 2.9s โ”‚ 6.0s โ”‚\nโ”‚ tcp.send.write_queue โ”‚ 8.5s โ”‚ 1.4s โ”‚ 3.2s โ”‚\nโ”‚ tcp.recv.queue โ”‚ 8.5s โ”‚ 1.4s โ”‚ 3.9s โ”‚\nโ”‚ tcp.send.compress โ”‚ 7.7s โ”‚ 1.3s โ”‚ 2.5s โ”‚\nโ”‚ tcp.send.serialize โ”‚ 6.2s โ”‚ 1.0s โ”‚ 2.1s โ”‚\nโ”‚ worker.transfer.recv โ”‚ 5.5s โ”‚ 0.9s โ”‚ 1.2s โ”‚\nโ”‚ worker.deserialize โ”‚ 4.4s โ”‚ 0.7s โ”‚ 1.0s โ”‚\nโ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜\nPer-wkr = mean over the workers that ran it; Max wkr = the busiest single worker. Max wkr โ‰ซ Per-wkr means a few workers carry the work (imbalance).\nโ†’ one op over time: observe timeline http://localhost:8787 --view detailed --prefix worker.exec.call raw: observe spans http://localhost:8787 --name worker.exec.call\n\n Workers unlike their peers\nโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“\nโ”ƒ Worker โ”ƒ Excess โ”ƒ Top deviations (vs median worker) โ”ƒ\nโ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ\nโ”‚ 127.0.0.1:58058 โ”‚ 3s โ”‚ tcp.recv.queue +3s (20ฯƒ) โ”‚\nโ”‚ 127.0.0.1:58053 โ”‚ 0s โ”‚ tcp.send.write +0s (4ฯƒ) โ”‚\nโ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜\nEach worker's top-3 most unusual span types vs the median worker; Excess = total extra seconds across them.\nโ†’ rank all: observe stragglers http://localhost:8787 one worker: observe timeline http://localhost:8787 --worker 127.0.0.1:58058\n```\n\nYou can see an example of this above and learn more at [the agents page](../ai/).\nAgents often exclaim upon seeing the results. Here are some examples taken from my session history:\n\nThis is rich.\n\nThis is a decisive win. The sparklines reveal what binary completely hidโ€ฆ\n\nThis is working beautifully\n\nExactly what I need to see the run shape\n\nIt immediately answered \"why isn't progress happening\".\n\nJackpot!\n\n-- Claude and Codex\n\nI think I'm decently smart (and obviously at least a little arrogant) which is why I was able to fit the entire Dask state machine in my head back in the day. Oh boy have I met my match. Agents are better than me at this, so I've structured Frisky to be developed by agents; giving them tons of feedback loops. I've got to say, they're doing a marvelous job. I hope that you enjoy the fruits of their labor.\n\nI also hope that you find a way to work this way too. Frisky can help you feed your agents really good context about your computations. Performance optimization is really easy with good feedback.\n\n## New Dashboard!\n\nOur mountains of telemetry also produce an entrancing dashboard (users loved Dask's dashboard, and Frisky's dashboard makes Dask's look like it was written in 2015 (it was!)).\n\nThere's nothing more fun than pumping mountains of telemetry onto a dashboard. Humans love charts, and Frisky's dashboard has more charts than even I understand. I actually had to switch to WebGL just to display things fast enough.\n\nI don't know how much you'll get from it (it needs more thought at the moment) but damn is it fun to watch. If you're viewing this on desktop (not mobile) you should see a live dashboard below. Click the buttons at the top for different pages.\n\nNote\n\nThis is a recorded loop, so the numbers repeat and live actions (opening\nlogs, drilling into a specific task) aren't wired up. To see your own\nworkload, run `frisky demo`\n\n(or any cluster) and open its dashboard.\n\n## Speed again! Fast Disk and Network!\n\nOK, so rarely is scheduling the actual bottleneck. It's more common to feel pain around four things:\n\n**Disk:** Running out of memory / writing to disk**Network:** Communication between workers**Graph Generation:** Large graph generation client-side and uploading it to the scheduler**Being Dumb:** Dumb user code\n\nLet's talk about 1-2 now, then we'll get to 3-4 later. This is about disk and sockets.\n\nI no longer worry about running out of memory. Modern disk is fast enough if used well, and Frisky is designed to keep disk and socket pipelines busy. Doing disk well is hard. You've got to do obvious things, like compression, but also lots of non-obvious things like:\n\n-\n**Pipeline** writes, don't write concurrently (even if the internet or LLMs tells you to)Instead pipeline a constant stream of data to your disk. Baby modern SSDs like the spinning disk of yore.\n\n-\n**Byteshuffle** your data before compressing (but only sometimes) **Sample blocks** to choose the right compression for a buffer**Write directly** to disk from network when you're memory bound**Account for every byte** in transit to avoid OOMing- โ€ฆ and so on\n\nMaybe you're saying \"I'm smart and know HPC. These are standard solutions.\" You're right, but they're also uncommon in modern frameworks.\n\nModern hardware is faster than I expected. My MacBook Air (the absolute cheapest model I could buy) runs 20x faster than the hardware I could get when I designed Dask. Hardware is faster than our software realizes. **It's time for our software to catch up**.\n\n## Graph Construction!\n\nOne of the most common complaints from Dask users is the delay between submitting a computation, and seeing anything happen on the dashbaord. For a large graph this can be minutes of time without any feedback. It's not always the slowest part, but it is the most anxiety inducing.\n\nIn these minutes of stillness, Dask is furiously doing the following:\n\n- Constructing millions of Python tasks to run\n- Optimizing that graph\n- Determining a memory-minimizing trajectory through that graph\n- Sending that graph to the scheduler\n- Integrating those Python tasks into scheduler state\n\nThis happens in libraries like Xarray and Dask Dataframe/Array before we get to Frisky.\n\nTo resolve this problem I've re-implemented Dask Array at\n[github.com/mrocklin/dask-array](https://github.com/mrocklin/dask-array).\nOriginally I built this for query optimization (another topic I won't get into\nhere) but, because it's my personal version of Dask Array it also moves a bit\nfaster, and so now does all of the above steps in Rust, and in a Frisky native\nway.\n\nYou can use it with Xarray too if you're on git `main`\n\nversions of Dask and\nXarray. Just do this.\n\n``` python\nfrom dask_array.xarray import register\nregister()\n```\n\nFor Legacy Dask collections (what I'm now calling everything else), Frisky\ncan't speed this up, but it can tell you what's going on. We now include\nClient activity on the Dashboard, which feels much better than staring at a\n`[*]`\n\ncell in Jupyter!\n\n## Lightweight!\n\nFrisky moves tremendous amounts of data, but it's quite compact in three ways:\n\n- The code is tight (it's not AI slop)\n- The processes are lightweight (start up is trivial)\n- The package is small (easy to depend on)\n\nDespite being AI engineered, it's about the same code complexity as the hand-crafted core of Dask.\n\nMeasured\n\nCode lines only (`cloc`\n\n, comments and blanks excluded, tests excluded),\nsplit by language. Frisky: Rust core 40,466 + Python package 6,781.\n`dask.distributed`\n\n: Python 44,177.\n\nYou can spin up an entire in-process Frisky cluster, do some work, and spin everything down all faster than you can blink an eye.\n\n``` python\nimport frisky\n\nwith frisky.LocalCluster(processes=False) as cluster:\n with cluster.get_client() as client:\n futures = client.map(lambda x: x + 1, range(10))\n results = client.gather(futures)\n```\n\nThat whole round trip - start the cluster, submit ten tasks, distribute them to\nthe workers, do work, gather results back, shut everything down, - takes about\n**6 ms**. The first cluster in a fresh process pays a one-time ~30 ms import\ncost; every one after that stays in the single-digit-millisecond range.\n\nMeasured\n\n~2.5 ms to start and stop a warm in-process cluster; running ten\n`client.map`\n\ntasks through it adds ~3 ms, for ~6 ms end to end (Apple M4).\nA human blink is ~100 ms.\n\nAnd as a dependency Frisky is trivial. Much smaller than standard libraries in the Python stack.\n\nMeasured\n\nCPython 3.12 / manylinux x86_64 wheel sizes reported by PyPI.\n\n## Moving Fast! Breaking things!\n\nAs Dask developed more users and more downstream dependencies and more companies involved it became, for me, less fun. I found that I enjoy the early-to-middle stages of software development where change is easy.\n\nChange is easy in Frisky, for both good and bad. If you're looking to base your Fortune 500 company's software stack on a distributed computing platform then Frisky is a terrible choice. If you're looking to have fun with rapidly changing distributing computing then Frisky may make sense.\n\n## But really, you don't need Frisky\n\nYou almost certainly don't need parallelism, and if you do, there are more pragmatic choices.\n\nFrisky isn't a need. Frisky is fun. I had fun building Frisky and I hope that you have fun playing with it.", "url": "https://wpnews.pro/news/why-frisky", "canonical_source": "https://getfrisky.dev/why/", "published_at": "2026-07-27 17:42:46+00:00", "updated_at": "2026-07-27 17:52:50.035910+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools"], "entities": ["Frisky", "Dask", "Rust", "Polars", "Spark", "Apple M4"], "alternates": {"html": "https://wpnews.pro/news/why-frisky", "markdown": "https://wpnews.pro/news/why-frisky.md", "text": "https://wpnews.pro/news/why-frisky.txt", "jsonld": "https://wpnews.pro/news/why-frisky.jsonld"}}