Originally published on tamiz.pro.
The serverless AI dream is finally becoming locally scalable — but the cost isn't negligible.
By early 2025, the pendulum had swung back hard against cloud-dependent AI. Every major outage, every data-leak headline, and every per-token pricing surprise pushed a growing cohort of engineers toward a simpler idea: run everything on your own machine. Products like ScreenPipe (by Agnostiq) and Headroom have emerged not as niche experiments but as proof that local-first AI agents can deliver real, production-grade capability — if you're willing to make tough architectural tradeoffs. This article breaks down what those products taught us, where the field is heading, and what the privacy-vs-performance equation actually looks like when you ship something that works.
Local-first AI wasn't born in 2025, but several converging forces made it the dominant design philosophy for agent builders this year:
The result: a wave of agents that observe, reason, and act without ever leaving the user's machine. And two products stood out as the clearest case studies.
ScreenPipe's core insight was deceptively simple: an AI agent needs persistent, multimodal context about the user's digital life, and the most reliable way to capture that context is through local screen recording and OCR.
ScreenPipe runs a continuous local loop:
┌─────────────────────────────────────────────┐
│ ScreenPipe Agent │
│ │
│ ┌──────────┐ ┌───────────┐ ┌──────────┐ │
│ │ Recorder │→ │ OCR / │→ │ Embedding │ │
│ │ (screen) │ │ Audio │ │ Model │ │
│ └──────────┘ └───────────┘ └────┬─────┘ │
│ │ │
│ ┌──────────┐ ┌───────────┐ ┌────▼─────┐ │
│ │ Agent │← │ Context │← │ Vector │ │
│ │ (local │ │ Retrieval│ │ DB │ │
│ │ LLM) │ └───────────┘ └──────────┘ │
│ └──────────┘ │
└─────────────────────────────────────────────┘
Every frame captured on screen is processed locally through OCR (typically using faster models like YOLO or custom Tesseract pipelines), audio from the mic is transcribed via Whisper.cpp, and the resulting text is embedded and stored in a local vector store (usually Chroma or Qdrant in WAL mode). The LLM query then retrieves the most relevant context before generating a response — all without a single network call to a third-party service.
Opt-out-by-design privacy: ScreenPipe stores everything locally by default. There's no telemetry by default, no cloud sync. The UI surfaces what data is being captured so the user can audit it. This transparency is what turns a creepy "always-on recorder" into a privacy-preserving tool — because you see exactly what's being captured and can delete it in one click.
Streaming context, not batch dumps: Instead of up hours of footage to a server, ScreenPipe continuously embeds short context windows (typically 5–15 second clips) and indexes them. This keeps the vector DB manageable (thousands to low millions of entries) and means retrieval is fast enough for real-time agent responses.
Multi-modal fusion: The system doesn't just do OCR — it combines screen text, system clipboard events, audio transcription, and even file metadata into a unified context retrieval pipeline. The agent can answer questions like "What was I looking at when I was researching that API?" by fusing screen captures with process logs.
ScreenPipe's biggest engineering challenge isn't building the pipeline — it's managing the compute cost of running it 24/7. A typical setup on an M2 Max with 64 GB RAM might consume 15–25 W just for continuous screen capture, OCR, and embedding generation. On CPU-only machines, inference for even small OCR models can bottleneck the entire pipeline.
The team's compromise: smart sampling. Instead of processing every frame, ScreenPipe detects screen changes and only processes frames where something actually changed. Combined with lower-resolution thumbnails for initial change detection, this reduces compute by ~70% with negligible quality loss for retrieval purposes.
Headroom takes a different but complementary approach. Rather than continuous screen recording, it focuses on conversational context retention with complete data sovereignty.
Headroom's design prioritizes three constraints simultaneously:
User Input
│
▼
┌──────────────┐
│ LLM (local) │ ← Ollama / llama.cpp
└──────┬───────┘
│
▼
┌──────────────┐
│ Structured │ ← Fact extraction & entity linking
│ Memory Store │
└──────┬───────┘
│
▼
┌──────────────┐
│ Retrieval │ ← Hybrid: keyword + semantic
│ Engine │
└──────┬───────┘
│
▼
┌──────────────┐
│ Response │ ← Synthesized with full context
│ (local) │
└──────────────┘
Most local AI apps treat the LLM as a stateless black box — you send a prompt with retrieved context and get a response. Headroom introduces structured memory parsing:
async def parse_and_store(conversation_turn: str):
facts = await local_ner_model.extract(conversation_turn)
structured = {
"entities": facts.entities,
"relations": facts.relations,
"timestamp": now(),
"embedding": await embed(conversation_turn)
}
await vector_db.upsert(structured.embedding, structured)
await graph_db.merge_entities(facts.entities, facts.relations)
This means Headroom doesn't just remember what you said — it understands what you meant, enabling questions like "What did I say last week about the API latency issue?" with high recall, because the retrieval uses both semantic embeddings and an entity graph built from your conversation history.
Headroom's structured memory approach has a storage tax. A typical user with a year of daily conversations might accumulate 50,000–200,000 structured entries. On a local device, this means:
ef_construction
values to keep query times under 50 ms even at scale.Both ScreenPipe and Headroom embody the same fundamental tension that defines the entire local-first AI movement. Here's what the engineering data shows:
| Dimension | Cloud-First | Local-First |
|---|---|---|
| Latency | ||
| 200–800 ms (network round-trip) | 50–500 ms (GPU) / 500 ms–10 s (CPU) | |
| Cost at scale | ||
| $0.01–$0.10 per complex query | Near-zero marginal cost; high upfront hardware | |
| Privacy | ||
| Data leaves your machine | Full data sovereignty | |
| Availability | ||
| Depends on provider uptime | Always-on (as long as the machine is on) | |
| Scalability | ||
| Unlimited (abuse limits apply) | Hard-limited by hardware | |
| Model freshness | ||
| Instant access to newest models | Manual update cycle | |
| Multimodal | ||
| Native (vision, audio APIs) | Requires self-hosted pipelines | |
| Customization | ||
| Limited to provider APIs | Full model control, fine-tuning possible |
The critical insight from both projects is that local-first works when the task profile matches the hardware profile:
Engineers who skip this analysis and try to run a ScreenPipe-style agent on a $300 laptop discover quickly that "local" doesn't mean "free" — it means "you pay in battery life, heat, and noise instead of in dollars."
If you're considering building a local-first AI agent in 2025, here are the distilled lessons from these projects:
Both ScreenPipe and Headroom made their biggest architectural decisions around what data moves where before choosing a single LLM. Define your data boundaries first:
A hybrid approach — local for sensitive data, cloud for non-sensitive enrichment — often gives the best user experience. But decide deliberately, don't let it be accidental.
ScreenPipe's use case (real-time assistance) demands the fastest path; Headroom's (journaling/conversation) comfortably fits the middle tier.
Local hardware is heterogeneous and unreliable. Your agent should degrade gracefully:
async def build_agent():
try:
return LocalAgent(model="llama-3.1-8b-instruct", device="gpu")
except DeviceNotFoundError:
pass
try:
return LocalAgent(model="llama-3.1-8b-instruct-q4", device="cpu")
except OOMError:
pass
return CloudAgent(provider="openai", consent_required=True)
Don't assume every user has an M-series Mac or an RTX 4090. The agent that fails silently on unsupported hardware loses trust faster than any privacy violation.
Both projects treat their vector stores as first-class infrastructure. Don't neglect this:
This is the lesson that separates hobby projects from production-ready local agents. Track and report:
The momentum behind local-first AI isn't slowing. Three trends are worth watching:
Edge deployment tools are maturing: Projects like Ollama, llama.cpp, and MLX are making local inference feel almost like a cloud API. The gap between "runs on my machine" and "runs reliably in production" is narrowing fast.
Smaller models are getting shockingly good: Llama 3.2 1B/3B, Phi-3.5 Mini, and Qwen 2.5 1.5B are competitive on narrow tasks. An agent doesn't always need a 70B model — sometimes a well-prompted 3B model with good retrieval beats a 70B model with poor context. This changes the hardware equation dramatically.
Privacy regulations are pushing adoption: GDPR enforcement, EU AI Act compliance requirements, and enterprise data residency mandates are making cloud-first architectures expensive in ways that go beyond token costs. Local-first isn't just a developer preference anymore — it's becoming a compliance strategy.
The local-first AI agent movement of 2025 isn't about nostalgia for offline computing. It's a pragmatic response to real constraints: unpredictable API pricing, genuine privacy concerns, and the observation that most agent workloads don't actually need the world's largest models.
ScreenPipe and Headroom show that the architecture is solvable — the question isn't whether local-first can work, but whether your use case justifies the hardware investment. For personal productivity tools, journaling assistants, and privacy-sensitive applications, the answer is increasingly "yes." For anything requiring real-time, multimodal, high-fidelity reasoning at scale, the cloud still has a role to play — and the smartest agents will know when to bridge both worlds.
The engineers who win in this space won't be the ones who choose local or cloud. They'll be the ones who architect for both, making the switch between them seamless and intentional. That's the real lesson from 2025's local-first wave.
For deeper insights on the evolving landscape of local-first AI tools and development patterns, check out Tamiz's Insights.
Q: Can I run a local-first AI agent on a MacBook Air with 8 GB of RAM?
A: It's possible for lightweight use cases — a 3B–7B quantized model via Ollama can run on 8 GB, but you'll need to close other applications and expect slower response times (2–5 seconds per query). For continuous multimodal agents like ScreenPipe, 16 GB is the practical minimum. Consider using an external SSD for your vector store to free up RAM.
Q: How do I handle model updates without losing my local data?
A: Keep your data stores (vector databases, structured memory) completely separate from your model files. Ollama stores models in ~/.ollama/models
and your application data should live elsewhere. When updating a model, only replace the model files — your data persists. Regular backups of your vector store directory are still recommended.
Q: What's the best local LLM for agent use cases in 2025?
A: For most agent workloads, Llama 3.1 8B Instruct (via Ollama) offers the best balance of capability, speed, and hardware compatibility. If you need stronger reasoning and have the hardware, Qwen 2.5 14B or Llama 3.1 70B (quantized to 4-bit) are worth considering. For ultra-low-resource environments, Phi-3.5 Mini or Qwen 2.5 3B can handle simple agent tasks competently.