{"slug": "activity-frames-give-your-ai-agent-eyes-on-your-day", "title": "Activity-frames – give your AI agent eyes on your day", "summary": "Nocta released activity-frames, an open-source tool that records a user's screen locally and compiles it into structured activity frames for AI agents, enabling episodic memory without cloud processing or LLM inference. The tool provides agent-ready context blocks and MCP integration, aiming to give AI assistants awareness of a user's actual workday activities.", "body_md": "- Nocta uses activity-frames to watch how you work and brief you daily on what needs your attention. 100% local.[Download the desktop app]\n\n**Episodic memory for AI agents.**\n\nYour agent can read your code, search the web, and call APIs - but it has no idea what you have been doing for the last 8 hours. It starts every conversation blind.\n\nactivity-frames gives your agent eyes. It records your screen locally, compiles what it sees into structured **activity frames** (bounded episodes of what you actually did), and serves them to any agent over MCP. No cloud, no LLM in the pipeline, no guessing.\n\n```\npip install activity-frames\naframes record      # start capturing (local, audio off by default)\naframes context     # your last 2 hours, agent-ready\n```\n\nCapture stores instants: thousands of snapshot rows a day, each one saying \"at 22:53:05, Chrome showed linkedin.com/in/...\". Useless to reason over.\n\nactivity-frames compiles those instants into episodes:\n\n```\n- id: f-0007\n  app: Google Chrome\n  site: linkedin.com\n  start: \"20:24:04\"\n  end: \"20:42:11\"\n  duration_min: 18.0\n  pages:\n    - {kind: people_search, entity: \"cto paris\", count: 2}\n    - {kind: profile, entity: najmuzzaman}\n    - {kind: company, entity: nexdotai}\n  input: {keys: 214, clicks: 31}\n  evidence: {frame_ids: \"99871..100147\"}\n```\n\nAnd into a compact context block for any system prompt:\n\n```\nUSER ACTIVITY (2026-07-04, local time; measured from screen capture, no interpretation):\ncoverage: 09:12-18:47, 342 active min, 11 apps\naway: 12:30-13:15 (45m)\n- 09:12-09:58 Cursor (46.2m): main.py - api\n- 10:01-10:44 Google Chrome/github.com (41.3m): pull_request:acme/api#412; code:acme/api\n- 20:24-20:42 Google Chrome/linkedin.com (18.0m): people_search:cto paris x2; profile:najmuzzaman; company:nexdotai\n```\n\nDrop that into a prompt and your agent knows your day. A full day compiles in under a second and costs zero tokens.\n\nAgent memory today means conversation memory: what you told the model. Episodic memory is what you actually *did* - and the hard part is representing it without lying.\n\nactivity-frames enforces a two-tier contract ([SPEC.md](/nossa-y/activity-frames/blob/main/SPEC.md)):\n\n**Tier 1, measured (this package):** everything is derivable by deterministic code from capture data. Sessions, durations, typed page entities, input volume, coverage gaps. Same input, same output, every time. There are no intent labels - code cannot know that 2 profile views + a people search was \"prospecting\". That is your agent's job; it is an LLM.**Tier 2, inferred (optional extension):** tools that add interpretation must namespace it, tag confidence (`high | medium | speculative`\n\n), and link evidence. Facts and guesses can never silently mix.\n\nEvery frame carries evidence pointers back to raw capture rows. Every document declares its blind spots. What the system did not see, it says it did not see.\n\n```\n# Claude Code\nclaude mcp add activity-frames -- aframes mcp\n```\n\nAny MCP client works: command `aframes`\n\n, args `[\"mcp\"]`\n\n. Four tools: `get_context`\n\n, `get_activity`\n\n, `get_day_summary`\n\n, `get_patterns`\n\n(repetitive-workflow detection: repeated clicks, URL loops, daily habits).\n\n``` python\nfrom activity_frames import ActivityLog\n\nlog = ActivityLog()\ndoc = log.day()                      # today, structured\ndoc = log.recent(hours=2)            # last 2 hours\nprint(log.context(hours=2))          # paste-ready context block\n```\n\n**Local only.** Capture, storage, and compilation all happen on your machine. Nothing is uploaded anywhere, ever.**Read-only compilation.** The compiler opens the capture database read-only.**Content opt-in at the output.** Compiled documents carry input*counts*by default; typed-text content appears only if you explicitly pass`--include-text`\n\n(this also gates the repeated-text pattern detector). Be clear about the boundary: the capture database itself does store what the recorder sees, locally, so protect it like any sensitive file (FileVault, permissions).**Audio off by default.**`aframes record --audio`\n\nto opt in.**No LLM in the compile path.** Compilation is plain code, so no language model, local or remote, is involved in producing memory. The capture engine does run on-device OCR to read what is on screen; that stays on your machine.**You choose what leaves**, when you paste a context block into an agent. Note that window titles and page entities originate from your screen and can contain third-party text; agents should treat them as data, not instructions.\n\n```\n capture engine          compiler (this package)         your agent\n ------------------      ---------------------------     -----------------\n screen snapshots   -->  sessionize (dwell, gaps,   -->  MCP tools /\n accessibility tree      flicker merge)                  context blocks /\n input events            entity typing (20+ sites)       JSON, YAML, md\n (local SQLite)          enrichment, patterns\n```\n\nThe default capture engine is [screenpipe](https://github.com/mediar-ai/screenpipe): `aframes record`\n\nprovisions a pinned, MIT-licensed build (v0.3.324), verifies its published sha512 before first run, and manages it for you (see [ACKNOWLEDGMENTS.md](/nossa-y/activity-frames/blob/main/ACKNOWLEDGMENTS.md)). Already running your own recorder? Point `$AFRAMES_DB`\n\nat any capture database with compatible `frames`\n\n/ `ui_events`\n\n/ `elements`\n\ntables and skip `aframes record`\n\nentirely.\n\n```\naframes record                   # start capture (--stop / --status / --audio)\naframes today                    # today's frames (YAML*)\naframes day 2026-07-03 -f json   # any day, JSON\naframes context --hours 3        # agent context block\naframes apps                     # per-app time ledger\naframes patterns --days 7        # repetitive workflow detection\naframes mcp                      # MCP stdio server\n```\n\n*YAML output uses PyYAML (`pip install \"activity-frames[yaml]\"`\n\n); without it the CLI falls back to JSON.\n\nv0.1. Developed and tested on macOS (Apple Silicon); Intel macOS and Linux x64 engine builds exist but are less exercised - reports welcome. Entity parsers cover LinkedIn, GitHub, Google (Search/Docs/Gmail/Maps/Meet/Calendar), YouTube, X, Instagram, Reddit, Luma, Partiful, Product Hunt, Vercel, Supabase, Stripe, Discord, Notion, Figma, Stack Overflow, Calendly, ChatGPT/Claude, localhost; unknown sites fall back to a generic page reference - always total, never lossy. Issues and parser PRs welcome.\n\nBuilt by [Nossa Iyamu](https://github.com/nossa-y), maker of [Nocta](https://usenocta.app). MIT.", "url": "https://wpnews.pro/news/activity-frames-give-your-ai-agent-eyes-on-your-day", "canonical_source": "https://github.com/nossa-y/activity-frames", "published_at": "2026-07-07 22:18:40+00:00", "updated_at": "2026-07-07 22:32:01.284199+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "machine-learning"], "entities": ["Nocta", "activity-frames", "MCP", "Claude Code", "Cursor", "Google Chrome", "LinkedIn", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/activity-frames-give-your-ai-agent-eyes-on-your-day", "markdown": "https://wpnews.pro/news/activity-frames-give-your-ai-agent-eyes-on-your-day.md", "text": "https://wpnews.pro/news/activity-frames-give-your-ai-agent-eyes-on-your-day.txt", "jsonld": "https://wpnews.pro/news/activity-frames-give-your-ai-agent-eyes-on-your-day.jsonld"}}