{"slug": "teaching-cognee-to-remember-your-background-jobs-building-the-trigger-dev", "title": "Teaching Cognee to Remember Your Background Jobs: Building the Trigger.dev Connector", "summary": "A developer built a Trigger.dev connector for Cognee, the open-source memory layer for AI agents, that syncs background job history — schedules, errors, failed runs and deployments — into searchable memory via a triggerdev_source(...) function returning a dlt resource passed to cognee.remember(). The connector deliberately omits run payloads, outputs, metadata, cost data and Git remote URLs to limit sensitive data exposure, excludes volatile fields so unchanged records produce identical documents, and only deletes a schedule after a confirmed HTTP 404 rather than treating missing listing data as deletion. It also stops cleanly on HTTP 429s or revoked API keys instead of retrying, and retains historical records within a configurable lookback_days window defaulting to seven days.", "body_md": "**Pull Request:** [topoteretes/cognee-community#464](https://github.com/topoteretes/cognee-community/pull/464)\n\n**Related Issue:** [topoteretes/cognee#6197](https://github.com/topoteretes/cognee/issues/6197)\n\nWhen a background job fails at 3 a.m., finding the reason is rarely straightforward. The answer is usually scattered across schedule configurations, error logs, stack traces, and the deployment that introduced the bug.\n\nI wanted an AI agent to see all of that information in one place.\n\nThat led me to build a **Trigger.dev connector for [Cognee](https://github.com/topoteretes/cognee)**, a system that turns background job history into searchable memory.\n\nThe goal was simple: help AI agents understand what happened, identify patterns in recurring failures, and connect those failures to recent deployments.\n\nThe connector exposes a `triggerdev_source(...)` function that returns a dlt resource, which can then be passed to `cognee.remember()`.\n\nWith a single integration, it syncs a Trigger.dev environment into Cognee's memory, collecting four types of records:\n\nEach record becomes a document associated with the node set `triggerdev:task:<task id>`.\n\nThis allows an AI agent to answer questions such as:\n\n*Why does the nightly invoice job keep failing, and what changed in the latest deployment?*\n\nInstead of searching through multiple dashboards and logs, the agent can retrieve relevant information while staying within the context of a specific task.\n\nBuilding the connector was not just about retrieving API data. I also needed to make synchronization reliable, protect sensitive information, and ensure that failures would not accidentally destroy existing memory.\n\nBackground job data can contain sensitive customer information, so the connector deliberately avoids storing run payloads, outputs, or metadata.\n\nIt also excludes cost information and the deployment's Git `remoteUrl`, which could contain a token.\n\nThe API key is never included in logs or error messages, and the `api_url` must be an HTTPS origin.\n\nThe principle was straightforward: **store the information needed for debugging without unnecessarily storing sensitive data.**\n\nRepeated synchronization should not create unnecessary updates.\n\nTo achieve this, I excluded volatile fields such as next-run timestamps, changing error counts, and the duration of an active run.\n\nAs a result, an unchanged record produces an identical document. When the connector runs again without any meaningful changes, Cognee does not need to process the same information repeatedly.\n\nDuring my live test, the connector correctly skipped unchanged records.\n\nOne of the most important requirements was making sure that missing data would not automatically be interpreted as deleted data.\n\nSchedules are retrieved through a complete listing. If a schedule is missing from that listing, the connector checks it individually by ID. Only a confirmed HTTP 404 results in removing that schedule from memory.\n\nIf the listing fails or returns an incomplete result, the synchronization fails instead of deleting existing records.\n\nThis prevents temporary API failures from turning into permanent data loss.\n\nErrors, failed runs, and deployments represent historical events. They should not disappear simply because they are no longer recent.\n\nThe connector retrieves records within a configurable `lookback_days` window, which defaults to seven days. Older historical records remain in memory rather than being deleted during synchronization.\n\nThis helps preserve the context an AI agent may need to identify recurring problems over time.\n\nThe connector also needs to handle API limitations and interruptions without making things worse.\n\nWhen it encounters HTTP 429 responses, exhausts its request budget, or discovers that an API key has been revoked during execution, it stops cleanly.\n\nIt does not sleep and repeatedly retry after hitting a rate limit. Instead, the next synchronization can resume the work.\n\nThis makes the connector more predictable and avoids wasting requests.\n\nMy first implementation was wrong, and the reason it failed taught me an important lesson.\n\nIt called API endpoints that did not exist. Even worse, it treated a failed listing as an empty listing, which could have removed every stored task and schedule from memory.\n\nThe problem was not immediately obvious because my tests replaced the entire HTTP client with a fake implementation.\n\nThe tests passed, but they were testing my assumptions about the API rather than verifying that the implementation actually worked with it.\n\nI had effectively built a system that confirmed its own assumptions.\n\nI rebuilt the connector around the documented Trigger.dev management API and changed the testing strategy.\n\nThe updated tests now:\n\nI also tested the integration through `cognee.add()` to verify the behavior beyond the API client itself.\n\nThe result was **83 passing tests**, with both `ruff check` and `ruff format --check` completing successfully.\n\nAfter rebuilding the implementation and strengthening the tests, I tested it against a real Trigger.dev Cloud development environment running SDK version 4.8.1.\n\nThe environment contained two schedules and three deliberately failing runs.\n\nThe live test confirmed that the listing, filtering, and pagination parameters worked as expected. I also verified schedule deletion: after a schedule was deleted, the next synchronization rechecked it, received a 404 response, and removed it from memory.\n\nHowever, the live test also uncovered a subtle issue that was not apparent from the documentation.\n\nThe endpoint `GET /api/v3/runs/{id}` returns error information at the top level of the response alongside `attemptCount`. The `attempts` field, meanwhile, can be empty.\n\nMy original implementation relied on `attempts` to retrieve error information. Consequently, failed-run documents were being created without their errors.\n\nThe fix was to read the top-level error first and fall back to the attempts data when necessary.\n\nThis was a good example of why testing against the real API matters. A response can look perfectly reasonable in a mocked test while behaving differently in production.\n\nWhile addressing the error-handling issue, I also made two improvements.\n\nFirst, very short run durations are now displayed in milliseconds rather than being rounded to `0.0s`.\n\nSecond, stack traces are limited to a maximum of five frames from the task's own code, with shortened file paths.\n\nThe original stack traces contained mostly runtime frames and exposed my local directory paths. Filtering them made the stored error information more useful and reduced unnecessary path disclosure.\n\nDeployments were not tested against the live development environment because it did not contain any deployments.\n\nThat part of the connector is currently covered by the HTTP-level fake tests, rather than a live deployment test.\n\nI wanted to be transparent about this limitation instead of treating the entire integration as fully validated against a live environment.\n\nI built the connector with Claude Code, and the commits include a `Co-Authored-By` trailer.\n\nI reviewed the implementation and ran the live tests myself.\n\nThe biggest lesson for me was that **a passing test suite is only as reliable as the assumptions behind it**.\n\nMocking is useful, but a mock that reproduces your own incorrect understanding of an API can make a broken implementation look correct.\n\nTesting against the real service early helped me discover issues that would have otherwise remained hidden, especially around response structures, deletion behavior, and error handling.\n\nFor integrations involving external APIs, I now see realistic HTTP-level tests and live validation as complementary parts of the development process, not optional extras.\n\nTo run the connector's tests, navigate to the connector directory and execute:\n\n```\ncd packages/connector/triggerdev\nuv run --with pytest --with pytest-asyncio pytest tests/\n```\n\nThe pull request is currently open for review:\n\n[View the Trigger.dev Connector PR — cognee-community#464](https://github.com/topoteretes/cognee-community/pull/464)\n\nThe related issue is available here: [topoteretes/cognee#6197](https://github.com/topoteretes/cognee/issues/6197).\n\nThis project started with a practical question: what if an AI agent could understand the history behind a failed background job instead of looking at an isolated error message?\n\nThe connector brings schedules, failures, run details, and deployment information into a shared memory layer, making it possible to connect events that would otherwise remain scattered across different parts of a system.\n\nBuilding it also reinforced an important engineering lesson for me: correctness is not just about writing code that passes tests. It is about verifying assumptions, handling failure safely, protecting sensitive data, and checking that the implementation behaves as expected against the real service.\n\nThat is what made this project valuable to me—not just connecting Trigger.dev to Cognee, but learning how to build an integration that can be trusted with real operational data.", "url": "https://wpnews.pro/news/teaching-cognee-to-remember-your-background-jobs-building-the-trigger-dev", "canonical_source": "https://dev.to/jakkulaveerababu/teaching-cognee-to-remember-your-background-jobs-building-the-triggerdev-connector-1oi1", "published_at": "2026-10-31 14:48:16+00:00", "updated_at": "2026-10-31 14:52:39.706176+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "mlops"], "entities": ["Cognee", "Trigger.dev", "topoteretes", "dlt"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/teaching-cognee-to-remember-your-background-jobs-building-the-trigger-dev", "markdown": "https://wpnews.pro/news/teaching-cognee-to-remember-your-background-jobs-building-the-trigger-dev.md", "text": "https://wpnews.pro/news/teaching-cognee-to-remember-your-background-jobs-building-the-trigger-dev.txt", "jsonld": "https://wpnews.pro/news/teaching-cognee-to-remember-your-background-jobs-building-the-trigger-dev.jsonld"}}