{"slug": "the-last-mile-of-a-scraper-is-a-human-copying-a-spreadsheet", "title": "The last mile of a scraper is a human copying a spreadsheet", "summary": "A developer built an Apify Actor that uses MCP connectors to write scraped product data directly into third-party services such as Notion or Supabase, eliminating the manual step of copying data from a spreadsheet. The Actor dynamically discovers available tools from the connected service and shapes its calls accordingly, with credentials securely stored on the Apify platform. The developer documented six failed runs, highlighting issues such as a missing asyncio.run() call and version mismatches.", "body_md": "I scrape product cards from online stores for a content bank with [an Actor of my own](https://apify.com/isolovyev/ru-product-cards). The Actor finishes, the dataset fills up, and then a person exports it, checks it, and pastes it into the tool where the work actually happens. That last step has no technology in it. It is somebody's Tuesday.\n\nWhen Apify shipped [MCP connectors](https://docs.apify.com/platform/integrations/mcp-connectors), I wanted to delete that step. A connector lets an Actor call a third-party service over the [Model Context Protocol](https://modelcontextprotocol.io/) using credentials the user authorized once, on the platform. The Actor never holds the token. It talks to an Apify proxy with its own run token, and the platform injects the real credentials server-side.\n\nI built a small Actor to try it: take product cards, write them into whatever service the user connected. The full code is at [github.com/isolovyev77/apify-card-sink](https://github.com/isolovyev77/apify-card-sink). It works now. Getting there took six failed runs, and every one of them failed differently.\n\nVersions matter in this article more than usual, because one of the failures is purely a version mismatch. Mine: `apify==3.4.1`\n\n, `mcp>=1.2.0`\n\n, `httpx==0.28.1`\n\n, on the `apify/actor-python:3.12`\n\nbase image. The platform reported the run as SDK 3.4.1, client 2.5.1, Crawlee 1.9.0.\n\nOne job: take cards by value, connect to the service the user picked, find a tool that can write, write the rows, and report what happened.\n\nThe design decision worth explaining is that the Actor does not know its destination. It does not have a Notion branch and a Supabase branch. It asks the connected service which tools it exposes, picks one that can write, reads the argument schema that service published, and shapes the call to match. Connect it to a Notion workspace and it creates pages. Connect it to a database and it inserts rows.\n\nA connector is an input field with `resourceType: \"mcpConnector\"`\n\n, which the [Actor-side guide](https://docs.apify.com/platform/integrations/mcp-connectors/use-in-actors) documents in full. The `mcpServers`\n\nlist does two jobs at once: it filters which of the user's connectors show up in the picker, and it caps what the Actor may call at runtime. The proxy holds you to the declaration, so a tool you did not declare is not just discouraged, it is unreachable.\n\n```\n{\n    \"outputConnector\": {\n        \"title\": \"Destination\",\n        \"type\": \"string\",\n        \"description\": \"Where to write the cards. Pick a connector you have authorized: a Notion workspace, a Supabase project, a Slack channel. Your credentials stay on the Apify side and never reach this Actor.\",\n        \"resourceType\": \"mcpConnector\",\n        \"editor\": \"resourcePicker\",\n        \"mcpServers\": [\n            { \"url\": \"*\", \"tools\": { \"required\": [\"insert*\"] } },\n            { \"url\": \"*\", \"tools\": { \"required\": [\"create_*\"] } },\n            { \"url\": \"*\", \"tools\": { \"required\": [\"post_*\"] } }\n        ]\n    }\n}\n```\n\nIn Apify Console the field renders as a picker, and the connector the user chose is what the\n\nrun receives:\n\n*The mcpConnector field as the user sees it. No token, no endpoint, just a choice.*\n\nThat declaration is where my first real mistake lives, and I will come back to it.\n\n*Nine runs in one night. The two red ones are where the MCP client crashed before reaching Notion.*\n\n**Run one lasted three seconds and produced nothing.** Exit code 0, empty dataset, no error anywhere. I had written `async def main()`\n\nand never called it. The module imported cleanly, defined a function, and exited. On a platform that reports success by exit code, forgetting `asyncio.run(main())`\n\nlooks exactly like an Actor with nothing to do.\n\n**Run two also produced an empty dataset**, this time correctly: no connector was selected, so the Actor logged a warning and returned. The behaviour was right and the reporting was wrong. I had just written an article arguing that a caller sees the dataset and never the log, and here I was, putting the one fact the caller needed into the log. Now every refusal is a row:\n\n``` python\nasync def refuse(status, detail):\n    Actor.log.warning(detail)\n    await Actor.push_data({\"delivered\": 0, \"status\": status, \"detail\": detail})\n\nif not connector_id:\n    return await refuse(\"no_connector\", \"no connector selected: nowhere to write\")\n```\n\n**Run three crashed inside the MCP client**, before a single byte reached Notion:\n\n```\nValueError: not enough values to unpack (expected 3, got 2)\n  File \"/usr/src/app/src/main.py\", line 102, in main\n    http_client=http_client) as (read, write, _):\n```\n\nThe Python example in the Apify documentation unpacks three streams from `streamable_http_client`\n\n. The version of the MCP SDK that installed in my image yields two. Both are correct for their own version; a fixed unpack is what breaks. Index the result instead:\n\n```\nasync with streamable_http_client(f\"{proxy_url}/{connector_id}\",\n                                  http_client=http_client) as streams:\n    read, write = streams[0], streams[1]\n    async with ClientSession(read, write) as session:\n        await session.initialize()\n```\n\n**Run four connected, and the tool matcher came up empty.** My write hints were written for the names I imagined: `create_page`\n\n, `insert`\n\n, `append`\n\n. Notion exposes `notion-create-pages`\n\n. Different word, different separator. The naive repair is to match on `create`\n\n, which then also matches `notion-create-attachment`\n\nand `notion-create-file-upload`\n\n, and product cards do not belong in either. You can see the full set the connector exposes in Console:\n\n*Twenty-seven tools, every name hyphenated. This screen is also where you limit what an Actor may call.*\n\nThe list is explicit now:\n\n```\nWRITE_HINTS = (\"create-pages\", \"create_pages\", \"create-page\", \"create_page\",\n               \"insert\", \"add_row\", \"append\", \"execute_sql\",\n               \"send_message\", \"post_message\")\n```\n\nThat is also why the `mcpServers`\n\ndeclaration above is wrong in a way you will not notice until a connector fails to appear in the picker. Patterns like `create_*`\n\nnever match `notion-create-pages`\n\n. While I was learning what services actually name their tools, I widened the declaration to `[{\"url\": \"*\"}]`\n\nand let the code do the filtering. Narrow it back once you know the names you need.\n\n**Run five wrote nothing because I could not see the argument shape.** My dry run reported the tool it would use and a `null`\n\nwhere the schema should be. `getattr(tool, \"inputSchema\", None)`\n\nreturned an object that did not survive the JSON dump. The SDK hands back pydantic models, and the field is `input_schema`\n\nin a dump, not `inputSchema`\n\n:\n\n``` python\ndef describe(tool):\n    \"\"\"Argument shape of a tool, in a form that survives a JSON dump.\"\"\"\n    for attr in (\"model_dump\", \"dict\"):\n        fn = getattr(tool, attr, None)\n        if callable(fn):\n            try:\n                return fn(mode=\"json\") if attr == \"model_dump\" else fn()\n            except TypeError:\n                return fn()\n    return {\"name\": getattr(tool, \"name\", None), \"note\": \"shape unavailable\"}\n```\n\nWith that fixed, the dry run returned all 27 tools the Notion connector exposes and the full schema of the one it picked. That output is the single most useful thing this Actor produces, which is why `dryRun`\n\nis a first-class input rather than a debug flag.\n\nReading that schema killed my original design. I had assumed a write is a write: hand the tool a table name and a list of rows. Notion wants pages, each with a title property and a Markdown body. A database wants rows. A chat wants a channel and a text blob.\n\nSo the Actor stopped guessing from the tool name and started reading the published schema:\n\n``` python\ndef build_arguments(tool, payload):\n    \"\"\"Shape the call for the tool we picked.\"\"\"\n    props = schema_of(tool)\n    if \"pages\" in props:\n        pages = [{\"properties\": {\"title\": c.get(\"title\") or \"Untitled product card\"},\n                  \"content\": as_markdown(c)} for c in payload[\"rows\"]]\n        return {\"pages\": pages}, \"notion-style pages\"\n    for key in (\"rows\", \"records\", \"values\"):\n        if key in props:\n            return {key: payload[\"rows\"], \"table\": payload[\"table\"]}, \"table rows\"\n    if \"text\" in props or \"message\" in props:\n        body = \"\\n\\n\".join(\"%s - %s\" % (c.get(\"title\"), c.get(\"url\"))\n                           for c in payload[\"rows\"])\n        return {\"text\": body, \"channel\": payload[\"table\"]}, \"chat message\"\n    return payload, \"unrecognised argument shape, sending our own\"\n```\n\nThe last branch matters as much as the first three. When the shape is unfamiliar, the Actor says so in the dataset instead of sending a hopeful payload into somebody's workspace.\n\nOne detail from Notion's schema saved me a whole feature: the parent is optional. Without it, created pages land as private workspace-level pages. I had been about to build a parent-search step, and the service documentation had already answered it.\n\nRun six wrote for real. One card in, one page out:\n\n```\n{\n  \"delivered\": 1,\n  \"status\": \"ok\",\n  \"tool\": \"notion-create-pages\",\n  \"argumentShape\": \"notion-style pages\",\n  \"response\": \"{\\\"pages\\\":[{\\\"id\\\":\\\"3b3b52ac-9795-818b-b31c-fcc9852969ce\\\",\\\"properties\\\":{\\\"title\\\":\\\"Lenovo IdeaPad Slim 3 15ABR8\\\"}}]}\"\n}\n```\n\n*The whole exchange in five log lines: connect, list tools, pick one, call it, get a page id back.*\n\n*The dataset carries what happened, including which argument shape was used.*\n\nThe page appeared in the workspace with the product title as its heading and the card body in Markdown underneath.\n\n*The end of the last mile: a card that arrived without anyone exporting anything.* Total time from the Actor starting to the page existing: under ten seconds, and my Actor never saw a Notion token.\n\nI started with Supabase, because I already run one. Apify accepted the server URL, then told me the server does not support dynamic client registration and recommended registering my own OAuth application. Notion, by contrast, has managed OAuth: pick it from the dropdown, authorize, done.\n\nThis is worth checking before you design around a service. Apify provides managed OAuth for Notion and Supabase; for GitHub, Slack, Google and others you bring your own OAuth client. Supabase also accepts a personal access token through the API key method, which is the shortcut if you need that one specifically.\n\nThree layers decide what a connector-enabled Actor can do, and they compose: the scopes granted at authorization, the tool allowlist the user sets on the connector in Console, and the Actor's own `mcpServers`\n\ndeclaration. The proxy filters `tools/list`\n\ndown to what the Actor declared and rejects calls outside it.\n\nFor a published Actor this is the difference between \"give me your API key\" and \"pick a connector\". My Actor never sees a Notion token. A user who wants to be stricter can allow only page creation on their connector, and the Actor cannot get around it. Access dies with the run.\n\nI deploy Actors by pushing source files through the [Apify API](https://docs.apify.com/api/v2). That stopped working here: bodies over roughly 20 KB left my machine intact and never came back with a response. Rather than fight it, I switched the Actor to build from Git.\n\n```\nsourceType: GIT_REPO\ngitRepoUrl: https://github.com/isolovyev77/apify-card-sink#main\n```\n\nThe tiny request body sidestepped the problem entirely, and the article got a public repository as a side effect. If you are debugging an Actor through repeated deploys, this is the better default anyway: the build log tells you which commit it built.\n\n`dryRun`\n\nbefore you ship the write path. List the tools, dump the schema, write nothing. Everything else in this article was discovered by that one code path.`{\"url\": \"*\"}`\n\nwhile you learn the names, then narrow. A pattern that does not match leaves the user staring at an empty picker with nothing to click.The last mile is closed now. Cards land where the work happens, and nobody exports a spreadsheet on Tuesday.", "url": "https://wpnews.pro/news/the-last-mile-of-a-scraper-is-a-human-copying-a-spreadsheet", "canonical_source": "https://dev.to/apify/the-last-mile-of-a-scraper-is-a-human-copying-a-spreadsheet-48oo", "published_at": "2026-09-02 17:07:05+00:00", "updated_at": "2026-09-02 17:24:01.092418+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents", "ai-infrastructure"], "entities": ["Apify", "Model Context Protocol", "Notion", "Supabase", "isolovyev77/apify-card-sink"], "alternates": {"html": "https://wpnews.pro/news/the-last-mile-of-a-scraper-is-a-human-copying-a-spreadsheet", "markdown": "https://wpnews.pro/news/the-last-mile-of-a-scraper-is-a-human-copying-a-spreadsheet.md", "text": "https://wpnews.pro/news/the-last-mile-of-a-scraper-is-a-human-copying-a-spreadsheet.txt", "jsonld": "https://wpnews.pro/news/the-last-mile-of-a-scraper-is-a-human-copying-a-spreadsheet.jsonld"}}