{"slug": "i-migrated-a-fivetran-pipeline-to-dlthub-for-0-65", "title": "I migrated a Fivetran pipeline to dltHub for $0.65", "summary": "A developer replaced a daily Fivetran MySQL sync with a dltHub pipeline in about an hour, spending $0.65 in AI tokens using OpenCode with Minimax M3. The migration required five manual inputs, including adding an SSH key and verifying the plan, and the new pipeline triggers a Snowflake Cortex Search refresh after load.", "body_md": "[Lakehouse](https://www.sfrt.io/tag/lakehouse/)\n\n# I migrated a Fivetran pipeline to dltHub for $0.65\n\nOne hour, five manual inputs, and $0.65 of tokens... enough to replace a daily Fivetran sync with a tested dltHub job\n\nTL/DR: I replaced one daily Fivetran MySQL sync with dltHub in about an hour. OpenCode running Minimax M3 (because multimodal and still cheap 😅) consumed $0.65 in tokens. I gave it five manual inputs. One was actual infrastructure work. The other four were decisions and verification.\n\nThe old connection pulled 12 CMS backend (our Drupal uses MySQL) tables from a webserver through an SSH tunnel into Snowflake. It worked. And it was billed via monthly-active-rows 🙄 And it felt like a reasonable candidate for a small experiment: could I recreate the operational behavior with dltHub, use OpenCode as the implementation environment, and stay in a range where an hour of AI usage costs less than a coffee in Zurich?\n\nTurns out... yes. With caveats, obviously.\n\n## The baseline\n\nI first had to log in to Fivetran for the first time in months (it is a reliable platform after all) to get the connector ID. Then I used the [Fivetran MCP](https://github.com/fivetran/fivetran-mcp?ref=sfrt.io) to inspect its MySQL configuration and how it connects to the webserver through an SSH tunnel. Its selected 12 tables are all part of a Drupal CMS, and it runs daily.\n\nI did not want a generic MySQL ingestion example. I wanted the existing job's behavior: same source scope, data landing in Snowflake, incremental loads where I could justify them, and a downstream task triggered (❄️ Cortex Search refresh) after a successful load.\n\n## Five manual inputs\n\nI did not sit back and accept generated code blindly. The migration needed five interventions.\n\n- Add a new SSH key: This was the major manual task. I created a dedicated key pair and added its public key to the servive user's\n`authorized_keys`\n\nfile on the webserver. I also captured the server's host key while I was there so the new job can pin it instead of trusting whatever answers on port 22. - Decide source scope: The first profile discovered 346 tables and roughly 6.2 million rows. That was technically fun and operationally wrong. The previous Fivetran configuration had 12 enabled tables, because those are what I actually use. So I told OpenCode to match that scope exactly.\n- Verify the plan: I checked the generated design against the source schema and test suite. In particular, I wanted primary keys reflected from MySQL, a conservative cursor whitelist (for incremental loads: I didn't define the PK and cursor for each of the 12 tables individually but rather used a\n[dltHub skill](https://github.com/dlt-hub/dlthub-ai-workbench?ref=sfrt.io)to annotate sources and identify usually useful cursors), and cleanup of the SSH tunnel. - Add a downstream task: Fivetran stopped after the load. I asked OpenCode to trigger the existing Snowflake Cortex Search refresh task after dlt completes. The new pipeline now does a little more work than its predecessor 😎\n- Verify the run: After everything was implemented and tested locally, I had to point OpenCode to a modification of the pipeline necessary for it to run on dltHub (cf. below).\n\n## What dltHub generated\n\ndlt's `sql_database`\n\nsource doesn't [open SSH tunnels itself](https://dlthub.com/docs/dlt-ecosystem/verified-sources/sql_database/configuration?ref=sfrt.io#connecting-to-a-remote-database-over-ssh). Hence, my new job opens a pinned tunnel, then creates a SQLAlchemy engine through its ephemeral local port, and passes that engine to dlt.\n\n`sshtunnel`\n\nconvenience wrapper around Paramiko recommended in the dltHub docs is not maintained since 2021 and not compatible with recent Paramiko versions. Hence, I use a custom Paramiko-based local TCP forwarder `ParamikoTunnel`\n\ninstead of the `SSHTunnelForwarder`\n\nmentioned in the docs 😜\n\n``` python\ndef get_ssh_tunnel(ssh_credentials):\n    private_key = paramiko.Ed25519Key.from_private_key(\n        io.StringIO(ssh_credentials[\"private_key\"])\n    )\n    return ParamikoTunnel(\n        hostname=ssh_credentials[\"server_ip_address\"],\n        username=ssh_credentials[\"username\"],\n        private_key=private_key,\n        host_key=parse_ssh_host_key(ssh_credentials[\"host_key\"]),\n        target_address=(\"127.0.0.1\", 3306),\n    )\n```\n\nThe database has composite primary keys everywhere. I reflected MySQL metadata and only configured incremental merge for tables with a primary key plus one of three mutation-oriented columns:\n\n```\nSAFE_CURSOR_PRIORITY = (\n    \"changed\",\n    \"content_translation_changed\",\n    \"revision_timestamp\",\n)\n\ndef apply_resource_hints(table_name, resource, table):\n    pk_columns = [column.name for column in table.primary_key.columns]\n    cursor = choose_cursor(table_name, [column.name for column in table.columns])\n    if cursor is None or not pk_columns:\n        resource.apply_hints(write_disposition=\"replace\")\n        return\n\n    resource.apply_hints(\n        write_disposition=\"merge\",\n        primary_key=pk_columns,\n        incremental=dlt.sources.incremental(\n            cursor, on_cursor_value_missing=\"include\"\n        ),\n    )\n```\n\nI deliberately rejected `created`\n\n, generic `timestamp`\n\n, and revision IDs as universal cursors. Drupal puts lots of time-related integers into its schema. A plausible name is not enough evidence for incremental correctness.\n\n## Two failures worth mentioning\n\nThe first remote run failed (and there goes my 100% pipeline-run-success-rate for the week 😤) because dltHub connected to Snowflake from an IP outside the account allowlist. I wrapped the job with my existing Snowflake proxy context used by other dltHub jobs. On the source-side of the pipeline, SSH and MySQL keep using raw TCP, so they are unaffected by those HTTP proxy variables.\n\n[static egress IPs](https://dlthub.com/docs/devel/hub/pipeline-operations/job-configuration?ref=sfrt.io#static-egress-ips)now 😎 I continue using my proxy as it also is a gateway of some other connections anyway.\n\nThe first all-table run also exhausted SQLAlchemy's connection pool. I had given dlt hundreds of resources and long-lived MySQL reads. A larger pool only delayed the failure. `NullPool`\n\nfixed it by opening and closing a connection per source read:\n\n```\nreturn sa.create_engine(url, poolclass=sa.pool.NullPool)\n```\n\nOne dependency issue also appeared here: The aforementioned `sshtunnel`\n\ncalls Paramiko's removed `DSSKey`\n\nAPI, so I build a custom `ParamikoTunnel`\n\ninstead. The smoke test found that before deployment, which is exactly why I keep smoke tests around.\n\n## Proof, then schedule\n\nThe scoped baseline run completed locally in 31 seconds. The second run restored state from Snowflake and completed in 16 seconds, confirming that dlt retained its cursors. The deployed dltHub job completed in 39 seconds with zero failed jobs. And I really don't care about the number of processed rows anymore, as the only measure relevant for billing now is runtime 😜\n\nI also kept the verification close to the implementation: 25 focused tests cover SSH host-key parsing, disabled SSH-agent lookup, exact table scope, composite keys, cursor selection, cleanup, incremental state, and downstream SQL.\n\n## What I got for $0.65\n\nI got a working dltHub job, focused tests, deployment configuration, a documented secret shape, and a better downstream behavior than the old Fivetran sync. I still had to make the data and security decisions. That part should remain human work.\n\nBut I spent an hour turning five pieces of manual context into a deployed pipeline instead of typing every tunnel option, test fixture, and metadata rule myself. That is a pretty good trade for $0.65 😎\n\n## The full pipeline:\n\n``` python\n\"\"\"Load 12 enabled Fivetran tables into Snowflake over SSH.\"\"\"\n\nimport base64\nimport io\nimport logging\nimport select\nimport socket\nimport sys\nimport threading\nimport time\n\nimport dlt\nimport paramiko\nfrom dlt.sources.sql_database import sql_database\nimport sqlalchemy as sa\n\nlogger = logging.getLogger(__name__)\n\nSAFE_CURSOR_PRIORITY = (\n    \"changed\",\n    \"content_translation_changed\",\n    \"revision_timestamp\",\n)\n\nTABLE_NAMES = (\n    \"media__field_image_caption\",\n    \"media_revision__field_image_caption\",\n    \"node__body\",\n    \"node__field_shared_paragraphs\",\n    \"node_field_data\",\n    \"paragraph__field_pg_imgtxt_text\",\n    \"paragraph__field_pg_media_text\",\n    \"paragraph__field_pg_testimonial_testimonial\",\n    \"paragraph__field_pg_text_text\",\n    \"path_alias\",\n    \"taxonomy_index\",\n    \"taxonomy_term_field_data\",\n)\n\ndef parse_ssh_host_key(host_key):\n    \"\"\"Parse an OpenSSH ed25519 host-key line in memory.\"\"\"\n    key_type, encoded_key, *_ = host_key.split()\n    if key_type != \"ssh-ed25519\":\n        raise ValueError(f\"unsupported SSH host-key type: {key_type}\")\n    return paramiko.Ed25519Key(data=base64.b64decode(encoded_key, validate=True))\n\nclass ParamikoTunnel:\n    \"\"\"Forward local TCP clients to a remote TCP service over SSH.\"\"\"\n\n    MAX_WORKERS = 64\n\n    def __init__(self, hostname, username, private_key, host_key, target_address):\n        self.hostname = hostname\n        self.username = username\n        self.private_key = private_key\n        self.target_address = target_address\n        self._client = paramiko.SSHClient()\n        self._client.get_host_keys().add(hostname, host_key.get_name(), host_key)\n        self._client.set_missing_host_key_policy(paramiko.RejectPolicy())\n        self._listener = None\n        self._accept_thread = None\n        self._workers = set()\n        self._connections = set()\n        self._lock = threading.Lock()\n        self._lifecycle_lock = threading.RLock()\n        self._stop_event = threading.Event()\n        self._stopping = False\n        self._worker_slots = threading.BoundedSemaphore(self.MAX_WORKERS)\n        self.local_bind_port = None\n\n    def start(self):\n        \"\"\"Connect SSH and start a local listener on an ephemeral port.\"\"\"\n        with self._lifecycle_lock:\n            if self._listener is not None:\n                return\n\n            try:\n                self._client.connect(\n                    hostname=self.hostname,\n                    port=22,\n                    username=self.username,\n                    pkey=self.private_key,\n                    allow_agent=False,\n                    look_for_keys=False,\n                )\n            except Exception:\n                self._client.close()\n                raise\n            listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n            try:\n                listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n                listener.bind((\"127.0.0.1\", 0))\n                listener.listen()\n                listener.settimeout(0.2)\n            except Exception:\n                listener.close()\n                self._client.close()\n                raise\n\n            with self._lock:\n                self._stop_event.clear()\n                self._stopping = False\n                self._listener = listener\n                self.local_bind_port = listener.getsockname()[1]\n                self._accept_thread = threading.Thread(\n                    target=self._accept_loop,\n                    name=\"paramiko-tunnel-accept\",\n                    daemon=True,\n                )\n                self._accept_thread.start()\n\n    def _accept_loop(self):\n        listener = self._listener\n        while not self._stop_event.is_set():\n            try:\n                client_socket, client_addr = listener.accept()\n            except socket.timeout:\n                continue\n            except OSError:\n                break\n\n            with self._lock:\n                stopping = getattr(self, \"_stopping\", False)\n            if stopping or self._stop_event.is_set() or not self._worker_slots.acquire(\n                blocking=False\n            ):\n                try:\n                    client_socket.close()\n                except Exception:\n                    pass\n                continue\n\n            worker = threading.Thread(\n                target=self._forward_connection,\n                args=(client_socket, client_addr),\n                name=\"paramiko-tunnel-forward\",\n                daemon=True,\n            )\n            with self._lock:\n                if getattr(self, \"_stopping\", False) or self._stop_event.is_set():\n                    self._worker_slots.release()\n                    reject = True\n                else:\n                    reject = False\n                    self._connections.add(client_socket)\n                    self._workers.add(worker)\n                    try:\n                        worker.start()\n                    except Exception:\n                        self._connections.discard(client_socket)\n                        self._workers.discard(worker)\n                        self._worker_slots.release()\n                        try:\n                            client_socket.close()\n                        except Exception:\n                            pass\n                        logger.exception(\"Paramiko tunnel worker failed to start\")\n            if reject:\n                try:\n                    client_socket.close()\n                except Exception:\n                    pass\n                continue\n\n    def _forward_connection(self, client_socket, client_addr):\n        channel = None\n        try:\n            try:\n                channel = self._client.get_transport().open_channel(\n                    \"direct-tcpip\", self.target_address, client_addr\n                )\n                with self._lock:\n                    self._connections.add(channel)\n                self._relay(client_socket, channel, self._stop_event)\n            except Exception:\n                if not self._stop_event.is_set():\n                    logger.exception(\"Paramiko tunnel worker failed\")\n        finally:\n            for connection in (client_socket, channel):\n                if connection is not None:\n                    try:\n                        connection.close()\n                    except Exception:\n                        pass\n                    with self._lock:\n                        self._connections.discard(connection)\n            with self._lock:\n                self._workers.discard(threading.current_thread())\n            self._worker_slots.release()\n\n    @staticmethod\n    def _relay(client_socket, channel, stop_event=None):\n        sources = {client_socket, channel}\n        while sources:\n            try:\n                readable, _, _ = select.select(list(sources), [], [], 0.2)\n            except Exception:\n                if stop_event is None or not stop_event.is_set():\n                    logger.exception(\"Paramiko tunnel relay select failed\")\n                return\n            if not readable:\n                continue\n            for source in readable:\n                try:\n                    payload = source.recv(65536)\n                except Exception:\n                    if stop_event is None or not stop_event.is_set():\n                        logger.exception(\"Paramiko tunnel relay receive failed\")\n                    return\n                if not payload:\n                    destination = channel if source is client_socket else client_socket\n                    try:\n                        if destination is channel:\n                            destination.shutdown_write()\n                        else:\n                            destination.shutdown(socket.SHUT_WR)\n                    except Exception:\n                        pass\n                    sources.remove(source)\n                    continue\n                destination = channel if source is client_socket else client_socket\n                try:\n                    destination.sendall(payload)\n                except Exception:\n                    if stop_event is None or not stop_event.is_set():\n                        logger.exception(\"Paramiko tunnel relay send failed\")\n                    return\n\n    def stop(self):\n        \"\"\"Stop accepting clients and close SSH, listener, and forwarded sockets.\"\"\"\n        with self._lifecycle_lock:\n            with self._lock:\n                self._stopping = True\n                self._stop_event.set()\n                listener = self._listener\n                self._listener = None\n            if listener is not None:\n                try:\n                    listener.close()\n                except Exception:\n                    pass\n\n            with self._lock:\n                connections = list(self._connections)\n            for connection in connections:\n                try:\n                    connection.close()\n                except Exception:\n                    pass\n\n            try:\n                self._client.close()\n            finally:\n                deadline = time.monotonic() + 5\n                while True:\n                    with self._lock:\n                        connections = list(self._connections)\n                        workers = list(self._workers)\n                    for connection in connections:\n                        try:\n                            connection.close()\n                        except Exception:\n                            pass\n                    threads = [self._accept_thread, *workers]\n                    for thread in threads:\n                        if thread is not None and thread is not threading.current_thread():\n                            thread.join(timeout=0.1)\n                    if not any(thread is not None and thread.is_alive() for thread in threads):\n                        break\n                    if time.monotonic() >= deadline:\n                        break\n                self._accept_thread = None\n                self.local_bind_port = None\n\ndef get_ssh_tunnel(ssh_credentials):\n    \"\"\"Build an SSH tunnel without opening the connection.\"\"\"\n    private_key = paramiko.Ed25519Key.from_private_key(\n        io.StringIO(ssh_credentials[\"private_key\"])\n    )\n    return ParamikoTunnel(\n        hostname=ssh_credentials[\"server_ip_address\"],\n        username=ssh_credentials[\"username\"],\n        private_key=private_key,\n        host_key=parse_ssh_host_key(ssh_credentials[\"host_key\"]),\n        target_address=(\"127.0.0.1\", 3306),\n    )\n\ndef create_mysql_engine(db_credentials, local_port):\n    \"\"\"Create a MySQL engine connected to the tunnel's local endpoint.\"\"\"\n    url = sa.URL.create(\n        drivername=db_credentials.get(\"drivername\", \"mysql+pymysql\"),\n        username=db_credentials[\"username\"],\n        password=db_credentials[\"password\"],\n        host=\"127.0.0.1\",\n        port=local_port,\n        database=db_credentials.get(\"database\", \"<hard_coded_db_name>\"),\n    )\n    return sa.create_engine(url, poolclass=sa.pool.NullPool)\n\ndef create_source(engine):\n    \"\"\"Reflect the tables enabled for the conformably_fancies scope.\"\"\"\n    metadata = sa.MetaData()\n    source = sql_database(\n        engine,\n        backend=\"pyarrow\",\n        metadata=metadata,\n        table_names=list(TABLE_NAMES),\n    )\n    source.metadata = metadata\n    return source\n\ndef choose_cursor(column_names):\n    \"\"\"Choose a known-safe incremental column.\"\"\"\n    columns = {name.lower(): name for name in column_names}\n    for candidate in SAFE_CURSOR_PRIORITY:\n        if candidate in columns:\n            return columns[candidate]\n    return None\n\ndef apply_resource_hints(resource, table):\n    \"\"\"Configure merge and incremental hints from reflected table metadata.\"\"\"\n    if table is None:\n        resource.apply_hints(write_disposition=\"replace\")\n        return\n\n    pk_columns = [column.name for column in table.primary_key.columns]\n    cursor = choose_cursor([column.name for column in table.columns])\n    if cursor is None or not pk_columns:\n        resource.apply_hints(write_disposition=\"replace\")\n        return\n\n    resource.apply_hints(\n        write_disposition=\"merge\",\n        primary_key=pk_columns,\n        incremental=dlt.sources.incremental(\n            cursor, on_cursor_value_missing=\"include\"\n        ),\n    )\n\ndef cleanup(engine, tunnel):\n    \"\"\"Dispose resources without masking a load exception.\"\"\"\n    active_exception = sys.exc_info()[0] is not None\n    cleanup_error = None\n\n    try:\n        if engine is not None:\n            engine.dispose()\n    except Exception as error:\n        cleanup_error = error\n        if active_exception:\n            print(f\"warning: engine disposal failed: {error}\")\n\n    try:\n        tunnel.stop()\n    except Exception as error:\n        if cleanup_error is None:\n            cleanup_error = error\n        if active_exception:\n            print(f\"warning: tunnel stop failed: {error}\")\n\n    if cleanup_error is not None and not active_exception:\n        raise cleanup_error\n\ndef create_pipeline():\n    \"\"\"Create the production pipeline.\"\"\"\n    return dlt.pipeline(\n        pipeline_name=\"webserver_cms\",\n        destination=\"snowflake\",\n        staging=dlt.destinations.filesystem(\n            bucket_url=\"az://stage/webserver_cms\"\n        ),\n        dataset_name=\"webserver_cms\",\n    )\n\ndef trigger_downstream_tasks(pipeline):\n    \"\"\"Refresh Cortex Search after a successful Snowflake load.\"\"\"\n    with pipeline.sql_client() as client:\n        client.execute_sql(\n            \"execute task raw.webserver_cms.ta_refresh_cortex_search;\"\n        )\n        print(\n            \"Executed task raw.webserver_cms.ta_refresh_cortex_search\",\n            flush=True,\n        )\n\ndef load_webserver_cms():\n    \"\"\"Load the enabled Fivetran table scope.\"\"\"\n    ssh_credentials = dlt.secrets[\"sources.webserver_cms.ssh\"]\n    db_credentials = dlt.secrets[\"sources.webserver_cms.credentials\"]\n    tunnel = get_ssh_tunnel(ssh_credentials)\n    engine = None\n\n    try:\n        tunnel.start()\n        engine = create_mysql_engine(db_credentials, tunnel.local_bind_port)\n        source = create_source(engine)\n        metadata = getattr(source, \"metadata\", None)\n        for table_name, resource in source.resources.items():\n            print(f\"discovered table: {table_name}\")\n            table = metadata.tables.get(table_name) if metadata is not None else None\n            apply_resource_hints(resource, table)\n\n        pipeline = create_pipeline()\n        load_info = pipeline.run(source, loader_file_format=\"jsonl\")\n        print(load_info)\n        trigger_downstream_tasks(pipeline)\n        return load_info\n    finally:\n        cleanup(engine, tunnel)\n\nif __name__ == \"__main__\":\n    load_webserver_cms()\n```\n\n", "url": "https://wpnews.pro/news/i-migrated-a-fivetran-pipeline-to-dlthub-for-0-65", "canonical_source": "https://www.sfrt.io/i-migrated-a-fivetran-pipeline-to-dlthub-for-0-65/", "published_at": "2026-07-22 06:45:47+00:00", "updated_at": "2026-08-13 17:36:11.135014+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-agents"], "entities": ["Fivetran", "dltHub", "OpenCode", "Minimax M3", "Snowflake", "Drupal", "MySQL", "Paramiko"], "alternates": {"html": "https://wpnews.pro/news/i-migrated-a-fivetran-pipeline-to-dlthub-for-0-65", "markdown": "https://wpnews.pro/news/i-migrated-a-fivetran-pipeline-to-dlthub-for-0-65.md", "text": "https://wpnews.pro/news/i-migrated-a-fivetran-pipeline-to-dlthub-for-0-65.txt", "jsonld": "https://wpnews.pro/news/i-migrated-a-fivetran-pipeline-to-dlthub-for-0-65.jsonld"}}