{"slug": "kcp-how-to-migrate-to-confluent-cloud-in-days-not-weeks", "title": "KCP: How to Migrate to Confluent Cloud in Days, Not Weeks", "summary": "Confluent released KCP, an open source tool that orchestrates migrations from Amazon Managed Streaming for Apache Kafka (MSK) to Confluent Cloud, letting teams complete the move in days rather than weeks. KCP discovers clusters across selected AWS regions in no more than five commands, inventories throughput, estimated costs, networking, authentication, topics, ACLs, schemas, connectors and clients, then generates Terraform scripts to provision the target Confluent Cloud environment, including AWS PrivateLink, while Confluent's Cluster Linking replicates topic data byte-for-byte with offsets preserved. Support for migrations from self-managed Kafka is coming soon.", "body_md": "Introducing Streamhouse: the open data architecture for AI | [Learn More](https://www.confluent.io/blog/aiven-confluent-redpanda-streamnative-and-ververica-form-streamhouse-working-group/)\n\nWhile Apache Kafka® is incredibly powerful, self-managing brokers, upgrades, capacity, security, and incidents can quickly distract teams from what matters most: building real-time applications and delivering business value.\n\n[Confluent Cloud](https://www.confluent.io/confluent-cloud/) can remove that [operational burden](https://www.confluent.io/learn/kafka-tradeoffs/), yet migration can still be seen as risky and tedious. That’s why we developed a simpler way for customers to migrate their Kafka workloads to Confluent Cloud using Cluster Linking for data replication and a new, open source tool called KCP. This blog post explores how KCP works and how this new approach lets you [migrate to Confluent Cloud](https://www.confluent.io/blog/migrate-hosted-kafka-to-confluent-cloud-kraft/) in days instead of weeks.\n\nKafka migrations of the past often required so much effort that many teams settled on “[good enough” Kafka platforms](https://www.confluent.io/blog/hosted-apache-kafka-vs-fully-managed/) instead of moving to something better.\n\nTeams spend weeks manually inventorying clusters, untangling access control lists (ACLs), schemas, and connectors, and handcrafting complex [infrastructure-as-code](https://www.confluent.io/learn/iac/) (IaC). Once this mountain of work is done, they still have to coordinate their risky, late‑night cutovers, for which they must stop, reconfigure, and restart every client.\n\nModern migrations don’t have to look like this. **A typical migration can be broken down into four steps: discovery and planning, infrastructure provisioning, data migration, and client migration.**\n\nConfluent offers [Cluster Linking](https://docs.confluent.io/cloud/current/multi-cloud/cluster-linking/index.html), a fully managed, byte-for-byte data replication service that preserves offsets. This eliminates the need to manage replication infrastructure and allows clients to switch over seamlessly, picking up exactly where they left off. Cluster Linking simplifies data movement from any self-managed Kafka environment or hosted Kafka service.\n\nBut migration isn’t just about data. What about connectors, ACLs, schemas, and the rest of the migration workflow? How do you make *every* step simpler? That’s where KCP comes in.\n\n**KCP is an open source tool built by Confluent to orchestrate the entire migration workflow.** KCP coordinates discovery, provisioning, and migration across clusters so you can **move to Confluent Cloud in** **days, not weeks**, without manual, error-prone steps.\n\nToday, KCP supports migrations from Amazon Managed Streaming for Apache Kafka (MSK) to Confluent Cloud. Support for migrations from self-managed Kafka is coming soon.\n\nKCP orchestrates cloud-native technologies and services under the hood, so it’s the only tool you need to interact with during migration. Here’s how it works:\n\n**Discovery and Planning:** With **no more than five commands**, KCP automatically discovers your hosted Kafka clusters across selected Amazon Web Services (AWS) regions and scans each cluster to build a detailed inventory.\nThis includes throughput, estimated costs, networking setup, authentication mechanisms, topics, ACLs, schemas, connectors, and clients, giving you a complete, data-driven view of your existing environment before migration. You can also visualize this data in the KCP user interface (UI) for easier analysis.\n\n**Provisioning Infrastructure:** Once you understand your existing setup, KCP helps map it to the equivalent Confluent Cloud architecture. It generates the required Terraform scripts to provision your Confluent Cloud environment, including networking components such as AWS PrivateLink, so your target infrastructure is ready before any data is moved. This lowers the risk and barrier to entry, letting you set up a secure, production-ready environment without needing deep networking expertise.\n\n**Data Migration:** KCP **handles Kafka topic data migrations** as well as the migration of **ACLs, schemas, and connectors**, mapping them to Confluent Cloud resources and automatically generating the Terraform scripts to deploy them. For topic data, it generates Terraform scripts that provision the necessary migration infrastructure that uses Cluster Linking to seamlessly move topic data from your existing clusters to Confluent Cloud.\n\n**Client Migration:** Coming in the first half of 2026, KCP will offer a single command to migrate clients. By leveraging Confluent Cloud Gateway, KCP enables client teams to migrate any group of clients—including their topics, offsets, consumer groups, ACLs, and more—with little to no changes. Simply change the gateway endpoint from your source cluster to your target cluster to cut over all clients simultaneously.\n\nYou don’t need to be an expert in Cluster Linking, Confluent Cloud infrastructure, or AWS networking. KCP handles all of that, letting you focus on executing the migration while it takes care of the infrastructure provisioning and orchestration behind the scenes.\n\nLet’s see KCP in action and explore how it orchestrates your migration to Confluent Cloud. We’ll use Amazon MSK as our hosted Kafka environment.\n\nStart by creating a complete, accurate view of your existing Kafka environment. **Use KCP to generate a single state file through its discovery process**, which will be used to power all remaining migration steps.\n\nTo get a full picture of your existing setup, KCP discovers clusters, topics, ACLs, connectors, schemas, and clients, ensuring that no detail is missed before you start migrating.\n\nFirst, discover your existing setup (clusters and MSK Connect connectors) in one or more regions simultaneously with the following command:\n\nThis command scans all existing clusters in the specified region(s), which is especially useful in environments with multiple teams using Kafka. It provides a high-level view of all available clusters in the different regions and produces:\n\n`kcp-state.json` – a shared state file used for reports, the UI, and migration assets. This file serves as the **single source of truth** for orchestrating the entire migration process across all steps.\n\n`cluster-credentials.yaml` – a template for connection details required for deeper Kafka scans. \n\nWith the `kcp-state.json`, you can run detailed cost reports.\n\nKCP uses the AWS Cost Explorer API and Amazon Cloudwatch APIs to generate markdown files providing a detailed view of source cluster costs, throughput, storage, and sizing—helping you define migration scope and prioritize clusters effectively.\n\nYou can also visualize this data in the KCP UI. Run the following command on the machine where KCP is installed and then upload the generated state file to the UI:\n\nNow that we have a high-level understanding of our existing clusters and costs, we can dive deeper to **discover topics, ACLs, schemas, clients, and connectors**. Using the state and credentials files generated in the discovery step, KCP can enrich your environment inventory across multiple dimensions.\n\n**Clusters:** KCP can gather additional cluster-level details, including **topics, cluster IDs, ACLs, retention policies, partition counts, and replication factors**. Use the following command to enrich your state file with this information; enter your cluster credentials here to enable KCP to perform detailed cluster discovery.\n\n**Schemas**: To discover schemas, KCP can scan your Confluent [Schema Registry](https://docs.confluent.io/cloud/current/sr/index.html) and capture all schema subjects along with their compatibility settings. It supports both unauthenticated and Basic Auth connections to the Schema Registry. Here is an example of connecting to an unauthenticated local schema registry.\n\n**Clients**: KCP can inventory all active Kafka clients in your environment, including producers and consumers. It does this by scanning Kafka broker logs stored in Amazon S3 to identify active clients. KCP parses KafkaApi TRACE lines for FETCH and PRODUCE requests, extracting client metadata such as client ID, topic, role, authentication type, and principal.\n\nRunning these three commands enriches the same `kcp-state.json` file with topics, ACLs, connectors, schemas, and active clients, giving you **a single source of truth to plan the rest of your migration**.\n\nCluster Linking makes migrations simpler by enabling data movement from any hosted Kafka cluster to any Confluent Cloud cluster, regardless of the networking method used. With the newly introduced [External Cluster Linking over PrivateLink](https://docs.confluent.io/cloud/current/multi-cloud/cluster-linking/private-networking.html#link-external-clusters-to-ccloud-over-a-private-network), you can now connect a private, self-managed Kafka or hosted cluster directly to a private Confluent Cloud cluster over AWS PrivateLink. [Virtual private cloud (VPC) peering](https://docs.confluent.io/cloud/current/networking/peering/aws-peering.html) connections, [Transit Gateway](https://docs.confluent.io/cloud/current/networking/aws-transit-gateway.html), and public cluster migrations are also supported. \n\nPreviously, migrations from private Kafka clusters to a private Confluent Cloud cluster backed by PrivateLink required creating a jump cluster and performing a two-step data transfer. **With** **external Cluster Linking over PrivateLink****, you can connect both clusters directly using** **reverse PrivateLink**. This involves deploying a PrivateLink endpoint service in front of your Kafka cluster and creating [Egress Endpoints](https://docs.confluent.io/cloud/current/connectors/networking/aws-eap-self-managed.html) in Confluent Cloud that have access to this service.\n\nEstablishing the Cluster Link, configuring PrivateLink, and creating Egress Endpoints can be tedious and complex if done manually. Luckily, **KCP automates the provisioning of all infrastructure with Terraform**, including the networking components and endpoint configuration so that you can focus on the migration itself.\n\nWith a clear picture of your existing setup, you can now use KCP to provision the target Confluent Cloud environment and supporting infrastructure. At a high level, you:\n\nProvision Confluent Cloud resources, including destination cluster(s), networking (PrivateLink), and access control\n\nStand up migration infrastructure, including bastion hosts, reverse proxies, and Cluster Linking plumbing where needed\n\n**With just two commands, KCP can generate the IaC templates for both your target Confluent Cloud environment and migration infrastructure**, allowing you to provision consistently instead of handcrafting resources. You choose the networking pattern that matches your source cluster and Confluent Cloud setup (i.e., public or PrivateLink); KCP emits the right building blocks for the supported combinations.\n\nTo generate the target infrastructure Terraform script, run the create-asset target-infra command:\n\nThis command takes the `kcp-state.json` file and the Amazon Resource Name (ARN) of the existing cluster you want to migrate and then generates Terraform scripts in the `confluent-cloud-infrastructure` directory. **These scripts provision a Confluent Cloud environment and an Enterprise cluster, forming the foundation of your target infrastructure.**\n\nYou can either use an existing network connection or provision PrivateLink as part of this step. In most cases, creating a new PrivateLink connection is recommended. KCP will configure the required VPC endpoints in the same VPC as your existing cluster; you simply need to provide the subnet CIDRs where those endpoints should be created.\n\nOnce you have the Terraform script, deploy them by running the following command from the `confluent-cloud-infrastructure` directory:\n\nNow that we have target infrastructure in place, we can use KCP to deploy the migration infrastructure. This includes Cluster Linking and AWS resources needed for external Cluster Linking over PrivateLink to work.\n\nTo generate the migration infrastructure Terraform script, run:\n\nThis command uses Confluent Cloud cluster details generated by the `target-infra` command, along with the MSK cluster ARN, to create **Terraform scripts** that provision a **cluster link** named `migration-link`.\n\nKCP supports multiple migration patterns to accommodate different networking and authentication configurations:\n\n**MSK with SASL/SCRAM authentication** (public or private endpoints). If you don’t have SASL/SCRAM configured, it’s simple to add a SASL/SCRAM listener exclusively for Cluster Linking traffic.\n\n**MSK with identity and access management (IAM) authentication using a** **jump cluster** **pattern**\n\nThe example above illustrates a direct cluster link approach with a private MSK cluster with SASL/SCRAM authentication. If SASL is not an option, you’ll need to go down the jump cluster path. Here KCP automates the jump cluster infrastructure as well.\n\nFor more information on how this works, see [KCP documentation](https://github.com/confluentinc/kcp/tree/main/docs#kcp-create-asset-migration-infra).\n\nWith the target and migration infrastructure in place, you can now migrate configuration and data: ACLs, connectors, schemas, and topics. KCP’s pattern is the same each time: Generate assets and then apply them in a controlled way.\n\nUsing the discovery data collected earlier, **KCP migrates both IAM-based permissions and** **Kafka ACLs** from your source cluster to Confluent Cloud. IAM principals are extracted from the previously captured client inventory, while Kafka ACLs are mapped directly from the cluster-level metadata.\n\nKCP reuses the same `kcp-state.json` state file to translate these existing permissions into their Confluent Cloud equivalents, **automatically generating the required** **Terraform assets**.\n\nYou can either apply all generated ACLs or selectively pick principals via the KCP UI.\n\nKCP discovers both MSK Connect and self‑managed connectors and then helps you move them to [fully managed Confluent Cloud connectors](https://www.confluent.io/product/confluent-connectors/). KCP doesn’t migrate the actual connectors; it generates the connector configs in JSON so that you can use a tool of your choice to migrate the connectors.\n\nThe recommended tool is the [Connect Migration Utility](https://www.confluent.io/blog/migrate-self-fully-managed-connectors/), a free, open source, CLI-based tool available on [GitHub](https://github.com/confluentinc/connect-migration-utility/). It provides a guided, end-to-end migration experience from MSK Connect or self-managed connectors to fully managed Confluent Cloud connectors. You start paying for managed connectors once cutover is complete, giving you a low-risk path to migration.\n\nTo generate the JSON configs, run the following:\n\nThis command produces **connector configurations in JSON format, which can then be consumed by the** **Connect Migration Utility**. The utility:\n\nTranslates configurations into their **fully managed Confluent Cloud connector equivalents**\n\nHandles **v1 to v2 connector class changes**\n\n**Preserves** **connector offsets** to ensure a smooth transition\n\nIf you prefer an IaC approach, **KCP can also generate per-connector Terraform modules for both MSK Connect and self-managed connectors**, allowing you to apply them using your existing IaC workflows.\n\nAfter the discovery step, **KCP can migrate schemas by generating Terraform assets that configure** **schema exporters** to replicate from your source Schema Registry into Confluent Cloud.\n\nGenerate the migration assets with:\n\nKCP creates a `migrate_schemas/` folder containing Terraform for an exporter that continuously syncs all schemas to Confluent Cloud, **preserving contexts and compatibility settings** as they exist in the source registry.\n\n**Topic migration is powered by Cluster Linking, with KCP generating the Terraform scripts and supporting the configuration needed for your specific topology.**\n\nKCP automates the setup of all necessary infrastructure, including Cluster Linking plumbing so that you can move topic data from MSK to Confluent Cloud seamlessly and reliably. We’ve already created the target and migration infrastructure; now we start moving data by running:\n\nThis command generates Terraform scripts in the `migration_infra/` folder that:\n\n**Creates cluster links** between your existing cluster and Confluent Cloud\n\n**Mirrors topics**, keeping your data in sync—including [consumer group offsets](https://www.confluent.io/blog/guide-to-consumer-offsets/)—until you’re ready to cut over\n\nClient migrations are often the most complicated part of the migration process, not because of technical complexity but because of the cross-team coordination required to plan and execute client cutovers.\n\nThere are two common approaches to migrating clients today, each with their own trade-offs.\n\n**For a seamless and predictable client cutover experience, users can leverage Cluster Linking** for the client migration. This approach preserves offsets but requires a small amount of client downtime to perform the cutover.\n\n**Users who need to perform a zero-downtime migration can instead dual-produce to both the source and target clusters.** This requires more involved management of offsets during migration.\n\nToday, KCP can assist with client migrations by scanning logs and creating an inventory of active clients using the `kcp scan client-inventory` command. However, much of the client migration process is still manual.\n\nIn the first half of 2026, KCP will greatly simplify this step with [Confluent Cloud Gateway](https://docs.confluent.io/cloud/current/cp-component/gateway/overview.html) compatibility. With Confluent Cloud Gateway support in KCP, you’ll be able to connect all clients to the Gateway endpoint and then cut over all clients at once by directing the Gateway to your target cluster.\n\nKCP transforms what is often a complex and manual migration process into a streamlined, automated workflow. By orchestrating steps like inventory discovery and cost assessment, infrastructure provisioning, and data and client migration, KCP both accelerates migration timelines and reduces the work and complexity involved.\n\nIt’s never been easier to move to Confluent Cloud. Get started today with KCP by exploring the [documentation](https://docs.confluent.io/cloud/current/clusters/migrate-kcp.html), visiting the [GitHub repo](https://github.com/confluentinc/kcp/tree/main/.github), and trying out our [migration workshop](https://github.com/confluentinc/migration-workshops/tree/master/hosted-kafka-to-enterprise-migration) to see the tool in action. Once you’ve taken those first steps, Confluent can help you even more: To receive a granular cost breakdown, simply navigate to the [Migration hub](https://confluent.cloud/migration-hub) in Confluent Cloud, upload your kcp-state.json file, and select “Request cost estimate” to have a Confluent representative reach out with a detailed analysis.\n\nIf you haven’t done so already, sign up for a [free trial of Confluent Cloud](https://www.confluent.io/get-started/). New sign-ups receive $400 to spend within Confluent Cloud during their first 30 days. Use the code **CLOUDBLOG60** for an additional $60 worth of free usage.\n\n*Apache®, Apache Kafka®, Kafka®, and the Kafka logo are registered trademarks of the* *Apache Software Foundation**. No endorsement by the Apache Software Foundation is implied by the use of these marks.*\n\nZooKeeper is going away. Learn why hosted Kafka migrations are accelerating—and how Confluent Cloud simplifies your move with KRaft and KCP.\n\nA behind-the-scenes look at why hosted Kafka falls short—and how Confluent Cloud’s architecture solves for cost, resilience, and operational simplicity at scale.", "url": "https://wpnews.pro/news/kcp-how-to-migrate-to-confluent-cloud-in-days-not-weeks", "canonical_source": "https://www.confluent.io/blog/automate-kafka-migration-with-kcp/", "published_at": "2026-09-14 07:00:00+00:00", "updated_at": "2026-09-29 19:19:36.134753+00:00", "lang": "en", "topics": ["ai-infrastructure", "developer-tools", "mlops"], "entities": ["Confluent", "KCP", "Confluent Cloud", "Apache Kafka", "Amazon Managed Streaming for Apache Kafka", "Amazon Web Services", "Cluster Linking", "Terraform"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/kcp-how-to-migrate-to-confluent-cloud-in-days-not-weeks", "markdown": "https://wpnews.pro/news/kcp-how-to-migrate-to-confluent-cloud-in-days-not-weeks.md", "text": "https://wpnews.pro/news/kcp-how-to-migrate-to-confluent-cloud-in-days-not-weeks.txt", "jsonld": "https://wpnews.pro/news/kcp-how-to-migrate-to-confluent-cloud-in-days-not-weeks.jsonld"}}