NetApp buys DataPelago to become full-stack AI data infrastructure provider

NetApp acquired DataPelago, a startup that developed the Nucleus Universal Data Processing Engine (UDPE) for accelerating data analytics and GenAI workloads, to become a full-stack AI data infrastructure provider. The acquisition aims to help enterprises process data more efficiently by embedding Nucleus into NetApp's data platform, offering up to 10.5x faster performance than Nvidia's cuDF for certain operations.

NetApp buys DataPelago to become full-stack AI data infrastructure provider NetApp is acquiring DataPelago https://www.blocksandfiles.com/ai-ml/2024/10/04/datapelago-unveils-software-accelerator-for-genai-and-lakehouse-analytics/1591502 whose Nucleus Universal Data Processing Engine UDPE accelerates heterogeneous compute for data analytics and GenAI models. Nucleus uses open source Gluten, Velox, and Substrait to turbocharge Spark and Trino, providing customers with “disruptive price/performance advantages.” It integrates into existing data stores and lakehouse platforms, SQL, Python, Airflow workflow automation, Tableau, Power BI, and more with no need for data migration and no lock-in. DataPelago was started up in 2021 by CEO Rajan Goyal and chief product officer Anand Iyer, and came out of stealth in October 2024. The company has raised more than $75 million with its latest round in 2024 providing $47 million George Kurian, NetApp’s CEO, said: ”As AI models and the chips that power them get ever more effective, enterprises need data infrastructure that is just as intelligent and powerful to harness the potential of their data. … With DataPelago, we are extending our ability to help customers understand and process their data with the agility required to unleash competitive advantage.” A brief DataPelago blog https://www.datapelago.ai/resources/DataPelago-is-joining-NetApp stated: “NetApp manages more enterprise data across more environments than anyone in the industry, and Nucleus will be embedded directly into their data platform. It is exactly where this technology belongs, and it reaches far more enterprises than we could have reached on our own.” In March this year DataPelago was named to Fast Company’s List of the World’s Most Innovative Companies of 2026, ranking 4 in Data Science. DataPelago provides accelerated compute at the data layer, processing data at the storage layer rather than moving it to external compute clusters with CPUs and/or GPUs and other accelerators. It is up to 10.5x faster https://www.datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF https://www.datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF for project operations, up to 10.1x faster for filter operations, and up to 4.3x faster for aggregate operations compared to Nvidia’s cuDF. DataPelago’s technology operates between query processing engines, such as Spark, Trino, and Flink, and Ray and Dask Python frameworks, being a software data processing engine for AI with three component layers; DataApp – pluggable layer that enables integration with platforms including Spark and Trino to deliver acceleration capabilities to these engines. DataOS – operating system layer mapping data operations to heterogeneous accelerated computing elements and managing them dynamically to optimize performance at scale. DataVM – a virtual machine with a domain-specific Instruction Set Architecture ISA for data operators providing a common abstraction for execution on CPU, GPU, FPGA, and custom silicon. Here is our understanding of the high-level data flow; Data Access via Host Framework Connectors: Nucleus integrates as a plug-in e.g., Spark plug-in jar that works with Spark's or similar engines' standard data source connectors. It supports common lakehouse formats and file types like Parquet, ORC, Iceberg, Delta Lake, JSON, etc., stored on object storage S3, GCS, ADLS , HDFS, or on-prem storage arrays. Reading happens through the framework's I/O layer e.g., Spark's DataFrameReader or equivalent , which fetches data into the worker nodes' memory. Nucleus extends acceleration to storage and data source integrations while preserving semantics. Plan Transformation and Optimization: The host engine's query optimizer produces a physical plan. Nucleus's DataApp layer with Apache Gluten/Substrait converts it into an intermediate representation. The Intelligent Execution Optimizer builds optimal pipelines and selects hardware CPU/GPU . This becomes a Data Flow Graph DFG executed by DataOS and DataVM. Efficient Movement to CPUs and GPUs Key Optimizations : I/O and Data Movement Minimization: Nucleus addresses GPU challenges like host-to-device transfers and I/O bandwidth limits through operator fusion, kernel fusions, and streaming execution no store-and-forward . Data flows in a pipelined/streaming fashion rather than full materialization. Zero-Copy Techniques: It uses zero-copy shared memory management, especially for strings and complex types, reducing copies between CPU memory and GPU memory. Hardware Mapping: DataVM's domain-specific ISA leveraging LLVM, CUDA, ROCm dynamically maps operators to the best backend. It executes using vectorized/columnar processing and advanced primitives, keeping data close to compute e.g., direct GPU execution where possible . Result: Reduced I/O overall, with GPUs pegged at 80-90 percent utilization by minimizing unnecessary data movement between domains. Nucleus works on-premises and in the main public clouds. As Databricks is Spark-based with its own optimizations like Photon , DataPelago integrates natively via its Accelerator for Spark. It has shown 3-4x gains vs. Databricks Photon in benchmarks. DataPelago works with Snowflake as part of broader lakehouse ecosystems by accelerating processing on data in open table formats e.g., Iceberg that Snowflake supports, or by handling upstream/downstream Spark/Trino workloads that feed into or read from Snowflake. DataPelago claims Nucleus reduces infrastructure costs by up to 80 percent and delivers performance up to 10 times faster than conventional approaches. Also, by not requiring customers to copy their data from their operational systems to AI-systems, DataPelago eliminates the single biggest bottleneck in enterprise AI deployment. Nucleus SW is in use at large enterprises across multiple industries. Read customer case studies here https://www.datapelago.ai/resources?categories=case-studies . Rajan Goyal, Founder and CEO of DataPelago said: "DataPelago is on a mission to eliminate the data processing bottlenecks that prevent AI innovation from reaching its full potential. Joining NetApp gives us the opportunity to combine our breakthrough processing technology with the industry's best data infrastructure portfolio. Enterprises have invested billions in GPUs and AI models, but their data remains fragmented, leaving valuable computing resources to sit idle rather than putting these investments to work. Together, we’re positioned to help customers simplify and accelerate AI deployment at scale.” NetApp says the acquisition marks a foundational expansion of its portfolio, enabling GPU-accelerated data processing aligned directly with the storage layer. With this acquisition, it says it establishes itself as the company that makes zero-copy activation of enterprise data for AI real. KV Cache How does DataPelago technology relate to KV Caching? DataPelago focuses on data movement, transformation, and querying at scale, especially for massive structured/semi-structured/unstructured datasets feeding AI systems. This takes place before data goes to CPU/GPU/FPGA compute. Nvidia’s KV Caching scheme, supported by many storage suppliers, is a real-time inference memory optimization that comes into play once data has reached a GPU server system. DataPelago gets the data to the GPU server system where KV caching schemes are activated. DataPelago makes the data feeding AI faster/cheaper; KV caching makes the AI model run more efficiently during response generation. AIDE NetApp’s AIDE https://www.blocksandfiles.com/ai-ml/2025/10/14/netapp-disaggregates-ontap-storage-and-provides-an-ai-data-engine/1590170 AI Data Engine pre-processes AFX-stored ONTAP data for AI LLM and agent use, and is ONTAP-specific, whereas Data Pelago’s UDE is agnostic to the storage systems underlying the Spark, Trino and other query engines it accelerates. AIDE povides an AI data pipeline to have NetApp AFX-stored data made deliverable to and usable by AI Large language Models LLMs and agents. It is a built-in AI-focussed ETL Extract, Transform and Load pipeline that uses the AFX metadata engine. The AIDE software can discover and characterize data spread across a customer’s NetApp data estate, both on-premises and in the three main public clouds. As fresh data comes in and/or old data is deleted the metadata is updated. AIDE contains a vector database so that the unstructured data it manages can be used by LLMs and agents in semantic searches. It uses the Nvidia AI Data Platform reference design, which includes Nvidia GPUs, AI Enterprise software including NIM microservices for vectorization and retrieval, which joins advanced compression, fast semantic discovery, and secure, policy-driven workflows. It handles metadata cataloging, data discovery, curation, governance, vector embeddings, and RAG serving with Nvidia integration . Adding Nucleus to AIDE would make it more powerful for data transformation, ETL, training, fine-tuning, and inference pipelines. It could enable accelerated Spark/Trino workloads running directly on NetApp AFX storage and we might envisage a possible AIDE + Nucleus bundle for an end-to-end AI data pipeline. NetApp’s Chief Product Officer, Syam Nair, added this thought: “DataPelago's Nucleus engine brings software-defined acceleration directly to the storage layer, processing data across CPUs and GPUs so enterprises can prepare, govern, and activate their data for AI without moving it. This is true zero-copy activation. NetApp manages more enterprise data across more environments than anyone in the industry. The next phase of AI will be won by those who make that data work at the source, and the DataPelago team brings the technical depth and velocity to get us there faster.” DataPelago will operate as a wholly owned subsidiary of NetApp. The acquisition price was not revealed. Given the good fit between Nucleus and AIDE, the use of Nucleus in large enterprises across multiple industries, and the rise of agentic AI, we’d not be surprised by NetApp paying a 4 to 5x multiple of DataPelago’s total funding.