{"slug": "the-data-platform-1password-needed-didn-t-exist-so-we-built-it", "title": "The data platform 1Password needed didn't exist. So we built it.", "summary": "1Password built a new internal data platform based on Apache Iceberg tables after its existing centralized data lake became a bottleneck for real-time access, AI workflows, and governance. The platform separates storage from compute, using Databricks for engineering and execution and Snowflake for governed consumption, enabling consistent trust and accessibility across the business.", "body_md": "Consider a few tasks that take place across every business, every day:\n\nA product team ships a feature and wants to know if customers are using it successfully.\n\nA finance team needs customer and account information for planning.\n\nAn analyst needs definitions to create a report.\n\nAn AI assistant needs operational context to answer a business question.\n\nThose sound like different workflows, but they all rely on the same underlying data. And each one of these actors, across each of these teams, needs that data to be both accessible and trustworthy.\n\nAt 1Password, trust is at the center of everything we build. Millions of people and businesses rely on us to protect the credentials, secrets, and access workflows that power modern work. The same principle applies to our own internal data.\n\nAs our products, systems, and use of AI evolved, data became a shared dependency across the business. It powers everything from [ Unified Access](https://1password.com/blog/introducing-1password-unified-access) and\n\nBut our data infrastructure did not respond well to this. We had built a centralized data lake supported by a growing collection of bespoke data pipelines and one-off solutions. Each new use case required another integration or transformation. Over time, the data platform became a bottleneck: teams turned to CSVs to move faster, and data engineers spent more time maintaining pipelines than enabling new capabilities.\n\nWhat we learned was that manually moving data was no longer enough. Data needs to be available in real time, accessible wherever it’s needed, and trusted through the forms our customers need, whether that is through SQL, APIs, dashboards or AI workflows. Privacy and governance need to be built into data the moment it’s created so every downstream use remains secure and unambiguous by design.\n\nThis is the foundation we’re building.\n\nThe issue was not that we lacked data. We had plenty of it. The problem was that storage, governance, and compute had become tightly coupled inside individual tools.\n\nDifferent systems stored their own copies of the data. Governance rules varied between tools, schema changes could break downstream models, and the teams producing data could not inspect or validate their own raw event streams.\n\nThat fragmentation also made trust harder to maintain consistently. Every new workflow introduced another ingestion path, another policy surface, or another duplicate copy of the data. It also increased cost and operational complexity as the number of workflows and consumers grew.\n\nAI systems amplified those problems because agents depend on machine-readable trust, consistent semantics, and governed operational context in ways traditional dashboards do not. Humans are adept at navigating this kind of ambiguity, but agents need more clarity.\n\nSo we changed the model. We standardized a shared open data foundation that multiple execution engines and access patterns could operate against consistently.\n\nWe built the platform around Apache Iceberg tables, an open table format for managing large analytical datasets, giving us a shared open storage layer owned by 1Password rather than any individual compute platform. In this model, compute engines operate against the same underlying data instead of maintaining isolated copies.\n\nThis separation gives us flexibility without bifurcation. It enables us to make the best architectural choices and select purpose-built platforms for the workloads they are best suited for.\n\nThat is the thesis behind our adoption of both Snowflake and Databricks. This model allows us to take advantage of their unique strengths without fragmenting the platform itself.\n\n**Databricks** is our engineering and execution plane, handling raw ingestion, distributed processing, and schema registration.\n\n**Snowflake** is our governed consumption plane, where data becomes business and AI ready.\n\n**Federated governance **ensures ownership, lineage, classification, and access controls remain consistent across both platforms. Ownership, governance, and metadata are managed where data is authored, while Iceberg provides a shared foundation across both platforms.\n\nThe important shift is that no individual platform becomes the system of record; the foundation itself does.That lets us evolve our tooling and workflows without continuous rebuilding and managing the data layers underneath.\n\nOur data lifecycle starts with the team that creates the data.\n\nA team implements an event as part of feature delivery or within an external system. Instead of filing a request for a downstream extraction later, the data producer defines the event schema using Protobuf and registers it through the schema registry. That schema becomes the contract between the producing service and the data platform.\n\nEvents flow through Kafka and land in Iceberg-backed tables. At this stage, the goal is speed and fidelity: make source-aligned data available quickly with schema enforcement, ownership metadata, and classification attached from the start.\n\nFrom there, datasets move through a Medallion-style lifecycle:\n\n**Bronze** tables preserve source-aligned event streams for validation, debugging and operational analysis.\n\n**Silver** tables standardize schemas and create reusable domain level entities.\n\n**Gold** tables expose curated business ready datasets for reporting, APIs, and AI assisted workflows.\n\nThose stages let different teams access data in the way that best fits their workflow. A developer may work directly against what’s in Silver with Cursor, Claude or Codex, while an analyst may consume data products built in the Gold layer through SQL or natural language interfaces.\n\nThis new data platform delivers greater speed, but more importantly, it creates a different relationship between teams and data.\n\nData producers define and own the data their systems create. The platform ensures that data is accessible and governed consistently. Data consumers and AI workflows all operate against the same trusted foundation instead of maintaining their own versions of the truth.\n\nThat changes the economics of the platform too. Instead of spending time recreating data and maintaining custom integrations, teams can build on shared data products that are reusable across the business.\n\nThat is the shift this project represents: reducing the distance between data creation and use while ensuring security and privacy are built in from the start. Data is owned once, used many times, and shared through a trusted foundation. The result is less operational overhead and greater trust in the data behind every decision.\n\nOperational data is no longer “the necessary cost of running our systems.” It’s now seen as one of the company’s most valuable assets, shaping how we operate the business and serve our customers in real time.\n\nWant to work with Data at 1Password? We're hiring. [Explore our open roles.](https://jobs.ashbyhq.com/1password)", "url": "https://wpnews.pro/news/the-data-platform-1password-needed-didn-t-exist-so-we-built-it", "canonical_source": "https://1password.com/blog/we-built-the-data-platform-1password-needed", "published_at": "2026-07-22 00:00:00+00:00", "updated_at": "2026-07-22 13:25:09.593830+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-infrastructure"], "entities": ["1Password", "Apache Iceberg", "Databricks", "Snowflake"], "alternates": {"html": "https://wpnews.pro/news/the-data-platform-1password-needed-didn-t-exist-so-we-built-it", "markdown": "https://wpnews.pro/news/the-data-platform-1password-needed-didn-t-exist-so-we-built-it.md", "text": "https://wpnews.pro/news/the-data-platform-1password-needed-didn-t-exist-so-we-built-it.txt", "jsonld": "https://wpnews.pro/news/the-data-platform-1password-needed-didn-t-exist-so-we-built-it.jsonld"}}