At Block, we're building Foundational Event Models (FEMs): deep learning models trained on sequences of product events. Because they learn directly from these sequences, FEMs can pick up patterns that are hard to capture with hand-built features, and a single pre-trained sequence model can be fine-tuned to support many predictive modelling use cases.
Traditional predictive ML models are built mostly on tabular features: hand-crafted aggregates such as "number of payments in the last 30 days". FEMs instead take sequences of individual product events as input, much as a language model reads a sequence of words.
That raises a data problem before any modeling starts. Relevant product events are spread across hundreds of streams, each published by a different team with its own schema, identifiers, and timestamps. To prepare selected streams for model training, we standardize their events into a consistent format and assemble time-ordered sequences.
This series covers the Sequential Store, the system that ingests, standardizes, and prepares event data for training FEMs at scale. In this first post, we explain what a sequence is, then walk through the problems of working with hundreds of raw event sources: inconsistent schemas, onboarding new events quickly, late and duplicate events, and monitoring data quality. Later posts cover event embeddings and experimentation.
#
The Sequential Store is a data store and ingestion pipeline built for offline model training. The models it trains also run online, scoring events as they happen, so every transformation we apply offline must be reproducible in the online path. Otherwise the model sees different data in production than it saw in training, a problem known as training-serving skew.
It runs on a purpose-built software framework with these components:
- Execution engine: a shared Python package that runs config-driven jobs in Databricks Spark SQL.
- Deploys new jobs, backfills data, and runs incremental updates for each pipeline
- Dataflow: transform and conform raw events into a unified format, that are ready for sequencing
- Includes data validation and monitoring layers
- Deploys new jobs, backfills data, and runs incremental updates for each pipeline
- Deploys new jobs, backfills data, and runs incremental updates for each pipeline
- A federated, multi-tenant store : business units share the software framework while maintaining separate pipelines and datasets.
- Each store supports multiple “versions”, which isolate datasets and experiments. Versions also make consistency harder, which we discuss later.
- A Git-based CI/CD setup to define, deploy, test, and promote jobs.
- Outputs are materialized into a standardized schema that downstream training jobs consume.
- AI agents with purpose-built skills that onboard new events, deploy and run jobs, and validate the results (more on this below).
#
A sequence is a time-ordered history of actions ("events") performed by an entity, such as a Cash App customer or a Square seller. Training an FEM is a lot like training a language model: each event is a word, and a customer's journey is the sentence the model learns to read.
At its most basic, each event in a sequence needs five things: an event label (the type of action performed by the user), a timestamp*, an event ID that identifies the event, an entity token that represents the user who performed the event (so that events can be grouped by user), and event attributes (the details of the event). The main job of the Sequential Store is to turn raw events from many different upstream sources into this one consistent shape, so that each customer's activity can be assembled into one sequence.
*We record two timestamps: when the event happened (“happened-at”), and when we received it. The happened-at time orders the customer's timeline. The received-at time tells our daily jobs which events are new, including events that happened days ago but arrived late.
Here’s an illustration of a customer sequence:
#
This section covers concrete challenges we faced while building the Sequential Store and how we solved them: standardizing raw data, onboarding new events quickly, handling late and duplicate events, and monitoring data quality at this volume.
Context: working with a large variety of raw input sources
Our goal is to ingest relevant product event streams to build time-ordered event sequences. These sequences then enable many downstream predictive use cases.
There are hundreds of distinct product events. Each comes from a Kafka topic published by a product team, with its own schema.
At Block, raw Kafka events are archived into tables in the open Apache Iceberg format, which we call Kafka Topic Archives (KTAs). The Sequential Store reads these archives in daily batches rather than consuming the Kafka streams directly.
The main challenge comes from the large variety of events, which lack a standard format and schema. Despite standardization efforts and shared SDKs, publishers sometimes make their own business assumptions (such as which identifier names the customer) and implementation choices. As a result, some events need bespoke handling.
One issue is that the same piece of information lives in a different place in each event's payload. We handle this with selectors: SQL expressions in each event type's config that say where to find each standard field. Every event type is then written to the Sequential Store in the same shape.
For instance, entity_token_selector in the example below tells the pipeline where to find the customer ID. The coalesce handles a schema migration: older events carry the ID in one field and newer events in another, and coalesce takes whichever is present.
Illustrative example: an event definition config containing custom selectors. The table and field names are made up.
1message_type: payment_completed
2source_table: event_archive.payment_completed
3source_event_id_selector: event_metadata.id # the source's ID for the event
4entity_token_selector: coalesce(value.customer_id, value.sender.customer_id)
5happened_at_selector: CAST(value.timestamp AS TIMESTAMP)
6event_attributes_selector: ...
7actions:
8 - event_label: payment_completed.OCCURRED
The customer ID deserves special attention, because every sequence is keyed on it. In a product event, though, the customer is not always the main subject: in some events the field is optional, and in others it moved to a different place in the payload after a schema migration.
Selectors, coalescing across old and new field locations, and filters that drop events with no customer ID together form a standardizing layer: one consistent view across all events.
This approach handles hundreds of different ingestion jobs in a unified, programmatic way. The framework is an abstracted layer over the native Databricks jobs, which helps avoid fragmented logic and error-prone ad hoc processes.
The framework also has built-in helper jobs for a consistent workflow. For example, some jobs create the views that transform and standardize each event type, and others swap a newly built table into production. Every job is defined in config checked into Git, so each run can be reproduced, and each run is recorded for auditability.
Here’s an overview of our job automation framework:
Some highlights:
- The backfill and incremental jobs share the same materialization process and data quality checks, as a result, they produce a common, continuous history that has consistent data quality.
- Shared configs mean all data jobs are set up in the same way, including how tables are clustered, how updates are merged, and how compute clusters are configured. This approach provides a shared, centralized set of sensible defaults for standard jobs.
- All jobs are instrumented with monitoring metrics that are linked to Datadog monitors, which provide automated alerting should any failures or data quality issues occur.
Experimentation needs grew fast, and the team often identified new batches of promising new events to onboard quickly. That meant balancing speed (onboarding new events rapidly) against reliability (making the process less error-prone and more consistent). A few things that are useful to speed up onboarding automation:
- AI agents: they assist with inspecting event schemas, drafting configurations, and validating job outputs. Deployment and execution follow our existing review and authorization process. Each repeatable step is packaged as an agent skill. Because jobs are defined by configs, they're also easy for agents to generate and review.
- Versioning: we built lightweight versioning for the Sequential Store so multiple versions can coexist. Each version holds a fixed set of ingested events. For example, an earlier batch of events is already used in experimental pipelines. We keep that batch isolated in one version and onboard new events in a separate one.
- Separate versions bring a new challenge: every version shares one vocabulary, the mapping from event labels to the numeric IDs the model is trained on, and it must stay consistent across versions. We made the vocabulary append-only, much like protobuf field numbers: once a label has an ID, the ID never changes and is never reused, and CI checks catch collisions where two changes claim the same ID. That check matters when several developers, or agents, are editing the same codebase at once.
Raw events don't arrive neatly. An event can land in the archive days after it happened, and the same event can be read more than once. We have to handle both without breaking the customer's timeline.
Late events are why we keep two timestamps. Each daily run picks up events by the date they were received, re-reading a trailing window of recent days, so a late event is found whenever it arrives. It is then stored under the date it happened, so the timeline stays in order.
Re-reading a window means seeing the same event again, so every write is deduplicated. The hard part is choosing the deduplication key, and a wrong key fails in two different ways:
Too coarse, and distinct events collapse into one. One lifecycle event reused the same ID for every state a deposit moved through. Deduplicating on that ID kept one row per deposit and silently dropped most of the rows. The fix was a key made of the ID plus the state.
Shared across customers, and whole customers disappear. On another topic, the publisher reused a single envelope ID for a whole batch of events from different customers. Trusting the obvious ID field would have dropped customers, so we now measure every candidate ID's uniqueness before relying on it.
Some of these failures are invisible to standard monitors. If a key merges a customer's distinct events, no customer goes missing and both sides of a count reconciliation agree. The only symptom is a shorter sequence, which is exactly the signal the model learns from.
Our deduplication key includes the event's date, which has a sharp edge: if a definition changes how it computes when an event happened, re-read events move to a new date and are inserted a second time. So we never edit a live version's definitions in place. Changes ship as a new version, rebuilt from a full backfill.
One of our most important strategies is building effective data monitors as the last line of defense.
Below are data quality problems we faced, and the monitors and monitoring approaches that addressed them:
Example 1: NULL data monitor: the key challenge is telling “genuine” upstream data issues apart from our own implementation issues (which are in our control to fix).
As a concrete example, these scenarios can all produce NULL alerts, but each has a different cause:
- An eventing topic stopped producing data (for example, it was decommissioned)
- An unexpected upstream schema change (a field has “moved” to a different location)
NULLissues propagated from upstream, even when the choice of selectors (i.e. the field selection logic) is correct- A bug in our own selector logic
A generic NULL alert isn't very useful, because it can't tell these four causes apart, and each one needs a different fix. So we push the causes we can prevent earlier in the process. Validating selectors at backfill time catches a buggy selector (4), and NULLs already present in upstream history (3), before the data ever reaches a monitor. Agent-based checks upstream catch breaking schema changes (2) and new NULLs (3) at the source. That leaves the alert for the case only a monitor can see: a topic that has stopped producing data (1).
Silent failures are the dangerous ones. When one upstream schema moved its customer and time fields into new sub-structures, our selectors kept reading the old location, which was now empty. Nothing errored; the event simply stopped producing rows, and more than a year of the event's rows were excluded from the training dataset due to empty-data filtering logic before we caught it. The fix was a fallback selector that reads the old location first and then the new ones, so history resolves exactly as it did before.
Example 2: Checking data that is still arriving
Events keep arriving in the source archives around the clock, while the Sequential Store loads them in daily batches. A check that reconciles row counts over the same window the job has just loaded raises false alarms: by the time the load finishes, more events have arrived at the source that the target does not have yet. When one daily job ran late enough that its check scanned a day that was still filling up, most event types were flagged at once.
The fix is to check only data that has stopped moving. As the diagram shows, the daily job reloads a trailing window that includes today, but the monitor checks only the days before it.
Example 3: Avoiding alert storms across hundreds of event types
With per-event alerts, one pipeline-wide problem, such as an infrastructure incident, used to fail every event type at once and flood the on-call channel with dozens of alerts. Now, when the number of event types failing in the same run crosses a threshold, we suppress the per-event alerts and send a single mass-failure alert instead. Smaller failures alert exactly as before.
The general takeaway: data quality monitors are essential when you work with a large number of event topics, and even more so as AI workflows do more of the ingestion and transformation. Effective monitors take iteration, because generic monitors usually aren’t enough, as illustrated in example 1. Balancing coverage against noise (too many false positives) is what makes human oversight work.
#
Below are the takeaway lessons that we believe would generalize.
Make onboarding a config change. When each event type is a short config file rather than a bespoke job, hundreds of event types share one tested code path, and agents can do much of the onboarding work.
Immutability is important: never reuse an ID. A model's vocabulary should be append-only, like protobuf field numbers: once a label has an ID, that ID never changes and is never given to anything else.
Good IDs are important - evaluate an ID before adopting. An ID that looks unique can be shared across states, batches, or customers, and the resulting data loss is often invisible to monitors.
Design each alert around the action it triggers. An alert that fires for four different causes is an alert people learn to ignore.
Check data only once it has stopped moving. Reconciling a window that is still filling up produces false alarms.
In the next post, we look at how we turn each event's free-form attributes into embeddings the model can learn from.