Real-time fraud detection with Feature Views

Databricks detailed a real-time fraud detection workflow in which a single Feature Views definition, part of the Databricks Feature Store, generates both point-in-time correct training data and managed online inference pipelines for a $900 payment streamed through Apache Kafka to a Model Serving endpoint. The walkthrough combines three feature types — recent customer activity, normal spending behavior and a comparison of the current payment to that baseline — to avoid data leakage and training-serving skew that arise when training and serving logic are implemented separately. A Fraud Detection application built with Databricks Apps scores each transaction's risk and displays live transaction volume, flagged transactions and country-to-country flows, with Unity Catalog governing feature definitions, permissions and lineage.

Follow a single $900 payment from an Apache Kafka stream to a live fraud score, with code for streaming, on-demand and point-in-time correct features on Databricks. Real-time fraud detection depends on three kinds of features: what a customer did seconds ago, how they normally spend and how the current payment compares. Those features have to be computed the same way for model training and for live scoring — otherwise data leakage and training-serving skew quietly erode model quality. Feature Views, part of Databricks Feature Store, address this. You define each feature once, and that definition produces both point-in-time training data and the managed pipelines for online inference. A $900 payment may be routine for one customer and unusual for another. When it arrives, a fraud model has only a moment to decide whether it looks normal, needs review or should be blocked. Approving a fraudulent transaction can cost the customer money. Blocking a legitimate one disrupts their payment experience and erodes their trust in the platform. The transaction alone doesn't tell the whole story. The model also needs to know how this customer normally transacts and what they did seconds, minutes, days or weeks ago. During training, features must use only information available when each payment occurred. During live scoring, they must reflect recent transactions quickly enough to catch suspicious activity. If the two paths implement the logic separately, small differences between them creep in. This walkthrough follows that $900 payment from a Kafka stream to a fraud score served by Model Serving, with code at each step. The central idea is that one feature definition, driving both training and serving, gives you point-in-time correct training data and consistent online features. You don't build a separate pipeline for each path. A Feature View is a Databricks Feature Store definition that specifies a feature's source, entity keys, time semantics and transformation. The same definition computes historical values for training and drives the managed pipelines that keep features fresh for serving. One API covers batch, streaming and on-demand features. Unity Catalog governs their definitions, permissions and lineage. Two failure modes motivate this design: Feature Views bring streaming feature computation into the same data science workflow as historical training and live scoring. You can evaluate features against past transactions, then use the same definitions to keep customer context fresh for live decisions. The goal is to flag suspicious payments as they arrive without raising false alarms on legitimate ones. On its own, the $900 payment may look harmless. But if it's the 10th payment in five minutes, it could be part of a suspicious burst. Any delay in recognizing that burst gives someone with access to a compromised account more time to make additional payments. A Fraud Detection application built with Databricks Apps processes a stream of financial transactions. Each event includes a customer ID, amount, payment type, timestamp and origin and destination countries. The application sends each transaction to a Model Serving endpoint. It then displays the transaction's risk score alongside live transaction volume, flagged transactions and country-to-country flows. The model combines three kinds of context: The architecture has a historical path for training and an online path for inference. Both start from the same feature definitions in Unity Catalog. The historical path reads the Stream's ingestion table to build point-in-time training sets. The online path materializes aggregates to the Online Feature Store, where Model Serving reads them at request time. Before deciding which features to serve, you can test alternatives against historical transactions. Does a five-minute transaction count give a stronger signal than a 15-minute count? Does comparing the current amount with the customer's spending variability help beyond using their average alone? In this example, customer transaction events arrive as Kafka streams. A registered Stream holds the connection, schema and ingestion settings, and a StreamSource references it to compute real-time features. Kafka topics don't retain enough history for model training. Databricks therefore manages an offline copy of the stream in an ingestion table. For training or backfilling an online store, Databricks calculates historical values from the ingestion table and uses the live Kafka feed for the most recent data. On ingest, transformation sql reshapes nested fields from the Kafka value payload into the schema the features need. The projection becomes part of the source definition, so you don't deploy a separate preprocessing pipeline. For our payment, the Kafka value payload contains: The demo uses demo user id alongside customer id to keep the customer histories of different app users separate. We define the five-minute count with RollingWindow . The companion notebook includes imports, configuration, and the full source projection. You can inspect its historical values in a notebook with compute features features= recent transaction count . The 10-minute total uses the same source and keys, with Sum over a 10-minute RollingWindow. A Sawtooth window is a long-lookback aggregation whose oldest boundary advances in daily steps. It combines batch-computed history with streaming updates for recent activity. For our $900 payment, SawtoothWindow adds a 30-day spending baseline to the short rolling windows. Both kinds of window keep recent activity fresh. Because the Sawtooth window's oldest boundary moves daily rather than continuously, its effective lookback is slightly longer than 30 days. The baseline uses the built-in Avg , StddevPop and Count aggregations on payment amounts over the same Sawtooth window. Databricks starts a managed ingestion pipeline when a Stream is created and keeps incoming events in its ingestion table. If you also need events from before the Stream existed, StreamBackfillSource lets you supply your own history during Stream setup. Once the table covers the full lookback, a new Sawtooth feature can use it right away instead of waiting 30 days for events to accumulate. An on-demand feature is computed at request time by combining values from the scoring request with stored features. Comparing a live transaction against a customer's historical baseline is a common way to spot abnormal behavior. In Feature Views, inference-time information is represented with a RequestSource — here, transaction amount from the application's request. Two customers can both average $300 per payment but spend very differently. With standard deviations of $100 and $600, the same $900 payment sits six standard deviations above one customer's average and only one above the other's. The model combines this comparison with recent activity and payment details to estimate fraud risk. The Sawtooth aggregations maintain the 30-day average and standard deviation. The on-demand feature combines them with the current amount to calculate current amount - average / standard deviation . We implement this calculation as a Python UDF registered in Unity Catalog: The feature definition uses registered upstream Feature objects. FeatureViewSource brings together the current amount and the three 30-day statistics, and CustomUDF invokes the function: When there isn't enough usable history, the function returns zero as a fallback. A separate indicator lets the model tell that fallback apart from a true zero, which means the payment matches the customer's average. The same derived feature is evaluated against historical inputs during training, and against the current request plus retrieved statistics during inference. During feature development, you can inspect feature definitions in a notebook with compute features https://api-docs.databricks.com/python/feature-engineering/latest/feature engineering.client.html databricks.feature engineering.client.FeatureEngineeringClient.compute features . This works for derived features whose inputs come entirely from historical source data. For request-dependent comparisons such as amount deviation , use create training set https://docs.databricks.com/aws/en/machine-learning/feature-store/train-with-declarative-features instead, which supplies the current payment fields from labeled examples. The Unity Catalog function must exist, but feature definitions can stay unregistered while you test candidates. This lets you check whether a feature improves the fraud model's offline performance before deploying it for live scoring. For example, you can compare models with and without amount deviation on later held-out payments. These experiments run on batch compute over transaction history, so you can evaluate each candidate without setting up online materialization. Once you've selected the features, register their definitions and materialize the rolling and Sawtooth aggregations for serving. Each historical training example needs features computed as of that payment's timestamp. For our $900 payment at 10:00 AM, the five-minute count should include transactions from 9:55 AM up to, but not including, 10:00 AM. A count retrieved at 10:03 AM could include a payment made at 10:02 AM, which wasn't available when the original payment was scored. That's data leakage: the model learns from future activity, and its offline performance looks better than it really is. Feature Views use each example's customer keys and timestamp to compute the correct historical window from events in the Stream's ingestion table. After registering the selected features, create training set builds the model's training data. The labeled DataFrame supplies the customer keys, event time, current payment fields and is fraud . This excerpt selects the walkthrough's registered features, including the 30-day baseline, and uses rows before a chosen cutoff for training: Keys and timestamps select customer history but are excluded as model inputs. Request fields feed the feature computations; the classifier uses the resulting named features. Later rows form a validation set using the same definitions. We train a random-forest classifier and evaluate average precision and ROC AUC on the latter. We then log the model through the Feature Engineering client. Passing training set=train set records the feature dependencies that serving will need: Moving the above selected features into live scoring requires pipelines to maintain streaming aggregates, process historical data, and keep the online store up to date. Building and operating these pipelines adds months of data engineering work beyond developing the fraud model. With Feature Views, the registered feature definitions drive this process: a call to materialize features creates managed pipelines that make the aggregations available in Databricks Online Feature Store, powered by Lakebase https://www.databricks.com/product/lakebase . The streaming path uses Spark Real-Time Mode RTM https://docs.databricks.com/aws/en/structured-streaming/real-time/concepts to process incoming events continuously and update rolling aggregates as they arrive. We materialize the rolling and Sawtooth aggregations. For the 30-day Sawtooth baseline, Databricks maintains the historical portion with batch computation and the recent activity with streaming. The two are merged on read into a single feature, and StreamingMode manages this hybrid path without a separate batch call. Check both materialization paths in Catalog Explorer before serving. The Sawtooth guidance https://docs.databricks.com/aws/en/machine-learning/feature-store/feature-views-api-reference sawtooth-window recommends waiting two full days before production use. RequestSource supplies values from each scoring request, and derived features are computed on demand. Once the model is deployed, the application sends the customer keys and current payment fields to its endpoint. Model Serving retrieves the latest customer features and evaluates the on-demand deviation, then passes the inputs to the fraud model. Here's our $900 payment in the app's request format, using the SDK call from the scorer: The app sends event time in this request, but online lookup uses the latest available customer features, not a historical snapshot selected by that timestamp. In this demo, the app scores each payment before publishing it to Kafka. That keeps the current payment out of its own history, which matches the training window's exclusion of that payment. Feature updates and scoring still run asynchronously, so other events may be in flight when the next payment is scored. The endpoint returns a probability in fraud probability and a flag in fraud prediction . The application stores scored transactions in Lakebase for the dashboard. Production fraud models raise governance questions: which models use a feature, where it comes from and who can read it. Unity Catalog brings feature definitions, permissions and lineage together to answer them. MLflow records which features each model uses, and Model Serving uses those dependencies when it handles an inference request. Feature privileges control the lifecycle: CREATE FEATURE controls registration. READ FEATURE grants access to feature metadata. MANAGE authorizes operations such as materialization. Feature privileges don't replace the SELECT permissions needed on the source or materialized tables for training and serving. In Catalog Explorer, Feature Views appear under Functions , with their definitions, source links, materialized tables and status. Streaming materializations link to their system-managed Lakeflow pipelines for inspection. Materialization tags https://docs.databricks.com/aws/en/machine-learning/feature-store/cost-management attribute-costs-with-tags-and-serverless-usage-policies let teams attribute pipeline or job compute costs to a specific team or project. Our $900 payment now has context: recent activity, a longer-term baseline and the current request. Three practices made that possible: The same patterns go beyond this walkthrough. LastN keeps sequences such as the destination countries of a customer's 10 most recent payments, which helps models spot changes in payment routes. Batch features can also use lifetime aggregations to summarize an entity's complete history. For a closer look at the infrastructure behind sub-second streaming freshness, read How Databricks Feature Store serves features with sub-second freshness https://www.databricks.com/blog/how-databricks-feature-store-serves-features-sub-second-freshness . For implementation details, see the docs for the Feature Views API https://docs.databricks.com/aws/en/machine-learning/feature-store/feature-views-api-reference , materialization https://docs.databricks.com/aws/en/machine-learning/feature-store/materialized-features and exploring Feature Views in Unity Catalog https://docs.databricks.com/aws/en/machine-learning/feature-store/explore-feature-views . What are Databricks Feature Views? Feature Views are a capability of Databricks Feature Store for defining ML features once and using them for both training and production. Each definition specifies a source, entity keys, time semantics and a transformation. The same API covers batch, streaming and on-demand features, and Unity Catalog governs them. How do Feature Views prevent data leakage when training a fraud model? Feature Views compute each training example's features as of that example's timestamp. For a payment at 10:00 AM, a five-minute transaction count includes payments from 9:55 AM up to, but not including, 10:00 AM. This keeps future activity out of the training data, so offline performance isn't inflated. What is training-serving skew and how do Feature Views reduce it? Training-serving skew occurs when features are computed one way for training and another way for live inference. Feature Views reduce it by using one registered definition for both historical computation and online materialization. Teams don't have to reimplement feature logic for production. Can Feature Views compute features from Kafka streams? Yes. A registered Stream holds the Kafka connection, schema and ingestion settings, and a StreamSource references it in feature definitions. Databricks keeps an offline copy of the stream in an ingestion table for training and backfills. Real-Time Mode updates streaming aggregates as new events arrive. How are real-time features served to a fraud model in production? Calling materialize features creates managed pipelines that keep aggregates current in the Online Feature Store, powered by Lakebase. When the model is logged with log model and deployed on Model Serving, the endpoint retrieves the latest customer features and computes on-demand features from the request. It then returns a fraud probability. Subscribe to our blog and get the latest posts delivered to your inbox.