cd /news/machine-learning/putting-a-transformer-inside-every-c… · home › topics › machine-learning › article
[ARTICLE · art-174208] src=mlnotes.substack.com ↗ pub= topic=machine-learning verified=true sentiment=↑ positive

Putting a Transformer Inside Every Card Authorization: How Ramp Achieves Sub-30ms Inference

Engineers at Ramp embedded a custom 3-layer sequence transformer directly inside live card authorizations, running on CPU within a 30-50ms decision budget. The model is pre-trained self-supervised on two years of transaction sequences to predict the next swipe's attributes, then paired with a dollar-weighted tree model, yielding 32% more fraud dollars caught at the same alert volume and 40% better detection of the first fraudulent swipe in an attack.

recommendation

When latency budgets rule out GPU serving, pre-train a compact transformer on sequences and run it on CPU alongside a tree model, as Ramp did to hit sub-30ms decisions.

more in playbook: Inference and self-hosting
by read8 min views3 publishedNov 22, 2026
Putting a Transformer Inside Every Card Authorization: How Ramp Achieves Sub-30ms Inference
Image: Mlnotes (auto-discovered)

Why real-time payment SLAs break standard deep learning deployments, how Ramp trained a sequence transformer to autocomplete card spending, and the systems tricks required to run it on CPU in millisec

When you tap a corporate credit card at a checkout terminal, a tight race starts running in the background.

An encrypted authorization payload travels from the merchant’s reader across the card network to the issuer. If the issuer’s risk engine does not respond within roughly 100 milliseconds, the request times out and the terminal declines the card. Once you subtract network round-trips, database lookups, and balance checks, the fraud model itself has only 30 to 50 milliseconds to make a decision.

Because of that latency ceiling, payment fraud teams have long relied on gradient-boosted decision trees (like XGBoost or LightGBM) fed by pre-computed tabular counters: How many times was this card swiped in the past hour? Has this employee ever purchased from this merchant category? What is the ratio of this charge to their 30-day average?

Counting is fast, but it throws away the story behind the spend.

A $1,400 charge at an electronics store is routine for an IT manager who bought five monitors and laptop docks earlier that morning, yet deeply suspicious for a sales rep whose history consists exclusively of $18 airport salads and $35 rideshares. Tabular counters also treat every unfamiliar merchant the same way: a traveling executive paying a taxi company in Zurich for the first time looks just as unfamiliar to a counter as a stolen card number tested at an overseas electronics reseller.

Recently, engineers at Ramp shared how they moved beyond flat counters by embedding a custom sequence transformer directly inside every live card authorization on CPU. By pairing sequence-level neural representations with a dollar-weighted tree model, Ramp caught 32% more fraud dollars at the exact same alert volume, and boosted detection on the first fraudulent swipe in an attack by 40%.

Why Tabular Counters Miss the Story Behind a Swipe #

In natural language processing, nobody classifies a word by counting how often it appeared in the previous paragraph. The meaning of “bank” depends on the sequence leading up to it: “walked along the muddy river bank” versus “wired funds to the investment bank”.

Card transactions have a grammar of their own:

  • A $5 coffee at JFK Terminal 4 at 7:00 a.m., followed by a $24 in-flight Wi-Fi purchase at 9:30 a.m., makes a $45 Uber charge in San Francisco at 2:00 p.m. completely predictable.
  • Without the airport coffee and in-flight Wi-Fi, an unexpected swipe three thousand miles away looks suspicious.

Instead of writing hundreds of brittle SQL interaction rules, Ramp asked a simpler question: What if we treat a cardholder’s transaction history as a sentence, treat each swipe as a token, and train a transformer to act like autocomplete for corporate spending?

If an incoming swipe is a natural next token given the cardholder’s recent history, the model assigns it high probability. If it looks out of place, the model’s surprise spikes immediately.

Teaching a Transformer to Autocomplete Corporate Spend #

Training a deep learning model directly on fraud labels is hard: confirmed fraud is rare, and chargeback labels often arrive 30 to 60 days later.

Ramp bypassed label scarcity by pre-training their transformer in a self-supervised manner on two years of card transactions. Just like a language model learns grammar by predicting the next word in unlabeled text, Ramp’s 3-layer transformer learns normal purchasing behavior by predicting the attributes of the next transaction in a cardholder’s sequence, training in under 90 minutes on four L40S GPUs.

Taming a Two-Million-Merchant Vocabulary

Unlike text models with a fixed vocabulary, payment systems encounter millions of distinct merchants globally. Giving two million merchants dedicated embedding vectors would bloat memory and slow down the final softmax layer during live scoring.

Ramp solved this with a hybrid 47,000-slot merchant vocabulary:

  • High-frequency merchants (AWS, Uber, Delta, Starbucks) get dedicated token IDs.
  • Long-tail merchants (roughly two million smaller vendors) are hashed into shared buckets partitioned by Merchant Category Code (MCC).
  • Brand-new merchants fall back to their category-level token, scoring unfamiliar vendors immediately without retraining.

Each sequence step encodes the merchant token, category, country, dollar amount bucket, card entry method (chip, tap, online, manual), and elapsed time since the previous swipe.

Three Signals From One Forward Pass

When a live swipe arrives, the transformer runs one forward pass over the cardholder’s recent history and extracts three signals:

  1. A Sequence History Embedding: A dense vector summarizing the cardholder’s spending trajectory.
  2. Field Surprisal Scores: Linear heads predict the probability distribution for thenext transaction’s merchant, category, amount, and country. Comparing those predictions against the actual swipe yields asurprisal score (-log P(observed | history) ).
  3. Swap Scores: Inspired by BERT’s next-sentence prediction objective, the model scores whether individual fields of the incoming swipe look like they were spliced in from another user’s transaction.

Combining Sequence Surprisal With Dollar-Weighted XGBoost #

Ramp feeds the transformer’s embeddings, surprisal scores, and swap scores into a downstream XGBoost classifier alongside their existing velocity counters. Tree models still excel at sharp threshold rules, while the transformer supplies the sequential context that flat tables miss.

At the same time, Ramp changed how the XGBoost ranker is trained. Standard binary cross-entropy treats a $4 fraudulent parking meter test the same as a $15,000 fraudulent wire transfer. By weighting the XGBoost training loss by the dollar amount of each transaction, Ramp aligned the classifier with actual financial exposure.

In ablations, dollar-weighting shifted the model toward higher-value fraud, while adding the transformer features lifted overall Precision-Recall AUC by 12% and caught 40% more fraud dollars on the first fraudulent swipe in an attack sequence.

The First Rollout Failed: Chasing Latency From 145ms Down to 54ms #

Running a 3-layer transformer inside a synchronous Python payment service on CPU is where the real systems engineering begins.

When Ramp first deployed the transformer to live traffic, authorization latency doubled. Median scoring time jumped to 145 milliseconds, and p99 hit 180 milliseconds, flirting with payment network timeouts. The team rolled the model back and opened the profiler.

Surprisingly, the attention layers were not the bottleneck; matrix multiplications inside the encoder accounted for less than a third of the runtime. Three systems traps were eating the latency budget:

  1. The Batch-Padding Tax: In offline GPU training, every sequence was padded to 32 tokens. During live inference (batch_size = 1 ), padding a 4-transaction card history out to 32 meant the CPU spent over half its cycles computing attention and 47,000-way merchant projections over 28 empty slots. Passing variable-length, C-contiguous NumPy arrays directly into the encoder eliminated that wasted compute.
  2. The Python Garbage Collector Walk: To map merchant strings to their 47,000 vocabulary IDs, the worker held a Python dictionary of two million merchant names in memory. Dictionary lookup is , but holding millions ofPyObject entries on the heap meant Python’s cyclic garbage collector d the worker to traverse all two million objects on every full GC sweep, causing 80ms p99 spikes. Moving the lookup table out of Python heap objects into flat memory buffers eliminated the GC s.
  3. Train/Serve Sequence Skew: Sequence models degrade if token ordering at inference time drifts from offline training SQL. Ramp’s Feature Platform compiles a single feature definition into both batch Snowflake SQL and an in-memory streaming view for live swipes.

After removing sequence padding and bypassing Python GC traversal, median CPU scoring latency dropped from 145ms to 54ms(with the transformer itself running in under 30ms), and p99 dropped from 180ms to 84ms, producing bit-for-bit identical predictions across 80,149 live authorizations.

Scoring Transaction Surprisal on CPU in PyTorch #

If you want to experiment with sequence-surprisal features in your own fraud or anomaly detection pipeline, here is a minimal PyTorch implementation of an unpadded sequence encoder that computes both a history embedding and the negative log-likelihood (surprisal) of an incoming event on CPU:

from typing import Dict, List
import torch
import torch.nn as nn

class EventSequenceScorer(nn.Module):
    def __init__(self, vocab_size: int = 47000, d_model: int = 64):
        super().__init__()
        self.token_embed = nn.Embedding(vocab_size, d_model)
        layer = nn.TransformerEncoderLayer(
            d_model=d_model, nhead=4, dim_feedforward=128, batch_first=True
        )
        self.encoder = nn.TransformerEncoder(layer, num_layers=3)
        self.next_token_head = nn.Linear(d_model, vocab_size)

    @torch.inference_mode()
    def score_live_event(self, history_ids: List[int], current_id: int) -> Dict[str, float]:
        seq = torch.tensor([history_ids], dtype=torch.long)  # Unpadded [1, seq_len]
        hidden = self.encoder(self.token_embed(seq))

        history_vec = hidden[:, -1, :]
        log_probs = torch.log_softmax(self.next_token_head(history_vec).squeeze(0), dim=-1)

        return {
            "surprisal_score": round(-float(log_probs[current_id].item()), 4),
            "history_norm": round(float(history_vec.norm().item()), 4),
        }

In production, exporting the trained encoder to ONNX Runtime or TorchScript and keeping vocabulary lookups inside memory-mapped buffers lets a standard CPU worker score sequences in single-digit milliseconds.

Small Models Belong in the Critical Path #

It is easy to equate modern AI with 70-billion-parameter language models running on racks of H100 GPUs. Ramp’s architecture is a reminder that some of the highest-leverage applications of transformers sit at the opposite end of the spectrum.

When you have a hard 50-millisecond budget, you do not need a model that can write poetry. You need a compact, domain-specific sequence encoder trained on your own event logs, paired with disciplined systems engineering that treats padding, memory layout, and garbage collection as first-class concerns.

Further Reading & Resources

── more in #machine-learning 4 stories · sorted by recency
── more on @ramp 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/putting-a-transforme…] indexed:0 read:8min 2026-11-22 · —