{"slug": "putting-a-transformer-inside-every-card-authorization-how-ramp-achieves-sub-30ms", "title": "Putting a Transformer Inside Every Card Authorization: How Ramp Achieves Sub-30ms Inference", "summary": "Engineers at Ramp embedded a custom 3-layer sequence transformer directly inside live card authorizations, running on CPU within a 30-50ms decision budget. The model is pre-trained self-supervised on two years of transaction sequences to predict the next swipe's attributes, then paired with a dollar-weighted tree model, yielding 32% more fraud dollars caught at the same alert volume and 40% better detection of the first fraudulent swipe in an attack.", "body_md": "# Putting a Transformer Inside Every Card Authorization: How Ramp Achieves Sub-30ms Inference\n\nWhy real-time payment SLAs break standard deep learning deployments, how Ramp trained a sequence transformer to autocomplete card spending, and the systems tricks required to run it on CPU in millisec\n\nWhen you tap a corporate credit card at a checkout terminal, a tight race starts running in the background.\n\nAn encrypted authorization payload travels from the merchant’s reader across the card network to the issuer. If the issuer’s risk engine does not respond within roughly 100 milliseconds, the request times out and the terminal declines the card. Once you subtract network round-trips, database lookups, and balance checks, **the fraud model itself has only 30 to 50 milliseconds to make a decision.**\n\nBecause of that latency ceiling, payment fraud teams have long relied on gradient-boosted decision trees (like XGBoost or LightGBM) fed by pre-computed tabular counters: *How many times was this card swiped in the past hour? Has this employee ever purchased from this merchant category? What is the ratio of this charge to their 30-day average?*\n\nCounting is fast, but it throws away the story behind the spend.\n\nA $1,400 charge at an electronics store is routine for an IT manager who bought five monitors and laptop docks earlier that morning, yet deeply suspicious for a sales rep whose history consists exclusively of $18 airport salads and $35 rideshares. Tabular counters also treat every unfamiliar merchant the same way: a traveling executive paying a taxi company in Zurich for the first time looks just as unfamiliar to a counter as a stolen card number tested at an overseas electronics reseller.\n\nRecently, engineers at **Ramp** shared how they moved beyond flat counters by embedding a **custom sequence transformer directly inside every live card authorization on CPU**. By pairing sequence-level neural representations with a dollar-weighted tree model, Ramp caught **32% more fraud dollars** at the exact same alert volume, and boosted detection on the first fraudulent swipe in an attack by **40%**.\n\n## Why Tabular Counters Miss the Story Behind a Swipe\n\nIn natural language processing, nobody classifies a word by counting how often it appeared in the previous paragraph. The meaning of *“bank”* depends on the sequence leading up to it: *“walked along the muddy river bank”* versus *“wired funds to the investment bank”*.\n\nCard transactions have a grammar of their own:\n\n- A $5 coffee at JFK Terminal 4 at 7:00 a.m., followed by a $24 in-flight Wi-Fi purchase at 9:30 a.m., makes a $45 Uber charge in San Francisco at 2:00 p.m. completely predictable.\n- Without the airport coffee and in-flight Wi-Fi, an unexpected swipe three thousand miles away looks suspicious.\n\nInstead of writing hundreds of brittle SQL interaction rules, Ramp asked a simpler question: **What if we treat a cardholder’s transaction history as a sentence, treat each swipe as a token, and train a transformer to act like autocomplete for corporate spending?**\n\nIf an incoming swipe is a natural next token given the cardholder’s recent history, the model assigns it high probability. If it looks out of place, the model’s surprise spikes immediately.\n\n## Teaching a Transformer to Autocomplete Corporate Spend\n\nTraining a deep learning model directly on fraud labels is hard: confirmed fraud is rare, and chargeback labels often arrive 30 to 60 days later.\n\nRamp bypassed label scarcity by pre-training their transformer in a **self-supervised** manner on two years of card transactions. Just like a language model learns grammar by predicting the next word in unlabeled text, Ramp’s 3-layer transformer learns normal purchasing behavior by predicting the attributes of the next transaction in a cardholder’s sequence, training in under 90 minutes on four L40S GPUs.\n\n### Taming a Two-Million-Merchant Vocabulary\n\nUnlike text models with a fixed vocabulary, payment systems encounter millions of distinct merchants globally. Giving two million merchants dedicated embedding vectors would bloat memory and slow down the final softmax layer during live scoring.\n\nRamp solved this with a hybrid **47,000-slot merchant vocabulary**:\n\n- **High-frequency merchants** (AWS, Uber, Delta, Starbucks) get dedicated token IDs.\n- **Long-tail merchants** (roughly two million smaller vendors) are hashed into shared buckets partitioned by Merchant Category Code (MCC).\n- **Brand-new merchants** fall back to their category-level token, scoring unfamiliar vendors immediately without retraining.\n\nEach sequence step encodes the merchant token, category, country, dollar amount bucket, card entry method (chip, tap, online, manual), and elapsed time since the previous swipe.\n\n### Three Signals From One Forward Pass\n\nWhen a live swipe arrives, the transformer runs one forward pass over the cardholder’s recent history and extracts three signals:\n\n1. **A Sequence History Embedding:** A dense vector summarizing the cardholder’s spending trajectory.\n2. **Field Surprisal Scores:** Linear heads predict the probability distribution for the*next* transaction’s merchant, category, amount, and country. Comparing those predictions against the actual swipe yields a**surprisal score** (`-log P(observed | history)` ).\n3. **Swap Scores:** Inspired by BERT’s next-sentence prediction objective, the model scores whether individual fields of the incoming swipe look like they were spliced in from another user’s transaction.\n\n## Combining Sequence Surprisal With Dollar-Weighted XGBoost\n\nRamp feeds the transformer’s embeddings, surprisal scores, and swap scores into a downstream **XGBoost classifier** alongside their existing velocity counters. Tree models still excel at sharp threshold rules, while the transformer supplies the sequential context that flat tables miss.\n\nAt the same time, Ramp changed how the XGBoost ranker is trained. Standard binary cross-entropy treats a $4 fraudulent parking meter test the same as a $15,000 fraudulent wire transfer. By weighting the XGBoost training loss by the **dollar amount** of each transaction, Ramp aligned the classifier with actual financial exposure.\n\nIn ablations, dollar-weighting shifted the model toward higher-value fraud, while adding the transformer features lifted overall Precision-Recall AUC by **12%** and caught **40% more fraud dollars** on the first fraudulent swipe in an attack sequence.\n\n## The First Rollout Failed: Chasing Latency From 145ms Down to 54ms\n\nRunning a 3-layer transformer inside a synchronous Python payment service on CPU is where the real systems engineering begins.\n\nWhen Ramp first deployed the transformer to live traffic, **authorization latency doubled**. Median scoring time jumped to **145 milliseconds**, and p99 hit **180 milliseconds**, flirting with payment network timeouts. The team rolled the model back and opened the profiler.\n\nSurprisingly, the attention layers were not the bottleneck; matrix multiplications inside the encoder accounted for less than a third of the runtime. Three systems traps were eating the latency budget:\n\n1. **The Batch-Padding Tax:** In offline GPU training, every sequence was padded to 32 tokens. During live inference (`batch_size = 1` ), padding a 4-transaction card history out to 32 meant the CPU spent over half its cycles computing attention and 47,000-way merchant projections over 28 empty slots. Passing variable-length, C-contiguous NumPy arrays directly into the encoder eliminated that wasted compute.\n2. **The Python Garbage Collector Walk:** To map merchant strings to their 47,000 vocabulary IDs, the worker held a Python dictionary of two million merchant names in memory. Dictionary lookup is , but holding millions of`PyObject` entries on the heap meant Python’s cyclic garbage collector paused the worker to traverse all two million objects on every full GC sweep, causing 80ms p99 spikes. Moving the lookup table out of Python heap objects into flat memory buffers eliminated the GC pauses.\n3. **Train/Serve Sequence Skew:** Sequence models degrade if token ordering at inference time drifts from offline training SQL. Ramp’s Feature Platform compiles a single feature definition into both batch Snowflake SQL and an in-memory streaming view for live swipes.\n\nAfter removing sequence padding and bypassing Python GC traversal, median CPU scoring latency dropped from **145ms to 54ms**(with the transformer itself running in under 30ms), and p99 dropped from **180ms to 84ms**, producing bit-for-bit identical predictions across 80,149 live authorizations.\n\n## Scoring Transaction Surprisal on CPU in PyTorch\n\nIf you want to experiment with sequence-surprisal features in your own fraud or anomaly detection pipeline, here is a minimal PyTorch implementation of an unpadded sequence encoder that computes both a history embedding and the negative log-likelihood (surprisal) of an incoming event on CPU:\n\n``` python\nfrom typing import Dict, List\nimport torch\nimport torch.nn as nn\n\nclass EventSequenceScorer(nn.Module):\n    def __init__(self, vocab_size: int = 47000, d_model: int = 64):\n        super().__init__()\n        self.token_embed = nn.Embedding(vocab_size, d_model)\n        layer = nn.TransformerEncoderLayer(\n            d_model=d_model, nhead=4, dim_feedforward=128, batch_first=True\n        )\n        self.encoder = nn.TransformerEncoder(layer, num_layers=3)\n        self.next_token_head = nn.Linear(d_model, vocab_size)\n\n    @torch.inference_mode()\n    def score_live_event(self, history_ids: List[int], current_id: int) -> Dict[str, float]:\n        seq = torch.tensor([history_ids], dtype=torch.long)  # Unpadded [1, seq_len]\n        hidden = self.encoder(self.token_embed(seq))\n\n        history_vec = hidden[:, -1, :]\n        log_probs = torch.log_softmax(self.next_token_head(history_vec).squeeze(0), dim=-1)\n\n        return {\n            \"surprisal_score\": round(-float(log_probs[current_id].item()), 4),\n            \"history_norm\": round(float(history_vec.norm().item()), 4),\n        }\n```\n\nIn production, exporting the trained encoder to ONNX Runtime or TorchScript and keeping vocabulary lookups inside memory-mapped buffers lets a standard CPU worker score sequences in single-digit milliseconds.\n\n## Small Models Belong in the Critical Path\n\nIt is easy to equate modern AI with 70-billion-parameter language models running on racks of H100 GPUs. Ramp’s architecture is a reminder that some of the highest-leverage applications of transformers sit at the opposite end of the spectrum.\n\nWhen you have a hard 50-millisecond budget, you do not need a model that can write poetry. You need a compact, domain-specific sequence encoder trained on your own event logs, paired with disciplined systems engineering that treats padding, memory layout, and garbage collection as first-class concerns.\n\n### Further Reading & Resources\n\n- **[Ramp Builders: Putting a Transformer Inside Every Card Authorization](https://builders.ramp.com/post/sequence-aware-card-fraud)** : The original engineering deep dive by Willi Gierke and the Ramp Risk team covering their sequence transformer and latency profiling.\n- **[PyTorch CPU Inference & Performance Tuning Guide](https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html)** : Official techniques for optimizing single-batch CPU inference and avoiding memory allocation overhead.\n- **[ONNX Runtime Performance Tuning](https://onnxruntime.ai/docs/performance/tune-performance.html)** : Guide to quantizing and executing small transformer encoders under strict p99 latency budgets.\n- **[Python](https://docs.python.org/3/library/gc.html#gc.freeze)** `gc.freeze()` **[Documentation](https://docs.python.org/3/library/gc.html#gc.freeze)** : How Python services exclude large static lookup structures from cyclic garbage collection sweeps in production workers.", "url": "https://wpnews.pro/news/putting-a-transformer-inside-every-card-authorization-how-ramp-achieves-sub-30ms", "canonical_source": "https://mlnotes.substack.com/p/putting-a-transformer-inside-every", "published_at": "2026-11-22 14:04:22+00:00", "updated_at": "2026-11-22 14:12:54.113126+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence", "ai-infrastructure", "neural-networks"], "entities": ["Ramp", "XGBoost", "LightGBM", "Nvidia L40S"], "also_reported_by": [], "takeaway": "Pre-training a small transformer self-supervised on transaction sequences and pairing its surprise signal with a dollar-weighted tree model caught 32% more fraud dollars at identical alert volume while fitting a 30-50ms CPU inference budget.", "recommendation": {"advice": "When latency budgets rule out GPU serving, pre-train a compact transformer on sequences and run it on CPU alongside a tree model, as Ramp did to hit sub-30ms decisions.", "playbook": "inference-serving", "playbook_url": "https://wpnews.pro/playbooks/inference-serving"}, "alternates": {"html": "https://wpnews.pro/news/putting-a-transformer-inside-every-card-authorization-how-ramp-achieves-sub-30ms", "markdown": "https://wpnews.pro/news/putting-a-transformer-inside-every-card-authorization-how-ramp-achieves-sub-30ms.md", "text": "https://wpnews.pro/news/putting-a-transformer-inside-every-card-authorization-how-ramp-achieves-sub-30ms.txt", "jsonld": "https://wpnews.pro/news/putting-a-transformer-inside-every-card-authorization-how-ramp-achieves-sub-30ms.jsonld"}}