# You Deleted BM25 the Day You Bought a Vector Database. Your Fraud Queue Noticed

> Source: <https://dev.to/fagundesv/you-deleted-bm25-the-day-you-bought-a-vector-database-your-fraud-queue-noticed-3d2j>
> Published: 2026-07-30 03:00:00+00:00

BM25 is a ranking function from the 90s. No GPU, no embedding API, no per-token bill. And in a fraud operation it still wins half the queries — because fraud analysts don't search in prose. They search in identifiers.

Watch a fraud analyst's search history for a day: decline code `4863`

. Rule `VEL-013`

. BIN `529904`

. Promo code `FIRSTBUY20`

. Order `#48213`

. These aren't concepts. They're strings — and "semantically similar" is the wrong answer when the user typed the literal thing.

Review-queue records from a mid-size e-commerce shop:

```
queue = [
    "Order 48213 held: BIN 529904, decline code 4863 on first attempt, "
    "retry succeeded, shipping to address seen in 3 prior chargebacks.",
    "Order 48377 held: velocity rule VEL-013, four orders in 11 minutes, "
    "same device fingerprint, different cards.",
    "Order 48391 released: false positive on gift purchase, billing and "
    "shipping mismatch explained by gift note. Customer verified via 3DS.",
    "Order 48402 held: promo FIRSTBUY20 redeemed by account created 8 "
    "minutes earlier, email domain registered yesterday.",
    "Order 48440 held: card declined with code 4863 twice, third card "
    "succeeded. Classic card-testing tail behavior.",
]
```

Dense retrieval on the identifier query:

``` python
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
E = model.encode(queue, normalize_embeddings=True)

q = model.encode(["decline code 4863"], normalize_embeddings=True)
for score, rec in sorted(zip((E @ q.T).ravel(), queue), reverse=True):
    print(f"[{score:.3f}] {rec[:62]}")
[0.471] Order 48391 released: false positive on gift purchase, billing
[0.446] Order 48213 held: BIN 529904, decline code 4863 on first atte
[0.412] Order 48440 held: card declined with code 4863 twice, third c
```

A record with *zero* occurrences of the code outranks one of the two records that contain it. The embedding is matching the general vibe of "payment trouble," not the token. Now BM25:

``` python
from rank_bm25 import BM25Okapi
bm25 = BM25Okapi([r.lower().split() for r in queue])

for score, rec in sorted(zip(bm25.get_scores("decline code 4863".split()), queue),
                         reverse=True)[:3]:
    print(f"[{score:.2f}] {rec[:62]}")
[4.87] Order 48440 held: card declined with code 4863 twice, third c
[4.12] Order 48213 held: BIN 529904, decline code 4863 on first atte
[0.31] Order 48391 released: false positive on gift purchase, billing
```

Exactly the two records containing the code, cleanly separated from the noise. `4863`

is a rare token, and rare tokens are precisely what the IDF term in BM25 was designed to reward — thirty years before anyone embedded anything.

Nobody's arguing for BM25-only. On the pattern queries — "gift purchase with mismatched addresses," "card testing behavior" — dense retrieval wins just as decisively, finding cases that share no vocabulary with the query.

The argument is against the silent deletion. Teams migrate to a vector database, the keyword index quietly dies, and nobody measures what happened to identifier queries — because the recall dashboard, if it exists at all, was built from pattern-style test queries. The failure mode is invisible by construction.

Run both. Fuse ranks (RRF, k=60, no tuning). The keyword index costs kilobytes per document and roughly nothing to serve.

The expensive part of your retrieval stack isn't the vector database. It's the queries it silently gets wrong — and in a fraud queue, a silently missed match is a pattern that gets to repeat.

What did your recall numbers do when you turned keyword search off — did anyone even measure?

*I'm Vinicius Fagundes — principal data engineer and MBA lecturer in São Paulo. I build fraud and risk analytics pipelines for e-commerce through vf-insights.com.*