# I tested 11 AI models on Indian GST, UPI and lakh-crore. Three famous ones got Puducherry wrong.

> Source: <https://dev.to/ankithm1006/i-tested-11-ai-models-on-indian-gst-upi-and-lakh-crore-three-famous-ones-got-puducherry-wrong-53e9>
> Published: 2026-10-06 10:34:02+00:00

*This is a submission for the [Kaggle Benchmarking Challenge](https://dev.to/challenges/kaggle-2026-09-23)*

I build software for Indian small businesses (POS, billing, payments), and this month I've been contributing Kestra workflow blueprints for Indian finance: GSTIN validation, UPI end-of-day close, GSTR-2B reconciliation. Every one of them exists because one small mistake costs real money:

So I asked: **can AI models handle everyday Indian business data?** I built a Kaggle benchmark with 6 tasks, 16 cases each:

| Task | What the model has to do | 
|---|---|
| `india_gstin_check_digit` | Validate a GSTIN and compute its Luhn mod 36 check character (algorithm given) | 
| `india_gstin_from_memory` | Same, without being told the algorithm | 
| `india_gst_tax_split` | Split GST into CGST + SGST, CGST + UTGST or IGST, with real traps: supplies to an SEZ unit in your own state, and Union Territories with and without a legislature | 
| `india_invoice_number_rules` | Apply GST invoice-number rules (16 characters, only letters/digits/ `-` /`/` , unique per financial year) with look-alike traps: en dash, division slash, trailing space, 17 characters, extra leading zero | 
| `india_lakh_crore_formats` | Indian digit grouping (12,34,56,789), lakh/crore conversions, cheque amounts in words | 
| `india_upi_reconciliation` | Match 18–30 UPI bills against a settlement report and list the bills whose money never arrived, arrived short, or share a UPI reference | 

How it's graded:

`python-stdnum` (0 disagreements). The other answer keys use the same logic as my Kestra blueprints.
I picked a mix to see what matters most: size, "thinking", or being open-weight.

(Grok 4.6 and gpt-oss-120b were unavailable through Kaggle's model proxy while I ran this, so they're not on the board.)

Overall (average of task scores): **GPT-5.5 1.00 · Gemini 3.7 Flash 1.00 · Gemini 3.5 Flash 0.99 · Claude Sonnet 5 0.98 · Gemma 4 0.97 · DeepSeek R1 0.96 · gpt-oss-20b 0.92 · Gemini 3.1 Flash-Lite 0.75 · Qwen3 235B 0.51 · Claude Haiku 4.5 0.45 · GPT-5.4 nano 0.34**

This was my favourite result. Under the CGST Act, section 2(103), a Union Territory **with its own legislature** (Puducherry, Delhi) counts as a *State*. So a sale inside Puducherry is **CGST + SGST**. UTGST is only for UTs without a legislature, like Chandigarh or Ladakh.

Bigger isn't uniformly better. Models fail on *different* local rules.

On the GSTIN check character, every model that reasons step by step scored **100%**, and every fast model scored **6–12%**. That's no better than guessing. Small open models that reason (gpt-oss-20b, Gemma 4) beat bigger fast ones. In an early test, one model needed about **15,000 thinking tokens and 84 seconds for a single GSTIN**. Correct isn't always cheap.

A supply to a Special Economic Zone unit in your *own* state is still an inter-state supply, so it's IGST (IGST Act, section 7(5)(b)). Claude Haiku, gpt-oss-20b and GPT-5.4 nano split it into CGST + SGST anyway.

The small models made the most dangerous kind of mistake, being off by exactly one digit group:

`500000000` (it's `50000000`)` 986.4` (it's `98.64`)` 8,263,425,718` instead of `8,26,34,25,718`
On GST invoice numbers, the look-alikes caught the small models: a trailing space, a 17-character number, and a number with an extra leading zero (a different number, so it's allowed) that they called a duplicate. All the big models were perfect here.

On days with 18–30 UPI bills, weaker models **missed** bills whose money never arrived (Claude Haiku 44%, Qwen 25%). A false alarm costs a minute. A missed bill is silent lost money.

On Kaggle's score-vs-cost chart, the efficient frontier runs through the open-weight **gpt-oss-20b** and **Gemma 4 31B**. They get most of the way to the top score at a small fraction of the cost.

👉 [Indian Business Data: GST, UPI and Lakh-Crore on Kaggle](https://www.kaggle.com/benchmarks/ankith111111111/indian-business-data-gst-upi-and-lakh-crore)

All 6 tasks and their notebooks are public, so you can run them on any model.

*I used an AI assistant to draft this post. The idea, benchmarking,  the Indian accounting rules and the final review are mine.*