# I Fine-Tuned Gemma 4 to Read Receipts. #Hactoberfest26

> Source: <https://dev.to/katsu/i-fine-tuned-gemma-4-to-read-receipts-hactoberfest26-56ic>
> Published: 2026-10-25 04:04:32+00:00

The project my team and I worked on during the MLH Hacktoberfest Open-Source AI Hack Day on October 24, 2026 was **ReceiptLens**.

We were a team of first-year engineering students, and we decided to take on one of the advanced challenges: fine-tuning Google's Gemma 4 model for receipt information extraction.

ReceiptLens is an AI-powered receipt extraction and expense analysis application.

The idea is straightforward. Instead of manually entering the information printed on a receipt, you upload an image and let the model extract it.

Our original task focused on four fields: merchant name, date, address, and total amount.

We used Google's Gemma 4 E2B vision-language model, fine-tuned with LoRA using Unsloth, to convert receipt images into structured JSON.

But as we continued building, we started thinking about what happens after the information is extracted.

An extracted total isn't particularly useful if it just sits inside a JSON object. And what happens when the model makes a mistake?

That led us to extend ReceiptLens beyond the original challenge.

We added support for extracting two more optional fields — the receipt or transaction number and merchant phone number — along with editable results, CSV/JSON exports, and a spending-analysis dashboard.

The expanded application is designed around this workflow:

Upload → Extract → Review → Correct → Save → Analyze

The idea is to turn a receipt extraction experiment into the foundation of a practical expense-management tool.

**What does the output look like?**

Our expanded data structure contains six fields:

`{`

  "company": "EXAMPLE STORE",

  "date": "2026-10-24",

  "address": "12 MAIN STREET",

  "total": "245.50",

  "receipt_number": "TXN-10482",

  "phone_number": "012-3456789"

}

This is an illustrative example of the expanded schema, not an actual model prediction.

There's an important distinction here.

Our fine-tuned model was trained and evaluated on the original four fields. The additional receipt and phone numbers are extracted through a separate prompted image-inference pass. Since the SROIE dataset doesn't provide labels for those additional fields, we haven't measured their extraction accuracy.

That means users still need to verify the output — which is exactly why we added manual correction.

**How we built it?**

I wanted ReceiptLens to be more than an API wrapper, so we organized it around an actual machine learning workflow.

**The dataset**

We used the ICDAR 2019 SROIE dataset, available on Hugging Face. It contains scanned receipts along with annotated information for company, date, address, and total.

The dataset version we used provides 626 training receipts and 361 test receipts. For our initial experiment, we configured 160 training examples, 20 validation examples, and a fixed 30-receipt evaluation sample.

**Turning receipt images into structured data with an open-weight vision-language model and LoRA fine-tuning.**

ReceiptLens is a Python-based experiment and interactive demo that extracts **merchant name, date, address, total amount, receipt/transaction number, and contact number** from scanned receipts. It fine-tunes **Google Gemma 4 E2B** using **Unsloth + LoRA**, then compares the adapted model with the original model on the same receipt images.

Built for **HacktoberFest Hack Day — Open-Source AI, Theme 3B: Receipt Information Extraction**.

**Experimental project, not production-ready OCR.** Results below are from a 30-receipt evaluation shared from a Colab run. On a 30-receipt test sample, complete extraction accuracy improved from 20% to 30%, with improvements in merchant, date, and total extraction.

**Tech Stack:** Python, Gemma 4 E2B, Unsloth, LoRA, Hugging Face, PyTorch, Google Colab, and Gradio

**Gemma 4 E2B** is an open-weight vision-language model. It can process an image alongside a text instruction, which makes it suitable for extracting information directly from receipt photographs.

Full fine-tuning would have been expensive and difficult with our available computing resources. So we used **LoRA** — Low-Rank Adaptation. Rather than updating every weight in the original model, LoRA trains small additional adapter weights.

We used **Unsloth** to manage the vision fine-tuning workflow, with 4-bit model loading to reduce memory requirements.

Our project configuration started with LoRA rank 8, a learning rate of 0.0002, and 40 training steps.

We also kept preprocessing and evaluation separate from the training code so we could compare the original and fine-tuned models under consistent conditions.

**Hugging Face** provided the receipt dataset and access to the open-weight model ecosystem.

**Google Colab** and **PyTorch** provided the environment for GPU-based inference and training

And **Gradio** helped us build a usable Python interface without needing a separate frontend framework.

This was a particularly interesting experience because we were all first-year engineering students, and most of us were still relatively new to practical AI development.

We chose an advanced problem despite that.

Of course, things didn't work perfectly from beginning to end.

There were dependency warnings, GPU memory considerations, and the occasional frustrating Colab error.

At one point, Colab reported that CUDA wasn't available. At another, our application couldn't find the expected project directory.

It was a reminder that building an AI application involves much more than getting a model to respond.

You have to understand your environment, manage dependencies, keep track of files and checkpoints, test the actual output, and make sure everything still works when you demonstrate it.

I also learned to be more careful about how we describe model performance.

An improvement from 20% to 30% sounds impressive as a relative percentage. But when you say that it means going from six correct receipts to nine, it becomes easier to understand both the improvement and how much work remains.

GitHub: [https://github.com/KAZUTO0/ReceiptLens](https://github.com/KAZUTO0/ReceiptLens)

Event: MLH Hacktoberfest Open-Source AI Hack Day, October 24, 2026

Team: PARADOX

Name: Sameer Kumar Karn