{"slug": "i-fine-tuned-gemma-4-to-read-receipts-hactoberfest26", "title": "I Fine-Tuned Gemma 4 to Read Receipts. #Hactoberfest26", "summary": "A team of first-year engineering students built ReceiptLens, an open-source receipt-extraction demo that fine-tunes Google's Gemma 4 E2B vision-language model with LoRA via Unsloth to convert receipt images into structured JSON. On a 30-receipt evaluation sample, complete extraction accuracy rose from 20% to 30% over the base model, with gains in merchant, date, and total fields. The team extended the app with editable results, CSV/JSON export, and a spending dashboard, though the added receipt-number and phone-number fields come from a separate prompted inference pass and remain unmeasured.", "body_md": "The project my team and I worked on during the MLH Hacktoberfest Open-Source AI Hack Day on October 24, 2026 was **ReceiptLens**.\n\nWe were a team of first-year engineering students, and we decided to take on one of the advanced challenges: fine-tuning Google's Gemma 4 model for receipt information extraction.\n\nReceiptLens is an AI-powered receipt extraction and expense analysis application.\n\nThe idea is straightforward. Instead of manually entering the information printed on a receipt, you upload an image and let the model extract it.\n\nOur original task focused on four fields: merchant name, date, address, and total amount.\n\nWe used Google's Gemma 4 E2B vision-language model, fine-tuned with LoRA using Unsloth, to convert receipt images into structured JSON.\n\nBut as we continued building, we started thinking about what happens after the information is extracted.\n\nAn extracted total isn't particularly useful if it just sits inside a JSON object. And what happens when the model makes a mistake?\n\nThat led us to extend ReceiptLens beyond the original challenge.\n\nWe added support for extracting two more optional fields — the receipt or transaction number and merchant phone number — along with editable results, CSV/JSON exports, and a spending-analysis dashboard.\n\nThe expanded application is designed around this workflow:\n\nUpload → Extract → Review → Correct → Save → Analyze\n\nThe idea is to turn a receipt extraction experiment into the foundation of a practical expense-management tool.\n\n**What does the output look like?**\n\nOur expanded data structure contains six fields:\n\n`{`\n\n  \"company\": \"EXAMPLE STORE\",\n\n  \"date\": \"2026-10-24\",\n\n  \"address\": \"12 MAIN STREET\",\n\n  \"total\": \"245.50\",\n\n  \"receipt_number\": \"TXN-10482\",\n\n  \"phone_number\": \"012-3456789\"\n\n}\n\nThis is an illustrative example of the expanded schema, not an actual model prediction.\n\nThere's an important distinction here.\n\nOur fine-tuned model was trained and evaluated on the original four fields. The additional receipt and phone numbers are extracted through a separate prompted image-inference pass. Since the SROIE dataset doesn't provide labels for those additional fields, we haven't measured their extraction accuracy.\n\nThat means users still need to verify the output — which is exactly why we added manual correction.\n\n**How we built it?**\n\nI wanted ReceiptLens to be more than an API wrapper, so we organized it around an actual machine learning workflow.\n\n**The dataset**\n\nWe used the ICDAR 2019 SROIE dataset, available on Hugging Face. It contains scanned receipts along with annotated information for company, date, address, and total.\n\nThe dataset version we used provides 626 training receipts and 361 test receipts. For our initial experiment, we configured 160 training examples, 20 validation examples, and a fixed 30-receipt evaluation sample.\n\n**Turning receipt images into structured data with an open-weight vision-language model and LoRA fine-tuning.**\n\nReceiptLens is a Python-based experiment and interactive demo that extracts **merchant name, date, address, total amount, receipt/transaction number, and contact number** from scanned receipts. It fine-tunes **Google Gemma 4 E2B** using **Unsloth + LoRA**, then compares the adapted model with the original model on the same receipt images.\n\nBuilt for **HacktoberFest Hack Day — Open-Source AI, Theme 3B: Receipt Information Extraction**.\n\n**Experimental project, not production-ready OCR.** Results below are from a 30-receipt evaluation shared from a Colab run. On a 30-receipt test sample, complete extraction accuracy improved from 20% to 30%, with improvements in merchant, date, and total extraction.\n\n**Tech Stack:** Python, Gemma 4 E2B, Unsloth, LoRA, Hugging Face, PyTorch, Google Colab, and Gradio\n\n**Gemma 4 E2B** is an open-weight vision-language model. It can process an image alongside a text instruction, which makes it suitable for extracting information directly from receipt photographs.\n\nFull fine-tuning would have been expensive and difficult with our available computing resources. So we used **LoRA** — Low-Rank Adaptation. Rather than updating every weight in the original model, LoRA trains small additional adapter weights.\n\nWe used **Unsloth** to manage the vision fine-tuning workflow, with 4-bit model loading to reduce memory requirements.\n\nOur project configuration started with LoRA rank 8, a learning rate of 0.0002, and 40 training steps.\n\nWe also kept preprocessing and evaluation separate from the training code so we could compare the original and fine-tuned models under consistent conditions.\n\n**Hugging Face** provided the receipt dataset and access to the open-weight model ecosystem.\n\n**Google Colab** and **PyTorch** provided the environment for GPU-based inference and training\n\nAnd **Gradio** helped us build a usable Python interface without needing a separate frontend framework.\n\nThis was a particularly interesting experience because we were all first-year engineering students, and most of us were still relatively new to practical AI development.\n\nWe chose an advanced problem despite that.\n\nOf course, things didn't work perfectly from beginning to end.\n\nThere were dependency warnings, GPU memory considerations, and the occasional frustrating Colab error.\n\nAt one point, Colab reported that CUDA wasn't available. At another, our application couldn't find the expected project directory.\n\nIt was a reminder that building an AI application involves much more than getting a model to respond.\n\nYou have to understand your environment, manage dependencies, keep track of files and checkpoints, test the actual output, and make sure everything still works when you demonstrate it.\n\nI also learned to be more careful about how we describe model performance.\n\nAn improvement from 20% to 30% sounds impressive as a relative percentage. But when you say that it means going from six correct receipts to nine, it becomes easier to understand both the improvement and how much work remains.\n\nGitHub: [https://github.com/KAZUTO0/ReceiptLens](https://github.com/KAZUTO0/ReceiptLens)\n\nEvent: MLH Hacktoberfest Open-Source AI Hack Day, October 24, 2026\n\nTeam: PARADOX\n\nName: Sameer Kumar Karn", "url": "https://wpnews.pro/news/i-fine-tuned-gemma-4-to-read-receipts-hactoberfest26", "canonical_source": "https://dev.to/katsu/i-fine-tuned-gemma-4-to-read-receipts-hactoberfest26-56ic", "published_at": "2026-10-25 04:04:32+00:00", "updated_at": "2026-10-25 04:11:15.845080+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "computer-vision", "ai-tools"], "entities": ["Google", "Gemma 4 E2B", "Unsloth", "Hugging Face", "PyTorch", "Gradio", "SROIE", "ReceiptLens"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-fine-tuned-gemma-4-to-read-receipts-hactoberfest26", "markdown": "https://wpnews.pro/news/i-fine-tuned-gemma-4-to-read-receipts-hactoberfest26.md", "text": "https://wpnews.pro/news/i-fine-tuned-gemma-4-to-read-receipts-hactoberfest26.txt", "jsonld": "https://wpnews.pro/news/i-fine-tuned-gemma-4-to-read-receipts-hactoberfest26.jsonld"}}