{"slug": "how-i-built-a-secure-3072-dim-ai-document-indexer-using-next-js-supabase", "title": "How I Built a Secure, 3,072-Dim AI Document Indexer Using Next.js & Supabase.", "summary": "Architecture of DocuIntel, a secure AI document indexer built with Next.js and Supabase that uses Gemini 2.0 Flash for multimodal parsing and a 3,072-dimensional vector embedding model. It details how the system enforces strict multi-tenant security through explicit SQL grants and user-specific storage folders to prevent unauthorized data access. The author also explains how a hardcoded system prompt forces the AI to extract clean Markdown text and smart metadata tags, improving semantic search accuracy.", "body_md": "Building a production-ready RAG (Retrieval-Augmented Generation) application from scratch is a very difficult and time consuming.\n\nYou first get a new idea for a project or business but before you can write a single line of your actual core business logic, you find yourself spending weeks fighting with multimodal file parsing, configuring vector extensions, designing complex database architecture, and securing multi-tenant storage.\n\nThe idea behind building DocuIntel was to solve that exact infrastructure headache. I wanted a template where I could just drop in my API keys, run a single setup script, and have a fully functioning, secure AI document portal ready to go.\n\nHere is a deep dive into the architecture, some of the technical hurdles I ran into, and how I solved them.\n\n**The Multimodal Architecture**\n\nDocuIntel needs to process a wide variety of inputs—PDFs, Word docs, images (JPG/PNG), and even audio transcriptions (MP3s).\n\nAt first I was routing different file types through different third party parsing libraries like I was using tesseract OCR for extracting text from from images and pdf-parser for PDF files.\n\nBut instead of doing that, I offloaded the heavy lifting directly to Gemini 2.0 Flash. Gemini’s native multimodal capabilities handle OCR and layout analysis beautifully.\n\nTo ensure consistent, high-quality search results, I hardcoded a rigorous master SYSTEM_PROMPT inside the backend processing route (/api/process/route.ts). It forces the AI to behave like a clean data architect:\n\n``` js\nconst SYSTEM_PROMPT = `\n  You are an expert document parser for DocuIntel. \n  Your goal is to extract text with 100% accuracy for semantic search indexing.\n\n  RULES:\n  1. PRESERVE STRUCTURE: Use Markdown for headings, subheadings, and lists.\n  2. TABLES: Convert all data tables into clean Markdown table format.\n  3. NO CHAT: Do not say \"Here is the text\" or \"I have processed the file.\" \n  4. NOISE REDUCTION: Ignore headers, footers, and page numbers.\n  5. SMART METADATA: At the very end of your output, add a section called '---METADATA---' \n     and list 5-10 key entities or topics (e.g., 'Company: Acme Corp', 'Date: 2024-01-01').\n`;\n```\n\nWhy the Automated Metadata Tag Matters?\n\nBy forcing Gemini to extract smart metadata tags right into the text, the vector embedding model (gemini-embedding-2) captures high-value concepts. If a user searches for a specific company or date, the semantic search will surface the document even if that detail only appeared once in fine print.\n\n**Navigating the New Supabase Security Standards**\n\nSupabase has shifted to a \"Secure by Default\" model which will come into force by 30th May 2026.\n\nPreviously, creating a table in the public schema automatically exposed it to the Data API. Now, new tables require explicit grants, or your frontend client libraries (supabase-js) will get a 42501 permission error—even if Row-Level Security (RLS) is enabled.\n\nTo future-proof the setup, my SQL initialization script applies explicit grants directly to the authenticated user roles right after creating the tables and vector search RPC functions:\n\n```\n-- Create the documents table with 3,072-dimensional vector support\ncreate table if not exists public.documents (\n  id uuid primary key default uuid_generate_v4(),\n  file_name text not null,\n  file_url text not null,\n  content text,               \n  user_id uuid references auth.users(id),\n  user_email text not null,   \n  embedding vector(3072),     -- Optimized for high-res gemini-embedding-2\n  created_at timestamp with time zone default timezone('utc'::text, now())\n);\n\n-- EXPLICIT GRANTS (Fixes 42501 Permission Errors)\ngrant select, insert, update, delete on table public.documents to authenticated;\ngrant all on table public.documents to service_role;\n```\n\n**Hardened Multi-Tenant Storage Policies**\n\nSince the app deals with private user documents i.e. User A must never be able to discover or access User B's files, I have structured the Supabase Storage bucket so that every uploaded file is dynamically sandboxed into a folder named exactly after the user's unique authenticated ID (auth.uid()).\n\nHere are the folder-level RLS storage policies that enforce that rule on every single request:\n\n```\n-- Restrict file uploads to the user's own UID folder\ncreate policy \"Users can upload their own documents\"\non storage.objects for insert to authenticated\nwith check (\n  bucket_id = 'documents' AND \n  (storage.foldername(name))[1] = auth.uid()::text\n);\n\n-- Restrict file reading to the user's own UID folder\ncreate policy \"Users can view their own documents\"\non storage.objects for select to authenticated\nusing (\n  bucket_id = 'documents' AND \n  (storage.foldername(name))[1] = auth.uid()::text\n);\n```\n\n**Keeping the Frontend Clean & Readable**\n\nOn the frontend (Next.js 14 + TypeScript + Tailwind CSS), I wanted a highly scalable UI. A minor but common mess I see in boilerplate projects is massive, nested ternary operators in the JSX to handle dynamic file icons.\n\nTo keep things clean and performant, I grouped the file extension arrays and used standard JavaScript .includes() methods to dynamically assign Lucide React icons using a single class utility string:\n\nWrapping Up:\n\nBuilding these layers from scratch took extensive testing, debugging environment variables, and reading through updated security documentation. But once it's configured, it works like magic: a user uploads an asset, Gemini reads and indexes it with a 3,072-dimension vector embedding, and you can instantly query your documents conceptually rather than just matching keywords.\n\nIf you are planning to build an AI document product for a client or launching your own micro-SaaS, you don't have to spend your weekend configuring this infrastructure from zero.\n\nI've packaged this exact production-ready foundation—complete with the 1-click database initialization script, Next.js frontend, and pre-configured API routes—into a developer-friendly boilerplate.\n\nSetting this up from scratch took me weeks of debugging. The boilerplate gets you there in 10 minutes.\n\n[https://dhritiman.gumroad.com/l/docuintel](https://dhritiman.gumroad.com/l/docuintel)", "url": "https://wpnews.pro/news/how-i-built-a-secure-3072-dim-ai-document-indexer-using-next-js-supabase", "canonical_source": "https://dev.to/dhritich20baruah/how-i-built-a-secure-3072-dim-ai-document-indexer-using-nextjs-supabase-1bbf", "published_at": "2026-05-20 03:20:19+00:00", "updated_at": "2026-05-20 03:32:35.574429+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "developer-tools", "data", "enterprise-software"], "entities": ["DocuIntel", "Gemini 2.0 Flash", "Next.js", "Supabase", "RAG", "OCR", "tesseract"], "alternates": {"html": "https://wpnews.pro/news/how-i-built-a-secure-3072-dim-ai-document-indexer-using-next-js-supabase", "markdown": "https://wpnews.pro/news/how-i-built-a-secure-3072-dim-ai-document-indexer-using-next-js-supabase.md", "text": "https://wpnews.pro/news/how-i-built-a-secure-3072-dim-ai-document-indexer-using-next-js-supabase.txt", "jsonld": "https://wpnews.pro/news/how-i-built-a-secure-3072-dim-ai-document-indexer-using-next-js-supabase.jsonld"}}