Candidate Compliance Agent: Building a Multilingual RAG System for Tamil Nadu Election Affidavits

A developer built Candidate Compliance Agent, a multilingual RAG system that lets users query Tamil Nadu election candidate affidavits in English or Tamil. The system extracts text from scanned PDFs using Tesseract OCR, creates semantic documents per candidate topic, and uses a multilingual sentence transformer for embeddings. It also features intent routing and a persistent memory service for context across sessions.

Every election cycle in India, candidates are not easily accessible to many so they don't the many information about candidate's they are voting for. Candidate Compliance Agent is an AI agent that lets anyone query Tamil Nadu election candidate affidavits in plain English or Tamil. Ask it about assets, criminal records, income declarations, education — it answers from the actual documents. Live demo: https://affidavit-rag-frontend.vercel.app https://affidavit-rag-frontend.vercel.app This wasn't a clean dataset project. The raw materials were: Scanned PDFs ~6MB each Mixed Tamil and English text Stamp paper backgrounds with notary seals overlaid on text Inconsistent formatting across candidates React Frontend Vercel ↓ Flask Backend Railway ↓ Hindsight Memory recall past context ↓ Intent Router ↓ Structured Query OR Semantic RAG ↓ Groq LLM Llama 3.3 70B ↓ Hindsight Memory retain new context ↓ Response + Follow-up Questions Technical Feature Implemnentaion Walkthrough The first challenge was extracting text from scanned PDFs. I used Tesseract with Tamil + English language models: python text = pytesseract.image to string image, lang="eng+tam", config="--psm 6" --psm 6 treats each page as a uniform text block — critical for structured affidavit forms. Each PDF was converted to images at 300 DPI using pdf2image, then OCR'd page by page. Output was stored as JSON with metadata: { "text": "...", "metadata": { "candidate": "MKStalin", "party": "DMK", "constituency": "KOLATHUR", "page": 3 } } The biggest architectural decision was how to chunk the data. Naive approach: Split each page into 500-token chunks with overlap. This gives you 600+ noisy chunks where assets, criminal records, and education are all mixed together. Better approach: Build one semantic document per topic per candidate. Each candidate gets 6 documents: Identity and contact Education Criminal cases structured Criminal detail raw OCR text Income tax Assets and liabilities python def build docs candidate, raw pages : docs = docs.append {"section": "criminal cases", "text": f""" Candidate: {name} Party: {party} Section: Criminal Cases Pending Cases: {pending} Convicted Cases: {convicted} Summary: {"No criminal record" if pending == 0 else f"{pending} pending"} """} ... repeat for each section return docs 35 candidates × 6 sections = 210 clean semantic documents instead of 600+ noisy chunks. Result: When someone asks "does certain candidate have criminal cases?" the retrieval hits the criminal cases document directly instead of a random page slice. I used paraphrase-multilingual-MiniLM-L12-v2 for embeddings — a sentence transformer model that handles Tamil and English in the same vector space. model = SentenceTransformer "paraphrase-multilingual-MiniLM-L12-v2" embeddings = model.encode texts, batch size=32 .tolist collection.add documents=texts, embeddings=embeddings, metadatas= { "candidate": d "candidate" , "party": d "party" , "section": d "section" , } for d in all docs , ids= f"{d 'candidate' } {d 'section' }" for d in all docs there are 5 intent category based on that the query is passed through python def detect intent query : q = query.lower for party in PARTIES: if party in q: return "party" for pattern in EDUCATION PATTERNS: if re.search pattern, q : return "education" for word in SEMANTIC KEYWORDS: if word in q: return "semantic" return "unknown" Hindsight is a persistent memory service that stores and recalls context across sessions. When a user comes back the next day and asks a follow-up question, the agent remembers what they asked before. python from hindsight client import Hindsight client = Hindsight base url="https://api.hindsight.vectorize.io", api key=os.getenv "HINDSIGHT API KEY" def recall memory session id, query : result = client.recall bank id="candidate-rag", query=f"Session: {session id}\nQuestion: {query}" return "\n".join m.text for m in result.results def retain memory session id, memory : client.retain bank id="candidate-rag", content=memory, metadata={"session id": session id} past memory = recall memory session id, query prompt = f""" Previous context: {past memory} Context from affidavits: {retrieved context} Question: {query} """ answer = ask groq prompt retain memory session id, f"User asked: {query}. Answer: {answer :200 }" Why this matters: A voter researching a candidate can ask multiple questions across multiple sessions and get progressively better, context-aware answers. After every answer, the LLM generates 4 follow-up questions based on the query, retrieved context, and memory: ANSWER: Poornima has no pending criminal cases. FOLLOW UP QUESTIONS: These appear as clickable buttons — clicking fills the input box directly. Backend: Flask + gunicorn on Railway Frontend: React on Vercel Vector DB: ChromaDB persistent, committed to GitHub LLM: Groq API Llama 3.3 70B Versatile Memory: Hindsight OCR quality determines everything downstream. Bad OCR produces bad embeddings. The --psm 6 config change improved Tamil extraction significantly. Semantic chunking beats token chunking. 210 topic-focused documents retrieve better than 600+ random page slices. Persistent memory changes the interaction model. Without Hindsight, every conversation starts cold. With it, the agent builds understanding over time. Real-world data is always messier than tutorials suggest. Stamp paper backgrounds, notary seal overlays, mixed scripts — none of this appears in RAG tutorials. What's Next Scale to all 234 Tamil Nadu constituencies ~4500 PDFs Add proactive alerts when new affidavits are filed Support voice queries in Tamil