{"slug": "candidate-compliance-agent-building-a-multilingual-rag-system-for-tamil-nadu", "title": "Candidate Compliance Agent: Building a Multilingual RAG System for Tamil Nadu Election Affidavits", "summary": "A developer built Candidate Compliance Agent, a multilingual RAG system that lets users query Tamil Nadu election candidate affidavits in English or Tamil. The system extracts text from scanned PDFs using Tesseract OCR, creates semantic documents per candidate topic, and uses a multilingual sentence transformer for embeddings. It also features intent routing and a persistent memory service for context across sessions.", "body_md": "Every election cycle in India, candidates are not easily accessible to many so they don't the many information about candidate's they are voting for.\n\nCandidate Compliance Agent is an AI agent that lets anyone query Tamil Nadu election candidate affidavits in plain English or Tamil. Ask it about assets, criminal records, income declarations, education — it answers from the actual documents.\n\nLive demo: [https://affidavit-rag-frontend.vercel.app](https://affidavit-rag-frontend.vercel.app)\n\nThis wasn't a clean dataset project. The raw materials were:\n\nScanned PDFs (~6MB each)\n\nMixed Tamil and English text\n\nStamp paper backgrounds with notary seals overlaid on text\n\nInconsistent formatting across candidates\n\nReact Frontend (Vercel)\n\n↓\n\nFlask Backend (Railway)\n\n↓\n\nHindsight Memory (recall past context)\n\n↓\n\nIntent Router\n\n↓\n\nStructured Query OR Semantic RAG\n\n↓\n\nGroq LLM (Llama 3.3 70B)\n\n↓\n\nHindsight Memory (retain new context)\n\n↓\n\nResponse + Follow-up Questions\n\nTechnical Feature Implemnentaion Walkthrough\n\nThe first challenge was extracting text from scanned PDFs. I used Tesseract with Tamil + English language models:\n\n```\npython\n\ntext = pytesseract.image_to_string(\n    image,\n    lang=\"eng+tam\",\n    config=\"--psm 6\"\n)\n```\n\n--psm 6 treats each page as a uniform text block — critical for structured affidavit forms.\n\nEach PDF was converted to images at 300 DPI using pdf2image, then OCR'd page by page. Output was stored as JSON with metadata:\n\n```\n{\n  \"text\": \"...\",\n  \"metadata\": {\n    \"candidate\": \"MKStalin\",\n    \"party\": \"DMK\",\n    \"constituency\": \"KOLATHUR\",\n    \"page\": 3\n  }\n}\n```\n\nThe biggest architectural decision was how to chunk the data.\n\nNaive approach: Split each page into 500-token chunks with overlap. This gives you 600+ noisy chunks where assets, criminal records, and education are all mixed together.\n\nBetter approach: Build one semantic document per topic per candidate.\n\nEach candidate gets 6 documents:\n\nIdentity and contact\n\nEducation\n\nCriminal cases (structured)\n\nCriminal detail (raw OCR text)\n\nIncome tax\n\nAssets and liabilities\n\n``` python\ndef build_docs(candidate, raw_pages):\n    docs = []\n    docs.append({\"section\": \"criminal_cases\", \"text\": f\"\"\"\nCandidate: {name}\nParty: {party}\nSection: Criminal Cases\n\nPending Cases: {pending}\nConvicted Cases: {convicted}\nSummary: {\"No criminal record\" if pending == 0 else f\"{pending} pending\"}\n\"\"\"})\n    # ... repeat for each section\n    return docs\n```\n\n35 candidates × 6 sections = 210 clean semantic documents instead of 600+ noisy chunks.\n\nResult: When someone asks \"does certain candidate have criminal cases?\" the retrieval hits the criminal_cases document directly instead of a random page slice.\n\nI used paraphrase-multilingual-MiniLM-L12-v2 for embeddings — a sentence transformer model that handles Tamil and English in the same vector space.\n\nmodel = SentenceTransformer(\"paraphrase-multilingual-MiniLM-L12-v2\")\n\nembeddings = model.encode(texts, batch_size=32).tolist()\n\n```\ncollection.add(\n    documents=texts,\n    embeddings=embeddings,\n    metadatas=[{\n        \"candidate\": d[\"candidate\"],\n        \"party\": d[\"party\"],\n        \"section\": d[\"section\"],\n    } for d in all_docs],\n    ids=[f\"{d['candidate']}_{d['section']}\" for d in all_docs]\n)\n```\n\nthere are 5 intent category based on that the query is passed through\n\n``` python\ndef detect_intent(query):\n    q = query.lower()\n    for party in PARTIES:\n        if party in q:\n            return \"party\"\n    for pattern in EDUCATION_PATTERNS:\n        if re.search(pattern, q):\n            return \"education\"\n    for word in SEMANTIC_KEYWORDS:\n        if word in q:\n            return \"semantic\"\n    return \"unknown\"\n```\n\nHindsight is a persistent memory service that stores and recalls context across sessions. When a user comes back the next day and asks a follow-up question, the agent remembers what they asked before.\n\n``` python\nfrom hindsight_client import Hindsight\n\nclient = Hindsight(\n    base_url=\"https://api.hindsight.vectorize.io\",\n    api_key=os.getenv(\"HINDSIGHT_API_KEY\")\n)\n\ndef recall_memory(session_id, query):\n    result = client.recall(\n        bank_id=\"candidate-rag\",\n        query=f\"Session: {session_id}\\nQuestion: {query}\"\n    )\n    return \"\\n\".join([m.text for m in result.results])\n\ndef retain_memory(session_id, memory):\n    client.retain(\n        bank_id=\"candidate-rag\",\n        content=memory,\n        metadata={\"session_id\": session_id}\n    )\n```\n\npast_memory = recall_memory(session_id, query)\n\nprompt = f\"\"\"\n\nPrevious context: {past_memory}\n\nContext from affidavits: {retrieved_context}\n\nQuestion: {query}\n\n\"\"\"\n\nanswer = ask_groq(prompt)\n\nretain_memory(session_id, f\"User asked: {query}. Answer: {answer[:200]}\")\n\nWhy this matters: A voter researching a candidate can ask multiple questions across multiple sessions and get progressively better, context-aware answers.\n\nAfter every answer, the LLM generates 4 follow-up questions based on the query, retrieved context, and memory:\n\nANSWER:\n\nPoornima has no pending criminal cases.\n\nFOLLOW_UP_QUESTIONS:\n\nThese appear as clickable buttons — clicking fills the input box directly.\n\nBackend: Flask + gunicorn on Railway\n\nFrontend: React on Vercel\n\nVector DB: ChromaDB (persistent, committed to GitHub)\n\nLLM: Groq API (Llama 3.3 70B Versatile)\n\nMemory: Hindsight\n\nOCR quality determines everything downstream. Bad OCR produces bad embeddings. The --psm 6 config change improved Tamil extraction significantly.\n\nSemantic chunking beats token chunking. 210 topic-focused documents retrieve better than 600+ random page slices.\n\nPersistent memory changes the interaction model. Without Hindsight, every conversation starts cold. With it, the agent builds understanding over time.\n\nReal-world data is always messier than tutorials suggest. Stamp paper backgrounds, notary seal overlays, mixed scripts — none of this appears in RAG tutorials.\n\nWhat's Next\n\nScale to all 234 Tamil Nadu constituencies (~4500 PDFs)\n\nAdd proactive alerts when new affidavits are filed\n\nSupport voice queries in Tamil", "url": "https://wpnews.pro/news/candidate-compliance-agent-building-a-multilingual-rag-system-for-tamil-nadu", "canonical_source": "https://dev.to/hari_babu_36078517bf8a8de/candidate-compliance-agent-building-a-multilingual-rag-system-for-tamil-nadu-election-affidavits-3mcm", "published_at": "2026-07-07 01:57:29+00:00", "updated_at": "2026-07-07 02:29:10.486966+00:00", "lang": "en", "topics": ["large-language-models", "natural-language-processing", "ai-agents", "developer-tools", "machine-learning"], "entities": ["Tesseract", "paraphrase-multilingual-MiniLM-L12-v2", "Groq", "Llama 3.3 70B", "Hindsight", "Vercel", "Railway", "Vectorize"], "alternates": {"html": "https://wpnews.pro/news/candidate-compliance-agent-building-a-multilingual-rag-system-for-tamil-nadu", "markdown": "https://wpnews.pro/news/candidate-compliance-agent-building-a-multilingual-rag-system-for-tamil-nadu.md", "text": "https://wpnews.pro/news/candidate-compliance-agent-building-a-multilingual-rag-system-for-tamil-nadu.txt", "jsonld": "https://wpnews.pro/news/candidate-compliance-agent-building-a-multilingual-rag-system-for-tamil-nadu.jsonld"}}