{"slug": "real-estate-scraper-token-efficient-structured-property-crawler", "title": "Real Estate Scraper– Token-efficient structured property crawler", "summary": "Kodomoppoi released Real Estate Scraper, an open-source pipeline that uses LLM Structured Outputs to discover real estate websites, crawl listing pages, and extract structured property data without site-specific scrapers. The tool features AI pre-curation, DOM token condensation (~75% noise reduction), and country-aware extraction, with support for Google Gemini (4,000,000 TPM) and Groq Cloud models, and is designed to handle portals in any language.", "body_md": "## An intelligent, token-efficient pipeline designed to automatically discover real estate websites, crawl listing pages, and extract structured property data using LLM Structured Outputs.\n\nVIDEO SHOWCASE : [https://youtu.be/xcjRcRTZt-I?si=thM-5ZTkQCM7sZmI](https://youtu.be/xcjRcRTZt-I?si=thM-5ZTkQCM7sZmI)\n\n[\n](/Kodomoppoi/Real-estate-Scrapper/blob/main/docs/screenshots/dashboard_overview.png)*Interactive Real Estate Explorer with KPI metrics, AI curated portals, and property listing tables.*\n\n⚠️ Important Note on Crawl Settings, Model Selection & Rate Limits (TPM):\n\nExponential Crawl Scaling: The total number of crawled pages, AI token expenditure, and overall execution time increaseexponentiallywith theCrawl Settings(`AI Curated Sites`\n\n×`Pages per Site`\n\n).\n\nFor Testing: It is strongly recommended to set1 Curated Siteand1 Page per Site (to verify location results quickly and conserve API quota.`1 / 1`\n\n)For Full Scrapes: Increase to higher limits (e.g., 2–5 sites, 2–3 pages) once you confirm portal accessibility.\n\nAI Provider Capacity & Extraction Yield (TPM Limits):\n\nGoogle Gemini (: Features a 4,000,000 TPM limit and 1M context window. It effortlessly extracts`gemini-3.6-flash`\n\n) [Recommended for Maximum Volume]20 to 25 complete listings per pagein seconds.Groq Cloud (Free Tier): Provides ultra-fast LPU inference, but large 70B/120B models (e.g.`openai/gpt-oss-120b`\n\n) have a tight ~6,000 TPM cap that can rate-limit full-page extraction down to only 1–2 listings per request.For Groq, use high-throughput modelslike`qwen/qwen3.6-27b`\n\n,`groq/compound-mini`\n\n, or`openai/gpt-oss-20b`\n\n(20,000+ TPM).\n\nTraditional web scrapers rely on brittle CSS/XPath selectors that constantly break whenever real estate portals change layouts, obfuscate class names, or render dynamic JavaScript feeds.\n\nBy leveraging an **AI semantic extraction engine** with Pydantic structured outputs, this pipeline extracts clean, structured property data across any portal worldwide in any language without writing or maintaining site-specific scrapers.\n\n```\nLocation & Filters (e.g. Ipanema, Rio de Janeiro / Brasil)\n       │\n       ▼\n1. Direct Listing Discovery (DuckDuckGo Engine - Scaled Candidates)\n       │\n       ▼\n2. AI Pre-Curation & Index Matching (Preserves exact deep routes & filters noise)\n       │\n       ▼\n3. Dual-Engine Crawler with Early Site Abandonment (Validates Page 1 first)\n       │\n       ▼\n4. DOM Token Condensation (~75% Noise Reduction)\n       │\n       ▼\n5. Country-Aware Structured Extraction (Suites, Amenities, Highlights, Financing)\n       │\n       ▼\n6. Fail-Fast Resiliency, Pandas Deduplication & CSV/JSON Export\n```\n\n**Problem Solved**: Real estate homepages are landing pages with complex search forms, while hardcoding portal URLs breaks across cities. The discovery engine performs targeted natural-language queries (e.g.,`apartamentos a venda em Ipanema Rio de Janeiro`\n\n), retrieving live deep listing search URLs dynamically without form automation.\n\n**Problem Solved**: Asking an LLM to rewrite or generate URLs causes link hallucinations or truncates deep paths to root domains (e.g. returning`zapimoveis.com.br/`\n\n). By numbering candidate URLs and having the LLM select 1-based integer indexes (`[1, 2]`\n\n), exact deep paths are preserved with 100% fidelity.\n\n**Problem Solved**: Crawling multi-page routes on dead or anti-bot blocked portals wastes network time and AI tokens. Page 1 is validated first; if 0 listings are found or access is blocked, all remaining pages for that site are aborted immediately.\n\n**Problem Solved**: Raw webpage HTML contains 80,000+ characters of SVGs, cookie banners, navigation menus, and ads. The regex cleaner strips noise and filters text to retain only property-relevant signals (`R$`\n\n,`m²`\n\n,`quartos`\n\n,`amenities`\n\n), fitting within fast LLM token windows.\n\n**Problem Solved**: Property listings are unstructured and written in diverse regional formats. The Pydantic schema extracts normalized attributes (`price`\n\n,`area_m2`\n\n,`bedrooms`\n\n,`suites`\n\n,`amenities`\n\n,`financing_accepted`\n\n) localized to the target country's official language.\n\n**Problem Solved**: Rate limits and quota exhaustion previously led to long wait loops. The engine enforces immediate fail-fast handling on critical errors and compiles extracted records into deduplicated Pandas DataFrames for instant CSV (`utf-8-sig`\n\n) and JSON export.\n\n```\n# 1. Clone the repository\ngit clone https://github.com/Kodomoppoi/Real-estate-Scrapper.git\ncd Real-estate-Scrapper\n\n# 2. Run the application\nstreamlit run app.py\n```\n\nConfigure your API key directly in the Web Dashboard sidebar or create a `.env`\n\nfile in the project root:\n\n```\n# Google Gemini (Recommended - Free Tier available)\nGEMINI_API_KEY=AIzaSy...\nLLM_MODEL=gemini-3.6-flash\n\n# Or OpenAI\nOPENAI_API_KEY=sk-...\nLLM_MODEL=gpt-4o-mini\n```\n\n-\n**Real-Time Terminal Activity**: Live streaming terminal box embedded directly inside the browser showing search, crawling, and AI steps. -\n**In-App API Key Manager**: Test and save Gemini, OpenAI, Groq, or OpenRouter API keys directly from the sidebar. -\n**KPI Metrics Cards**: Total listings, estimated average market price, median area ($m^2$ ), and top neighborhood. -\n**Interactive Listings Table**: Client-side keyword search, neighborhood filters, bedroom filters, price range filters, and direct links to original ads. -\n**Extra Details Tab**: Amenity frequency rankings, financing status breakdown, and Price-per-$m^2$ calculation rankings. -\n**One-Click Export**: Export consolidated datasets to CSV (Excel compatible with`utf-8-sig`\n\n) and JSON.\n\nRun the automated test suite with pytest:\n\n```\npytest tests/\n```\n\n", "url": "https://wpnews.pro/news/real-estate-scraper-token-efficient-structured-property-crawler", "canonical_source": "https://github.com/Kodomoppoi/Real-estate-Scrapper", "published_at": "2026-08-31 11:21:48+00:00", "updated_at": "2026-08-31 11:53:21.744686+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-tools", "ai-products"], "entities": ["Kodomoppoi", "Google Gemini", "Groq Cloud", "DuckDuckGo", "Pydantic"], "alternates": {"html": "https://wpnews.pro/news/real-estate-scraper-token-efficient-structured-property-crawler", "markdown": "https://wpnews.pro/news/real-estate-scraper-token-efficient-structured-property-crawler.md", "text": "https://wpnews.pro/news/real-estate-scraper-token-efficient-structured-property-crawler.txt", "jsonld": "https://wpnews.pro/news/real-estate-scraper-token-efficient-structured-property-crawler.jsonld"}}