{"slug": "how-we-built-scraping-ai-turning-500-enterprise-projects-into-a-self-serve-api", "title": "How We Built Scraping AI: Turning 500+ Enterprise Projects Into a Self-Serve API", "summary": "PigData, a Japanese managed data extraction provider, has launched Scraping AI, a self-serve developer API built on Django REST Framework, Celery, RabbitMQ, and PostgreSQL. The system codifies more than 500 prior enterprise scraping projects into a versioned state-machine pipeline that chains modular crawlers, BM25 and vector-based page rankers, and LLM extractors using OpenAI and Gemini models. The company says the architecture can run anywhere from 10 to 10,000 concurrent crawling jobs on the same infrastructure.", "body_md": "**From managed enterprise scraping at PigData to a high-scale developer API in six months.**\n\n[!NOTE]\n\n**TL;DR / Engineering Retrospective:**\n\n**Origin:** PigData delivered 500+ custom enterprise scraping projects (spanning Tier-1 automotive, e-commerce, and mega-bank financial institutions) via managed services before codifying core scraping patterns into a self-serve developer API.\n**Tech Stack:** Django REST Framework + Celery + RabbitMQ + PostgreSQL (`VersionedModel` optimistic locking) + S3 / MinIO storage.\n**Key Innovation:** A versioned state-machine pipeline (`InputState`) powering modular Crawlers, LLM Extractors (OpenAI / Gemini), and BM25 + Vector Rankers.\n**Zero-Risk Trial:** Get **200 free tokens** (no credit card required) at [https://pig-data.jp/service/scraping-ai/](https://pig-data.jp/service/scraping-ai/).\n\nFor years, PigData operated as a managed data extraction service in Japan, building bespoke scrapers for enterprise data pipelines. Whether extracting product catalogs or market intelligence, our engineers handled the end-to-end process.\n\nThe problem? **Every project started from scratch.** Even when two clients needed similar data (e.g., e-commerce product listings), we were rebuilding identical parsing logic, browser automation routines, and anti-bot retry loops.\n\nWe faced four core engineering bottlenecks:\n\nWe needed an architecture capable of running 10 jobs or 10,000 concurrent crawling jobs on the exact same infrastructure.\n\nWe chose a Python stack centered around **Django REST Framework (DRF)**, **Celery**, **RabbitMQ**, and **PostgreSQL**:\n\n```\n┌─────────────────┐       ┌─────────────────┐       ┌─────────────────┐\n│   Django API    │ ─────▶│   RabbitMQ      │ ─────▶│ Celery Workers  │\n│   (DRF Layer)   │       │ (Message Queue) │       │ (Distributed)   │\n└─────────────────┘       └─────────────────┘       └─────────────────┘\n         │                                                   │\n         ▼                                                   ▼\n┌─────────────────┐                                 ┌─────────────────┐\n│ PostgreSQL State│ ◀───────────────────────────────│ S3 / MinIO      │\n│ (Optimistic Lock│                                 │ Data Exports    │\n└─────────────────┘                                 └─────────────────┘\n```\n\n`httpx`, `BeautifulSoup`, `pydantic`, `openai`, and `google-genai` directly without cross-language serialization overhead.\nEvery data extraction job follows a predictable lifecycle:\n\n```\n[Keywords / Search Query]\n          │\n          ▼\n   ┌─────────────┐\n   │ URL Finder  │ (Discovers link graph up to max_depth)\n   └─────────────┘\n          │\n          ▼\n   ┌─────────────┐\n   │  Crawler    │ (Fetches HTML via httpx or headless browser)\n   └─────────────┘\n          │\n          ▼\n   ┌─────────────┐\n   │ AI Ranker   │ (Ranks pages via BM25 + Vector embeddings)\n   └─────────────┘\n          │\n          ▼\n   ┌─────────────┐\n   │ LLM Extractor│ (Applies JSON Schema via GPT-4o / Gemini)\n   └─────────────┘\n          │\n          ▼\n   [Structured JSON / CSV Export]\n```\n\nWe codified this workflow into a single state machine backed by our central `InputState` model:\n\n```\nclass InputState(VersionedModel):\n    \"\"\"Central state machine model for an extraction task.\"\"\"\n\n    # Configuration & Instructions\n    base_url = models.URLField(max_length=2048)\n    user_instruction = models.TextField()\n    schema_instruction = models.TextField()\n\n    # Pipeline Execution State\n    site_type = models.CharField(max_length=20, choices=[('general', 'General'), ('ec', 'E-Commerce')])\n    auto_flow = models.BooleanField(default=True)\n    current_step = models.CharField(max_length=50, choices=PIPELINE_STEPS)\n\n    # Step Status Trackers\n    keyword_generator_status = models.CharField(max_length=20, choices=STATUS_CHOICES, default='PENDING')\n    url_finder_status = models.CharField(max_length=20, choices=STATUS_CHOICES, default='PENDING')\n    url_crawler_status = models.CharField(max_length=20, choices=STATUS_CHOICES, default='PENDING')\n    url_ranker_status = models.CharField(max_length=20, choices=STATUS_CHOICES, default='PENDING')\n    schema_generator_status = models.CharField(max_length=20, choices=STATUS_CHOICES, default='PENDING')\n    extraction_status = models.CharField(max_length=20, choices=STATUS_CHOICES, default='PENDING')\n```\n\nWith dozens of Celery workers processing URLs concurrently, multiple workers attempted to update `InputState` status simultaneously, causing lost updates.\n\n**Solution:** We built optimistic locking into `VersionedModel`:\n\n```\nclass VersionedModel(models.Model):\n    version = models.IntegerField(default=0)\n\n    class Meta:\n        abstract = True\n\n    def save(self, *args, **kwargs):\n        if self.pk:\n            affected = self.__class__.objects.filter(\n                pk=self.pk, version=self.version\n            ).update(version=models.F('version') + 1, **kwargs.get('update_fields_dict', {}))\n\n            if not affected:\n                raise ConcurrencyError(f\"Version conflict on {self.__class__.__name__} ID {self.pk}\")\n            self.version += 1\n            return\n        super().save(*args, **kwargs)\n```\n\n`ConcurrencyManager`)\nCalling ORM `.save()` inside loops on 10,000 discovered URLs overwhelmed PostgreSQL. We implemented a custom `ConcurrencyManager`:\n\n``` python\nclass ConcurrencyManager(models.Manager):\n    def bulk_claim_and_create(self, urls_data: list, state_id: int):\n        \"\"\"Batch upserts URLs using PostgreSQL bulk ON CONFLICT handling.\"\"\"\n        existing_urls = set(\n            self.filter(input_state_id=state_id, url__in=[u['url'] for u in urls_data])\n            .values_list('url', flat=True)\n        )\n        new_objects = [\n            self.model(input_state_id=state_id, url=u['url'], status='PENDING')\n            for u in urls_data if u['url'] not in existing_urls\n        ]\n        self.bulk_create(new_objects, batch_size=1000, ignore_conflicts=True)\n```\n\nInstead of complex billing per CPU second, we implemented a real-time transactional token ledger:\n\n```\nclass TokenLedger(models.Model):\n    user = models.ForeignKey(User, on_delete=models.CASCADE)\n    amount = models.IntegerField()  # Negative for debits, positive for credits\n    action = models.CharField(max_length=50)  # e.g., 'task.start', 'url.extractor'\n    balance_after = models.IntegerField()\n    timestamp = models.DateTimeField(auto_now_add=True)\n```\n\nWhile our backend handles complex async state machines, celery queues, and token ledgers, developers interact with our official published PyPI package ([`scraping-ai`](https://pypi.org/project/scraping-ai/)):\n\n```\npip install scraping-ai\npython\nfrom scraping_ai import ScrapingAIClient\n\nclient = ScrapingAIClient(api_key=\"YOUR_API_KEY\")\n\n# Extract web data directly in one step (polls until finished)\ndata = client.extract(\n    url=\"https://example.com/products\",\n    schema={\"title\": \"string\", \"price\": \"number\", \"in_stock\": \"boolean\"}\n)\n\nprint(data.results)\n```\n\nLooking back at our 6-month journey:\n\nStop writing fragile scrapers and fixing broken CSS selectors.\n\n`https://pypi.org/project/scraping-ai/`\n**Scraping AI** ([https://pig-data.jp/service/scraping-ai/](https://pig-data.jp/service/scraping-ai/)) is developed and operated by **indigodata Inc.**, an AI venture subsidiary of **SMS DataTech Co., Ltd.** (Tokyo, Japan). Built upon PigData's track record of 500+ enterprise data extraction projects, Scraping AI provides a self-serve LLM extraction API for developers worldwide.", "url": "https://wpnews.pro/news/how-we-built-scraping-ai-turning-500-enterprise-projects-into-a-self-serve-api", "canonical_source": "https://dev.to/amandeep-sms/how-we-built-scraping-ai-turning-500-enterprise-projects-into-a-self-serve-api-oii", "published_at": "2026-09-15 01:27:15+00:00", "updated_at": "2026-09-15 02:00:48.240473+00:00", "lang": "en", "topics": ["ai-tools", "ai-products", "ai-agents", "developer-tools", "ai-infrastructure"], "entities": ["PigData", "Scraping AI", "Django REST Framework", "Celery", "RabbitMQ", "PostgreSQL", "OpenAI", "Gemini"], "alternates": {"html": "https://wpnews.pro/news/how-we-built-scraping-ai-turning-500-enterprise-projects-into-a-self-serve-api", "markdown": "https://wpnews.pro/news/how-we-built-scraping-ai-turning-500-enterprise-projects-into-a-self-serve-api.md", "text": "https://wpnews.pro/news/how-we-built-scraping-ai-turning-500-enterprise-projects-into-a-self-serve-api.txt", "jsonld": "https://wpnews.pro/news/how-we-built-scraping-ai-turning-500-enterprise-projects-into-a-self-serve-api.jsonld"}}