# I Got Tired of Checking 15 AI Sites Every Morning, So I Built a Pipeline That Does It While I Sleep > Source: > Published: 2026-10-24 02:37:35+00:00 *This is a submission for the [Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady](https://dev.to/challenges/hacktoberfest-week3-2026-10-19)* ## What I Built **Global Intelligence Graph** is a data ingestion engine that keeps an eye on the AI world so I don't have to. Staying current in AI means opening arXiv, Papers with Code, GitHub, a handful of news sites and several job boards, then copy-pasting the interesting bits somewhere. I dreaded that chore, so I automated it. One command now: - **Scrapes research:** the latest AI papers from arXiv and Papers with Code, plus GitHub star counts for papers that have repos. - **Finds startups and products:** it queries the GitHub API for repositories and extracts startup and product entities from them. - **Reads the news:** it pulls articles from 5 major AI news sources and keeps only those published in the last 24 hours. - **Watches the job market:** it scrapes 5 AI job boards with Playwright and converts relative dates ("3 hours ago") into absolute ISO timestamps, again keeping only the last 24 hours. - **Cleans up the mess:** it resolves entity names against a canonical dictionary using fuzzy matching, so "Open AI" and "OpenAI" become one entity. - **Hands over tidy data:** everything lands in SQLite and is exported as structured CSVs in`output/` . Anyone who tracks the AI ecosystem (students, researchers, recruiters, founders) gets to be a little lazier: the morning tab-hopping becomes opening a folder of CSVs. ## Code # Global Intelligence Graph: Data Ingestion Engine This repository contains the data ingestion pipeline for the AI Engineer demo task. It is designed for massive bulk extraction, 24-hour freshness tracking, and resilient LLM extraction using a multi-tier fallback architecture. ## Setup Instructions 1. **Python Environment** : Ensure you have Python 3.11+ installed. 2. **Install Dependencies** 3. **Configure Environment Variables** : Open the`.env` file in the root directory and add your API keys:*(Note: You can run the pipeline with just one of these keys, but the fallback chain works best when all three are present.)* ## Running the Pipeline To execute the pipeline: ### Pipeline Flow 1. **Phase I (Arxiv, Papers with Code & GitHub)** : Scrapes the latest AI research papers from Arxiv and Papers with Code. Queries the GitHub API for repositories to extract Startup and Product… ## How I Built It The pipeline runs in four phases: 1. **Phase I, research and GitHub.** Scrapers collect papers from arXiv and Papers with Code. The GitHub API supplies repositories (the source of startup and product entities) and star counts. 2. **Phase II, news and jobs.** News articles are fetched from 5 sources. Job listings come from 5 JavaScript-heavy boards, so I used Playwright to render them. 3. **Phase III, LLM extraction.** An`LLMExtractor` turns messy raw payloads into validated**Pydantic** models. It also enforces the 24-hour freshness rule, so stale items never reach the database. 4. **Phase IV, entity resolution.** Company names across startups, products and jobs are normalized with fuzzy matching against a canonical dictionary. Results are stored in an intermediate **SQLite** database (`data_ingestion.db`) that preserves relationships and raw payloads, then exported to CSV. ### The multi-tier LLM fallback Free-tier LLM APIs hit `429` rate limits constantly when you're doing bulk extraction. Rather than crash or pay for a bigger plan, the extractor falls back across providers: **Gemini → Groq (Llama 3) → DeepSeek** If one provider rate-limits, the next takes over, and a 5-second processing delay keeps the load gentle. You can run it with just one key, but it's most resilient with all three. Two of the three tiers are open-weight model families (Llama 3 and DeepSeek), so the extraction layer is not tied to a single closed vendor. ## Why Does Open Innovation Matter? Bulk extraction is exactly where closed APIs hurt: thousands of small calls, strict quotas, and a bill that grows with every page scraped. - **No single point of failure.** Because Llama 3 and DeepSeek are open-weight, many providers can serve them. My pipeline can swap models and hosts without rewriting the extraction logic, and a rate limit on one provider is a detour, not an outage. - **Cost stays near zero.** Open-weight models on free or cheap tiers let me run a daily pipeline without a subscription. - **Swappable by design.** The fallback chain is just an ordered list. Adding a locally hosted open-weight model (for example through Ollama) as another tier, or the first one, is a small change that would make the pipeline work fully offline with no data leaving my machine. - **Open tools all the way down.** Playwright, SQLite, Pydantic and the GitHub and arXiv ecosystems are all open, which made the whole thing easy to build and inspect.