{"slug": "feed-your-rag-pipeline-full-news-articles-not-snippets-python", "title": "Feed your RAG pipeline full news articles, not snippets (Python)", "summary": "A Python tutorial published on NestDaddy's API documentation shows how to build a roughly 100-line retrieval-augmented generation pipeline that fetches full news article text via the NestDaddy API's news/search and extract endpoints rather than relying on two-line search snippets. The guide chunks articles by paragraph at about 800 characters, carries the last paragraph of each chunk into the next to preserve facts spanning paragraphs, filters boilerplate such as newsletter prompts, and skips pages under 500 characters to drop paywalled or JavaScript-rendered content. It requires only the requests library and a NestDaddy API key, with a free 7-day key offering 5,000 calls, and the retrieval step can later be swapped for embeddings.", "body_md": "# Feed your RAG pipeline full news articles, not snippets (Python)\n\nSearch results give a RAG system two lines per article, which is not enough to answer real questions. This guide fetches recent news, pulls the full readable text, chunks it by paragraph, retrieves the best passages and builds a cited context block. About 100 lines, no vector database needed to start.\n\nMost retrieval-augmented generation (RAG) demos over the news stop at search results: a title, a URL and a two-line snippet. That is fine for \"what happened?\" and useless for \"what exactly did they announce, and who disagreed?\". The answer is in paragraph nine, and the snippet never had it.\n\nThis guide builds a small pipeline that works on the **full text** of recent articles:\n\n1. Find recent articles on a topic.\n2. Fetch each article's readable body, without menus, ads and comment sections.\n3. Clean and chunk it by paragraph, keeping the source and date with every chunk.\n4. Retrieve the best chunks for a question.\n5. Build a numbered context block a model can cite.\n\nEverything runs in one Python file with `requests` only. You can swap the retrieval step for embeddings later without touching the rest.\n\n## Setup\n\n```\npip install requests\nexport NESTDADDY_API_KEY=\"your_key\"\n```\n\nThe examples use the [NestDaddy API](https://nestdaddy.com/apis): `news/search` for recent articles and `extract` for the full text of any URL. A free 7-day key (5,000 calls, no card) is enough to follow along.\n\n## Step 1: find recent articles\n\n``` python\nimport math\nimport os\nimport re\nfrom collections import Counter\n\nimport requests\n\nAPI = \"https://nestdaddy.com/api/v1\"\nKEY = os.environ[\"NESTDADDY_API_KEY\"]\nHEADERS = {\"X-API-Key\": KEY}\n\ndef find_articles(query, days=3, country=None, limit=5):\n    \"\"\"Recent news articles about a topic: title, url, source, date.\"\"\"\n    params = {\"q\": query, \"days\": days, \"limit\": limit}\n    if country:\n        params[\"country\"] = country\n    r = requests.get(f\"{API}/news/search\", headers=HEADERS, params=params, timeout=30)\n    r.raise_for_status()\n    return r.json().get(\"results\", [])\n```\n\nKeep `days` small for questions about current events. A wider window brings in older articles that confidently contradict newer ones.\n\n## Step 2: fetch the full text\n\n``` python\ndef full_text(url):\n    \"\"\"The readable body of one article, without menus, ads or comments.\"\"\"\n    r = requests.get(f\"{API}/extract\", headers=HEADERS, params={\"url\": url}, timeout=30)\n    r.raise_for_status()\n    data = r.json()\n    return (data.get(\"text\") or \"\") if data.get(\"success\") else \"\"\n```\n\nSome pages will not give you much: paywalls, consent walls and pages that build their content with JavaScript. Step 4 skips anything shorter than 500 characters, which catches most of them.\n\n## Step 3: clean and chunk by paragraph\n\nFixed-size chunks cut sentences in half and split a quote from the person who said it. Chunking by paragraph keeps meaning together. Each new chunk also starts with the last paragraph of the previous one, so a fact that spans two paragraphs is never lost.\n\n```\nBOILERPLATE = re.compile(\n    r\"sign up for|newsletter|follow us on|subscribe to|read more:|all rights reserved|\"\n    r\"click here|share this article\", re.I)\n\ndef chunk(text, size=800):\n    \"\"\"Pack whole paragraphs into ~size-character chunks.\n\n    Each new chunk starts with the last paragraph of the previous one, so a\n    fact that spans two paragraphs is never cut in half.\n    \"\"\"\n    paras = [p.strip() for p in text.split(\"\\n\")\n             if len(p.strip()) > 40 and not BOILERPLATE.search(p)]\n    chunks, cur = [], []\n    for p in paras:\n        if cur and sum(len(x) for x in cur) + len(p) > size:\n            chunks.append(\"\\n\".join(cur))\n            cur = cur[-1:]\n        cur.append(p)\n    if cur:\n        chunks.append(\"\\n\".join(cur))\n    return chunks\n```\n\nThe `BOILERPLATE` filter is small on purpose. Publishers drop newsletter prompts and social links in the middle of articles, and those lines are noise to retrieval. Add patterns as you meet them.\n\n## Step 4: build the corpus with metadata\n\nEvery chunk carries its title, URL, source and publish date. That metadata is what lets the model cite, and lets you filter by date or source later.\n\n``` python\ndef build_corpus(query, **kw):\n    corpus = []\n    for a in find_articles(query, **kw):\n        text = full_text(a[\"url\"])\n        if len(text) < 500:  # paywall stub or failed extraction: skip it\n            continue\n        for i, c in enumerate(chunk(text)):\n            corpus.append({\n                \"text\": c,\n                \"title\": a[\"title\"],\n                \"url\": a[\"url\"],\n                \"source\": a.get(\"source_domain\"),\n                \"published\": a.get(\"published_timestamp\"),\n                \"chunk\": i,\n            })\n    return corpus\n```\n\n## Step 5: retrieve the best chunks\n\nTo keep the example dependency-free, here is BM25, the keyword ranking behind most search engines. For a few dozen chunks from fresh news it works well, because questions about events usually share exact words with the answer (names, places, products).\n\n``` python\nWORD = re.compile(r\"[a-z0-9]+\")\n\ndef bm25_top(corpus, question, k=5, k1=1.5, b=0.75):\n    \"\"\"Tiny BM25 so the example has no dependencies; swap in embeddings later.\"\"\"\n    docs = [Counter(WORD.findall(d[\"text\"].lower())) for d in corpus]\n    avg = sum(sum(d.values()) for d in docs) / max(len(docs), 1)\n    terms = set(WORD.findall(question.lower()))\n    df = {t: sum(1 for d in docs if t in d) for t in terms}\n    n = len(docs)\n    scores = []\n    for i, d in enumerate(docs):\n        length = sum(d.values())\n        s = 0.0\n        for t in terms:\n            if t not in d:\n                continue\n            idf = math.log(1 + (n - df[t] + 0.5) / (df[t] + 0.5))\n            s += idf * d[t] * (k1 + 1) / (d[t] + k1 * (1 - b + b * length / avg))\n        scores.append((s, i))\n    return [corpus[i] for s, i in sorted(scores, reverse=True)[:k] if s > 0]\n```\n\nWhen your corpus grows or questions get vaguer, add embeddings and combine both scores (hybrid search). Keyword matching stays useful for names and numbers, which embeddings often blur.\n\n## Step 6: a context block the model can cite\n\n``` python\ndef context_block(chunks):\n    \"\"\"Numbered sources the model can cite as [1], [2]...\"\"\"\n    lines = []\n    for n, c in enumerate(chunks, 1):\n        lines.append(f\"[{n}] {c['title']} ({c['source']}, {c['published']})\\n{c['url']}\\n{c['text']}\")\n    return \"\\n\\n\".join(lines)\n\nif __name__ == \"__main__\":\n    corpus = build_corpus(\"AI\", days=3, country=\"GB\", limit=4)\n    print(len(corpus), \"chunks from full articles\")\n    top = bm25_top(corpus, \"What are employers being told to prepare for?\")\n    print(context_block(top)[:1500])\n```\n\nPut that block in your prompt under an instruction such as:\n\n```\nAnswer using only the sources below. Cite them as [1], [2]. Mention the date of\neach fact. If the sources do not answer the question, say so.\n```\n\nOn a test run, four full articles became about 50 chunks, and the top result for the question above was the paragraph that actually quoted the minister's guidance, not the headline.\n\n## Things that matter in production\n\n- **Deduplicate before chunking.** The same story appears on several sites; keep one copy per URL and consider near-duplicate titles.\n- **Cache the full text.** Articles rarely change after the first hours; store the extracted text by URL and skip the fetch next time.\n- **Keep dates in the prompt.** Models are bad at knowing today's date. \"As of 12 November\" in the context prevents stale answers.\n- **Respect copyright.** Use full text to ground answers and quote short passages with a link, not to republish articles.\n- **Check what you skipped.** Log URLs that returned under 500 characters. If one source always fails, drop it from your query.\n\n## Use it with an agent instead\n\nIf your RAG pipeline is really an agent, you can skip most of this code: the [NestDaddy MCP server](https://nestdaddy.com/mcp) exposes news search, page reading and research papers as tools, and news tools can return full article text directly. Setup for Claude, Cursor and local models: [Claude](https://nestdaddy.com/blog/add-live-web-search-to-claude-mcp), [Cursor](https://nestdaddy.com/blog/cursor-web-search-mcp), [Ollama](https://nestdaddy.com/blog/local-llm-web-search-ollama).\n\nOne All-in-One key covers news search, page extraction, web search and the MCP server, from $9.99 a month for 10,000 calls after the free 7-day trial: [get a key](https://nestdaddy.com/apis).", "url": "https://wpnews.pro/news/feed-your-rag-pipeline-full-news-articles-not-snippets-python", "canonical_source": "https://nestdaddy.com/blog/news-rag-full-article-text", "published_at": "2026-11-13 00:00:00+00:00", "updated_at": "2026-11-13 16:52:06.400723+00:00", "lang": "en", "topics": ["ai-tools", "generative-ai", "large-language-models", "developer-tools"], "entities": ["NestDaddy", "NestDaddy API", "Python", "requests"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/feed-your-rag-pipeline-full-news-articles-not-snippets-python", "markdown": "https://wpnews.pro/news/feed-your-rag-pipeline-full-news-articles-not-snippets-python.md", "text": "https://wpnews.pro/news/feed-your-rag-pipeline-full-news-articles-not-snippets-python.txt", "jsonld": "https://wpnews.pro/news/feed-your-rag-pipeline-full-news-articles-not-snippets-python.jsonld"}}