Feed your RAG pipeline full news articles, not snippets (Python)

A Python tutorial published on NestDaddy's API documentation shows how to build a roughly 100-line retrieval-augmented generation pipeline that fetches full news article text via the NestDaddy API's news/search and extract endpoints rather than relying on two-line search snippets. The guide chunks articles by paragraph at about 800 characters, carries the last paragraph of each chunk into the next to preserve facts spanning paragraphs, filters boilerplate such as newsletter prompts, and skips pages under 500 characters to drop paywalled or JavaScript-rendered content. It requires only the requests library and a NestDaddy API key, with a free 7-day key offering 5,000 calls, and the retrieval step can later be swapped for embeddings.

Feed your RAG pipeline full news articles, not snippets Python Search results give a RAG system two lines per article, which is not enough to answer real questions. This guide fetches recent news, pulls the full readable text, chunks it by paragraph, retrieves the best passages and builds a cited context block. About 100 lines, no vector database needed to start. Most retrieval-augmented generation RAG demos over the news stop at search results: a title, a URL and a two-line snippet. That is fine for "what happened?" and useless for "what exactly did they announce, and who disagreed?". The answer is in paragraph nine, and the snippet never had it. This guide builds a small pipeline that works on the full text of recent articles: 1. Find recent articles on a topic. 2. Fetch each article's readable body, without menus, ads and comment sections. 3. Clean and chunk it by paragraph, keeping the source and date with every chunk. 4. Retrieve the best chunks for a question. 5. Build a numbered context block a model can cite. Everything runs in one Python file with requests only. You can swap the retrieval step for embeddings later without touching the rest. Setup pip install requests export NESTDADDY API KEY="your key" The examples use the NestDaddy API https://nestdaddy.com/apis : news/search for recent articles and extract for the full text of any URL. A free 7-day key 5,000 calls, no card is enough to follow along. Step 1: find recent articles python import math import os import re from collections import Counter import requests API = "https://nestdaddy.com/api/v1" KEY = os.environ "NESTDADDY API KEY" HEADERS = {"X-API-Key": KEY} def find articles query, days=3, country=None, limit=5 : """Recent news articles about a topic: title, url, source, date.""" params = {"q": query, "days": days, "limit": limit} if country: params "country" = country r = requests.get f"{API}/news/search", headers=HEADERS, params=params, timeout=30 r.raise for status return r.json .get "results", Keep days small for questions about current events. A wider window brings in older articles that confidently contradict newer ones. Step 2: fetch the full text python def full text url : """The readable body of one article, without menus, ads or comments.""" r = requests.get f"{API}/extract", headers=HEADERS, params={"url": url}, timeout=30 r.raise for status data = r.json return data.get "text" or "" if data.get "success" else "" Some pages will not give you much: paywalls, consent walls and pages that build their content with JavaScript. Step 4 skips anything shorter than 500 characters, which catches most of them. Step 3: clean and chunk by paragraph Fixed-size chunks cut sentences in half and split a quote from the person who said it. Chunking by paragraph keeps meaning together. Each new chunk also starts with the last paragraph of the previous one, so a fact that spans two paragraphs is never lost. BOILERPLATE = re.compile r"sign up for|newsletter|follow us on|subscribe to|read more:|all rights reserved|" r"click here|share this article", re.I def chunk text, size=800 : """Pack whole paragraphs into ~size-character chunks. Each new chunk starts with the last paragraph of the previous one, so a fact that spans two paragraphs is never cut in half. """ paras = p.strip for p in text.split "\n" if len p.strip 40 and not BOILERPLATE.search p chunks, cur = , for p in paras: if cur and sum len x for x in cur + len p size: chunks.append "\n".join cur cur = cur -1: cur.append p if cur: chunks.append "\n".join cur return chunks The BOILERPLATE filter is small on purpose. Publishers drop newsletter prompts and social links in the middle of articles, and those lines are noise to retrieval. Add patterns as you meet them. Step 4: build the corpus with metadata Every chunk carries its title, URL, source and publish date. That metadata is what lets the model cite, and lets you filter by date or source later. python def build corpus query, kw : corpus = for a in find articles query, kw : text = full text a "url" if len text < 500: paywall stub or failed extraction: skip it continue for i, c in enumerate chunk text : corpus.append { "text": c, "title": a "title" , "url": a "url" , "source": a.get "source domain" , "published": a.get "published timestamp" , "chunk": i, } return corpus Step 5: retrieve the best chunks To keep the example dependency-free, here is BM25, the keyword ranking behind most search engines. For a few dozen chunks from fresh news it works well, because questions about events usually share exact words with the answer names, places, products . python WORD = re.compile r" a-z0-9 +" def bm25 top corpus, question, k=5, k1=1.5, b=0.75 : """Tiny BM25 so the example has no dependencies; swap in embeddings later.""" docs = Counter WORD.findall d "text" .lower for d in corpus avg = sum sum d.values for d in docs / max len docs , 1 terms = set WORD.findall question.lower df = {t: sum 1 for d in docs if t in d for t in terms} n = len docs scores = for i, d in enumerate docs : length = sum d.values s = 0.0 for t in terms: if t not in d: continue idf = math.log 1 + n - df t + 0.5 / df t + 0.5 s += idf d t k1 + 1 / d t + k1 1 - b + b length / avg scores.append s, i return corpus i for s, i in sorted scores, reverse=True :k if s 0 When your corpus grows or questions get vaguer, add embeddings and combine both scores hybrid search . Keyword matching stays useful for names and numbers, which embeddings often blur. Step 6: a context block the model can cite python def context block chunks : """Numbered sources the model can cite as 1 , 2 ...""" lines = for n, c in enumerate chunks, 1 : lines.append f" {n} {c 'title' } {c 'source' }, {c 'published' } \n{c 'url' }\n{c 'text' }" return "\n\n".join lines if name == " main ": corpus = build corpus "AI", days=3, country="GB", limit=4 print len corpus , "chunks from full articles" top = bm25 top corpus, "What are employers being told to prepare for?" print context block top :1500 Put that block in your prompt under an instruction such as: Answer using only the sources below. Cite them as 1 , 2 . Mention the date of each fact. If the sources do not answer the question, say so. On a test run, four full articles became about 50 chunks, and the top result for the question above was the paragraph that actually quoted the minister's guidance, not the headline. Things that matter in production - Deduplicate before chunking. The same story appears on several sites; keep one copy per URL and consider near-duplicate titles. - Cache the full text. Articles rarely change after the first hours; store the extracted text by URL and skip the fetch next time. - Keep dates in the prompt. Models are bad at knowing today's date. "As of 12 November" in the context prevents stale answers. - Respect copyright. Use full text to ground answers and quote short passages with a link, not to republish articles. - Check what you skipped. Log URLs that returned under 500 characters. If one source always fails, drop it from your query. Use it with an agent instead If your RAG pipeline is really an agent, you can skip most of this code: the NestDaddy MCP server https://nestdaddy.com/mcp exposes news search, page reading and research papers as tools, and news tools can return full article text directly. Setup for Claude, Cursor and local models: Claude https://nestdaddy.com/blog/add-live-web-search-to-claude-mcp , Cursor https://nestdaddy.com/blog/cursor-web-search-mcp , Ollama https://nestdaddy.com/blog/local-llm-web-search-ollama . One All-in-One key covers news search, page extraction, web search and the MCP server, from $9.99 a month for 10,000 calls after the free 7-day trial: get a key https://nestdaddy.com/apis .