Source Score: Continuing Exploration of LLM Usage in Automated Workflows

The article describes a system that automates the extraction and verification of falsifiable claims from news articles. It uses NewsData.io to fetch the ten newest articles from a source, then employs LLMs via OpenRouter to select the best falsifiable claim, fill in missing summaries, and find independent proofs. The workflow runs weekly using Python scripts, custom OpenRouter skills, and GitHub Actions, ultimately generating pull requests with ready-to-ingest claim and proof files.

TL;DR– I built a weekly pipeline that pulls the ten newest articles from a news outlet, picks the one that actually makes a falsifiable claim, fills in any missing summary, finds two independent proofs, and opens PRs with ready‑to‑ingest claims/ and proofs/ files. All of this runs on a couple of Python scripts, 2 custom OpenRouter skills, and two GitHub Actions. In my last post I covered how I automated ingestion of top news sources by combining Firecrawl, OpenRouter API, and Github Action workflows. In this post I'll implement the same pattern for news source claims and their proofs. The first challenge I ran into was to figure out a way to fetch recent articles published by a news source. Luckily, I found NewsData.IO https://newsdata.io/ which provides an API to search, collect and track worldwide news. The NewsData.io free tier gives me 200 API credits per day, more than enough for a weekly run across 12 sources for now at least 😌 . 1️⃣ Fetch the latest articles - newsdata io.py https://github.com/SatyaLens/sources/blob/main/scripts/newsdata io.py The first step is to write a thin wrapper around the NewsData.io latest endpoint. It pulls the 10 most recent articles for a given domain. Free‑tier‑friendly query params CATEGORY = "environment,technology,world" LANGUAGE = "en" REMOVE DUPLICATE = "1" SIZE = "10" DATATYPE = "news,research,analysis,pressRelease" NEWSDATA API BASE URL = os.getenv "NEWSDATA API BASE URL", "https://newsdata.io/api/1" NEWSDATA API KEY = os.environ "NEWSDATA API KEY" def get claims src domain url: str : """ Call the NewsData.io /latest endpoint for a specific domain. Returns a list of article dicts or None on error . """ endpoint = f"{NEWSDATA API BASE URL}/latest" params = { "category": CATEGORY, "language": LANGUAGE, "removeduplicate": REMOVE DUPLICATE, "size": SIZE, "datatype": DATATYPE, "apikey": NEWSDATA API KEY, "domainurl": src domain url, } response = requests.get endpoint, params=params, timeout=10 if response.status code = 200: print f"Error: couldn't fetch claims for {src domain url}: {response.status code}" Show suggestions if the API knows a better domain resp body = response.json if resp body.get "results" is not None and resp body.get "results" 0 .get "suggestion" is not None : print f"Suggested domain url s for {src domain url}: {resp body 'results' 0 'suggestion' }" return None return response.json "results" Why it matters : The free tier only gives 200 credits per day, so we keep the request lightweight: single domain, ten results, and a narrow set of categories. The DATATYPE filter is intentionally broad; we prune non‑falsifiable items later. 2️⃣ Filter for a falsifiable claim - the claim‑verification skill Given the way I'm currently calculating scores for a news source, only claims that are falsifiable matter. So out of all the claims returned by NewsData API I'm keeping only one that matches the falsifiable claim criteria best. I'm using LLMs to perform this classification, to be specific, I'm forwarding an array of objects returned by NewsData API to OpenRouter API. { "article id": "40305aa160787297dd3f9cc15faa8637", "link": "https://www.theguardian.com/us-news/2026/may/22/kansas-bird-nest-truck", "title": "Federally protected bird’s nest holds up sale of Ford truck in Kansas", "description": "A robin built a nest on a Ford-F-250’s tire and laid its eggs in it; a law prohibits removing it while inhabited by bird brood A truck sold by a Kansas dealership cannot be taken from the lot by its new owner because a family of robins is living atop one of the vehicle’s tires. The relatively novel situation has gained widespread attention after the dealership in the Kansas community of Olathe wrote about it on its Facebook page – and it perhaps taught many that active robin nests are protected by federal law from the US. Continue reading...", "keywords": "birds", "kansas", "wildlife", "ford", "animals", "us news", "law us " , "creator": "josé olivares" , "language": "english", "country": "united states of america" , "category": "top", "environment" , "datatype": "news", "pubDate": "2026-05-22 19:03:07", "pubDateTZ": "UTC", "fetched at": "2026-05-22 19:32:47", "image url": "https://i.guim.co.uk/img/media/c9e972eb2d494c4a9c713a7b5550f0fa9efcae1f/0 503 1536 1229/master/1536.jpg?width=140&quality=85&auto=format&fit=max&s=ad95c6dcdf71df9bc3461b683effe424", "video url": null, "source id": "theguardian", "source name": "The Guardian", "source priority": 106, "source url": "https://www.theguardian.com", "source icon": "https://n.bytvi.com/theguardian.jpg", "duplicate": false } To help with the classification I wrote an AI agent skill . It tells the LLM to: - Visit each article URL with the web‑search tool. - Detect whether the article contains a claim that can be objectively proven true or false. - Return exactly one JSON object that matches the criteria. CLAIM FILTER PROMPT = "Use web search tool to visit the link for each article, access the content and then assess if it is a falsifiable claim." "Out of these 10 articles, only return 1 article that best fits the falsifiable claim criterion." "Prefer claims that have been made by the news source directly." "Keep the json structure of the claims the same as the original schema in the input. Do not add, remove, or modify any key or value." "Only output the plain json array string that I can safely unmarshal." "Do not format the string. Do not output anything else." req content = "Following is a list of 10 articles published by the same news outlet. Each article is represented by a json string type element in the array" f"\n\n{claims}\n\n" f"{CLAIM FILTER PROMPT}" filtered claims = openrouter.req w addons req content, skill=falsifiable claim skill, tools= openrouter.WEB SEARCH TOOL Result : A single, well‑structured claim with all the fields required to create a claim ingestion doc. I'm only selecting one claim for now, once I've verified the stability of ingestion workflows and the reliability of OpenRouter responses I'll the increase the count and the ingestion frequency. 3️⃣ Fill in missing descriptions - a quick summarization pass NewsData.io sometimes returns "null" for the description field. When that happens I'm asking OpenRouter to summarize the article in under 500 characters. CLAIM SUMMARY PROMPT = "Use web search tool to visit the link to the article and access its content." "Summarize the article in under 500 characters." "Return only the summary without any additional text." req content = "Following is the url to an article published by a news media outlet." f"\n\n{claim 'link' }\n\n" f"{CLAIM SUMMARY PROMPT}" claim summary = openrouter.req w addons req content, tools= openrouter.WEB SEARCH TOOL claim "description" = claim summary Now every claim has a concise, human‑readable description, even when the source API left it blank. 4️⃣ Weekly claim ingestion - GitHub Actions workflow https://github.com/SatyaLens/sources/actions/workflows/fetch ingest claims.yml The whole thing runs on a GitHub Actions schedule once a week. The workflow https://github.com/SatyaLens/sources/blob/main/.github/workflows/fetch ingest claims.yml checks out the repo, installs dependencies, runs ingest claims.py , and opens a PR. Outcome : A PR https://github.com/SatyaLens/sources/pull/49 appears every Sunday with fresh claim documents, ready for review. 5️⃣ Fetch proofs - ingest proofs.py https://github.com/SatyaLens/sources/blob/main/scripts/ingest proofs.py + workflow https://github.com/SatyaLens/sources/blob/main/.github/workflows/fetch ingest proofs.yml Once a claim has been ingested, I need proofs that either support or refute it. The proof‑verification prompt asks the LLM to find two independent sources and label each with a boolean supports claim . To help LLMs search for supporting or refuting proofs for a given claim using web search tool, I wrote another skill, claim-verification https://github.com/semmet95/agent-skills/blob/main/claim-verification/SKILL.md , which performs the following: - Extracts and validates falsifiable claims : Takes a URL to a media article/post and identifies the core, testable claim from its content, ensuring it's concrete and can be proven true or false. - Performs targeted web searches : Uses time-aware search queries to find up to two high-quality external documents that directly support or refute the claim, with strict attention to temporal relevance matching the claim's timeframe . - Returns verifiable evidence URLs : Outputs a JSON array of retrieved URLs with boolean indicators supports claim: true/false showing whether each document confirms or contradicts the original claim, prioritizing official/authoritative sources over opinion pieces. CLAIM VERIFICATION PROMPT = "Use web search tool to access the claim link, fetch the content and process it." "Use the web search tool again to look for proofs in the form of official statements, press releases, or reports from reputable sources to prove the claim right or wrong conclusively." "Ensure that the proofs belong to the same timeline as the claim. Do not include outdated sources." "Output links to the 2 sources that prove the claim right or wrong and specify as a boolean whether they support the claim or not." "The output format should be a json array with each element being a json object corresponding to a source supporting or refuting the claim." "Each json element should follow the following schema: {\"uri\": \"string\", \"supports claim\": boolean}" req content = "Following is a link to a falsifiable claim by a news media outlet as an article" f"\n\n{claim 'uri' }\n\n" f"{CLAIM VERIFICATION PROMPT}" claim proofs = openrouter.req w addons req content, skill=claim verification skill, tools= openrouter.WEB SEARCH TOOL Proof workflow : Just like claim ingestion workflow, I'm running proof ingestion workflow once a week. It runs the ingest proofs.py script, creates the proof documents in a new branch, then creates a PR from this branch to the main branch. Outcome : A PR https://github.com/SatyaLens/sources/pull/48 appears every Sunday with fresh proof documents for ingested claims, ready for review. 6️⃣ OpenRouter reliability improvements Since my OpenRouter API usage has been increasing, my list of free tier models is shortened to the following: FREE MODELS DOC = "google/gemma-4-31b-it:free", "nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free", "openrouter/free" These three models consistently give good results while staying within the free tier. Back‑off retry loop openrouter.py https://github.com/SatyaLens/sources/blob/main/scripts/openrouter.py I also added incremental delays between retries to reduce the odds of running into server side errors. for i in range 1, OPENROUTER MAX RETRIES + 1 : status, body = helper.post request ... if status in 0, 429 or 500 <= status < 600: print f"OpenRouter API returned status {status}, retrying...", file=sys.stderr time.sleep 10 i continue success handling... Impact : 500‑error rate dropped dramatically, and weekly API spend stayed well under the free‑tier limits. I really like this OpenRouter dashboard btw. 7️⃣ Trade‑offs & limitations | Aspect | Trade‑off | |---|---| Free‑tier limits | 200 NewsData.io credits/day restricts us to a single domain per run. Scaling to many outlets will need a paid plan or smarter batching. | Single claim per source | I deliberately return only the “best” falsifiable claim to keep the workflow simple. Future work will support multiple high‑quality claims per article. | LLM hallucinations | Even with the claim‑verification skill, the model can surface outdated links. Temporal awareness helps, but it’s not a silver bullet. | Proof quality | Proofs are fetched automatically; a human review is still recommended for high‑stakes claims. | 8️⃣ Conclusion & next steps By stitching together NewsData.io , custom falsifiable-claim and claim‑verification skills, OpenRouter with incremental back‑off and free‑tier model rotation , and GitHub Actions , I built a cost‑effective, fully automated pipeline that turns raw news into structured, falsifiable claim documents and their supporting proofs. What’s next? - Add confidence scores to claims and proofs so downstream consumers can weigh evidence. - Move from weekly to daily ingestion for high‑volume outlets once the retry logic proves rock‑solid. - Negative tests because I'm yet to see a case where a claim is proven wrong, I guess the LLMs are playing it safe 🫣 If you’re curious, the full code lives in the SatyaLens/sources https://github.com/SatyaLens/sources repo. If you have any suggestions or questions for me, feel free to drop them in the comment section.