{"slug": "source-score-continuing-exploration-of-llm-usage-in-automated-workflows", "title": "Source Score: Continuing Exploration of LLM Usage in Automated Workflows", "summary": "The article describes a system that automates the extraction and verification of falsifiable claims from news articles. It uses NewsData.io to fetch the ten newest articles from a source, then employs LLMs via OpenRouter to select the best falsifiable claim, fill in missing summaries, and find independent proofs. The workflow runs weekly using Python scripts, custom OpenRouter skills, and GitHub Actions, ultimately generating pull requests with ready-to-ingest claim and proof files.", "body_md": "TL;DR– I built a weekly pipeline that pulls the ten newest articles from a news outlet, picks the one that actually makes a falsifiable claim, fills in any missing summary, finds two independent proofs, and opens PRs with ready‑to‑ingest`claims/`\n\nand`proofs/`\n\nfiles. All of this runs on a couple of Python scripts, 2 custom OpenRouter skills, and two GitHub Actions.\n\nIn my last post I covered how I automated ingestion of top news sources by combining Firecrawl, OpenRouter API, and Github Action workflows. In this post I'll implement the same pattern for news source claims and their proofs.\n\nThe first challenge I ran into was to figure out a way to fetch recent articles published by a news source. Luckily, I found [NewsData.IO](https://newsdata.io/) which provides an API to search, collect and track worldwide news. The NewsData.io free tier gives me 200 API credits per day, more than enough for a weekly run across 12 sources (*for now at least* 😌).\n\n## 1️⃣ Fetch the latest articles - [newsdata_io.py](https://github.com/SatyaLens/sources/blob/main/scripts/newsdata_io.py)\n\nThe first step is to write a thin wrapper around the **NewsData.io** `latest`\n\nendpoint. It pulls the **10 most recent articles** for a given domain.\n\n```\n# Free‑tier‑friendly query params\nCATEGORY = \"environment,technology,world\"\nLANGUAGE = \"en\"\nREMOVE_DUPLICATE = \"1\"\nSIZE = \"10\"\nDATATYPE = \"news,research,analysis,pressRelease\"\n\nNEWSDATA_API_BASE_URL = os.getenv(\n    \"NEWSDATA_API_BASE_URL\", \"https://newsdata.io/api/1\"\n)\nNEWSDATA_API_KEY = os.environ[\"NEWSDATA_API_KEY\"]\n\ndef get_claims(src_domain_url: str):\n    \"\"\"\n    Call the NewsData.io `/latest` endpoint for a specific domain.\n    Returns a list of article dicts (or None on error).\n    \"\"\"\n    endpoint = f\"{NEWSDATA_API_BASE_URL}/latest\"\n    params = {\n        \"category\": CATEGORY,\n        \"language\": LANGUAGE,\n        \"removeduplicate\": REMOVE_DUPLICATE,\n        \"size\": SIZE,\n        \"datatype\": DATATYPE,\n        \"apikey\": NEWSDATA_API_KEY,\n        \"domainurl\": src_domain_url,\n    }\n    response = requests.get(endpoint, params=params, timeout=10)\n    if response.status_code != 200:\n        print(\n            f\"Error: couldn't fetch claims for {src_domain_url}: {response.status_code}\"\n        )\n        # Show suggestions if the API knows a better domain\n        resp_body = response.json()\n        if (\n            resp_body.get(\"results\") is not None\n            and resp_body.get(\"results\")[0].get(\"suggestion\") is not None\n        ):\n            print(\n                f\"Suggested domain url(s) for {src_domain_url}: {resp_body['results'][0]['suggestion']}\"\n            )\n        return None\n    return response.json()[\"results\"]\n```\n\n*Why it matters*: The free tier only gives 200 credits per day, so we keep the request lightweight: single domain, ten results, and a narrow set of categories. The `DATATYPE`\n\nfilter is intentionally broad; we prune non‑falsifiable items later.\n\n## 2️⃣ Filter for a falsifiable claim - the **claim‑verification** skill\n\nGiven the way I'm currently calculating scores for a news source, only claims that are falsifiable matter. So out of all the claims returned by NewsData API I'm keeping only one that matches the **falsifiable claim** criteria best.\n\nI'm using LLMs to perform this classification, to be specific, I'm forwarding an array of objects returned by NewsData API to OpenRouter API.\n\n```\n[{\n\"article_id\": \"40305aa160787297dd3f9cc15faa8637\",\n\"link\": \"https://www.theguardian.com/us-news/2026/may/22/kansas-bird-nest-truck\",\n\"title\": \"Federally protected bird’s nest holds up sale of Ford truck in Kansas\",\n\"description\": \"A robin built a nest on a Ford-F-250’s tire and laid its eggs in it; a law prohibits removing it while inhabited by bird brood A truck sold by a Kansas dealership cannot be taken from the lot by its new owner because a family of robins is living atop one of the vehicle’s tires. The relatively novel situation has gained widespread attention after the dealership in the Kansas community of Olathe wrote about it on its Facebook page – and it perhaps taught many that active robin nests are protected by federal law from the US. Continue reading...\",\n\"keywords\": [\n\"birds\",\n\"kansas\",\n\"wildlife\",\n\"ford\",\n\"animals\",\n\"us news\",\n\"law (us)\"\n],\n\"creator\": [\n\"josé olivares\"\n],\n\"language\": \"english\",\n\"country\": [\n\"united states of america\"\n],\n\"category\": [\n\"top\",\n\"environment\"\n],\n\"datatype\": \"news\",\n\"pubDate\": \"2026-05-22 19:03:07\",\n\"pubDateTZ\": \"UTC\",\n\"fetched_at\": \"2026-05-22 19:32:47\",\n\"image_url\": \"https://i.guim.co.uk/img/media/c9e972eb2d494c4a9c713a7b5550f0fa9efcae1f/0_503_1536_1229/master/1536.jpg?width=140&quality=85&auto=format&fit=max&s=ad95c6dcdf71df9bc3461b683effe424\",\n\"video_url\": null,\n\"source_id\": \"theguardian\",\n\"source_name\": \"The Guardian\",\n\"source_priority\": 106,\n\"source_url\": \"https://www.theguardian.com\",\n\"source_icon\": \"https://n.bytvi.com/theguardian.jpg\",\n\"duplicate\": false\n}]\n```\n\nTo help with the classification I wrote an AI ** agent skill**. It tells the LLM to:\n\n-\n**Visit each article URL** with the web‑search tool. -\n**Detect** whether the article contains a claim that can be objectively proven true or false. -\n**Return exactly one** JSON object that matches the criteria.\n\n```\nCLAIM_FILTER_PROMPT = (\n    \"Use web search tool to visit the link for each article, access the content and then assess if it is a falsifiable claim.\"\n    \"Out of these 10 articles, only return 1 article that best fits the falsifiable claim criterion.\"\n    \"Prefer claims that have been made by the news source directly.\"\n    \"Keep the json structure of the claims the same as the original schema in the input. Do not add, remove, or modify any key or value.\"\n    \"Only output the plain json array string that I can safely unmarshal.\"\n    \"Do not format the string. Do not output anything else.\"\n)\n\nreq_content = (\n    \"Following is a list of 10 articles published by the same news outlet. Each article is represented by a json string type element in the array\"\n    f\"\\n\\n{claims}\\n\\n\"\n    f\"{CLAIM_FILTER_PROMPT}\"\n)\n\nfiltered_claims = openrouter.req_w_addons(\n    req_content, skill=falsifiable_claim_skill, tools=[openrouter.WEB_SEARCH_TOOL]\n)\n```\n\n*Result*: A single, well‑structured claim with all the fields required to create a claim ingestion doc.\n\nI'm only selecting one claim for now, once I've verified the stability of ingestion workflows and the reliability of OpenRouter responses I'll the increase the count and the ingestion frequency.\n\n## 3️⃣ Fill in missing descriptions - a quick summarization pass\n\nNewsData.io sometimes returns `\"null\"`\n\nfor the `description`\n\nfield. When that happens I'm asking **OpenRouter** to summarize the article in under 500 characters.\n\n```\nCLAIM_SUMMARY_PROMPT = (\n    \"Use web search tool to visit the link to the article and access its content.\"\n    \"Summarize the article in under 500 characters.\"\n    \"Return only the summary without any additional text.\"\n)\n\nreq_content = (\n    \"Following is the url to an article published by a news media outlet.\"\n    f\"\\n\\n{claim['link']}\\n\\n\"\n    f\"{CLAIM_SUMMARY_PROMPT}\"\n)\n\nclaim_summary = openrouter.req_w_addons(\n    req_content, tools=[openrouter.WEB_SEARCH_TOOL]\n)\n\nclaim[\"description\"] = claim_summary\n```\n\nNow every claim has a concise, human‑readable description, even when the source API left it blank.\n\n## 4️⃣ Weekly claim ingestion - GitHub Actions [workflow](https://github.com/SatyaLens/sources/actions/workflows/fetch_ingest_claims.yml)\n\nThe whole thing runs on a **GitHub Actions** schedule once a week. The [workflow](https://github.com/SatyaLens/sources/blob/main/.github/workflows/fetch_ingest_claims.yml) checks out the repo, installs dependencies, runs `ingest_claims.py`\n\n, and opens a PR.\n\n*Outcome*: A [PR](https://github.com/SatyaLens/sources/pull/49) appears every Sunday with fresh claim documents, ready for review.\n\n## 5️⃣ Fetch proofs - [ingest_proofs.py](https://github.com/SatyaLens/sources/blob/main/scripts/ingest_proofs.py) + [workflow](https://github.com/SatyaLens/sources/blob/main/.github/workflows/fetch_ingest_proofs.yml)\n\nOnce a claim has been ingested, I need **proofs** that either support or refute it. The proof‑verification prompt asks the LLM to find **two** independent sources and label each with a boolean `supports_claim`\n\n.\n\nTo help LLMs search for supporting or refuting proofs for a given claim using web search tool, I wrote another skill, [claim-verification](https://github.com/semmet95/agent-skills/blob/main/claim-verification/SKILL.md), which performs the following:\n\n-\n**Extracts and validates falsifiable claims**: Takes a URL to a media article/post and identifies the core, testable claim from its content, ensuring it's concrete and can be proven true or false. -\n**Performs targeted web searches**: Uses time-aware search queries to find up to two high-quality external documents that directly support or refute the claim, with strict attention to temporal relevance (matching the claim's timeframe). -\n**Returns verifiable evidence URLs**: Outputs a JSON array of retrieved URLs with boolean indicators (supports_claim: true/false) showing whether each document confirms or contradicts the original claim, prioritizing official/authoritative sources over opinion pieces.\n\n```\nCLAIM_VERIFICATION_PROMPT = (\n    \"Use web search tool to access the claim link, fetch the content and process it.\"\n    \"Use the web search tool again to look for proofs in the form of official statements, press releases, or reports from reputable sources to prove the claim right or wrong conclusively.\"\n    \"Ensure that the proofs belong to the same timeline as the claim. Do not include outdated sources.\"\n    \"Output links to the 2 sources that prove the claim right or wrong and specify as a boolean whether they support the claim or not.\"\n    \"The output format should be a json array with each element being a json object corresponding to a source supporting or refuting the claim.\"\n    \"Each json element should follow the following schema: {\\\"uri\\\": \\\"string\\\", \\\"supports_claim\\\": boolean}\"\n)\n\nreq_content = (\n    \"Following is a link to a falsifiable claim by a news media outlet as an article\"\n    f\"\\n\\n{claim['uri']}\\n\\n\"\n    f\"{CLAIM_VERIFICATION_PROMPT}\"\n)\n\nclaim_proofs = openrouter.req_w_addons(\n    req_content, skill=claim_verification_skill, tools=[openrouter.WEB_SEARCH_TOOL]\n)\n```\n\n** Proof workflow**: Just like claim ingestion workflow, I'm running proof ingestion workflow once a week. It runs the\n\n`ingest_proofs.py`\n\nscript, creates the proof documents in a new branch, then creates a PR from this branch to the `main`\n\nbranch.*Outcome*: A [PR](https://github.com/SatyaLens/sources/pull/48) appears every Sunday with fresh proof documents for ingested claims, ready for review.\n\n## 6️⃣ OpenRouter reliability improvements\n\nSince my OpenRouter API usage has been increasing, my list of free tier models is shortened to the following:\n\n```\nFREE_MODELS_DOC = [\n    \"google/gemma-4-31b-it:free\",\n    \"nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free\",\n    \"openrouter/free\"\n]\n```\n\nThese three models consistently give good results while staying within the free tier.\n\n### Back‑off retry loop ([openrouter.py](https://github.com/SatyaLens/sources/blob/main/scripts/openrouter.py))\n\nI also added incremental delays between retries to reduce the odds of running into server side errors.\n\n```\nfor i in range(1, OPENROUTER_MAX_RETRIES + 1):\n    status, body = helper.post_request(...)\n    if status in (0, 429) or 500 <= status < 600:\n        print(f\"OpenRouter API returned status {status}, retrying...\", file=sys.stderr)\n        time.sleep(10 * i)\n        continue\n    # success handling...\n```\n\n*Impact*: 500‑error rate dropped dramatically, and weekly API spend stayed well under the free‑tier limits.\n\nI really like this OpenRouter dashboard btw.\n\n## 7️⃣ Trade‑offs & limitations\n\n| Aspect | Trade‑off |\n|---|---|\nFree‑tier limits |\n200 NewsData.io credits/day restricts us to a single domain per run. Scaling to many outlets will need a paid plan or smarter batching. |\nSingle claim per source |\nI deliberately return only the “best” falsifiable claim to keep the workflow simple. Future work will support multiple high‑quality claims per article. |\nLLM hallucinations |\nEven with the claim‑verification skill, the model can surface outdated links. Temporal awareness helps, but it’s not a silver bullet. |\nProof quality |\nProofs are fetched automatically; a human review is still recommended for high‑stakes claims. |\n\n## 8️⃣ Conclusion & next steps\n\nBy stitching together **NewsData.io**, custom **falsifiable-claim** and **claim‑verification** skills, **OpenRouter** (with incremental back‑off and free‑tier model rotation), and **GitHub Actions**, I built a **cost‑effective, fully automated pipeline** that turns raw news into structured, falsifiable claim documents and their supporting proofs.\n\n**What’s next?**\n\n-\n**Add confidence scores** to claims and proofs so downstream consumers can weigh evidence. -\n**Move from weekly to daily ingestion** for high‑volume outlets once the retry logic proves rock‑solid. -\n**Negative tests** because I'm yet to see a case where a claim is proven wrong, I guess the LLMs are playing it safe 🫣\n\nIf you’re curious, the full code lives in the [SatyaLens/sources](https://github.com/SatyaLens/sources) repo. If you have any suggestions or questions for me, feel free to drop them in the comment section.", "url": "https://wpnews.pro/news/source-score-continuing-exploration-of-llm-usage-in-automated-workflows", "canonical_source": "https://dev.to/semmet/source-score-continuing-exploration-of-llm-usage-in-automated-workflows-eoi", "published_at": "2026-05-24 06:17:50+00:00", "updated_at": "2026-05-24 06:32:39.093653+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "large-language-models", "open-source", "data"], "entities": ["NewsData.IO", "Firecrawl", "OpenRouter", "GitHub Actions", "Python"], "alternates": {"html": "https://wpnews.pro/news/source-score-continuing-exploration-of-llm-usage-in-automated-workflows", "markdown": "https://wpnews.pro/news/source-score-continuing-exploration-of-llm-usage-in-automated-workflows.md", "text": "https://wpnews.pro/news/source-score-continuing-exploration-of-llm-usage-in-automated-workflows.txt", "jsonld": "https://wpnews.pro/news/source-score-continuing-exploration-of-llm-usage-in-automated-workflows.jsonld"}}