cd /news/ai-tools/how-to-pull-structured-public-data-i… · home › topics › ai-tools › article
[ARTICLE · art-169498] src=dev.to ↗ pub= topic=ai-tools verified=true sentiment=↑ positive

How to Pull Structured Public Data into n8n Without Community Nodes (2 Free Templates)

A developer open-sourced two n8n workflow templates that pull public web data and podcast transcripts into automation pipelines using only native core nodes — Schedule Trigger, Edit Fields, HTTP Request, Code, Google Sheets and Slack — with no community node packages. The templates call Apify Actors through the synchronous run-sync-get-dataset-items endpoint and pass a maxTotalChargeUsd parameter to enforce a hard spending cap at the API gateway, defaulting to $0.10 per run. One template extracts Podcasting 2.0 <podcast:transcript> tags from RSS feeds to feed RAG pipelines without paying for speech-to-text, with the first 25 saved items per run free and pay-per-result pricing of $2.00 per 1,000 items beyond that.

by read5 min views3 publishedNov 14, 2026

When automating workflows in n8n, bringing in public web data is often where setups become fragile. Custom Puppeteer or Playwright scripts break on site updates, headless browsers consume heavy system memory on self-hosted instances, and custom community nodes can create dependency issues or fail security reviews in production.

A cleaner architectural pattern is to decouple data collection from workflow orchestration: let specialized, serverless cloud extractors handle the fetching, parsing, and schema normalization, while n8n handles what it does best: conditional logic, deduplication, and routing data into Google Sheets, Slack, databases, or AI vector pipelines.

In this tutorial, we will look at how to connect n8n directly to public Apify Actors using only native core n8n nodes (Schedule Trigger, Edit Fields / Set, HTTP Request, Code, Google Sheets, and Slack). Zero custom community nodes required. Everything runs seamlessly on both self-hosted n8n and n8n Cloud.

We have open-sourced two production-ready workflow templates on GitHub that you can import with a single click:

<podcast:transcript> tag) for RAG pipelines and content research—without paying for audio speech-to-text models. Full Transparency & Disclosure: I am the author of these free, open-source n8n templates (GitHub Repository) and the maintainer of the two Apify Actors they invoke on the Apify Store. Every run's first 25 saved items are 100% free with $0 in Actor run charges. The default template configurations stay within this free tier. Beyond 25 items per run, usage is pay-per-result ($2.00 per 1,000 items; the Actor's Pricing tab on Apify is the authoritative source). Both workflows include hard cost caps to guarantee no unexpected charges occur.

Many developers assume integrating a scraping platform into n8n requires installing a community node package. While community nodes exist, relying on them introduces several tradeoffs:

By using n8n's native HTTP Request node calling the synchronous Apify API endpoint, your workflow remains completely lightweight and portable:

https://api.apify.com/v2/acts/{actorId}/run-sync-get-dataset-items

Typically, running an asynchronous cloud task requires a webhook receiver or polling loop. However, for batch extractions that complete within standard timeout windows (under 60-120 seconds), Apify's run-sync-get-dataset-items API runs the Actor, waits for completion, and returns the final JSON dataset array directly in the HTTP response body.

Furthermore, by appending ?maxTotalChargeUsd={{ $json.maxCostPerRunUsd }}, Apify's execution engine enforces a hard spending ceiling at the API gateway level. Even if you accidentally configure a scrape over thousands of pages, the run automatically stops when reaching your specified limit (defaulting to $0.10).

Feeding podcast knowledge into LLMs or RAG systems usually requires down audio files and running them through Whisper or another speech-to-text API. At typical API pricing ($0.36 to $1.50+ per hour of audio), transcribing daily podcasts across several shows quickly becomes expensive and computationally heavy.

However, a large and growing number of podcast publishers (hosted on Buzzsprout, Omny Studio, Podbean, etc.) already attach human-verified or pre-generated transcripts directly to their RSS feeds using the standardized Podcasting 2.0 <podcast:transcript> tag (in VTT, SRT, JSON, HTML, or plain text formats).

[Schedule Trigger (Daily 07:00)]
       │
       ▼
[Podcast Settings (Edit Fields)]
       │
       ▼
[Validate Settings (Code)]
       │
       ▼
[Fetch Transcripts (HTTP Request -> Apify)]
       │
       ▼
[Read Saved GUIDs (Google Sheets)]
       │
       ▼
[Filter New Episodes & Format (Code)]
       │
       ▼
[Append New Transcripts (Google Sheets)]

feedUrls), backfill episode limits ( maxEpisodesPerFeed, default 3), and maximum items per run ( maxTranscriptsPerRun, default 25). ahmed.fouaad82/podcast-published-transcripts. It parses the feeds, resolves the linked transcript files, converts VTT/SRT timestamps into clean clean text paragraphs with speaker labels, and returns structured episode objects.guid column. A Code node compares incoming episode GUIDs against existing entries, ensuring only brand-new episodes are processed.textTruncated = true), while always preserving the permanent transcriptUrl link to the full raw file. Create a tab named Transcripts with the following 15-column header row:

guid, publishedAt, podcastTitle, episodeTitle, episodeNumber, durationMinutes, wordCount, speakers, language, transcriptUrl, audioUrl, feedUrl, textTruncated, text, savedAt

Keeping tabs on competitive hiring markets and compensation trends requires monitoring dozens of company career pages. Most modern job boards are JavaScript Single Page Applications (SPAs) where raw HTML scraping fails. Furthermore, only a subset of postings include explicit numeric pay ranges.

[Schedule Trigger (Weekdays 08:00)]
       │
       ▼
[Job Watch Settings (Edit Fields)]
       │
       ▼
[Validate Board Slugs (Code)]
       │
       ▼
[Fetch Lever Jobs (HTTP Request -> Apify)]
       │
       ▼
[Read Known Job IDs (Google Sheets)]
       │
       ▼
[Filter New Postings & Format (Code)]
       │
       ▼
[Append Postings (Google Sheets)]
       │
       ▼
[Send Digest (Slack)]

leverdemo, or direct URLs https://jobs.lever.co/company), minimum salary threshold ( minSalary), and remote filters ( remoteOnly). ahmed.fouaad82/lever-salary-jobs. The Actor interfaces with Lever's public career APIs, extracts structured compensation objects (currency, minimum, maximum, interval), and returns only qualifying postings.jobId column from Google Sheets. The first run populates the sheet; subsequent runs filter strictly for newly discovered postings.#jobs (up to 15 highlighted roles with direct application links). Create a tab named Salary watch with the following header row:

jobId, board, title, team, department, location, workplaceType, commitment, salaryMin, salaryMax, salaryCurrency, salaryInterval, salaryRange, postedAt, jobUrl, firstSeenAt

Setting up either template takes approximately 5 minutes. Here is the complete checklist:

Authorization Bearer YOUR_APIFY_API_TOKEN Apify Header Auth. podcast-transcripts-to-google-sheets/workflow.json``lever-salary-watch-sheets-slack/workflow.json`` Ctrl+V / Cmd+V). Podcast settings or Job watch settings)./d/ and /edit). When automating scraping, safeguarding against accidental runaway costs is critical. These templates implement a two-layer defense:

maxTotalChargeUsd is dynamically bound to {{ $json.maxCostPerRunUsd }} (defaulting to $0.10). If a run attempts to backfill hundreds of episodes or thousands of postings beyond your budget, Apify terminates the job automatically before the charge can exceed 10 cents. By combining n8n's visual orchestration engine with Apify's serverless scraping actors through standard HTTP requests, you get the reliability of a managed scraping platform without the operational overhead of maintaining custom headless browsers or third-party community nodes.

All resources are free and open source under the MIT license:

If you build on top of these templates—such as connecting the transcript output directly to a LangChain / Pinecone vector node for interactive Q&A—feel free to share your fork or open an issue on the GitHub repository!

── more in #ai-tools 4 stories · sorted by recency
── more on @n8n 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/how-to-pull-structur…] indexed:0 read:5min 2026-11-14 · —