Originally published at linkdigest.dev, where I build this.
Paste a Xiaohongshu, Douyin, TikTok, YouTube or X link into an AI agent and it fetches the URL, gets an app-download shell or a login wall, and tells you there is nothing there. It is not wrong. The content of those posts is video, images, and text printed inside images, behind tokenised share links β none of it is in the HTML a fetch returns. Strip the <script> tags from a real Xiaohongshu note and 264 characters of navigation are left.
LinkDigest is a hosted reader: one call turns the link into text an LLM can use β a transcript with timecodes, the on-screen text, a description and OCR of every image, the caption and metadata β as Markdown or JSON. It is an MCP server (claude mcp add --transport http linkdigest https://linkdigest.dev/mcp --header "Authorization: Bearer $KEY"), a REST API, and a web console. Measured on a 17-image Xiaohongshu note: 17 images described and read, 381 on-screen text fragments, 13 key points, 119 seconds. Three digests are free, no card.
What it does not do, stated up front: Bilibili refuses our server's address (HTTP 412), Instagram is wired but not verified, Facebook is out of scope. A digest that could not read part of a post says so in a degraded field instead of returning a thin result quietly β and a digest that read nothing costs nothing.
Every claim here was checked against a live link. Where something doesn't work, it says so.
Paste a Xiaohongshu link into an AI coding agent and it sees nothing. Same for Douyin. The usual answer β "just use yt-dlp" β is half true in a way that wastes an afternoon, so here is the whole picture.
XiaoHongShuIE reads note.video.media.stream. That is a video note. Image notes β εΎζ, a caption plus a stack of photos β are the majority of the platform and usually the ones worth reading, and the extractor has nothing to say about them.
Getting those means parsing the note payload out of the page itself: the state blob the page ships, and the imageList inside it. Reusing yt-dlp's own js_to_json and traverse_obj for that keeps you aligned with upstream when the page shape shifts, which it does.
This one cost me an embarrassing amount of time.
Request a note with a mobile user agent and you get 202KB of app-download shell whose <title> is just the site name. Request the same URL with a desktop user agent and you get 85KB containing the real note.
Both are HTTP 200. Nothing tells you that you got the wrong one except that the content isn't there.
[Diagram: The same note URL returns 202KB of app shell to a mobile user agent, and 85KB containing the note to a desktop one.]
Fetching a note URL directly, with no prior session, does not work. Fetch the /explore feed first, keep the cookie jar it gives you β acw_tc, abRequestId β and then the note loads.
No account, no credentials, no API key. Just the same two-step a browser performs without you noticing.
Coverage and output for Douyin: platforms/douyin. Douyin refuses anonymous requests outright: captcha on the web page, 403 from the APIs. There is no user-agent trick here. That path needs cookies from a logged-in session, and yt-dlp does not support the platform at all.
Coverage and output for YouTube: platforms/youtube. The single most common "it worked locally and broke in production" report. YouTube blocks datacenter IP ranges, so a yt-dlp fetch that is perfect on your machine fails from EC2, App Runner, Lambda or anywhere else you deploy β and no amount of configuration fixes an IP-range block.
The workable fallback is a model that watches the video and returns a transcript, which costs meaningfully more than parsing captions and is worth measuring separately.
Being honest about this saves everyone time:
The bug that taught me the most had nothing to do with extraction.
A note was digested during a provider rate-limit storm. Every vision batch failed, the per-batch error handler swallowed each one, and the job returned images: 0, ocr: 0 β a perfectly well-formed, completely empty result. It was then written to a shared cache with a 30-day TTL.
Long after the underlying problem was fixed, that link still returned nothing, because a cached answer is cheaper to serve than to recompute. The pipeline was healthy. The cache was serving a fossil, and nothing downstream could tell the difference between an empty result and an easy one.
Two things came out of that:
If you are building anything that caches derived content, that second one is the trap. The obvious fix is the expensive one. The findings are the same whether you use a hosted service or write your own.
For how this fits into an agent's workflow end to end β the MCP tool, the job id for long media, what the output looks like β see Reading links your agent can't open.