What it actually takes to read a Xiaohongshu post from a server

A developer built LinkDigest, a hosted reader and MCP server that converts links from Xiaohongshu, Douyin, TikTok, YouTube and X into LLM-usable text, including transcripts with timecodes, on-screen text, image descriptions and OCR. The project parses note payloads from page state blobs rather than relying on yt-dlp, which only handles video notes, and works around platform blocks such as mobile user-agent app shells, session cookies and datacenter IP restrictions. On a 17-image Xiaohongshu note, the service reported 17 images described, 381 on-screen text fragments and 13 key points in 119 seconds, while flagging unsupported platforms like Bilibili and Instagram in a degraded field.

Originally published at linkdigest.dev https://linkdigest.dev/blog/reading-xiaohongshu-from-a-server?ref=devto , where I build this. Paste a Xiaohongshu, Douyin, TikTok, YouTube or X link into an AI agent and it fetches the URL, gets an app-download shell or a login wall, and tells you there is nothing there. It is not wrong. The content of those posts is video, images, and text printed inside images, behind tokenised share links — none of it is in the HTML a fetch returns. Strip the <script tags from a real Xiaohongshu note and 264 characters of navigation are left. LinkDigest https://linkdigest.dev/?ref=devto is a hosted reader: one call turns the link into text an LLM can use — a transcript with timecodes, the on-screen text, a description and OCR of every image, the caption and metadata — as Markdown or JSON. It is an MCP server claude mcp add --transport http linkdigest https://linkdigest.dev/mcp --header "Authorization: Bearer $KEY" , a REST API, and a web console. Measured on a 17-image Xiaohongshu note: 17 images described and read, 381 on-screen text fragments, 13 key points, 119 seconds. Three digests are free, no card. What it does not do, stated up front: Bilibili refuses our server's address HTTP 412 , Instagram is wired but not verified, Facebook is out of scope. A digest that could not read part of a post says so in a degraded field instead of returning a thin result quietly — and a digest that read nothing costs nothing. Every claim here was checked against a live link. Where something doesn't work, it says so. Paste a Xiaohongshu https://linkdigest.dev/platforms/xiaohongshu?ref=devto link into an AI coding agent and it sees nothing. Same for Douyin. The usual answer — "just use yt-dlp" — is half true in a way that wastes an afternoon, so here is the whole picture. XiaoHongShuIE reads note.video.media.stream . That is a video note. Image notes — 图文, a caption plus a stack of photos — are the majority of the platform and usually the ones worth reading, and the extractor has nothing to say about them. Getting those means parsing the note payload out of the page itself: the state blob the page ships, and the imageList inside it. Reusing yt-dlp's own js to json and traverse obj for that keeps you aligned with upstream when the page shape shifts, which it does. This one cost me an embarrassing amount of time. Request a note with a mobile user agent and you get 202KB of app-download shell whose <title is just the site name. Request the same URL with a desktop user agent and you get 85KB containing the real note. Both are HTTP 200. Nothing tells you that you got the wrong one except that the content isn't there. Diagram: The same note URL returns 202KB of app shell to a mobile user agent, and 85KB containing the note to a desktop one. Fetching a note URL directly, with no prior session, does not work. Fetch the /explore feed first, keep the cookie jar it gives you — acw tc , abRequestId — and then the note loads. No account, no credentials, no API key. Just the same two-step a browser performs without you noticing. Coverage and output for Douyin: platforms/douyin https://linkdigest.dev/platforms/douyin?ref=devto . Douyin refuses anonymous requests outright: captcha on the web page, 403 from the APIs. There is no user-agent trick here. That path needs cookies from a logged-in session, and yt-dlp does not support the platform at all. Coverage and output for YouTube: platforms/youtube https://linkdigest.dev/platforms/youtube?ref=devto . The single most common "it worked locally and broke in production" report. YouTube blocks datacenter IP ranges, so a yt-dlp fetch that is perfect on your machine fails from EC2, App Runner, Lambda or anywhere else you deploy — and no amount of configuration fixes an IP-range block. The workable fallback is a model that watches the video and returns a transcript, which costs meaningfully more than parsing captions and is worth measuring separately. Being honest about this saves everyone time: The bug that taught me the most had nothing to do with extraction. A note was digested during a provider rate-limit storm. Every vision batch failed, the per-batch error handler swallowed each one, and the job returned images: 0, ocr: 0 — a perfectly well-formed, completely empty result. It was then written to a shared cache with a 30-day TTL. Long after the underlying problem was fixed, that link still returned nothing, because a cached answer is cheaper to serve than to recompute. The pipeline was healthy. The cache was serving a fossil, and nothing downstream could tell the difference between an empty result and an easy one. Two things came out of that: If you are building anything that caches derived content, that second one is the trap. The obvious fix is the expensive one. The findings are the same whether you use a hosted service or write your own. For how this fits into an agent's workflow end to end — the MCP tool, the job id for long media, what the output looks like — see Reading links your agent can't open https://linkdigest.dev/blog/reading-social-links-from-an-agent?ref=devto .