A few weeks ago, a TikTok dataset published on Hugging Face caught my eye: it contained metadata of 5.6B videos from the platform. That’s a huge dataset, and the data collection immediately appeared to me to be the most interesting part. Using a browser or an emulator to do so would have been too expensive. In fact, the write-up from DataSocial explained the general shape of the approach: it was completely headless, with no emulators in the loop.
I was able to read it before it disappeared (thanks Web Archive for the snapshot) and was intrigued by replicating it, but it didn’t reveal the ingredients of the secret sauce. They were put behind a paywall costing 1699 USD. Which, I mean, is totally fine, but then I asked myself: “I have this cool reverse engineering setup, made by Hermes Agent, Reverse Engineer Anything repo and GLM 5.3, maybe I can find the answers by myself”.
So I did. Or better, we did. Well, most of the work was done by GLM; I just set the direction and checked the results.
Give your AI a web data layer – Decodo’s Web Data API turns any site into clean, structured data your models can actually use.
The workstation: a model, an agent, and a tool #
I have limited experience in reverse engineering, so I needed to delegate most of the work to LLMs while making sure I wasn’t fooled by them. Western models (ChatGPT, Claude) have very strict guardrails and block any request that resembles a cybersecurity one. For this reason, even in past articles, we used GLM or Manus for our investigation on anti-bots. This time I’m using GLM 5.3, hosted on OpenRouter, called by Hermes agent.
Hermes gives all the remaining tools we need to complete the job and I’ve integrated them with the Reverse Engineer Anything repo, which turns a general agent into a reverse engineer.
It is an open source project that runs as an MCP server, the standard way an agent plugs into external tools. You install it with npx rea-agents setup, pick your agent, approve the changes it proposes, and restart. REA drives the programs this field has always used: Ghidra, Hopper or IDA for native binaries, jadx for Android, adb for talking to a device, TShark for captured packets. Instead of choosing the tool yourself and gluing together the results, the agent asks REA for the pseudocode of a function or the strings in a library; REA runs the right program and returns a clean answer, along with its evidence and limitations, which also helps non-specialists like me. On top of REA and its tools, I also installed two classics like Frida for watching a running app and mitmproxy for reading its traffic.
Tired of getting blocked while scraping the web? ScrapingBee handles proxies, browsers, anti-bot systems, and retries so you can focus on your data. Get clean Markdown, JSON, or HTML from the web with up to a 99.9% success rate. ScrapingBee is SOC 2 Type II and GDPR compliant, and trusted by 4,000+ developers.
Why TikTok is different #
Most of the time, when I find an internal API behind an app, the hard part is already over. You intercept the traffic, you read the request, you replay it from Python, and you are done. The app was just a nicer way to call an endpoint that happily answers anyone who asks. I have written about this before, from reverse-engineering a mobile API to pulling data from apps with HTTP Toolkit and chasing JWT tokens.
TikTok’s mobile API doesn’t work like that, but it’s way more strict. Every request to its private API carries a signature, a short blob the app computes and attaches to prove the request came from a real, untampered copy of the app. The problem is that if the signature is wrong, the server does not argue but it answers with a normal HTTP 200 and an empty body. No error, no code, nothing to say what you broke.
The app is really checking four things at once, and all four have to be right on the same request:
- A device identity that TikTok issued, so the request looks like it comes from a device it already knows.
- The request signature, computed by a native library embedded inside the app.
- The right regional endpoint for what you are asking, because the API is split by region and route.
- A network fingerprint that looks like a phone and not a script.
Miss any one, and you get the same empty 200. There is no partial success and no hint about which of the four failed. The consequence is that you cannot trust the status line at all, but the only honest signal is the body: a short or empty body means a gate is wrong, and a real payload means all four passed.
So the first useful thing to build, before touching any of the gates, is a small judge that determines success based on the content of the response rather than its status code. In the real work, it is a few lines, and this is the one piece of code worth showing because everything else depends on it:
def classify(http_status: int, body: bytes) -> Outcome:
if http_status != 200:
return Outcome(False, f"upstream HTTP {http_status}",
retry_would_help=(http_status == 429))
if len(body) < MIN_BODY_BYTES:
return Outcome(False, f"empty upstream body ({len(body)} bytes)",
retry_would_help=True)
...
Starting with the heavy lifting #
The agent started reading the app without running it: an Android app hides its logic as bytecode that decompiles back to something close to Java, bundled with compiled native libraries that do not decompile to anything readable. The first part names things while the native part hides them. So what Hermes did was to use REA, and in particular Jadx, over the bytecode, so that we could create a map: the name of the signing library, the headers the signature carries, the app ID that turns up everywhere. At least now we have something to have a look at.
Then it ran the app, on a rooted emulator, with Frida attached so it could watch the signing code from the inside. Frida works by hooking functions in a running app and reading what goes in and out of them, so the natural place to attach is the boundary where the app’s Java code calls down into its native libraries. If you put a listener there, you can see every call that crosses from the managed side into native code.
The first real finding was that the signing we wanted to replicate, never crossed this border.
Let me explain that better: an Android app runs on two levels, the Java side on top and the compiled native libraries underneath, and the signer has a piece on each. The Java-facing piece, the one that sets the device id and starts the SDK, is called from Java, so the hook sees it. But the signature on an actual data request is made on the other side, by the app’s own networking stack calling the signing library directly, all inside the native code. For this reason, the hook sitting in the middle never sees it. The work we needed was happening under the floor we were standing on, which is worth knowing early because it kills the obvious plan of hooking the signer and reading its output.
Another problem we faced was the following: to read an app’s encrypted traffic, you put a proxy in the middle and make the app trust its certificate. TikTok’s API refuses: it pins its own certificate and rejects anything else before a single request goes out, so the proxy sees nothing.
The agent’s first tried the usual textbook move: replace the app’s certificate check with one that always says yes. The app crashed instantly, so the agent spent an afternoon treating it as a tamper-detection issue. But it was not: a pass through the decompiler showed that the real fix was not to replace the check but to let it run normally and flip only its final verdict from reject to accept. The check does its bookkeeping, the stack stays happy, and the pinning is bypassed anyway. With that, the traffic opened up, and the agent captured a batch of real signed requests.
After reviewing the traffic, we captured some signatures and wanted to understand whether they could be reused. Of course not.
The signature is computed over the exact bytes of the request, so changing a single byte, say to ask for the next page, gets the request rejected. You cannot sign once and paginate. Every request you ever send needs its own fresh signature.
So we had two options: reimplement the app’s signing math yourself, which is the exact piece the paid write-up sells and which depends on keys that are not public, or take the app’s own signing library and run it outside the app.
We choose the second, using a tool called unidbg, which can load that one native library on its own and run its functions on a plain server, with no phone involved, which produces signatures identical to the app’s for the simple reason that they come from the app’s own code. With that running, a request assembled in a script and signed by the borrowed library comes back full of data. Finally, the app is out of the loop.
The last piece was identity. A signature has to be made for a specific device that TikTok knows about, and the one captured from the emulator eventually ages out. So the agent learned to register fresh identities the way the app does on its first launch, warm them up, and prove each one works before trusting it.
From there it is ordinary engineering: a pool of working identities, the content judge from earlier reading every response, and a rotating residential IP so a blocked request retries from a clean address. The numbers from the real runs are modest, but the shape is the point. One machine, no phone, collecting the same public data the app shows, at a steady rate, and scaling by running more signers in parallel.
“Data is the new oil.” You’ve heard it a hundred times, but do you know who does the drilling?With AI, it’s more true than ever. Models are trained on data scraped from the web. Agents need browser automation that doesn’t fall over at the first CAPTCHA. But who’s doing the heavy lifting? Proxies that actually work.
That’s why we partnered with Geonode for an aperitivo on 5 November in Milan. No talks to sit through but just people who build scrapers, data pipelines and AI agents for a living, and a drink. If you’re one of them, here’s the link to join: https://luma.com/5c4v7zjk
The code behind it, and the proof it ran #
“Ok, nice work, but we want the code; otherwise, we won’t believe you,” you would say. I’m not super convinced about sharing the code; I’d like to avoid requests to take down the article or even being deindexed.
The code lives in the Lab’s repository, at github.com/TheWebScrapingClub/thelab, directory 118.TikTok. The work is a set of small Python modules, each doing one job, and they fall into two groups. Only the first group is in the repo. The code that actually touches TikTok stays out of it on purpose.