4.5B Posts Scraped from TikTok

A developer scraped 4.5 billion TikTok videos and uploaded the dataset to Hugging Face, collected via TikTok's private mobile API over three weeks. The system gathered 3.23 billion creator profiles, 5.94 billion videos, and 2.8 billion comments, with the dataset including captions, view counts, like counts, comment counts, save counts, sound, country, and posting time. The developer documented the technical process, including device registration, request signing, regional hosts, and TLS fingerprinting, and made 24 API endpoints available.

Scraping TikTok's Mobile API TikTok's Android app talks to a private HTTP+JSON API that is faster than the web endpoints and returns considerably more. This is a technical guide to reaching it: how devices are registered, how requests are signed, how the regional hosts are partitioned, and how the TLS handshake is fingerprinted. A system built on it collected 3.23 billion creator profiles, 5.94 billion videos and 2.8 billion comments in three weeks. Free dataset. I uploaded 4.5 billion of those videos to Hugging Face: captions, view, like, comment and save counts, the sound, the country and the posting time. huggingface.co/datasets/kuben-developer/tiktok-videos-4b https://huggingface.co/datasets/kuben-developer/tiktok-videos-4b What you can pull. Creator profiles, every video a creator has posted, followers and following lists, and TikTok's own similar-creator graph. Full video detail with the complete statistics block. Comments and comment replies, each with the commenter's account. Sounds, the videos using them, and the trending sounds chart. Hashtags and their videos, newest or most popular. Keyword search across videos, creators and sounds. Trending shelves and camera effects. 24 endpoints in all api , each with a measured success rate. What this is Almost every TikTok scraper you will find drives a headless browser or hits the public web endpoints. Both are the wrong layer: slow, fragile, and missing most of the interesting fields. The Android app does not use either. It talks to a private HTTP+JSON API, the same one com.zhiliaoapp.musically hits when you scroll, and that API is fast, stable, and returns far more. Getting into it is the hard part, and it is hard in a specific way. Four completely unrelated things have to be right at once: a device credential TikTok issued, a valid request signature, the correct regional host, and a TLS handshake that looks like a phone. Get any one of them wrong and you receive the identical response: a clean HTTP 200 with an empty body . No error message. No status code. Your HTTP client reports success, your logs stay green, and your database fills with nothing. There is no signal telling you which of the four you are standing at. This article walks through all four, then documents the 24 endpoints that come out the other side. It names the primitives, shows the real pipeline and gives measured numbers rather than claims. None of the four has a feedback loop. A wrong rotation constant, a wrong byte order, a wrong host and a wrong cipher suite in the handshake all produce the same well-formed request and the same empty response, so there is no error to bisect on and no partial credit. Everything below is anonymous device traffic. There is no login anywhere in this system, no account, no session cookie. That also means anything genuinely account-gated your own DMs, private videos, who liked what is out of reach and stays out of reach. No amount of tuning gets you there. Anatomy of a request Before anything else, here is what one of these requests actually looks like. This is a real call, with the identifying values shortened: Three things to notice, because each one bites later: - Two thirds of the URL is device identity. Thirty-eight common parameters describe the handset, the carrier, the region and the app build. They are not decoration. The signature covers them. - , not chosen by you. device id and iid are issued by TikTok cdid and openudid you generate and submit at registration. Getting the distinction wrong is the first wall. - Parameter order is fixed. The signature hashes the query string as a literal, so url.Values.Encode , which sorts keys alphabetically, silently produces an invalid signature. In Go you have to build the query by hand. The vocabulary, since it recurs throughout: | Field | What it is | Origin | |---|---|---| | aid | Application id. 1233 is the main app musically , 473824 is Lite, 1340 is musically go. Different aid means a different signing key and a different endpoint set. | Constant | | device id | The durable device identity. 19 digits. | TikTok, at register | | iid | Install id. Pairs with device id . | TikTok, at register | | cdid | Client device id. A UUID you generate. | You | | openudid | 16 hex characters you generate. | You | | license id | Feeds the X-Ladon key schedule. | Constant per app | | version code | App build. Gates which endpoints answer at all. | You choose | The first empty 200 A correctly implemented signer produces output that verifies against captured traffic, with parameters matching byte for byte. The response is still this: This is TikTok's soft block, and it is the single most important thing to understand about this API. It is not a 403. It is not a 429. It is not a challenge page. It is a successful HTTP response containing nothing. Which means this code, which is what everyone writes first, is silently broken: res = requests.get url, headers=signed if res.ok: True. Always true. store res.json {} stored, no exception six hours later: 400,000 rows in the database, all empty, nothing in the error log, dashboard green It is expensive to debug because four unrelated failures produce it : - Your device was never activated § activation activate - Your signature is wrong § X-Argus argus - You are talking to the wrong regional host § regions regions - Your TLS handshake looks like a server, not a phone § JA3 ja3 There is nothing in the response to tell you which. You cannot bisect it by reading errors, because there are none. The only way through is to fix all four and measure each one in isolation. Where device IDs come from You cannot invent a device id . TikTok issues it, from /service/2/device register/ on its logging host, in exchange for a plausible handset. The request body is a JSON document app header, device header, custom block encrypted with TTEncrypt TikTok's own body cipher, a simple byte-level transform with a fixed key schedule and posted as application/octet-stream;tt-data=a . It goes out with the full signature set, so you need working signing before you can get a device, and the signing needs a device . You bootstrap with the client-generated fields and zeros where the issued ones go. The body's shape, with the parts that matter: { "magic tag": "ss app log", "header": { // app identity: must agree with the aid in the query string "aid": 473824, "package": "com.ss.android.ugc.tiktok.lite", "app version": "32.8.2", "version code": 320820, "sdk version": "...", "git hash": "...", "sig hash": "...", // hardware: every field here has to be internally consistent "device model": "SM-A136U", "device brand": "Samsung", "device manufacturer": "samsung", "cpu abi": "arm64-v8a", "os version": "12", "os api": 30, "resolution": "2280 1080", "density dpi": 440, "rom": "...", "rom version": "...", // identity you generate and are about to trade in "cdid": "<uuid4 ", "openudid": "<16 hex ", "clientudid": "<uuid4 ", "google aid": "<uuid4 ", // region: carrier must plausibly exist in this country "region": "SG", "sim region": "sg", "carrier": "Singtel", "mcc mnc": "52506", "tz name": "Asia/Singapore", "tz offset": 25200, "custom": { "screen width dp": 408, "screen height dp": 883, "web ua": "Dalvik/2.1.0 Linux; U; Android 12; SM-A136U Build/... ", "apk last update time": 1788361409271 }, "apk first install time": 1788360902118 }, " gen time": 1788361402240 } Every field there is checked against the others. A Samsung SM-A136U has a specific screen resolution, a specific DPI, a specific ABI, and shipped with a specific range of Android versions. It is sold on carriers in some countries and not others. A flagship handset on a network that never carried it is not a real phone, and the registration is refused. Rather than generating these procedurally, I build them from a catalogue of ~250 real Android device profiles crossed with a carrier table of MCC/MNC pairs roughly 2,000 rows, derived from public numbering-plan data . Pick a handset, pick a carrier that actually exists in the target country, fill in the coherent values. A successful registration comes back with the two ids you needed: { "device id str": "7680616891110524437", "install id str": "7680617333853718293", "new user": 1 } Most implementations stop here. The activation call With registration working, most endpoints answered. Video listings, search, hashtags, sounds, all fine. But /aweme/v1/user/profile/other/ , the full profile record, returned the empty 200 every single time , on every device I made, forever. The obvious suspect is the signature, and it is the wrong one. The tell is that an older pool of devices, generated months earlier by different code, worked fine on that same endpoint with the same signer and the same parameters. The only difference was in how the devices had been created, and it came down to one extra HTTP call: GET /service/2/app alert check/?<common params &cronet version=...&ttnet version=... &tt info=<base64url TTEncrypt <60-field key=value blob → {"message":"success"} That is it. It returns nothing you need. It looks like telemetry, and functionally it is telemetry. It is the call the real app makes on launch, before it requests any data. That is what the call is for. A device that registered and then immediately started querying the API is, from ByteDance's side, an install that never launched . Registration alone does not make you a running app. The startup call does. | Device generation | Profile endpoint | | |---|---|---| | Register only | 0 / 360 | Correct signature. Empty body, every time, indefinitely. | | Register + startup call | 100 / 100 | Same code, same signature, one extra request. | Zero to a hundred percent, from a call whose response you throw away. It is not documented anywhere. It is not visible in a signature dump. It does not fail loudly. And because the symptom is the empty 200, it is indistinguishable from a broken signer. The tt info blob is the interesting part of the request: about sixty key=value pairs GAID, timezone, install id, device id, carrier, screen, ABI, locale, a request UUID TTEncrypt-ed and base64url-encoded. It is the app reporting its full environment on startup. My guess, and it is only a guess, is that this is where the device gets marked as a real install rather than a bare registration; I have not tried to prove it, because the empirical result is unambiguous. Proving a device before you use it The activation fixed the profile endpoint, but it introduced a second-order problem: activation itself sometimes fails silently, and a device that failed activation looks exactly like a device that succeeded until you use it. So generation does not end at activation. It ends with a real read against a known creator. If real content comes back, the device joins the pool. If not, it is thrown away. Not retried, not quarantined. Discarded . func GenerateDevice client http.Client, country string map string any, error { tmpl, err := NewAndroidTemplate // handset × carrier ... if err := registerDevice client, tmpl ; err = nil { return nil, fmt.Errorf "register: %w", err } // Without this TikTok will not serve profile detail to a fresh device. if err := appAlertCheck client, tmpl ; err = nil { return nil, fmt.Errorf "activate: %w", err } // Survivorship filter: only provably-capable devices enter the pool. if profileCapable client, tmpl { return nil, errors.New "profile probe failed: device not capable" } return tmpl, nil } The three-stage pipeline. Roughly 60-95% of attempts survive it, depending almost entirely on proxy quality. Without the filter you get a pool that is a mixture of working and quietly dead devices, and because dead devices return the empty 200, the same as every other failure, the pool degrades invisibly. Your success rate drifts down over days and there is nothing in the logs to explain it. With the filter, the pool is uniformly capable by construction. Live health is visible from the running server: bash $ curl -s localhost:8080/v1/devices | jq { "live": 43, "generated total": 43, "rejected total": 2, "evicted total": 0, "success total": 177, "failure total": 74, "generation survival rate": 0.9555 } The five headers Every request carries a family of headers that TikTok verifies before it looks at your query. The names are public; knowing them is worth nothing. | Header | Binds | Difficulty | |---|---|---| | X-Khronos | Unix seconds. Bounds replay. | None, it is a timestamp | | X-Ss-Stub | MD5 of the request body | None, and only on POSTs | | X-Gorgon | Legacy digest over URL, body, time | Low. Public write-ups exist. | | X-Ladon | Timestamp + license id + app id | Moderate. Speck-128/256. | | X-Argus | Everything, bound to the device | High. Protobuf, two cipher layers, no feedback. | Two properties of the scheme shape everything downstream: The signature covers the query string, not just the path. There is no signing a template and varying the arguments. Change count=20 to count=21 and you recompute from scratch. The upside is that a retry is a genuinely fresh cryptographic operation and never a replay. The signature is bound to one device. You cannot sign with device A and send device B's identifiers. Every retry against a different device re-signs. Inside X-Argus X-Argus is not a hash of a string. Its plaintext is a protobuf message in proto3 wire format, varints and length-delimited fields, which is then run through a two-stage encryption pipeline. The message carries, among other fields: type Argus struct { Magic int32 // fixed marker Version int32 Rand int64 // per-request random, 0x10000000..0xFFFFFFFF MsAppID string // "1233" / "473824" LicenseID string DeviceID string SdkVersion int32 SdkVersionStr string AppVersion string EnvCode byte CreateTime int64 // X-Khronos, again, inside the blob BodyHash byte // SM3 of the body 16 zero bytes on GET QueryHash byte // SM3 of the literal query string AlgorithmCount struct { SignCount int32 // how many signatures this install has made ReportCount int32 SettingCount int32 Timestamp int64 } SecDeviceToken string IsAppLicense int64 PskHash byte CallType int32 ChannelInfo struct { PhoneInfo, Channel string; ... } } The subset the signer actually populates. Establishing the field numbering is most of the reverse-engineering work. AlgorithmCount.SignCount is a counter of how many requests this install has signed. A real phone's counter climbs steadily over the life of the install. A scraper that emits a constant, or resets to zero on every request, is producing a statistically obvious pattern even when every individual signature verifies. I seed it randomly per device in a plausible range and it has never been a problem, but it is the kind of field that exists specifically so that naive replay is detectable in aggregate rather than at the individual request. The pipeline Once the protobuf is serialised, it goes through this, in order: 1. pb = proto3 serialize Argus{...} 2. padded = pkcs7 pb, 16 // key derivation: the signing key is a per-aid 32-byte constant 3. xmKey = SM3 signKey 0:32 || f rand lo, rand hi || signKey 0:32 4. enc1 = Simon-128/256-ECB key = xmKey, padded 5. enc1 = reverse bytes enc1 6. enc1 = xor mix enc1, derived from rand // bit-level, order-sensitive // framing: a version byte, entropy, and a 3-byte marker built from // the first bytes of two separate SM3 digests 7. framed = hexFirstByte aid || rand bytes || append array || enc1 8. enc2 = AES-128-CBC key = MD5 signKey 0:16 , iv = MD5 signKey 16:32 , framed 9. X-Argus = base64 rand lo || enc2 Two encryption layers with different primitives and different key derivations, with a byte reversal and an XOR mix sandwiched between them. None of the individual steps is hard. The difficulty is entirely that there is no feedback . Get step 6 wrong and you produce a perfectly well-formed, correctly-sized, base64-clean header that TikTok answers with an empty 200. Which is why the implementation ships with independent test vectors for every primitive. You verify Simon, Speck, SM3 and TTEncrypt separately against known input/output pairs, so that when a request fails you already know the crypto is right and the bug is in composition. Simon, Speck and SM3 The choice of primitives is deliberate, and it says something about the threat model. ARX ciphers Simon and Speck are lightweight block ciphers published by the NSA in 2013. Both are ARX constructions, built entirely from modular A ddition, bitwise R otation and X or. No S-boxes. No lookup tables. No multiplication. Speck's round function, in full, is two lines: x = ROR x, α + y ⊕ k y = ROL y, β ⊕ x // for the 128-bit block size: α = 8, β = 3, 64-bit words // 128/256 configuration: 256-bit key, 34 rounds Simon is the same idea with the addition swapped for AND, which makes it cheaper in hardware and slightly more expensive in software: x' = y ⊕ ROL x,1 & ROL x,8 ⊕ ROL x,2 ⊕ k y' = x // 128/256 configuration: 256-bit key, 128-bit block, 72 rounds, // round constants from the Z4 sequence a 62-bit LFSR period Why these and not AES? Three reasons, and they all point the same way: - They compile to almost nothing. A few hundred bytes of ARM, no tables, no data-dependent memory access. That matters when the code lives inside an obfuscated native library that has to be small and has to avoid cache-timing side channels that would make it easy to locate. - They are not in your standard library. AES is a function call in every language. Simon and Speck you have to implement, and the parameterisation space is large block size, key size, round count, rotation constants, key schedule so a wrong guess produces plausible ciphertext and no error. - They are easy to get subtly wrong. Speck's key schedule reuses the round function itself. Get the word order or the endianness wrong and you get 32 valid-looking round keys that are all incorrect. Note that AES-128-CBC is in the pipeline, as the outer layer. The interesting design choice is that the inner layer, the one actually protecting the protobuf, is the one you can't just call. SM3 SM3 is the Chinese national cryptographic hash standard GB/T 32905-2016 . 256-bit output, 512-bit blocks, Merkle-Damgård construction with a compression function structurally similar to SHA-256 but with two parallel message expansion schedules and a different round function: // two boolean functions, switching at round 16 FF x,y,z = x ⊕ y ⊕ z // j < 16 = x&y | x&z | y&z // j ≥ 16 GG x,y,z = x ⊕ y ⊕ z // j < 16 = x&y | ~x&z // j ≥ 16 // IV 7380166F 4914B2B9 172442D7 DA8A0600 A96F30BC 163138AA E38DEE4D B0FB0E4E SM3 shows up in ByteDance's stack for the obvious reason. It is also, usefully for them, absent from every Western standard library. And the two message expansion arrays W and W' are trivially transposable, so a large fraction of the reference implementations floating around are wrong in ways that only show up on certain inputs. TTEncrypt The body cipher, used for the registration payload and the activation blob. Not a standard construction, just a fixed-key byte transform with a small table. It is not cryptographically serious and is not meant to be; it exists to stop casual traffic inspection, and it is the easiest of the four to reimplement. X-Ladon Much simpler than Argus, and worth showing in full because it is a good illustration of how these schemes are layered: a cheap gate in front of an expensive one. plaintext = "<khronos -<license id -<aid " key = ascii hex MD5 rand bytes 4 || aid // 32 bytes cipher = Speck-128/256-ECB key, pkcs7 plaintext X-Ladon = base64 rand bytes || cipher Four random bytes, an MD5, and a Speck encryption of a dash-joined string. The random bytes are prepended to the output so the server can rederive the key. That is the whole construction. It filters out anyone who hasn't looked at the app at all, and costs approximately nothing to verify at scale. Argus is the expensive check that runs after. Version gating A correct signature is necessary and not sufficient. Some endpoints are gated on the client build, and the gate is server-side. The clearest case is comments. Same device, same signer, same second, same everything. Only version code differs: | App build | /aweme/v2/comment/list/ | |---|---| | 32.8.2 320802 | empty 200 | | 35.5.4 350504 | 178 KB of comments | The version bump also unlocked comment replies and follower listing. It is not that the older build's signature is rejected, because it verifies fine. It is that the endpoint is simply not served to that client version. Practically this means the app version is a per-endpoint property, not a global setting. In my catalogue each endpoint records the build it needs and the server swaps the four version fields transparently before signing: func WithAppVersion dev DevInfo, version, code string DevInfo { c := dev c.App.AppVersion = version c.App.AppVersionCode = code c.App.ManifestVersionCode = code c.App.UpdateVersionCode = code return &c } Pinning the newest build everywhere is not the answer, because newer builds tighten other checks. The catalogue exists so that each endpoint sits on the build that works for it. Region partitioning The third gate, and the one with nothing to go on: no error, no redirect, no hint in the response. TikTok does not run one API. It runs several regional data centres: alisg Singapore , useast1a , useast5 and others. And they do not serve the same endpoints to the same devices . With activation fixed, profile detail still failed on freshly generated devices while working on an older pool. Same code, same signer. The difference turns out to be the host: | Host | Fresh device, profile detail | Response | |---|---|---| | api16-normal-useast5.tiktokv.us | 50 / 50 | 9,517 bytes | | api16-normal-c-alisg.tiktokv.com | 1 / 50 | empty 200 | | api16-normal-c-useast1a.tiktokv.com | 0 / 50 | empty 200 | Same second, same credential, same signed request, three hosts, one answer. And music/detail is the reverse: it answers on useast1a and returns nothing on the Singapore host that serves almost everything else. So the host is part of the endpoint definition. Not a global base URL but a per-route property, established by measurement, because there is no documentation to consult: { ID: "user.info", Route: "/v1/user/info", Host: tiktok.HostUSEast5, // the ONLY host that serves this to fresh devices Path: "/aweme/v1/user/profile/other/", ... }, { ID: "music.info", Route: "/v1/music/info", Host: tiktok.HostUSEast1A, // and this one is the only host for THIS Path: "/aweme/v1/music/detail/", ... }, There is a useful second-order effect here. The device's registered region also influences content on the region-scoped endpoints: trending sounds and trending category shelves. Running one pool registered in US and another in BR gives you genuinely different charts from the identical call, which is how you get per-country data without any per-country code. The TLS fingerprint The fourth gate, and the one that is invisible at every layer an application developer normally inspects. Before any of your bytes arrive, your TLS client sends a ClientHello. Everything in it, and crucially the order of everything in it, is a fingerprint. JA3, the standard way of capturing this, is an MD5 of five comma-joined fields: TLSVersion , Ciphers , Extensions , EllipticCurves , ECPointFormats 771,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53, 0-23-65281-10-11-35-16-5-13-18-51-45-43-27-21,29-23-24,0 ↓ MD5 cd08e31494f9531f560d64c695473da9 A JA3 string and its hash. The cipher list and the extension list are ordered, and libraries order them differently. That fingerprint identifies your TLS library , and often its version, with high precision. OpenSSL, BoringSSL, NSS, Go's crypto/tls , Java's JSSE are all distinguishable, before a single byte of HTTP is exchanged. Go's crypto/tls has a very distinctive one. And no Android app has ever emitted it, because Android apps use BoringSSL through OkHttp. TikTok's useast5 edge checks. The experiment that isolated it The same request works from Python and fails from Go. Signature byte-identical, parameters byte-identical, cookies irrelevant it works with and without . Dump the exact headers Python just used, replay them from Go, and hold everything else constant. Same URL, same signature, same device, same second: // identical request, three clients, back to back python urllib3 / OpenSSL → 9,517 bytes curl OpenSSL → 9,519 bytes go crypto/tls → 0 bytes ← HTTP 200 Nothing about the request was different. The handshake was. The fix is uTLS https://github.com/refraction-networking/utls , which lets you specify the exact ClientHello to emit instead of accepting the one Go builds for you: cfg := &utls.Config{ServerName: host, NextProtos: string{"http/1.1"}} conn := utls.UClient raw, cfg, utls.HelloAndroid 11 OkHttp if err := conn.HandshakeContext ctx ; err = nil { return nil, fmt.Errorf "utls handshake: %w", err } One line of profile selection. Profile detail on fresh devices went from 0% to 100% . NextProtos is pinned to http/1.1 on purpose. The Android profile advertises h2, but the transport underneath this is HTTP/1.1 only. Negotiate h2 and you get a connection nothing can speak on. The Go proxy trap Short, and specific to Go. Wire up uTLS, test it directly, confirm the fingerprint has changed, then put it behind the rotating proxy. The failures come straight back. The reason is that http.Transport ignores DialTLSContext when Proxy is set . It dials the proxy, issues CONNECT itself, and then runs its own standard-library handshake over the resulting tunnel. Your custom dialer is silently discarded. No error, no warning, no log line. You have to do the tunnel by hand: dialTLS := func ctx context.Context, network, addr string net.Conn, error { // 1. plain TCP to the proxy raw, err := d.DialContext ctx, "tcp", proxyURL.Host ... // 2. CONNECT by hand: this is the part Transport would have done req := &http.Request{Method: "CONNECT", URL: &url.URL{Opaque: addr}, Host: addr, ...} req.Write raw resp, := http.ReadResponse bufio.NewReader raw , req if resp.StatusCode = 200 { return nil, fmt.Errorf "CONNECT: %s", resp.Status } // 3. NOW run the uTLS handshake over the tunnel u := utls.UClient raw, cfg, utls.HelloAndroid 11 OkHttp return u, u.HandshakeContext ctx } tr := &http.Transport{ DialTLSContext: dialTLS, DisableKeepAlives: true, // see the next section // note: NO Proxy field. Setting it would bypass all of the above. } The Proxy field is deliberately absent from the transport. Setting it is what silently discards the dialer. Detecting the empty 200 With all four gates passed you still need to know, per response, whether you actually got data. Status codes will not tell you. The check has to be on content: if resp.StatusCode = http.StatusOK { return body, fmt.Errorf "upstream HTTP %d", resp.StatusCode } if len body < minBodyBytes { // 64 // The soft block: 200 with almost nothing in it. return body, fmt.Errorf "empty upstream body %d bytes ", len body } var probe map string json.RawMessage if err := json.Unmarshal body, &probe ; err = nil { // HTML, usually a proxy error page rather than TikTok return body, errors.New "upstream body is not a JSON object" } if raw, ok := probe "status code" ; ok { var n int if json.Unmarshal raw, &n == nil && n = 0 { return body, fmt.Errorf "upstream status code %d", n } } Four conditions, in order: HTTP status, length floor, parseable JSON object, clean internal status code . Anything that fails one is retried against a different device from a different IP. Except when retrying is pointless Some non-zero status code values are TikTok answering rather than refusing. Retrying those four times is a waste of four devices and four IPs: status code | Message | Treated as | |---|---|---| | 2065 | User doesn't exist. | 404, no retry | | 3170 | user not exists | 404, no retry | | 3002060 | Profile user is hiding following list | 403, no retry | Which surfaces to the caller as a real answer instead of a gateway failure: bash $ curl -s localhost:8080/v1/user/following?user id=6744630345964389381 | jq { "error": { "code": "hidden by user", "message": "This creator has hidden their following list. Most accounts do; there is no way around it.", "upstream status code": 3002060, "upstream status msg": "Profile user is hiding following list", "retried": false, "retry would not help": true } } Everything else keeps its full retry budget, and when it exhausts it you get the per-attempt breakdown rather than a generic failure, which is what makes this debuggable in production: { "error": { "code": "upstream failed", "attempts": 4, "attempt failures": {"attempt": 1, "reason": "empty upstream body 0 bytes "}, {"attempt": 2, "reason": "empty upstream body 0 bytes "}, {"attempt": 3, "reason": "transport: ... EOF"}, {"attempt": 4, "reason": "upstream HTTP 429"} } } Real output from the weakest endpoint in the catalogue. Two soft blocks, a dropped connection, and an honest rate limit. Keep-alive pins the exit IP With all four gates passed, the highest-volume endpoint ran at 88.2% over 174 million attempts . Good, and at that volume the missing 12% is twenty million lost records. The obvious move is more retries. It does nothing, because of how rate limiting and connection reuse interact. Rate limiting here is per exit IP. A rotating proxy gateway assigns an exit IP per TCP connection . HTTP keep-alive, which every client does by default and which is normally exactly what you want, pins you to one exit IP for the life of that connection. So the retry went out from the address that had just been refused. And the next one. And the next: // keep-alive on a rotating proxy attempt 1 → exit 203.0.113.44 → empty 200 attempt 2 → exit 203.0.113.44 → empty 200 ← same IP attempt 3 → exit 203.0.113.44 → empty 200 ← same IP attempt 4 → exit 203.0.113.44 → empty 200 ← same IP // fresh connection per attempt attempt 1 → exit 203.0.113.44 → empty 200 attempt 2 → exit 198.51.100.7 → ok Four attempts, one IP, four identical failures. The retry budget bought nothing at all. It was structurally incapable of helping. Why the naive fix stalls at 96% Setting DisableKeepAlives: true everywhere took it to 96.2% and then stopped. The cause is that at full concurrency you are now paying a TLS handshake for every attempt, including the ~88% that were going to succeed first time. The proxy gateway, not TikTok, became the bottleneck and started refusing tunnels: {"attempt": 1, "reason": "transport: proxy CONNECT: 466 Too Many Requests"} The failure had moved, not gone. The production shape is a hybrid of the two, which comes down to two pools and one policy switch: // First-attempt pool: keep-alive, so the common case costs no handshake. r.proxyPool, = httpclient.New httpclient.Config{ProxyURL: cfg.ProxyURL} // Retry pool: DisableKeepAlives = fresh TCP = NEW exit IP. r.proxyPoolFresh, = httpclient.New httpclient.Config{ ProxyURL: cfg.ProxyURL, DisableKeepAlives: true, } // ...and in the request path: pool := r.proxyPool if attempt 0 && r.proxyPoolFresh = nil { pool = r.proxyPoolFresh // rotation exactly where it matters } | Configuration | Success | Bottleneck | |---|---|---| | Keep-alive everywhere | 88.2% | Retries reuse the blocked IP | | Keep-alive nowhere | 96.2% | Proxy gateway, handshake storm | | Keep-alive on first attempt only | 99.3% | none | Measured over hundreds of millions of calls across four shards. The self-hosted server described below keeps the simpler always-fresh form, because a single instance is nowhere near the load where the second bottleneck appears. Proxies: the one running cost Everything up to here is a software problem you solve once. The proxy is the single external dependency and the only recurring cost, and the requirement for one is structural rather than incidental. What a proxy is, briefly A proxy is a machine that makes the request on your behalf. You connect to it, it connects to TikTok, and TikTok sees the proxy's IP address instead of yours. A rotating gateway is one where each new connection comes out of a different address in a large pool, which is the property that matters here. Why it is mandatory rather than recommended Two reasons, and the second is the one people underestimate. Rate limiting is per exit IP. One address gets a budget and it is not a large one. Without a proxy every request in your system shares a single address, and you exhaust it in minutes. Retries are structurally useless without rotation. This is the point from the previous section rotation . When a request is soft blocked, the retry has to leave from a different address or it fails identically. Rotation per connection is the entire mechanism behind the jump from 88% to 99.3%. A static proxy gives you one IP and therefore gives you nothing. So the requirement is a rotating gateway, and you can verify yours actually rotates in one line before you commit to anything: bash $ for i in 1 2 3; do curl -s --proxy "$PROXY URL" https://api.ipify.org; echo; done 203.0.113.44 198.51.100.7 different IP each time = rotating, good 192.0.2.19 If the same address comes back three times, your retry budget is decorative and your success rate will sit near the first-try rate no matter what you set MAX ATTEMPTS to. What it actually costs The first thing to know is that you should not be paying by the gigabyte . Metered plans are the default recommendation in this space and they are the wrong shape for this workload, because the endpoints that return the most useful data are the ones measured in megabytes. A page of videos is 1.1 MB. A page of recommended creators is 2.9 MB. Metered billing turns every one of those into a line item. Flat monthly subscriptions exist for both proxy types, and they are what you want. Two tiers cover essentially everyone: | Tier | What you get | Cost | Realistic for | |---|---|---|---| Rotating datacenter | 100 concurrent threads at 200 Mbit/s | ~$150 / month | Millions of records. Where almost everyone should start. | Unlimited residential | Unmetered residential pool | ~$950 / month | Billions. What the six-day run at the top of this page used. | For enriching a few hundred thousand creators, tracking sounds daily, or mapping a niche, the $150 tier is sufficient rather than a compromise: a rotating datacenter pool registers devices, passes activation and sustains the success rates in the table further down. The residential tier is what you escalate to once you are saturating the datacenter one. What the $150 tier buys On a flat plan the two limits are threads how many requests can be in flight and line speed how many bytes per second . Which one binds depends entirely on response size, and the endpoints here differ by two orders of magnitude: a profile is 9.5 KB, a page of videos is 1.1 MB. The figures below are arithmetic from the measured response sizes and latencies in the benchmark, at 100 threads and 200 Mbit/s. They are ceilings at full saturation, so treat them as an upper bound rather than a promise. | Collecting | Per response | Binding limit | Ceiling | |---|---|---|---| | Creator profiles user.info | 9.5 KB | Threads | ~140/s · ~12M/day | | Comments 20 per page | 174 KB | Threads | ~1,600/s · ~140M/day | | Followers 20 per page | 178 KB | Threads | ~1,600/s · ~140M/day | | Videos with full metadata 20 per page | 1.1 MB | Line speed | ~450/s · ~39M/day | | Creator graph walk user.recommended | 2.9 MB | Line speed | ~350/s · ~30M/day | On the small endpoints you run out of threads long before bandwidth, so the fix is a higher thread count. On the video endpoints you saturate the line at around 22 requests a second, and more threads buy you nothing at all. That is the number MAX CONCURRENT exists to control, and setting it above what your plan can carry produces proxy CONNECT: 466 Too Many Requests in the attempt failures rather than more throughput. One $150 subscription comfortably collects millions of records , and tens of millions on the small endpoints. It is not enough for billions. The six-day run at the top of this page needed the unlimited residential tier at roughly $950 a month, sharded across four instances, and at that scale the proxy bill is the dominant cost of the entire operation. Scaling is horizontal either way: another subscription, another instance of the server pointed at it. Nothing in the code changes. What to look for when buying one Rotating, with a single gateway endpoint. Verify rotation with the loop above before you pay for a month. A published thread limit. If it is not stated, assume it is low. This is the number you actually plan around. Flat rate over metered , unless you know your volume is small and stays small. Metered plans punish exactly the endpoints that return the most useful data. Country targeting , if you want regional charts. The device region and the exit region should agree. A trial or one month first. Registration success rate is the real test and it varies between providers advertising the same product. Generate 30 devices and read generation survival rate before committing. Measured success rates Every endpoint ships with a real success rate rather than a claim: 100 calls each, at most 4 attempts, against a freshly generated pool over a rotating proxy gateway. Seeds are discovered live by walking the API rather than hardcoded, which changes the numbers. The note below explains why. | Endpoint | Success | Avg attempts | Avg response | |---|---|---|---| | user.info | 100% | 1.00 | 9 KB | | user.recommended | 100% | 1.08 | 2.9 MB | | music.posts | 100% | 1.06 | 1.7 MB | | music.posts fresh | 100% | 1.34 | 1.7 MB | | music.trending | 100% | 1.00 | 85 KB | | music.related | 100% | 1.00 | 139 KB | | hashtag.info | 100% | 1.00 | 3.8 KB | | hashtag.posts fresh | 100% | 1.00 | 1.3 MB | | search.general | 100% | 1.06 | 527 KB | | search.music | 100% | 1.11 | 100 KB | | search.users | 100% | 1.00 | 86 KB | | trending.categories | 100% | 1.07 | 380 KB | | trending.effects | 100% | 1.10 | 232 KB | | video.comment replies | 100% | 1.05 | 8 KB | | video.info | 94% | 1.96 | 58 KB | | user.following | 93% | 1.90 | 23 KB | | user.followers | 92% | 2.12 | 178 KB | | video.comments | 92% | 2.02 | 174 KB | | search.videos | 92% | 1.82 | 639 KB | | user.posts | 90% | 2.26 | 1.1 MB | | music.info | 90% | 2.15 | 12 KB | | hashtag.posts | 89% | 2.17 | 1.3 MB | | hashtag.search | 78% | 2.16 | 11 KB | | feed.recommended | 10% | 3.93 | 248 KB | Average attempts is the more informative column. A 100% endpoint at 1.00 attempts succeeds first time, every time. A 92% endpoint at 2.12 attempts is being soft-blocked on roughly half its first tries and recovering on retry, which means a wider budget moves it, whereas nothing moves a first-try-clean endpoint because there is nothing to move. feed.recommended is genuinely weak, at 10-25% across runs, and it is dominated by honest 429 s rather than soft blocks. An anonymous device with no watch history asking for a personalised feed is precisely the traffic shape TikTok most wants to throttle. It ships documented as weak with the two 100% alternatives named in its place. Seeds are discovered live rather than hardcoded: creator, then video, then a comment that actually has replies, then a sound that actually has videos, then a hashtag. This matters for accuracy. Point a follower benchmark at a creator who hides their following list and you measure TikTok correctly answering "nothing here" and score it as a failure. The 24 endpoints All of the above is packaged as a self-hosted Go service. One binary, no database, no queue, no emulator, no native library. Reference data is compiled in. bash $ cp .env.example .env set PROXY URL $ docker compose up -d $ docker compose logs -f TikTok Open API 1.0.0 starting config: port=8080 country=SG pool=15/30 attempts=4 concurrency=32 proxy=http:// @gw:9000 auth=true pool: 0 device s live, filling to 30 ... listening on http://0.0.0.0:8080 GET /healthz, GET /v1/endpoints pool: initial fill complete, 43 device s live Real startup output. Cold start is 15 to 60 seconds; the pool persists to disk so restarts after that are instant. All 24 routes are GET, all take query parameters, all return TikTok's JSON unmodified. Creators | Route | Parameters | Returns | |---|---|---| | /v1/user/posts | user id, count, max cursor | Videos, each with the full author object | | /v1/user/info | user id, sec user id | Full profile, incl. bio email , links, commerce flags | | /v1/user/followers | user id, sec user id, count, max time | Follower list | | /v1/user/following | user id, sec user id, count, max time | Following list, where published | | /v1/user/recommended | user id, sec user id, count | TikTok's own similar-creators graph | Videos | Route | Parameters | Returns | |---|---|---| | /v1/video/info | aweme id | Media, stats, sound, tags, author | | /v1/video/comments | aweme id, count, cursor | Comments with the commenter's user object | | /v1/video/comments/replies | aweme id, comment id, count, cursor | Second level of the comment tree | Sounds | Route | Parameters | Returns | |---|---|---| | /v1/music/info | music id | Sound detail incl. user count | | /v1/music/posts | music id, count, cursor | Popular videos using the sound | | /v1/music/posts/fresh | music id, count, cursor | Newest videos using the sound | | /v1/music/trending | count, cursor | Trending sounds chart, per device region | | /v1/music/related | aweme id, count, cursor | Sounds suggested for a video | Hashtags and search | Route | Parameters | Returns | |---|---|---| | /v1/hashtag/search | keyword, count, cursor | Hashtag ids with view counts | | /v1/hashtag/info | hashtag id | Hashtag detail | | /v1/hashtag/posts | hashtag id, count, cursor | Popular videos under the tag | | /v1/hashtag/posts/fresh | hashtag id, count, cursor | Newest videos under the tag | | /v1/search/videos | keyword, count, offset | Videos | | /v1/search/general | keyword, count, offset | Blended creators, videos and tags | | /v1/search/music | keyword, count, cursor | Sounds | | /v1/search/users | keyword, count, cursor | Handle or name → numeric user id | Discovery | Route | Parameters | Returns | |---|---|---| | /v1/trending/categories | count, cursor | The app's what-is-hot shelves | | /v1/trending/effects | count, cursor | Videos carrying sticker detail for trending effects | | /v1/feed | count, max cursor | Anonymous For You feed weak, see above | A real response Trimmed to the interesting fields. The raw object has several hundred keys: bash $ curl -s "localhost:8080/v1/user/posts?user id=6744630345964389381&count=20" \ | jq '{has more, max cursor, first: .aweme list 0 | {aweme id, desc, statistics, music, author} }' { "has more": 1, "max cursor": 1751028792000, "first": { "aweme id": "7678101694902832397", "desc": "Who Remembers 2022? fortnite piececontrolkyle dogwater", "statistics": { "play count": 19438, "digg count": 2461, "comment count": 39, "share count": 176 }, "music": { "id str": "7245172246876227585", "title": "Need 2 Instrumental " }, "author": { "uid": "6744630345964389381", "unique id": "freakynaughty", "nickname": "freaky", "follower count": 1277258 } } } Note that the author object is embedded in every video. One request gives you twenty videos and the full creator record. On the web that is twenty-one requests. Request metadata comes back in headers rather than polluting the body: HTTP/1.1 200 OK X-Endpoint-Id: user.posts X-Attempts: 2 ← first try was soft-blocked X-Elapsed-Ms: 1874 X-Upstream-Region: sg X-Device-Region: SG Fields the web does not give you | Field | Where | Populated | |---|---|---| | statistics.collect count | any video | always. Saves, often the earliest movement signal | | music.user count | any sound | always. Videos made with the sound | | author.ins id | video author object | ~26% of creators | | author.youtube channel id | video author object | ~19% | | bio email | user.info only | ~1% | | commerce user level | user.info | always | | bio link | nowhere | 0%. Not in the mobile API at all | The last row was measured across 579 creators on both endpoints that could plausibly carry it. The outbound profile link is a web-surface field only. Tutorial: a sound-trend detector Concrete worked example, because the endpoint list on its own does not tell you what the data is good for. The goal: find sounds that are taking off right now , before they are obviously trending. The signal is music.user count , how many videos have been made with a sound. The absolute number tells you a sound is big. The rate of change tells you it is moving, which is the part you want. Step 1. Snapshot the chart curl -s "$API/v1/music/trending?count=50" \ | jq -r '.music list | .id str, .user count, .title | @tsv' \ "sounds-$ date +%s .tsv" Run it hourly from cron. Each row is id, uses, title . Step 2. Diff consecutive snapshots python import glob, csv, collections snaps = sorted glob.glob "sounds- .tsv" -2: prev, curr = {r 0 : int r 1 , r 2 for r in csv.reader open f , delimiter="\t" } for f in snaps movers = for mid, n, title in curr.items : was = prev.get mid, 0, title 0 if was 0: movers.append n / was - 1, n - was, title, mid for growth, delta, title, mid in sorted movers, reverse=True :10 : print f"{growth:6.1%} +{delta: 8,} {title :40 :<40} {mid}" Step 3. Confirm it is actually accelerating Growth in the chart is a candidate, not a confirmation. The check that separates a real acceleration from a chart-placement artefact is the time spread of recent videos . Pull the newest videos using that sound and look at how tightly their upload times cluster: curl -s "$API/v1/music/posts/fresh?music id=$MID&count=30" \ | jq ' .aweme list .create time | max - min / 3600' 2.4 Thirty videos in a 2.4-hour window means thirty people picked up that sound this afternoon. Compare with the popular ordering, which tells you whether it has already landed: curl -s "$API/v1/music/posts?music id=$MID&count=30" \ | jq ' .aweme list .statistics.play count | add' High fresh-clustering plus low cumulative plays is the interesting quadrant. Lots of people using it, not much accumulated reach yet. That is a sound on the way up rather than one on the way down. Step 4. Find who is driving it curl -s "$API/v1/music/posts/fresh?music id=$MID&count=30" \ | jq -r '.aweme list .author | .follower count, .unique id | @tsv' \ | sort -rn | head Because the author object is embedded, this costs no extra requests. If one large account is at the top and everyone else is small, you are looking at a sound that one creator kicked off, which is a different and usually shorter-lived phenomenon than organic uptake across many mid-sized accounts. Step 5. Widen it music.trending is region-scoped to the device. Run a second instance with POOL COUNTRY=US and a third with POOL COUNTRY=BR and you get three independent charts from identical code. Sounds frequently break in one market days before another. The whole loop above is 4 requests per candidate sound per cycle. At 50 candidates hourly that is 200 requests an hour, which is nothing. The expensive version is walking user.recommended outward to map a niche, where responses run ~3 MB each and bandwidth, not rate limiting, becomes the constraint. Getting the code Everything described here is a private Go repository. One-time payment, permanent access, complete source. Get access https://buy.stripe.com/5kQ6oH7sn9s573b5C95J60q - Full signing stack Simon, Speck, SM3, TTEncrypt, Argus, Ladon with per-primitive test vectors - Device registration, activation and proving pipeline - uTLS transport with the manual proxy tunnel - All 24 endpoints, measured and documented - Pool management: health, eviction, rotation, persistence - Docker image and compose file - Live integration test suite with seed discovery - Python, Node, curl and .http clients - Generated reference docs for every endpoint - Reliability, pagination and error engineering guides - Lifetime updates to the same repository - Direct line during setup Checkout asks for your GitHub username. The repository invitation goes to that account automatically, normally within a minute of payment. Before you run it The one hard requirement is a rotating proxy gateway proxies . Rate limiting is per exit IP and the retry design assumes a new connection gets a new address, so a static proxy is no better than none. It is the single external dependency and it is not optional at volume. Maintenance is yours once you self-host. The repository is structured so that when something moves it is usually one struct literal in one table, but it is your struct literal. If you would rather skip the setup entirely, there is a done-for-you tier setup where I build it on your server and hand it over running. Done for you The repository is one component of a collection system, and on its own it answers one request at a time. Getting from there to billions of records is a different piece of work: deciding what to fetch next, keeping the queue moving through failures, landing the results somewhere that is still queryable at that size, and running the whole thing across enough shards and proxy capacity to sustain the rate. This tier is that system, built on your infrastructure and handed over running. Not a demo pointed at a few creators. The same shape as the one that produced the figures at the top of this page, sized to what you are collecting. Get in touch setup - Everything in the $699 tier, including lifetime updates - The whole collection system built on your server, from a clean box - Database architecture sized to your volume: engine choice, partition and sort keys, the update path for records that change, and the denormalisation your queries actually need - ClickHouse installed, tuned and sized, with retention set on its own system tables - The crawl pipeline: discovery, work queues, resume after failure, and deduplication so you are not paying to re-collect what you already hold - Sharded across multiple instances and proxy capacity, which is what billion scale actually requires - Proxy plan chosen with you, configured and rotation-verified - Concurrency and retry budget tuned to the plan you actually bought - Device pool generated, sized to your workload and persisting across restarts - Monitoring on freshness, write failures, pool health and disk, so a stall is visible instead of silent - Full smoke test run on your instance, all 24 endpoints returning live data - Live walkthrough of the endpoints you care about and how to paginate each - 7 days of support after handover The server and the proxy subscription are yours and are not included in the price. Both stay in your name and under your control. See the proxy section proxies for which tier your volume needs. Storage, if you are keeping the data Collecting the data is what the repository solves. Keeping it queryable once there are billions of rows is a separate problem with its own failure modes, and it is the half that decides whether the collection was worth doing. If you are building a store rather than running a one-off pull, that design is part of the handover: table engines, partition and sort keys, the update path for records that change, and the denormalisation that keeps a creator-to-video-to-sound question answerable without a join across billions of rows. Four decisions that determine whether it holds, each of them measured on a production ClickHouse store at billion-row scale: - Partition keys. A bare modulus of an id looks even and is not, because platform ids are not uniformly distributed. One table partitioned on author id % 8 ended up with a 43x spread between its largest and smallest partition, the largest heading for the size where merges stop keeping up. Hashing the id before the modulus flattens it. - The update path. ReplacingMergeTree keeps the newest whole row, not the newest value per column. Write a partial update and every field you left out is silently blanked. Nothing errors, and you find out much later. - Duplicate control. Re-collecting the same creator is normal and the storage cost of that compounds quietly. One table was carrying three times the rows it needed before anything made it visible, and rebuilding it returned several terabytes. - The database's own logs. ClickHouse writes text log and trace log by default with no retention. They reached 243 GB on one instance and took a 7 TB volume to full, which stops writes for everything sharing it. A TTL on the system tables is not optional at this scale. How it goes - You tell me what you are collecting. Creators in a niche, sounds on a schedule, comments on a set of videos. This determines the proxy tier and the pool size, so it is worth being concrete. - You provide a server and a proxy subscription. The collection layer is light and runs comfortably on two cores. If you are storing what you pull, the database is the part that needs real disk and real memory, and sizing it is part of step one rather than a surprise later. - I deploy and tune it. Usually the same week. You get the running instance, the repository access, and the reasoning behind each setting rather than just the settings. - We run the integration test together. You watch 24 endpoints come back with live data on your own hardware before you consider it delivered. Questions Do I need TikTok accounts? No. There is no login anywhere. Every device is an anonymous app install TikTok issued credentials to. Nothing to get banned, no credentials to rotate, no 2FA. Why won't it work without a proxy? It works fine for a look around. It does not work at volume, because rate limiting is per exit IP and the entire retry design assumes a new connection gets a new IP. Rotation is the requirement; a static proxy is no better than none. A rotating datacenter gateway at around $150 a month, flat rate, covers millions of records. Billions is a different tier at roughly $950. The proxy section proxies works through both and where each one runs out. Is it legal? It is against TikTok's terms of service. It is sold for research and educational use.