{"slug": "4-5b-posts-scraped-from-tiktok", "title": "4.5B Posts Scraped from TikTok", "summary": "A developer scraped 4.5 billion TikTok videos and uploaded the dataset to Hugging Face, collected via TikTok's private mobile API over three weeks. The system gathered 3.23 billion creator profiles, 5.94 billion videos, and 2.8 billion comments, with the dataset including captions, view counts, like counts, comment counts, save counts, sound, country, and posting time. The developer documented the technical process, including device registration, request signing, regional hosts, and TLS fingerprinting, and made 24 API endpoints available.", "body_md": "# Scraping TikTok's Mobile API\n\nTikTok's Android app talks to a private HTTP+JSON API that is faster than the web endpoints and returns considerably more. This is a technical guide to reaching it: how devices are registered, how requests are signed, how the regional hosts are partitioned, and how the TLS handshake is fingerprinted. A system built on it collected 3.23 billion creator profiles, 5.94 billion videos and 2.8 billion comments in three weeks.\n\n**Free dataset.** I uploaded 4.5 billion of those videos to Hugging Face:\ncaptions, view, like, comment and save counts, the sound, the country and the\nposting time.\n[huggingface.co/datasets/kuben-developer/tiktok-videos-4b](https://huggingface.co/datasets/kuben-developer/tiktok-videos-4b)\n\n**What you can pull.** Creator profiles, every video a creator has posted,\nfollowers and following lists, and TikTok's own similar-creator graph. Full video\ndetail with the complete statistics block. Comments and comment replies, each with\nthe commenter's account. Sounds, the videos using them, and the trending sounds\nchart. Hashtags and their videos, newest or most popular. Keyword search across\nvideos, creators and sounds. Trending shelves and camera effects.\n[24 endpoints in all](#api), each with a measured success rate.\n\n## What this is\n\nAlmost every TikTok scraper you will find drives a headless browser or hits the\npublic web endpoints. Both are the wrong layer: slow, fragile, and missing most of\nthe interesting fields. The Android app does not use either. It talks to a private\nHTTP+JSON API, the same one `com.zhiliaoapp.musically`\n\nhits when you\nscroll, and that API is fast, stable, and returns far more.\n\nGetting into it is the hard part, and it is hard in a specific way. Four completely unrelated things have to be right at once: a device credential TikTok issued, a valid request signature, the correct regional host, and a TLS handshake that looks like a phone.\n\nGet any one of them wrong and you receive the identical response:\n**a clean HTTP 200 with an empty body**. No error message.\nNo status code. Your HTTP client reports success, your logs stay green, and your\ndatabase fills with nothing. There is no signal telling you which of the four you\nare standing at.\n\nThis article walks through all four, then documents the 24 endpoints that come out the other side. It names the primitives, shows the real pipeline and gives measured numbers rather than claims.\n\nNone of the four has a feedback loop. A wrong rotation constant, a wrong byte order, a wrong host and a wrong cipher suite in the handshake all produce the same well-formed request and the same empty response, so there is no error to bisect on and no partial credit.\n\nEverything below is anonymous device traffic. There is no login anywhere in this system, no account, no session cookie. That also means anything genuinely account-gated (your own DMs, private videos, who liked what) is out of reach and stays out of reach. No amount of tuning gets you there.\n\n## Anatomy of a request\n\nBefore anything else, here is what one of these requests actually looks like. This is a real call, with the identifying values shortened:\n\nThree things to notice, because each one bites later:\n\n-\n**Two thirds of the URL is device identity.** Thirty-eight common parameters describe the handset, the carrier, the region and the app build. They are not decoration. The signature covers them. -\n, not chosen by you.`device_id`\n\nand`iid`\n\nare issued by TikTok`cdid`\n\nand`openudid`\n\nyou generate and submit at registration. Getting the distinction wrong is the first wall. -\n**Parameter order is fixed.** The signature hashes the query string as a literal, so`url.Values.Encode()`\n\n, which sorts keys alphabetically, silently produces an invalid signature. In Go you have to build the query by hand.\n\nThe vocabulary, since it recurs throughout:\n\n| Field | What it is | Origin |\n|---|---|---|\n| aid | Application id. `1233` is the main app (musically), `473824` is Lite, `1340` is musically_go. Different `aid` means a different signing key and a different endpoint set. | Constant |\n| device_id | The durable device identity. 19 digits. | TikTok, at register |\n| iid | Install id. Pairs with `device_id` . | TikTok, at register |\n| cdid | Client device id. A UUID you generate. | You |\n| openudid | 16 hex characters you generate. | You |\n| license_id | Feeds the X-Ladon key schedule. | Constant per app |\n| version_code | App build. Gates which endpoints answer at all. | You choose |\n\n## The first empty 200\n\nA correctly implemented signer produces output that verifies against captured traffic, with parameters matching byte for byte. The response is still this:\n\nThis is TikTok's soft block, and it is the single most important thing to understand about this API. It is not a 403. It is not a 429. It is not a challenge page. It is a successful HTTP response containing nothing.\n\nWhich means this code, which is what everyone writes first, is silently broken:\n\n```\nres = requests.get(url, headers=signed)\nif res.ok:                      # True. Always true.\n    store(res.json())           # {} stored, no exception\n\n# six hours later: 400,000 rows in the database, all empty,\n# nothing in the error log, dashboard green\n```\n\nIt is expensive to debug because *four unrelated failures produce it*:\n\n- Your device was never activated (\n[§ activation](#activate)) - Your signature is wrong (\n[§ X-Argus](#argus)) - You are talking to the wrong regional host (\n[§ regions](#regions)) - Your TLS handshake looks like a server, not a phone (\n[§ JA3](#ja3))\n\nThere is nothing in the response to tell you which. You cannot bisect it by reading errors, because there are none. The only way through is to fix all four and measure each one in isolation.\n\n## Where device IDs come from\n\nYou cannot invent a `device_id`\n\n. TikTok issues it, from\n`/service/2/device_register/`\n\non its logging host, in exchange for a\nplausible handset.\n\nThe request body is a JSON document (app header, device header, custom block) encrypted with TTEncrypt (TikTok's own body cipher, a simple byte-level transform\nwith a fixed key schedule) and posted as\n`application/octet-stream;tt-data=a`\n\n. It goes out with the full\nsignature set, so **you need working signing before you can get a device, and\nthe signing needs a device**. You bootstrap with the client-generated fields\nand zeros where the issued ones go.\n\nThe body's shape, with the parts that matter:\n\n```\n{\n  \"magic_tag\": \"ss_app_log\",\n  \"header\": {\n    // app identity: must agree with the aid in the query string\n    \"aid\": 473824, \"package\": \"com.ss.android.ugc.tiktok.lite\",\n    \"app_version\": \"32.8.2\", \"version_code\": 320820,\n    \"sdk_version\": \"...\", \"git_hash\": \"...\", \"sig_hash\": \"...\",\n\n    // hardware: every field here has to be internally consistent\n    \"device_model\": \"SM-A136U\", \"device_brand\": \"Samsung\",\n    \"device_manufacturer\": \"samsung\", \"cpu_abi\": \"arm64-v8a\",\n    \"os_version\": \"12\", \"os_api\": 30,\n    \"resolution\": \"2280*1080\", \"density_dpi\": 440,\n    \"rom\": \"...\", \"rom_version\": \"...\",\n\n    // identity you generate and are about to trade in\n    \"cdid\": \"<uuid4>\", \"openudid\": \"<16 hex>\",\n    \"clientudid\": \"<uuid4>\", \"google_aid\": \"<uuid4>\",\n\n    // region: carrier must plausibly exist in this country\n    \"region\": \"SG\", \"sim_region\": \"sg\", \"carrier\": \"Singtel\",\n    \"mcc_mnc\": \"52506\", \"tz_name\": \"Asia/Singapore\", \"tz_offset\": 25200,\n\n    \"custom\": {\n      \"screen_width_dp\": 408, \"screen_height_dp\": 883,\n      \"web_ua\": \"Dalvik/2.1.0 (Linux; U; Android 12; SM-A136U Build/...)\",\n      \"apk_last_update_time\": 1788361409271\n    },\n    \"apk_first_install_time\": 1788360902118\n  },\n  \"_gen_time\": 1788361402240\n}\n```\n\nEvery field there is checked against the others. A Samsung `SM-A136U`\n\nhas a specific screen\nresolution, a specific DPI, a specific ABI, and shipped with a specific range of\nAndroid versions. It is sold on carriers in some countries and not others. A\nflagship handset on a network that never carried it is not a real phone, and the\nregistration is refused.\n\nRather than generating these procedurally, I build them from a catalogue of ~250 real Android device profiles crossed with a carrier table of MCC/MNC pairs (roughly 2,000 rows, derived from public numbering-plan data). Pick a handset, pick a carrier that actually exists in the target country, fill in the coherent values.\n\nA successful registration comes back with the two ids you needed:\n\n```\n{\n  \"device_id_str\":  \"7680616891110524437\",\n  \"install_id_str\": \"7680617333853718293\",\n  \"new_user\": 1\n}\n```\n\nMost implementations stop here.\n\n## The activation call\n\nWith registration working, most endpoints answered. Video listings, search,\nhashtags, sounds, all fine. But `/aweme/v1/user/profile/other/`\n\n, the\nfull profile record, returned the empty 200 *every single time*, on every\ndevice I made, forever.\n\nThe obvious suspect is the signature, and it is the wrong one. The tell is that an older pool of devices, generated months earlier by different code, worked fine on that same endpoint with the same signer and the same parameters. The only difference was in how the devices had been created, and it came down to one extra HTTP call:\n\n```\nGET /service/2/app_alert_check/?<common params>\n    &cronet_version=...&ttnet_version=...\n    &tt_info=<base64url(TTEncrypt(<60-field key=value blob>))>\n\n→ {\"message\":\"success\"}\n```\n\nThat is it. It returns nothing you need. It looks like telemetry, and functionally\nit *is* telemetry. It is the call the real app makes on launch, before it\nrequests any data.\n\nThat is what the call is for. A device that registered and then immediately started\nquerying the API is, from ByteDance's side, an install that **never\nlaunched**. Registration alone does not make you a running app. The startup\ncall does.\n\n| Device generation | Profile endpoint | |\n|---|---|---|\n| Register only | 0 / 360 | Correct signature. Empty body, every time, indefinitely. |\n| Register + startup call | 100 / 100 | Same code, same signature, one extra request. |\n\nZero to a hundred percent, from a call whose response you throw away. It is not documented anywhere. It is not visible in a signature dump. It does not fail loudly. And because the symptom is the empty 200, it is indistinguishable from a broken signer.\n\nThe `tt_info`\n\nblob is the interesting part of the request: about sixty\n`key=value`\n\npairs (GAID, timezone, install id, device id, carrier, screen, ABI, locale, a request UUID) TTEncrypt-ed and base64url-encoded. It is the\napp reporting its full environment on startup. My guess, and it is only a guess, is\nthat this is where the device gets marked as a real install rather than a bare\nregistration; I have not tried to prove it, because the empirical result is\nunambiguous.\n\n## Proving a device before you use it\n\nThe activation fixed the profile endpoint, but it introduced a second-order problem: activation itself sometimes fails silently, and a device that failed activation looks exactly like a device that succeeded until you use it.\n\nSo generation does not end at activation. It ends with a real read against a known\ncreator. If real content comes back, the device joins the pool. If not, it is\nthrown away. Not retried, not quarantined. *Discarded*.\n\n```\nfunc GenerateDevice(client *http.Client, country string) (map[string]any, error) {\n    tmpl, err := NewAndroidTemplate()          // handset × carrier\n    ...\n    if err := registerDevice(client, tmpl); err != nil {\n        return nil, fmt.Errorf(\"register: %w\", err)\n    }\n    // Without this TikTok will not serve profile detail to a fresh device.\n    if err := appAlertCheck(client, tmpl); err != nil {\n        return nil, fmt.Errorf(\"activate: %w\", err)\n    }\n    // Survivorship filter: only provably-capable devices enter the pool.\n    if !profileCapable(client, tmpl) {\n        return nil, errors.New(\"profile probe failed: device not capable\")\n    }\n    return tmpl, nil\n}\n```\n\nThe three-stage pipeline. Roughly 60-95% of attempts survive it, depending almost entirely on proxy quality.\n\nWithout the filter you get a pool that is a mixture of working and quietly dead devices, and because dead devices return the empty 200, the same as every other failure, the pool degrades invisibly. Your success rate drifts down over days and there is nothing in the logs to explain it.\n\nWith the filter, the pool is uniformly capable by construction. Live health is visible from the running server:\n\n``` bash\n$ curl -s localhost:8080/v1/devices | jq\n{\n  \"live\": 43,\n  \"generated_total\": 43,\n  \"rejected_total\": 2,\n  \"evicted_total\": 0,\n  \"success_total\": 177,\n  \"failure_total\": 74,\n  \"generation_survival_rate\": 0.9555\n}\n```\n\n## The five headers\n\nEvery request carries a family of headers that TikTok verifies before it looks at your query. The names are public; knowing them is worth nothing.\n\n| Header | Binds | Difficulty |\n|---|---|---|\n| X-Khronos | Unix seconds. Bounds replay. | None, it is a timestamp |\n| X-Ss-Stub | MD5 of the request body | None, and only on POSTs |\n| X-Gorgon | Legacy digest over URL, body, time | Low. Public write-ups exist. |\n| X-Ladon | Timestamp + license id + app id | Moderate. Speck-128/256. |\n| X-Argus | Everything, bound to the device | High. Protobuf, two cipher layers, no feedback. |\n\nTwo properties of the scheme shape everything downstream:\n\n**The signature covers the query string, not just the path.** There is\nno signing a template and varying the arguments. Change `count=20`\n\nto\n`count=21`\n\nand you recompute from scratch. The upside is that a retry is\na genuinely fresh cryptographic operation and never a replay.\n\n**The signature is bound to one device.** You cannot sign with device A\nand send device B's identifiers. Every retry against a different device re-signs.\n\n## Inside X-Argus\n\n`X-Argus`\n\nis not a hash of a string. Its plaintext is a\n**protobuf message** in proto3 wire format, varints and length-delimited fields, which is then run through a two-stage encryption\npipeline.\n\nThe message carries, among other fields:\n\n```\ntype Argus struct {\n    Magic          int32      // fixed marker\n    Version        int32\n    Rand           int64      // per-request random, 0x10000000..0xFFFFFFFF\n    MsAppID        string     // \"1233\" / \"473824\"\n    LicenseID      string\n    DeviceID       string\n    SdkVersion     int32\n    SdkVersionStr  string\n    AppVersion     string\n    EnvCode        []byte\n    CreateTime     int64      // X-Khronos, again, inside the blob\n    BodyHash       []byte     // SM3 of the body (16 zero bytes on GET)\n    QueryHash      []byte     // SM3 of the literal query string\n    AlgorithmCount struct {\n        SignCount    int32   // how many signatures this install has made\n        ReportCount  int32\n        SettingCount int32\n        Timestamp    int64\n    }\n    SecDeviceToken string\n    IsAppLicense   int64\n    PskHash        []byte\n    CallType       int32\n    ChannelInfo    struct { PhoneInfo, Channel string; ... }\n}\n```\n\nThe subset the signer actually populates. Establishing the field numbering is most of the reverse-engineering work.\n\n`AlgorithmCount.SignCount`\n\nis a counter of how\nmany requests this install has signed. A real phone's counter climbs steadily over\nthe life of the install. A scraper that emits a constant, or resets to zero on every\nrequest, is producing a statistically obvious pattern even when every individual\nsignature verifies. I seed it randomly per device in a plausible range and it has\nnever been a problem, but it is the kind of field that exists specifically so that\nnaive replay is detectable in aggregate rather than at the individual request.\n\n### The pipeline\n\nOnce the protobuf is serialised, it goes through this, in order:\n\n```\n1.  pb        = proto3_serialize(Argus{...})\n2.  padded    = pkcs7(pb, 16)\n\n    // key derivation: the signing key is a per-aid 32-byte constant\n3.  xmKey     = SM3( signKey[0:32] || f(rand_lo, rand_hi) || signKey[0:32] )\n\n4.  enc1      = Simon-128/256-ECB( key = xmKey, padded )\n5.  enc1      = reverse_bytes(enc1)\n6.  enc1      = xor_mix(enc1, derived_from(rand))      // bit-level, order-sensitive\n\n    // framing: a version byte, entropy, and a 3-byte marker built from\n    // the first bytes of two separate SM3 digests\n7.  framed    = hexFirstByte(aid) || rand_bytes || append_array || enc1\n\n8.  enc2      = AES-128-CBC( key = MD5(signKey[0:16]),\n                          iv  = MD5(signKey[16:32]), framed )\n\n9.  X-Argus   = base64( rand_lo || enc2 )\n```\n\nTwo encryption layers with different primitives and different key derivations,\nwith a byte reversal and an XOR mix sandwiched between them. None of the individual\nsteps is hard. The difficulty is entirely that **there is no feedback**\n. Get step 6 wrong and you produce a perfectly well-formed, correctly-sized,\nbase64-clean header that TikTok answers with an empty 200.\n\nWhich is why the implementation ships with independent test vectors for every primitive. You verify Simon, Speck, SM3 and TTEncrypt separately against known input/output pairs, so that when a request fails you already know the crypto is right and the bug is in composition.\n\n## Simon, Speck and SM3\n\nThe choice of primitives is deliberate, and it says something about the threat model.\n\n### ARX ciphers\n\n**Simon** and **Speck** are lightweight block ciphers\npublished by the NSA in 2013. Both are *ARX* constructions, built entirely\nfrom modular **A** ddition, bitwise **R** otation and **X** or. No S-boxes.\nNo lookup tables. No multiplication.\n\nSpeck's round function, in full, is two lines:\n\n```\nx = (ROR(x, α) + y) ⊕ k\ny =  ROL(y, β)     ⊕ x\n\n// for the 128-bit block size: α = 8, β = 3, 64-bit words\n// 128/256 configuration: 256-bit key, 34 rounds\n```\n\nSimon is the same idea with the addition swapped for AND, which makes it cheaper in hardware and slightly more expensive in software:\n\n```\nx' = y ⊕ (ROL(x,1) & ROL(x,8)) ⊕ ROL(x,2) ⊕ k\ny' = x\n\n// 128/256 configuration: 256-bit key, 128-bit block, 72 rounds,\n// round constants from the Z4 sequence (a 62-bit LFSR period)\n```\n\nWhy these and not AES? Three reasons, and they all point the same way:\n\n-\n**They compile to almost nothing.** A few hundred bytes of ARM, no tables, no data-dependent memory access. That matters when the code lives inside an obfuscated native library that has to be small and has to avoid cache-timing side channels that would make it easy to locate. -\n**They are not in your standard library.** AES is a function call in every language. Simon and Speck you have to implement, and the parameterisation space is large (block size, key size, round count, rotation constants, key schedule) so a wrong guess produces plausible ciphertext and no error. -\n**They are easy to get subtly wrong.** Speck's key schedule reuses the round function itself. Get the word order or the endianness wrong and you get 32 valid-looking round keys that are all incorrect.\n\nNote that AES-128-CBC *is* in the pipeline, as the outer layer. The\ninteresting design choice is that the inner layer, the one actually protecting the protobuf, is the one you can't just call.\n\n### SM3\n\n**SM3** is the Chinese national cryptographic hash standard\n(GB/T 32905-2016). 256-bit output, 512-bit blocks, Merkle-Damgård construction with\na compression function structurally similar to SHA-256 but with two parallel message\nexpansion schedules and a different round function:\n\n```\n// two boolean functions, switching at round 16\nFF(x,y,z) = x ⊕ y ⊕ z                      // j < 16\n          = (x&y) | (x&z) | (y&z)          // j ≥ 16\nGG(x,y,z) = x ⊕ y ⊕ z                      // j < 16\n          = (x&y) | (~x&z)                 // j ≥ 16\n\n// IV\n7380166F 4914B2B9 172442D7 DA8A0600 A96F30BC 163138AA E38DEE4D B0FB0E4E\n```\n\nSM3 shows up in ByteDance's stack for the obvious reason. It is also, usefully for\nthem, absent from every Western standard library. And the two message expansion\narrays (`W`\n\nand `W'`\n\n) are trivially transposable, so a large\nfraction of the reference implementations floating around are wrong in ways that\nonly show up on certain inputs.\n\n### TTEncrypt\n\nThe body cipher, used for the registration payload and the activation blob. Not a standard construction, just a fixed-key byte transform with a small table. It is not cryptographically serious and is not meant to be; it exists to stop casual traffic inspection, and it is the easiest of the four to reimplement.\n\n## X-Ladon\n\nMuch simpler than Argus, and worth showing in full because it is a good illustration of how these schemes are layered: a cheap gate in front of an expensive one.\n\n```\nplaintext = \"<khronos>-<license_id>-<aid>\"\nkey       = ascii_hex( MD5( rand_bytes(4) || aid ) )   // 32 bytes\ncipher    = Speck-128/256-ECB( key, pkcs7(plaintext) )\nX-Ladon   = base64( rand_bytes || cipher )\n```\n\nFour random bytes, an MD5, and a Speck encryption of a dash-joined string. The random bytes are prepended to the output so the server can rederive the key. That is the whole construction.\n\nIt filters out anyone who hasn't looked at the app at all, and costs approximately nothing to verify at scale. Argus is the expensive check that runs after.\n\n## Version gating\n\nA correct signature is necessary and not sufficient. Some endpoints are gated on the client build, and the gate is server-side.\n\nThe clearest case is comments. Same device, same signer, same second, same\neverything. Only `version_code`\n\ndiffers:\n\n| App build | `/aweme/v2/comment/list/` |\n|---|---|\n| 32.8.2 (320802) | empty 200 |\n| 35.5.4 (350504) | 178 KB of comments |\n\nThe version bump also unlocked comment replies and follower listing. It is not that the older build's signature is rejected, because it verifies fine. It is that the endpoint is simply not served to that client version.\n\nPractically this means the app version is a per-endpoint property, not a global setting. In my catalogue each endpoint records the build it needs and the server swaps the four version fields transparently before signing:\n\n```\nfunc WithAppVersion(dev *DevInfo, version, code string) *DevInfo {\n    c := *dev\n    c.App.AppVersion          = version\n    c.App.AppVersionCode      = code\n    c.App.ManifestVersionCode = code\n    c.App.UpdateVersionCode   = code\n    return &c\n}\n```\n\nPinning the newest build everywhere is not the answer, because newer builds tighten other checks. The catalogue exists so that each endpoint sits on the build that works for it.\n\n## Region partitioning\n\nThe third gate, and the one with nothing to go on: no error, no redirect, no hint in the response.\n\nTikTok does not run one API. It runs several regional data centres:\n`alisg`\n\n(Singapore), `useast1a`\n\n, `useast5`\n\nand\nothers. And **they do not serve the same endpoints to the same devices**.\n\nWith activation fixed, profile detail still failed on freshly generated devices while working on an older pool. Same code, same signer. The difference turns out to be the host:\n\n| Host | Fresh device, profile detail | Response |\n|---|---|---|\n| api16-normal-useast5.tiktokv.us | 50 / 50 | 9,517 bytes |\n| api16-normal-c-alisg.tiktokv.com | 1 / 50 | empty 200 |\n| api16-normal-c-useast1a.tiktokv.com | 0 / 50 | empty 200 |\n\nSame second, same credential, same signed request, three hosts, one answer. And\n`music/detail`\n\nis the reverse: it answers on `useast1a`\n\nand\nreturns nothing on the Singapore host that serves almost everything else.\n\nSo the host is part of the endpoint definition. Not a global base URL but a per-route property, established by measurement, because there is no documentation to consult:\n\n```\n{\n    ID: \"user.info\", Route: \"/v1/user/info\",\n    Host: tiktok.HostUSEast5,    // the ONLY host that serves this to fresh devices\n    Path: \"/aweme/v1/user/profile/other/\",\n    ...\n},\n{\n    ID: \"music.info\", Route: \"/v1/music/info\",\n    Host: tiktok.HostUSEast1A,   // and this one is the only host for THIS\n    Path: \"/aweme/v1/music/detail/\",\n    ...\n},\n```\n\nThere is a useful second-order effect here. The device's registered region also\ninfluences *content* on the region-scoped endpoints: trending sounds and trending category shelves. Running one pool registered in `US`\n\nand\nanother in `BR`\n\ngives you genuinely different charts from the identical\ncall, which is how you get per-country data without any per-country code.\n\n## The TLS fingerprint\n\nThe fourth gate, and the one that is invisible at every layer an application developer normally inspects.\n\nBefore any of your bytes arrive, your TLS client sends a ClientHello. Everything in\nit, and crucially the *order* of everything in it, is a fingerprint.\nJA3, the standard way of capturing this, is an MD5 of five comma-joined fields:\n\n```\nTLSVersion , Ciphers , Extensions , EllipticCurves , ECPointFormats\n\n771,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,\n0-23-65281-10-11-35-16-5-13-18-51-45-43-27-21,29-23-24,0\n       ↓ MD5\ncd08e31494f9531f560d64c695473da9\n```\n\nA JA3 string and its hash. The cipher list and the extension list are ordered, and libraries order them differently.\n\nThat fingerprint identifies your TLS *library*, and often its version, with\nhigh precision. OpenSSL, BoringSSL, NSS, Go's `crypto/tls`\n\n, Java's JSSE\nare all distinguishable, before a single byte of HTTP is exchanged.\n\nGo's `crypto/tls`\n\nhas a very distinctive one. And no Android app has ever\nemitted it, because Android apps use BoringSSL through OkHttp. TikTok's\n`useast5`\n\nedge checks.\n\n### The experiment that isolated it\n\nThe same request works from Python and fails from Go. Signature byte-identical, parameters byte-identical, cookies irrelevant (it works with and without). Dump the exact headers Python just used, replay them from Go, and hold everything else constant. Same URL, same signature, same device, same second:\n\n```\n// identical request, three clients, back to back\n\npython  urllib3 / OpenSSL     →  9,517 bytes\ncurl    OpenSSL               →  9,519 bytes\ngo      crypto/tls            →      0 bytes   ← HTTP 200\n```\n\nNothing about the request was different. The handshake was.\n\nThe fix is [uTLS](https://github.com/refraction-networking/utls), which\nlets you specify the exact ClientHello to emit instead of accepting the one Go\nbuilds for you:\n\n```\ncfg := &utls.Config{ServerName: host, NextProtos: []string{\"http/1.1\"}}\nconn := utls.UClient(raw, cfg, utls.HelloAndroid_11_OkHttp)\nif err := conn.HandshakeContext(ctx); err != nil {\n    return nil, fmt.Errorf(\"utls handshake: %w\", err)\n}\n```\n\nOne line of profile selection. Profile detail on fresh devices went from\n**0% to 100%**.\n\n`NextProtos`\n\nis pinned to `http/1.1`\n\non purpose.\nThe Android profile advertises h2, but the transport underneath this is HTTP/1.1\nonly. Negotiate h2 and you get a connection nothing can speak on.\n\n## The Go proxy trap\n\nShort, and specific to Go.\n\nWire up uTLS, test it directly, confirm the fingerprint has changed, then put it behind the rotating proxy. The failures come straight back.\n\nThe reason is that `http.Transport`\n\n**ignores\nDialTLSContext when Proxy is set**. It dials the\nproxy, issues\n\n`CONNECT`\n\nitself, and then runs its own standard-library\nhandshake over the resulting tunnel. Your custom dialer is silently discarded. No\nerror, no warning, no log line.\nYou have to do the tunnel by hand:\n\n```\ndialTLS := func(ctx context.Context, network, addr string) (net.Conn, error) {\n    // 1. plain TCP to the proxy\n    raw, err := d.DialContext(ctx, \"tcp\", proxyURL.Host)\n    ...\n    // 2. CONNECT by hand: this is the part Transport would have done\n    req := &http.Request{Method: \"CONNECT\", URL: &url.URL{Opaque: addr}, Host: addr, ...}\n    req.Write(raw)\n    resp, _ := http.ReadResponse(bufio.NewReader(raw), req)\n    if resp.StatusCode != 200 { return nil, fmt.Errorf(\"CONNECT: %s\", resp.Status) }\n\n    // 3. NOW run the uTLS handshake over the tunnel\n    u := utls.UClient(raw, cfg, utls.HelloAndroid_11_OkHttp)\n    return u, u.HandshakeContext(ctx)\n}\n\ntr := &http.Transport{\n    DialTLSContext:    dialTLS,\n    DisableKeepAlives: true,   // see the next section\n    // note: NO Proxy field. Setting it would bypass all of the above.\n}\n```\n\nThe `Proxy`\n\nfield is deliberately absent from the transport. Setting it is\nwhat silently discards the dialer.\n\n## Detecting the empty 200\n\nWith all four gates passed you still need to know, per response, whether you actually got data. Status codes will not tell you. The check has to be on content:\n\n```\nif resp.StatusCode != http.StatusOK {\n    return body, fmt.Errorf(\"upstream HTTP %d\", resp.StatusCode)\n}\nif len(body) < minBodyBytes {                    // 64\n    // The soft block: 200 with (almost) nothing in it.\n    return body, fmt.Errorf(\"empty upstream body (%d bytes)\", len(body))\n}\nvar probe map[string]json.RawMessage\nif err := json.Unmarshal(body, &probe); err != nil {\n    // HTML, usually a proxy error page rather than TikTok\n    return body, errors.New(\"upstream body is not a JSON object\")\n}\nif raw, ok := probe[\"status_code\"]; ok {\n    var n int\n    if json.Unmarshal(raw, &n) == nil && n != 0 {\n        return body, fmt.Errorf(\"upstream status_code %d\", n)\n    }\n}\n```\n\nFour conditions, in order: HTTP status, length floor, parseable JSON object,\nclean internal `status_code`\n\n. Anything that fails one is retried against\na different device from a different IP.\n\n### Except when retrying is pointless\n\nSome non-zero `status_code`\n\nvalues are TikTok *answering* rather\nthan refusing. Retrying those four times is a waste of four devices and four IPs:\n\n`status_code` | Message | Treated as |\n|---|---|---|\n| 2065 | User doesn't exist. | 404, no retry |\n| 3170 | user not exists | 404, no retry |\n| 3002060 | Profile user is hiding following list | 403, no retry |\n\nWhich surfaces to the caller as a real answer instead of a gateway failure:\n\n``` bash\n$ curl -s localhost:8080/v1/user/following?user_id=6744630345964389381 | jq\n{\n  \"error\": {\n    \"code\": \"hidden_by_user\",\n    \"message\": \"This creator has hidden their following list. Most accounts do; there is no way around it.\",\n    \"upstream_status_code\": 3002060,\n    \"upstream_status_msg\": \"Profile user is hiding following list\",\n    \"retried\": false,\n    \"retry_would_not_help\": true\n  }\n}\n```\n\nEverything else keeps its full retry budget, and when it exhausts it you get the per-attempt breakdown rather than a generic failure, which is what makes this debuggable in production:\n\n```\n{\n  \"error\": {\n    \"code\": \"upstream_failed\",\n    \"attempts\": 4,\n    \"attempt_failures\": [\n      {\"attempt\": 1, \"reason\": \"empty upstream body (0 bytes)\"},\n      {\"attempt\": 2, \"reason\": \"empty upstream body (0 bytes)\"},\n      {\"attempt\": 3, \"reason\": \"transport: ... EOF\"},\n      {\"attempt\": 4, \"reason\": \"upstream HTTP 429\"}\n    ]\n  }\n}\n```\n\nReal output from the weakest endpoint in the catalogue. Two soft blocks, a dropped connection, and an honest rate limit.\n\n## Keep-alive pins the exit IP\n\nWith all four gates passed, the highest-volume endpoint ran at\n**88.2% over 174 million attempts**. Good, and at that volume the\nmissing 12% is twenty million lost records.\n\nThe obvious move is more retries. It does nothing, because of how rate limiting and connection reuse interact.\n\nRate limiting here is per exit IP. A rotating proxy gateway assigns an exit IP\n**per TCP connection**. HTTP keep-alive, which every client does by default and which is normally exactly what you want, pins you to one exit IP for\nthe life of that connection.\n\nSo the retry went out from the address that had just been refused. And the next one. And the next:\n\n```\n// keep-alive on a rotating proxy\nattempt 1  →  exit 203.0.113.44  →  empty 200\nattempt 2  →  exit 203.0.113.44  →  empty 200     ← same IP\nattempt 3  →  exit 203.0.113.44  →  empty 200     ← same IP\nattempt 4  →  exit 203.0.113.44  →  empty 200     ← same IP\n\n// fresh connection per attempt\nattempt 1  →  exit 203.0.113.44  →  empty 200\nattempt 2  →  exit 198.51.100.7  →  ok\n```\n\nFour attempts, one IP, four identical failures. The retry budget bought nothing at all. It was structurally incapable of helping.\n\n### Why the naive fix stalls at 96%\n\nSetting `DisableKeepAlives: true`\n\neverywhere took it to 96.2% and then\nstopped. The cause is that at full\nconcurrency you are now paying a TLS handshake for *every* attempt, including\nthe ~88% that were going to succeed first time. The proxy gateway, not TikTok, became the bottleneck and started refusing tunnels:\n\n```\n{\"attempt\": 1, \"reason\": \"transport: proxy CONNECT: 466 Too Many Requests\"}\n```\n\nThe failure had moved, not gone. The production shape is a hybrid of the two, which comes down to two pools and one policy switch:\n\n```\n// First-attempt pool: keep-alive, so the common case costs no handshake.\nr.proxyPool, _      = httpclient.New(httpclient.Config{ProxyURL: cfg.ProxyURL})\n\n// Retry pool: DisableKeepAlives => fresh TCP => NEW exit IP.\nr.proxyPoolFresh, _ = httpclient.New(httpclient.Config{\n    ProxyURL: cfg.ProxyURL, DisableKeepAlives: true,\n})\n\n// ...and in the request path:\npool := r.proxyPool\nif attempt > 0 && r.proxyPoolFresh != nil {\n    pool = r.proxyPoolFresh          // rotation exactly where it matters\n}\n```\n\n| Configuration | Success | Bottleneck |\n|---|---|---|\n| Keep-alive everywhere | 88.2% | Retries reuse the blocked IP |\n| Keep-alive nowhere | 96.2% | Proxy gateway, handshake storm |\n| Keep-alive on first attempt only | 99.3% | none |\n\nMeasured over hundreds of millions of calls across four shards. The self-hosted server described below keeps the simpler always-fresh form, because a single instance is nowhere near the load where the second bottleneck appears.\n\n## Proxies: the one running cost\n\nEverything up to here is a software problem you solve once. The proxy is the single external dependency and the only recurring cost, and the requirement for one is structural rather than incidental.\n\n### What a proxy is, briefly\n\nA proxy is a machine that makes the request on your behalf. You connect to it, it\nconnects to TikTok, and TikTok sees the proxy's IP address instead of yours. A\n*rotating* gateway is one where each new connection comes out of a different\naddress in a large pool, which is the property that matters here.\n\n### Why it is mandatory rather than recommended\n\nTwo reasons, and the second is the one people underestimate.\n\n**Rate limiting is per exit IP.** One address gets a budget and it is\nnot a large one. Without a proxy every request in your system shares a single\naddress, and you exhaust it in minutes.\n\n**Retries are structurally useless without rotation.** This is the\npoint from the [previous section](#rotation). When a request is soft\nblocked, the retry has to leave from a different address or it fails identically.\nRotation per connection is the entire mechanism behind the jump from 88% to 99.3%.\nA static proxy gives you one IP and therefore gives you nothing.\n\nSo the requirement is a **rotating** gateway, and you can verify yours\nactually rotates in one line before you commit to anything:\n\n``` bash\n$ for i in 1 2 3; do curl -s --proxy \"$PROXY_URL\" https://api.ipify.org; echo; done\n203.0.113.44\n198.51.100.7      # different IP each time = rotating, good\n192.0.2.19\n```\n\nIf the same address comes back three times, your retry budget is decorative and your\nsuccess rate will sit near the first-try rate no matter what you set\n`MAX_ATTEMPTS`\n\nto.\n\n### What it actually costs\n\nThe first thing to know is that **you should not be paying by the\ngigabyte**. Metered plans are the default recommendation in this space and\nthey are the wrong shape for this workload, because the endpoints that return the\nmost useful data are the ones measured in megabytes. A page of videos is\n1.1 MB. A page of recommended creators is 2.9 MB. Metered billing turns\nevery one of those into a line item.\n\nFlat monthly subscriptions exist for both proxy types, and they are what you want. Two tiers cover essentially everyone:\n\n| Tier | What you get | Cost | Realistic for |\n|---|---|---|---|\nRotating datacenter |\n100 concurrent threads at 200 Mbit/s | ~$150 / month | Millions of records. Where almost everyone should start. |\nUnlimited residential |\nUnmetered residential pool | ~$950 / month | Billions. What the six-day run at the top of this page used. |\n\nFor enriching a few hundred thousand creators, tracking sounds daily, or mapping a niche, the $150 tier is sufficient rather than a compromise: a rotating datacenter pool registers devices, passes activation and sustains the success rates in the table further down.\n\nThe residential tier is what you escalate to once you are saturating the datacenter one.\n\n### What the $150 tier buys\n\nOn a flat plan the two limits are **threads** (how many requests can be\nin flight) and **line speed** (how many bytes per second). Which one\nbinds depends entirely on response size, and the endpoints here differ by two orders\nof magnitude: a profile is 9.5 KB, a page of videos is 1.1 MB.\n\nThe figures below are arithmetic from the measured response sizes and latencies in the benchmark, at 100 threads and 200 Mbit/s. They are ceilings at full saturation, so treat them as an upper bound rather than a promise.\n\n| Collecting | Per response | Binding limit | Ceiling |\n|---|---|---|---|\n| Creator profiles (user.info) | 9.5 KB | Threads | ~140/s · ~12M/day |\n| Comments (20 per page) | 174 KB | Threads | ~1,600/s · ~140M/day |\n| Followers (20 per page) | 178 KB | Threads | ~1,600/s · ~140M/day |\n| Videos with full metadata (20 per page) | 1.1 MB | Line speed | ~450/s · ~39M/day |\n| Creator graph walk (user.recommended) | 2.9 MB | Line speed | ~350/s · ~30M/day |\n\nOn the small endpoints you run out of *threads* long before bandwidth, so the\nfix is a higher thread count. On the\nvideo endpoints you saturate the *line* at around 22 requests a second, and\nmore threads buy you nothing at all. That is the number `MAX_CONCURRENT`\n\nexists to control, and setting it above what your plan can carry produces\n`proxy CONNECT: 466 Too Many Requests`\n\nin the attempt failures rather\nthan more throughput.\n\nOne $150 subscription comfortably collects **millions of records**,\nand tens of millions on the small endpoints. It is not enough for billions. The\nsix-day run at the top of this page needed the unlimited residential tier at\nroughly $950 a month, sharded across four instances, and at that scale the proxy\nbill is the dominant cost of the entire operation.\n\nScaling is horizontal either way: another subscription, another instance of the server pointed at it. Nothing in the code changes.\n\n### What to look for when buying one\n\n**Rotating, with a single gateway endpoint.** Verify rotation with the loop above before you pay for a month.**A published thread limit.** If it is not stated, assume it is low. This is the number you actually plan around.**Flat rate over metered**, unless you know your volume is small and stays small. Metered plans punish exactly the endpoints that return the most useful data.**Country targeting**, if you want regional charts. The device region and the exit region should agree.** A trial or one month first.**Registration success rate is the real test and it varies between providers advertising the same product. Generate 30 devices and read`generation_survival_rate`\n\nbefore committing.\n\n## Measured success rates\n\nEvery endpoint ships with a real success rate rather than a claim: 100 calls each, at most 4 attempts, against a freshly generated pool over a rotating proxy gateway. Seeds are discovered live by walking the API rather than hardcoded, which changes the numbers. The note below explains why.\n\n| Endpoint | Success | Avg attempts | Avg response |\n|---|---|---|---|\n| user.info | 100% | 1.00 | 9 KB |\n| user.recommended | 100% | 1.08 | 2.9 MB |\n| music.posts | 100% | 1.06 | 1.7 MB |\n| music.posts_fresh | 100% | 1.34 | 1.7 MB |\n| music.trending | 100% | 1.00 | 85 KB |\n| music.related | 100% | 1.00 | 139 KB |\n| hashtag.info | 100% | 1.00 | 3.8 KB |\n| hashtag.posts_fresh | 100% | 1.00 | 1.3 MB |\n| search.general | 100% | 1.06 | 527 KB |\n| search.music | 100% | 1.11 | 100 KB |\n| search.users | 100% | 1.00 | 86 KB |\n| trending.categories | 100% | 1.07 | 380 KB |\n| trending.effects | 100% | 1.10 | 232 KB |\n| video.comment_replies | 100% | 1.05 | 8 KB |\n| video.info | 94% | 1.96 | 58 KB |\n| user.following | 93% | 1.90 | 23 KB |\n| user.followers | 92% | 2.12 | 178 KB |\n| video.comments | 92% | 2.02 | 174 KB |\n| search.videos | 92% | 1.82 | 639 KB |\n| user.posts | 90% | 2.26 | 1.1 MB |\n| music.info | 90% | 2.15 | 12 KB |\n| hashtag.posts | 89% | 2.17 | 1.3 MB |\n| hashtag.search | 78% | 2.16 | 11 KB |\n| feed.recommended | 10% | 3.93 | 248 KB |\n\nAverage attempts is the more informative column. A 100% endpoint at 1.00 attempts succeeds first time, every time. A 92% endpoint at 2.12 attempts is being soft-blocked on roughly half its first tries and recovering on retry, which means a wider budget moves it, whereas nothing moves a first-try-clean endpoint because there is nothing to move.\n\n`feed.recommended`\n\nis genuinely weak, at 10-25% across runs, and it is\ndominated by honest `429`\n\ns rather than soft blocks. An anonymous device\nwith no watch history asking for a personalised feed is precisely the traffic shape\nTikTok most wants to throttle. It ships documented as weak with the two 100%\nalternatives named in its place.\n\nSeeds are discovered live rather than hardcoded: creator, then video, then a comment that actually has replies, then a sound that actually has videos, then a hashtag. This matters for accuracy. Point a follower benchmark at a creator who hides their following list and you measure TikTok correctly answering \"nothing here\" and score it as a failure.\n\n## The 24 endpoints\n\nAll of the above is packaged as a self-hosted Go service. One binary, no database, no queue, no emulator, no native library. Reference data is compiled in.\n\n``` bash\n$ cp .env.example .env        # set PROXY_URL\n$ docker compose up -d\n$ docker compose logs -f\n\nTikTok Open API 1.0.0 starting\nconfig: port=8080 country=SG pool=15/30 attempts=4 concurrency=32 proxy=http://***@gw:9000 auth=true\npool: 0 device(s) live, filling to 30 ...\nlistening on http://0.0.0.0:8080  (GET /healthz, GET /v1/endpoints)\npool: initial fill complete, 43 device(s) live\n```\n\nReal startup output. Cold start is 15 to 60 seconds; the pool persists to disk so restarts after that are instant.\n\nAll 24 routes are GET, all take query parameters, all return TikTok's JSON unmodified.\n\n#### Creators\n\n| Route | Parameters | Returns |\n|---|---|---|\n| /v1/user/posts | user_id, count, max_cursor | Videos, each with the full author object |\n| /v1/user/info | user_id, sec_user_id | Full profile, incl. `bio_email` , links, commerce flags |\n| /v1/user/followers | user_id, sec_user_id, count, max_time | Follower list |\n| /v1/user/following | user_id, sec_user_id, count, max_time | Following list, where published |\n| /v1/user/recommended | user_id, sec_user_id, count | TikTok's own similar-creators graph |\n\n#### Videos\n\n| Route | Parameters | Returns |\n|---|---|---|\n| /v1/video/info | aweme_id | Media, stats, sound, tags, author |\n| /v1/video/comments | aweme_id, count, cursor | Comments with the commenter's user object |\n| /v1/video/comments/replies | aweme_id, comment_id, count, cursor | Second level of the comment tree |\n\n#### Sounds\n\n| Route | Parameters | Returns |\n|---|---|---|\n| /v1/music/info | music_id | Sound detail incl. `user_count` |\n| /v1/music/posts | music_id, count, cursor | Popular videos using the sound |\n| /v1/music/posts/fresh | music_id, count, cursor | Newest videos using the sound |\n| /v1/music/trending | count, cursor | Trending sounds chart, per device region |\n| /v1/music/related | aweme_id, count, cursor | Sounds suggested for a video |\n\n#### Hashtags and search\n\n| Route | Parameters | Returns |\n|---|---|---|\n| /v1/hashtag/search | keyword, count, cursor | Hashtag ids with view counts |\n| /v1/hashtag/info | hashtag_id | Hashtag detail |\n| /v1/hashtag/posts | hashtag_id, count, cursor | Popular videos under the tag |\n| /v1/hashtag/posts/fresh | hashtag_id, count, cursor | Newest videos under the tag |\n| /v1/search/videos | keyword, count, offset | Videos |\n| /v1/search/general | keyword, count, offset | Blended creators, videos and tags |\n| /v1/search/music | keyword, count, cursor | Sounds |\n| /v1/search/users | keyword, count, cursor | Handle or name → numeric `user_id` |\n\n#### Discovery\n\n| Route | Parameters | Returns |\n|---|---|---|\n| /v1/trending/categories | count, cursor | The app's what-is-hot shelves |\n| /v1/trending/effects | count, cursor | Videos carrying `sticker_detail` for trending effects |\n| /v1/feed | count, max_cursor | Anonymous For You feed (weak, see above) |\n\n### A real response\n\nTrimmed to the interesting fields. The raw object has several hundred keys:\n\n``` bash\n$ curl -s \"localhost:8080/v1/user/posts?user_id=6744630345964389381&count=20\" \\\n    | jq '{has_more, max_cursor, first: (.aweme_list[0] | {aweme_id, desc, statistics, music, author})}'\n{\n  \"has_more\": 1,\n  \"max_cursor\": 1751028792000,\n  \"first\": {\n    \"aweme_id\": \"7678101694902832397\",\n    \"desc\": \"Who Remembers 2022? #fortnite #piececontrolkyle #dogwater\",\n    \"statistics\": {\n      \"play_count\": 19438,  \"digg_count\": 2461,\n      \"comment_count\": 39, \"share_count\": 176\n    },\n    \"music\": {\n      \"id_str\": \"7245172246876227585\",\n      \"title\": \"Need 2 (Instrumental)\"\n    },\n    \"author\": {\n      \"uid\": \"6744630345964389381\",\n      \"unique_id\": \"freakynaughty\",\n      \"nickname\": \"freaky\",\n      \"follower_count\": 1277258\n    }\n  }\n}\n```\n\nNote that the author object is embedded in every video. One request gives you twenty\nvideos *and* the full creator record. On the web that is twenty-one requests.\n\nRequest metadata comes back in headers rather than polluting the body:\n\n```\nHTTP/1.1 200 OK\nX-Endpoint-Id:     user.posts\nX-Attempts:        2                 ← first try was soft-blocked\nX-Elapsed-Ms:      1874\nX-Upstream-Region: sg\nX-Device-Region:   SG\n```\n\n### Fields the web does not give you\n\n| Field | Where | Populated |\n|---|---|---|\n| statistics.collect_count | any video | always. Saves, often the earliest movement signal |\n| music.user_count | any sound | always. Videos made with the sound |\n| author.ins_id | video author object | ~26% of creators |\n| author.youtube_channel_id | video author object | ~19% |\n| bio_email | `user.info` only | ~1% |\n| commerce_user_level | `user.info` | always |\n| bio link | nowhere | 0%. Not in the mobile API at all |\n\nThe last row was measured across 579 creators on both endpoints that could plausibly carry it. The outbound profile link is a web-surface field only.\n\n## Tutorial: a sound-trend detector\n\nConcrete worked example, because the endpoint list on its own does not tell you what\nthe data is good for. The goal: find sounds that are taking off *right now*,\nbefore they are obviously trending.\n\nThe signal is `music.user_count`\n\n, how many videos have been made with a sound. The absolute number tells you a sound is big. The *rate of change*\ntells you it is moving, which is the part you want.\n\n#### Step 1. Snapshot the chart\n\n```\ncurl -s \"$API/v1/music/trending?count=50\" \\\n  | jq -r '.music_list[] | [.id_str, .user_count, .title] | @tsv' \\\n  > \"sounds-$(date +%s).tsv\"\n```\n\nRun it hourly from cron. Each row is `id, uses, title`\n\n.\n\n#### Step 2. Diff consecutive snapshots\n\n``` python\nimport glob, csv, collections\n\nsnaps = sorted(glob.glob(\"sounds-*.tsv\"))[-2:]\nprev, curr = [{r[0]: (int(r[1]), r[2])\n               for r in csv.reader(open(f), delimiter=\"\\t\")} for f in snaps]\n\nmovers = []\nfor mid, (n, title) in curr.items():\n    was = prev.get(mid, (0, title))[0]\n    if was > 0:\n        movers.append((n / was - 1, n - was, title, mid))\n\nfor growth, delta, title, mid in sorted(movers, reverse=True)[:10]:\n    print(f\"{growth:6.1%}  +{delta:>8,}  {title[:40]:<40} {mid}\")\n```\n\n#### Step 3. Confirm it is actually accelerating\n\nGrowth in the chart is a candidate, not a confirmation. The check that separates a\nreal acceleration from a chart-placement artefact is the *time spread of recent\nvideos*. Pull the newest videos using that sound and look at how tightly their\nupload times cluster:\n\n```\ncurl -s \"$API/v1/music/posts/fresh?music_id=$MID&count=30\" \\\n  | jq '[.aweme_list[].create_time] | (max - min) / 3600'\n2.4\n```\n\nThirty videos in a 2.4-hour window means thirty people picked up that sound this afternoon. Compare with the popular ordering, which tells you whether it has already landed:\n\n```\ncurl -s \"$API/v1/music/posts?music_id=$MID&count=30\" \\\n  | jq '[.aweme_list[].statistics.play_count] | add'\n```\n\n**High fresh-clustering plus low cumulative plays is the interesting\nquadrant.** Lots of people using it, not much accumulated reach yet. That is a sound on the way up rather than one on the way down.\n\n#### Step 4. Find who is driving it\n\n```\ncurl -s \"$API/v1/music/posts/fresh?music_id=$MID&count=30\" \\\n  | jq -r '.aweme_list[].author | [.follower_count, .unique_id] | @tsv' \\\n  | sort -rn | head\n```\n\nBecause the author object is embedded, this costs no extra requests. If one large account is at the top and everyone else is small, you are looking at a sound that one creator kicked off, which is a different (and usually shorter-lived) phenomenon than organic uptake across many mid-sized accounts.\n\n#### Step 5. Widen it\n\n`music.trending`\n\nis region-scoped to the device. Run a second instance\nwith `POOL_COUNTRY=US`\n\nand a third with `POOL_COUNTRY=BR`\n\nand\nyou get three independent charts from identical code. Sounds frequently break in one\nmarket days before another.\n\nThe whole loop above is 4 requests per candidate sound per cycle. At 50 candidates\nhourly that is 200 requests an hour, which is nothing. The expensive version is walking\n`user.recommended`\n\noutward to map a niche, where responses run ~3 MB\neach and bandwidth, not rate limiting, becomes the constraint.\n\n## Getting the code\n\nEverything described here is a private Go repository. One-time payment, permanent access, complete source.\n\n[Get access](https://buy.stripe.com/5kQ6oH7sn9s573b5C95J60q)\n\n- Full signing stack (Simon, Speck, SM3, TTEncrypt, Argus, Ladon) with per-primitive test vectors\n- Device registration, activation and proving pipeline\n- uTLS transport with the manual proxy tunnel\n- All 24 endpoints, measured and documented\n- Pool management: health, eviction, rotation, persistence\n- Docker image and compose file\n- Live integration test suite with seed discovery\n- Python, Node, curl and\n`.http`\n\nclients - Generated reference docs for every endpoint\n- Reliability, pagination and error engineering guides\n- Lifetime updates to the same repository\n- Direct line during setup\n\nCheckout asks for your GitHub username. The repository invitation goes to that account automatically, normally within a minute of payment.\n\n### Before you run it\n\nThe one hard requirement is a [rotating proxy gateway](#proxies). Rate\nlimiting is per exit IP and the retry design assumes a new connection gets a new\naddress, so a static proxy is no better than none. It is the single external\ndependency and it is not optional at volume.\n\nMaintenance is yours once you self-host. The repository is structured so that when something moves it is usually one struct literal in one table, but it is your struct literal.\n\nIf you would rather skip the setup entirely, there is a\n[done-for-you tier](#setup) where I build it on your server and hand it\nover running.\n\n## Done for you\n\nThe repository is one component of a collection system, and on its own it answers one request at a time. Getting from there to billions of records is a different piece of work: deciding what to fetch next, keeping the queue moving through failures, landing the results somewhere that is still queryable at that size, and running the whole thing across enough shards and proxy capacity to sustain the rate.\n\nThis tier is that system, built on your infrastructure and handed over running. Not a demo pointed at a few creators. The same shape as the one that produced the figures at the top of this page, sized to what you are collecting.\n\n[Get in touch](#setup)\n\n- Everything in the $699 tier, including lifetime updates\n- The whole collection system built on your server, from a clean box\n- Database architecture sized to your volume: engine choice, partition and sort keys, the update path for records that change, and the denormalisation your queries actually need\n- ClickHouse installed, tuned and sized, with retention set on its own system tables\n- The crawl pipeline: discovery, work queues, resume after failure, and deduplication so you are not paying to re-collect what you already hold\n- Sharded across multiple instances and proxy capacity, which is what billion scale actually requires\n- Proxy plan chosen with you, configured and rotation-verified\n- Concurrency and retry budget tuned to the plan you actually bought\n- Device pool generated, sized to your workload and persisting across restarts\n- Monitoring on freshness, write failures, pool health and disk, so a stall is visible instead of silent\n- Full smoke test run on your instance, all 24 endpoints returning live data\n- Live walkthrough of the endpoints you care about and how to paginate each\n- 7 days of support after handover\n\nThe server and the proxy subscription are yours and are not included in the\nprice. Both stay in your name and under your control. See the\n[proxy section](#proxies) for which tier your volume needs.\n\n### Storage, if you are keeping the data\n\nCollecting the data is what the repository solves. Keeping it queryable once there are billions of rows is a separate problem with its own failure modes, and it is the half that decides whether the collection was worth doing. If you are building a store rather than running a one-off pull, that design is part of the handover: table engines, partition and sort keys, the update path for records that change, and the denormalisation that keeps a creator-to-video-to-sound question answerable without a join across billions of rows.\n\nFour decisions that determine whether it holds, each of them measured on a production ClickHouse store at billion-row scale:\n\n-\n**Partition keys.** A bare modulus of an id looks even and is not, because platform ids are not uniformly distributed. One table partitioned on`author_id % 8`\n\nended up with a 43x spread between its largest and smallest partition, the largest heading for the size where merges stop keeping up. Hashing the id before the modulus flattens it. -\n**The update path.** ReplacingMergeTree keeps the newest whole row, not the newest value per column. Write a partial update and every field you left out is silently blanked. Nothing errors, and you find out much later. -\n**Duplicate control.** Re-collecting the same creator is normal and the storage cost of that compounds quietly. One table was carrying three times the rows it needed before anything made it visible, and rebuilding it returned several terabytes. -\n**The database's own logs.** ClickHouse writes`text_log`\n\nand`trace_log`\n\nby default with no retention. They reached 243 GB on one instance and took a 7 TB volume to full, which stops writes for everything sharing it. A TTL on the system tables is not optional at this scale.\n\n### How it goes\n\n-\n**You tell me what you are collecting.** Creators in a niche, sounds on a schedule, comments on a set of videos. This determines the proxy tier and the pool size, so it is worth being concrete. -\n**You provide a server and a proxy subscription.** The collection layer is light and runs comfortably on two cores. If you are storing what you pull, the database is the part that needs real disk and real memory, and sizing it is part of step one rather than a surprise later. -\n**I deploy and tune it.** Usually the same week. You get the running instance, the repository access, and the reasoning behind each setting rather than just the settings. -\n**We run the integration test together.** You watch 24 endpoints come back with live data on your own hardware before you consider it delivered.\n\n## Questions\n\n## Do I need TikTok accounts?\n\nNo. There is no login anywhere. Every device is an anonymous app install TikTok issued credentials to. Nothing to get banned, no credentials to rotate, no 2FA.\n\n## Why won't it work without a proxy?\n\nIt works fine for a look around. It does not work at volume, because rate limiting is per exit IP and the entire retry design assumes a new connection gets a new IP. Rotation is the requirement; a static proxy is no better than none.\n\nA rotating datacenter gateway at around $150 a month, flat rate, covers millions of records. Billions is a different tier at roughly $950. The [proxy section](#proxies) works through both and where each one runs out.\n\n## Is it legal?\n\nIt is against TikTok's terms of service. It is sold for research and educational use.", "url": "https://wpnews.pro/news/4-5b-posts-scraped-from-tiktok", "canonical_source": "https://tiktok-api.seeksocial.io/", "published_at": "2026-09-03 11:25:49+00:00", "updated_at": "2026-09-03 11:52:35.680044+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research"], "entities": ["TikTok", "Hugging Face", "kuben-developer"], "alternates": {"html": "https://wpnews.pro/news/4-5b-posts-scraped-from-tiktok", "markdown": "https://wpnews.pro/news/4-5b-posts-scraped-from-tiktok.md", "text": "https://wpnews.pro/news/4-5b-posts-scraped-from-tiktok.txt", "jsonld": "https://wpnews.pro/news/4-5b-posts-scraped-from-tiktok.jsonld"}}