ThinkBit: Search Codewords, Recalibrate Scales, Select by Reasoning

ThinkBit is a two-bit quantization method for reasoning language models that aims to cut inference cost by up to 8x versus bfloat16 by storing weights in two bits, addressing the memory-bandwidth limit on generation speed for models that emit thousands of tokens per answer. The work argues that two-bit methods should not be judged solely by perplexity and multiple-choice question answering, and instead combines search over codewords, recalibrated scales, and selection by reasoning performance.

Reasoning language models generate thousands of tokens per answer, and generation speed is limited by memory bandwidth, so storing weights in two bits instead of bfloat16 can potentially make inference 8times cheaper. Two-bit methods are usually judged by perplexity and multiple-choice question-answ