Surging token usage as artificial intelligence moves into large-scale deployment spurs search for new approaches
Chinese AI companies are optimising software to cope with surging demand for inference, as part of that workload still relies on computing power from a limited pool of high-end chips amid restricted access to Nvidia processors.
Compared with training an artificial intelligence model, which relies on high-end chips, inference – a later phase in which the trained model applies its knowledge to process responses – can be adapted to domestic hardware. However, industry insiders said complex tasks like coding still required Nvidia chips, which meant the sector was facing acute compute constraints as AI moved from model development to large-scale deployment.
“The demand side is now showing a bipolarisation,” said Guan Jiawei, vice-president of inference optimisation start-up Approaching.AI, noting that demand for high-quality tokens – the basic units of data that models process and generate – far outstripped supply.
High-tier tasks required stringent performance metrics that domestic processors could not yet reliably deliver, Guan said, adding that advanced Chinese models “place high demands on chips … especially in scenarios like coding, where users are willing to pay a premium”.
“If we rely solely on domestic chips for inference, they can only handle the low-quality tier – the tier with weak demand and weak monetisation,” Guan said. “That makes it very hard to find a viable commercial path. That’s why high-quality tokens still depend on Nvidia.”
Skyrocketing token usage, as AI turns more agentic – performing real-world tasks rather than just answering questions – has exacerbated the compute squeeze. China’s average daily token calls exceeded 140 trillion in March, up more than 1,000-fold from the beginning of 2024, according to the National Data Administration.