In the summer of 2017, Google's AI researchers released a paper titled "Attention Is All You Need", which described a new type of neural network called the Transformer.
It later turned out that the Transformer network performed exceptionally well in handling long sequence data, particularly text.
Today, in 2026, the Transformer can be said to be the core engine of all mainstream large language models (LLMs) on the market. Justin Dangel, co-founder and CEO of AI startup Subquadratic, said: "The entire AI industry is built on top of Transformers. They are one of the most important innovations in the history of computer science and have changed the world."
However, the Transformer technology is also showing signs of fatigue. Many recent advances in large language models, such as the development of reasoning models and the ability to process massive inputs at one time, are not a natural extension of the core technology, but rather a makeshift solution to patch up some of its fundamental flaws.
More and more scientists and engineers are pondering how technology will evolve, what form the next generation of AI will take, and what the future holds for Large Language Models (LLMs). We believe that LLMs themselves will not disappear, but it remains to be seen how they will be constructed in the future.
A batch of startups are vying to get in on the action, seeking to break new ground on the cutting edge of existing technology. While some competitors are bound to fail, these up-and-coming players have enormous potential and may even have a competitive advantage over current industry leaders.
The Twilight of the Web #
First, we need to clarify the existing technical difficulties. The core advantage of the Transformer comes from the so-called "dense attention" mechanism, which encodes the semantic meaning of a piece of text through a series of numbers. During the processing, each word, or part of a word, i.e., token, in the text is compared with all other words or tokens through multiplication operations.
Dense attention mechanisms can capture the meaning of text with extremely high precision, but as the length of the text increases, the amount of computation required surges. A document of 10,000 English words may require the Transformer to perform 50 million multiplication operations, which is the main reason why large language models consume a huge amount of energy.
Computing power costs are extremely high. According to OpenAI President Greg Brockman, OpenAI's computing power expenditure will reach $50 billion this year. The International Energy Agency predicts that the total power consumption of data centers will double by 2030.
What's worse, when it comes to handling the tasks required to deal with numerous new models, Transformer networks often fall short. Limited by their word-by-word processing of text, Transformers are not good at maintaining a large amount of information at the same time, in other words, their context window cannot be expanded too large.
However, for LLMs to undertake more challenging tasks, they must receive larger-scale data, such as an entire library of documents, a complete codebase, or output from other large language models.
The reasoning model works by first using the thought process chain as a draft to take notes for itself, and then continuing to reason by reading the notes. This approach also increases the amount of data that the model needs to maintain.
As LLMs continue to grow in size and strength, the bottlenecks of Transformers are becoming increasingly prominent, with their former core advantages now turning into limitations.
The following introduces four new approaches aimed at addressing the limitations of Transformer networks. These innovations could fundamentally reshape large language models, making them faster, significantly more efficient, and perhaps even more intelligent.
Redesigning Attention Mechanism #
To make large language models run faster and at a lower cost, it may be effective to directly address the pain points and change the attention mechanism, using sparse attention instead of dense attention.
The sparse attention mechanism only performs calculations on a portion of the word pairs in the text block, rather than all of them, which can greatly reduce the computational requirements of the LLM.
Over the years, researchers have proposed a multitude of sparse attention mechanisms, but they have always fallen short of dense attention mechanisms in terms of capturing semantics. However, times may have changed.
The aforementioned startup Subquadratic claims that its developed sparse attention mechanism has, for the first time, demonstrated performance comparable to that of mainstream top-tier LLMs in multiple tasks such as search and code generation. This assertion is a bombshell, but some industry insiders remain skeptical.
In terms of Subquadratic, its model SubQ can judge in real time which words are important or not for each paragraph of input text. It is said that tens of thousands of users have signed up for the waiting list to try SubQ, and the day when the model is open to the general public will arrive soon.
Another startup, Manifest AI, has taken a different approach, choosing not to reform the working mechanism of attention, but instead replacing it with a completely new solution.
The company has developed a mechanism called "power retention". The power retention mechanism only retains the information most relevant to the current task, ensuring that the amount of data that large models need to maintain does not grow exponentially.
Traditional attention mechanisms require LLMs to remember all content within the context window. In contrast, sparse attention models, such as SubQ, discard many individual words but still retain an overview of the content they have seen. By comparison, the power retention mechanism provides the model with a rolling summary of the context window content, which is all the text the model can currently see, and discards less relevant old information when new information is added.
The fundamental principle of order preservation has existed for about ten years. Manifest AI claims that they have completed a technological iteration, building a product that can rival large models based on Transformer for the first time. Additionally, they can modify Transformer models into order preservation models with only a small amount of retraining.
To demonstrate the aforementioned transformation capabilities, the technology team modified the existing open-source large model StarCoder to create PowerCoder, which is equipped with a power retention mechanism, and also released the Brumby model, claiming its performance can rival certain versions of Alibaba's Turing Question Answering.
Manifest AI's vision is that when large language models need to handle massive data tasks, their power-preserving technology will be the go-to solution. The company's co-founder and Chief Technology Officer, Carles Gelada, has stated that their technology has a wide range of applications, from analyzing videos that are several hours long to building AI entities that can perform tasks continuously for several weeks.
Building Smaller, More Flexible Models #
A startup incubated by the Massachusetts Institute of Technology, Liquid AI, has not completely overhauled or abandoned the Transformer, but instead combined its self-developed liquid neural networks with the Transformer to build what it calls "liquid foundation models," abbreviated as LFM.
Liquid AI's models have a much smaller size than most LLMs and consume less energy. The company has developed models for automakers such as Mercedes that can be adapted to small chips for in-vehicle use; its latest version can even run on a Raspberry Pi development board that costs $50 and has low power consumption.
Companies with annual revenues of less than $10 million can allegedly use Liquid AI's products for free, with model downloads having reached tens of millions and receiving widespread acclaim.
Liquid neural networks are an extension of convolutional neural networks, inspired by the brains of worms; convolutional neural networks are another type of neural network that existed before the emergence of Transformers. The key to liquid neural networks standing out lies in their adaptive mechanisms, which enable the model to dynamically adjust its behavior according to new information, achieving learning while running. In contrast, models based on Transformers lack this ability, with their behavior patterns fixed and unchanging once training is complete.
Liquid AI's early models, although relatively basic, were already capable of controlling drones and driving vehicles. Now, with the help of its liquid foundation model, the company is attempting to scale up its technology to compete with mainstream large language models. The new model's performance can match that of competitors that are four times its size, including certain versions of Alibaba's Tongyi Qianwen and Google's open-source model Gemma.
A typical LLM is built by stacking multiple Transformers. Liquid AI's latest LFM features a hybrid architecture, consisting of 20% Transformer and 80% liquid neural network.
The calculation of the aforementioned ratio is not done by humans, but by another AI system developed by the company, which is specifically used for auxiliary model design. It is said that this AI system, which is capable of designing AI, combines various different neural networks, such as liquid networks, convolutional networks, and Transformers, and screens combination schemes to find the best balance between performance and efficiency.
Ramin Hasani, co-founder and CEO of Liquid AI, believes the future of neural networks holds endless possibilities, with Transformer being just the beginning.
The human brain is essentially a universal artificial intelligence system, yet its power consumption is only 20 watts, so how does it achieve this? We are fully capable of making even more innovations.
——Ramin Hassani
Doubao and Nubia Partner with ByteDance to Revolutionize AI-Powered Content Creation #
Almost all large language models output content word by word, which is in line with human speaking and writing habits, but is extremely inefficient for computers. If LLMs can generate text in one go - spitting out entire sentences or even paragraphs at once - speed and cost will be optimized.
One of the practitioners of the above idea is Inception, which is using diffusion technology to build large models.
Diffusion technology is perhaps better known for its role as the driving core of the vast majority of image and video generation models. The training method for large diffusion models is as follows: first, it receives a random pixel grid, similar to the snowflake noise on old TVs, and then simultaneously processes all pixels, determining which ones need to be modified, and turning the snowflake noise into a high-definition image.
It turns out that the above process also applies to text. The Inception team, through specific training, enables the diffusion large model to receive a string of random word sequences and convert them into semantically coherent sentences. This diffusion LLM still uses Transformer to encode semantics, but outputs a whole block of text at once, completing more tasks at a lower cost.
The company's co-founder and CEO, Stefano Ermon, said that their innovation still belongs to large Transformer models, but it can predict multiple tokens at the same time. That's why, compared to most solutions on the market, this model generates text much faster and has a higher cost-effectiveness.
The challenge lies in how to migrate the technology originally designed for image generation to the text domain.
When processing images, a blue pixel can be changed to a red pixel by gradually adjusting through intermediate colors. However, this does not apply to text, as there are no intermediate words between "cat" and "dog".
——Stefano Elmoni
Elon has another identity as a researcher at Stanford University. In 2024, he and two Stanford colleagues discovered a mathematical method to bridge the text gap, and based on this, they trained a diffusion model adapted to text, whose performance is comparable to OpenAI's GPT-2 released in 2019, with a generation speed 10 times faster.
Based on this achievement, Elrom founded Inception, and his ambition is now aimed at the industry leaders. Inception claims that its latest model, Mercury 2, has performance comparable to some versions of GPT-4 released by OpenAI in 2023, and is 10 times faster.
We are very optimistic about this route, which has the potential for large-scale expansion. Ultimately, speed and cost are the core standards for measuring the value of large models - how much intelligence can you get for every dollar you invest in them?
Companies betting on diffusion technology are not limited to Inception. Google is also actively exploring the field and has built a prototype LLM called Diffusion Gemma.
Break Free from the Text World #
Among startups exploring innovative solutions to address the limitations of Transformers, Pathway's concept is the most disruptive. They aim to free large language models from linguistic constraints, and have developed the Dragon Hatchling large model with this goal in mind.
To date, its most impressive achievement comes from a benchmark test where it competed against more than 250,000 extremely difficult Sudoku puzzles. Mainstream large models from several top laboratories were unable to solve even a single puzzle, while Dragon Hatchling overcame more than 97% of the challenges.
Pathway's point is that while mainstream large language models excel in many tasks, they are still powerless in dealing with many key types of problems, with Sudoku being just one example.
The company's co-founder and CEO, Zuzanna Stamirowska, said that in order for large models to produce truly original solutions to real-world problems, they must move beyond the Transformer framework, as it forces large models to rely entirely on text for reasoning, but language is not the optimal vehicle for certain forms of reasoning.
Pathway's solution modifies the underlying mathematical logic of the Transformer by replacing the attention mechanism with a mathematical structure known as "state space".
The state space no longer encodes information word by word, but instead compresses it into more abstract expressions. Based on this, Dragon Hatchling can still read and write text, while also simulating reasoning patterns that are not dependent on word sequences. This not only improves the efficiency of the model, but also theoretically enables it to handle tasks that other LLMs are unable to accomplish.
Chess and math problems do not exist in the human brain in the form of long sentences. The brain's flashes of insight do not necessarily appear in linguistic form. We believe that if reasoning must be completed relying on language, this is a limitation in itself. Mainstream large language models have the ability to read Sudoku solution tutorials and then write solution code, but this is not what we are looking for. The model we idealize is not limited to book knowledge.
AI's vast universe is not about solving Sudoku, but about conquering cancer - however, there is no ready-made manual for curing cancer in this world.
——Zuzanna Stamirowska
Source:
Several startups are currently focusing on the development of Large Language Models (LLMs), with the goal of achieving breakthroughs in this field. Companies such as Alibaba's Damo Academy, Baidu, and Huawei are investing heavily in LLM research, while startups like ByteDance and Tencent are also actively exploring this area. In recent years, the application of LLMs has become increasingly widespread, with tech giants like Google and Microsoft achieving significant advancements. However, Chinese startups are now also making notable strides, with companies like Zhipu AI and Mobvoi developing their own LLMs. For instance, Zhipu AI's LLM has already demonstrated impressive capabilities in natural language processing and generation. Meanwhile, Mobvoi's LLM has shown promising results in areas such as language translation and text summarization. The development of LLMs is expected to have a major impact on various industries, including customer service, content creation, and language translation. As such, the competition among startups to develop the most advanced LLMs is likely to intensify in the coming years. According to a report by IDC, the global LLM market is projected to reach $43.8 billion by 2025, with the Chinese market accounting for a significant share