and it's actually a really good idea.
Thinking Machines released their first model, Inkling, a 975B parameter MoE, with 41B of those active, and a 1M token context length. It’s fair to say it’s the strongest open-weight model from a US lab so far, albeit a step behind the best Chinese models on agentic work. But it should be commended for strong omni-modal performance, including a rare foray into audio. The architecture is mostly familiar, it wasn’t exactly the wholesale rethink that some commentators had been expecting, but more a thorough example of best practice. There was one exception, however, Inkling does not use RoPE. And not in the way some other labs have done with substitutes, they do have positional encodings, but they introduce them via a combination of learnt local attention bias and a positionless far field.
The limited range of the attention bias means long context pairs carry no positional information at all, essentially becoming a content-only attention score outside the learnt bias. But because the bias is learnt at short range it gives a unique positional pattern for queries to keys in the window.
This is more than just an alternative way to reference position, they get at the uniqueness of how individual tokens relate to others in the sequence, how this is both content and context dependent. This is a fundamental shift in how positional information is thought of, and brings to bear the Bitter Lesson once again. The data knows these patterns better than you do.
This essay focuses on the positional encoding implemented in Inkling, but it has lots of other details worth checking out as well. The Inkling trending entry covers the model itself, with a full question set.
Attention can be thought of as a set operation, where if we permute the order of the keys and values the output permutes with them. i.e. nothing about the product knows which token came first. So, every transformer has to inject position somewhere, and broadly we can categorise most methods into three broad buckets.
Option 1: add positional information into the input vectors This is very much the original recipe, take the input token embeddings and concatenate the absolute or learnt position embeddings on top of the tokens. This basically makes every token
This is simple to implement but as should be clear looking at the shapes above, as soon as the sequence extends beyond the training window the sequence position loses meaning. It’s not even obvious how to extend this to a sliding window, does each window step need all tokens to be re-embedded with position? How do you efficiently use this in a KV-cache? These are just some of the reasons the move was broadly made to relative position encodings.
Option 2: Relative embeddings The logic with relative embeddings is that the thing we care about in a sequence is how far apart two tokens are, that is what informs their meaning, and if these are at the start of a sequence or a million tokens into a stream, the same relationship should hold. It is a natural fit to streaming long context data, sliding windows, and the way this is implemented in RoPE through the properties of rotation matrices is a lovely clean bit of maths. The core idea is that in RoPE we rotate each query and key vector by an angle determined by the position in the sequence across a spectrum of frequencies, but once we see these rotations in the dot product only the difference is required. (mechanics in the RoPE essay).
As is clear in the equation, the dot product of two rotated vectors depends only on the difference of positions not the absolute positions, so relative position falls out of absolute rotations. Even better, the keys can be cached once, and it doesn’t need any trainable parameters.
The problem is that while the model performs well at short to medium ranges, the simplicity of the maths doesn’t translate into simple patterns being learnt at longer ranges.
Option 3: position as a bias to the attention
The idea is elegant here, the attention mechanism already gives us a array of pairwise scores, so similarly to how option 1 concatenates the positional information, we can do the same with the pairwise matrix. This could be absolute difference in distance (simple, and exactly what you can do with graphs to encode position there), through to a custom pattern as we see in models like T5. All this comes down to is computing with no positional information, then adding a scalar bias that depends on the distance between the two tokens. In T5 there was a small learnt table, quantised by distance, which is uniformly applied to all tokens. In ALiBi they used a fixed linear penalty, which was steeper for some heads than others, but gave a uniform decay with respect to distance for all tokens. This should feel fairly natural, we already put a bias into the attention with the causal mask as it is, this just offers further modulation to the scores.
As a testament to how trends move the field, most labs generally converged on RoPE style rotations, and the attention biases were relegated largely to pure encoders, but has always played a very important role in the molecular / graph space.
The idea that a transformer might not need explicit position has been around for a while. The most famous example of PE free training is NoPE (Haviv et al. (2022)) which is used in a decoder only model allowing all combinations to be learnt by content. And there’s been research (Kazemnejad et al. (2023)) showing that for length-generalisation tasks small NoPE models beat the same models with RoPE or ALiBi. No frontier model is fully position-free, but many have been incorporating parts of the NoPE philosophy in the long range interactions. Often following the example of Llama 4, which has no position information in the global attention layers, and uses RoPE only in the local ones. Command A interleaves sliding-window RoPE layers with position-free global layers, SmolLM3 drops RoPE from every fourth layer, Kimi Linear uses full-attention layers which get no position, and relies on its linear-attention layers to carry PEs. We could go on, but the common thread is that long-range retrieval layers (where you’d expect RoPE to extrapolate the worst), are frequently able to go without. There’s a sound argument for this, when reading a passage we care greatly about the exact ordering of the local words in the current sentence, but when there is a reference to something earlier in the text, or an idea that ties back in, we don’t really care about the ordering, just that the idea was present. This same logic seems to apply to long range interactions in LLMs.
There is another axis where position has been increasingly turned off as well. Various models in the last year or so have started to employ a method called Partial RoPE, where only a quarter (usually) of the vector dimensions are rotated. This was a niche choice for a while (GPT-NeoX did it in 2022) but increasingly seen on SOTA open releases from big labs, including the Qwen, Deepseek and Gemma series of models. The argument is that it focuses the model capacity in the highest frequency elements, where the model can learn the positional dependence, and then reserve capacity in the low frequency elements to improve model expressivity.
The positional bias term in Inkling is a learnt function that’s somewhat reminiscent of how Fourier series can be used to approximate any function. Instead of learning a single function as you could with an MLP (universal approximations after all) they build a bank of functions with MLPs, and combine them in a linear sum with weighting coefficients.
The advantage of this approach is the functions can be reused for each head, with just the coefficients needing to change. This is more computationally efficient and (we suspect) easier to train.
The way it works is that each attention layer carries two extra pieces.
r_proj
: reads the hidden state for each token, and produces weighting coefficients for each of the 16 functions from the bank, which is done per head.
proj
: is the bank of 16 learnt functions, each mapping a bias to every backward distance in the position-aware window (1024 tokens for the global layers, and 512 for the sliding-window layers). The bank is shared by every head and token in the layer, so those 16 curves make up the layer's entire set of primitives to describe the behaviour of tokens over distance. This leaves the weighting component to mix these to account for context for each token.
We can see the flow of the positional information in the diagram, but we should also go through how this is reflected in the maths as well.
1. Generate the coefficients from the token embedding. Each token's hidden state goes through r_proj
to give 16 weights per head:
There is an elegance to this. In the simplest version, the first layer has a set of raw, unmixed token embeddings (cat is always the same token regardless of where it comes in the sequence) and therefore the model learns a set of coefficients associated with a query token i.e. for this token where is useful context likely to be? Then at later layers because attention mixes the information between tokens (including via the bias we introduce in earlier layers) we get a new token representation, which then includes content about that token, but also mixed with the context from the sequence. This mixed token then is used to pick out what pattern of position dependence to apply to this token.
2. Build the total function from the components. The bank holds the shared functions and a token's personal bias curve, per head, is a weighted sum of them:
These functions are not learnt per token, but are instead shared between all the heads in a given layer. Subsequent layers do learn a new bank of functions though.
3. Read off the curve at each distance. For a query at position and a key at position , we just read the value of the bias curve at the distance between the query and key tokens:
This only requires a single gather to perform the lookup.
4. Zero everything outside of the window. Any key more than tokens away from the query gets a bias of zero (as does any key ahead of the query incidentally, but that’s because of the usual causal mask removing those anyway).
Remember this is a hardcoded value, not something the model learns. There are no parameters for distances beyond , so it can't express a positional preference at those distances.
5. Add the bias to the logits. The bias is added to the attention score before the softmax, next to the causal mask:
In code this is very simple.
1def relative_position_bias(relative_states, proj, positions):
2 # relative_states: (T, H, 16) r_proj output: coefficients per token per head
3 # proj: (16, R) the shared bank: 16 profiles over R distances
4 # positions: (T,) token positions
5 T, H, C = relative_states.shape
6 R = proj.shape[1]
7
8 curves = (relative_states @ proj).transpose(1, 0, 2) # (H, T, R)
9
10 dist = positions[:, None] - positions[None, :] # (T, T)
11 idx = np.clip(dist, 0, R - 1) # safe gather indices
12 bias = curves[:, np.arange(T)[:, None], idx] # (H, T, T)
13
14 return np.where((dist < 0) | (dist >= R), 0.0, bias) # hard zero outside the band
In many ways it’s surprising that this method didn’t appear sooner. It’s striking, when you look at the position maps between tokens, that we typically expect the gap between all tokens to be modelled by the same set of curves. That the position embedding should in fact be a product of the query token itself makes logical sense.
Because Inkling's bias is a function of the token, r_proj
lets one token require a sharp locality while the next token might reach back through the full window. It feels like a natural extension. The 16-profile bank means the model learns a good basis of distance functions, rather than a full curve per head, and can use these reusable functions in a simple linear combination. The Fourier analogy here is a great way to conceptualise this, it allows the model to learn the sort of shape required for each token in each kind of context. There’s some argument that it might lead to even greater chance of memorisation, but for most applications this might also just improve real world performance. The difference is visible in the bias fields themselves. Here, if we compare the detail of the structure present in the Inkling local position bias, the increase in expressivity is clear. And we think these plots are also just aesthetically beautiful, which doesn’t hurt.
There are a few other details that help this new approach to positional encodings. It’s not completely clear from their technical report exactly how important each of these are, but we expect this to become clearer as other labs do their own ablations on the learnt attention bias.
The logit scaling is , not .
Inkling RMS-normalises q and k per head before the dot product (QK norm, as in Gemma and Qwen and many other modern architectures). The standard (that we all know and love) assumes independent random features, meaning the dot product of two zero mean and unit-variance vectors has std . So dividing by this factor ensures we recover the unit-variance, essential for stable training. But RMSNorm pins each norm to ~, so , which we can see will scale linearly with . So dividing by leaves whatever the head dim. Because the logits and bias are added before the softmax, they must share a scale, otherwise one or the other would dominate. Content contributes a bounded term, and the bias complements it at the same scale.
The temperature scale is dependent on context length
Softmax dilutes at long context because while the difference above a baseline for a significant token might stay the same, the total probability of the important token becomes a smaller fraction of the total. To mitigate this, global layers scale both the queries and the position bias by :
Where = keys the query can see. is flat until 128k tokens, then grows logarithmically, which sharpens the softmax at the rate the new tokens dilute it. This is the log-n scaling idea from the length-extrapolation literature, and the temperature half of YaRN.
Convolution windows on the query and key path
This is also quite unusual, borrowing the idea from linear attention type or state-space models (and some Transformers as well, to give them their due). Every layer runs four small depth-wise convolutions over the sequence, on the K and V projections and both residual-branch outputs. A three-token lookback on every key allows the representation to mix extremely local tokens; this is an example of almost the opposite notion to the learnt embeddings. This is a strong inductive bias to introduce, that says the architecture must look at these preceding three tokens in context as the immediate environment. This complements a position scheme that's otherwise purely about logits, but perhaps is a counter weight to the degree of model flexibility allowed in positional dependence.
Obviously this is not free, so we should compare just what the trade offs are here compared to RoPE, which maybe reminds us exactly why it has been such an attractive option for so long.
The trainable parameters are the easy part: r_proj
(hidden × heads × 16
per layer) plus the shared bank, a trivial addition at the scale of model we are talking about. The real cost is in the attention kernel. To see why, we should think again about where RoPE is applied, RoPE acts on q and k before they mix. The rotation is element-wise, and happens before the expensive dot product, so the fast kernels never have to adapt to the change. The relative part just falls out due to the maths of rotation matrices. FlashAttention never forms the full q·k matrix, instead it tiles over keys, while keeping a running softmax, and holds just a small block in fast memory. A logit bias cannot circumvent this, it acts on the attention matrix before softmax, exactly inside the operation the kernel is trying not to materialise. To add a per-query, per-key bias you have to produce it inside the tiled loop, block by block, as the scores form. This is not a fundamental problem, but should make us think of if not the Hardware Lottery, maybe the Kernel Lottery. Thinking Machines developed their own FlashAttention variant that makes this fast, but at release this was not public. Obviously the Open Source community quickly produced a viable option that recovered most of the performance, but there is a big difference between something tested to death in production through thousands of runs, and something adapted for this specific task. Not just in terms of reliability, but simply the number of engineer hours spent on optimising the last FLOPS out of these kernels.
Concretely the standard fast kernels for transformers have had the best part of five years of tuning across hardware generations. Exactly what tile sizes are best for particular cache hierarchies, warp specialisation, async copies, careful online-softmax numerics, coverage of every awkward shape (GQA, paged KV, variable seq len, odd head dims). The list could go on. Requiring more data to be in memory for the kernel will roughly double the I/O cost, at a key point in the computation that is usually bandwidth constrained as well. A fresh fused-bias kernel starts from scratch on all of that. Obviously Thinking Machines has their own (a FlashAttention-4 variant we understand) and some support was available, although notably not on HuggingFace, on release day. But an OSS kernel existing isn't the same as one with thousands of GPU-years of production behind it. There are all the edge cases that only get discovered in real use, unsupported shapes, perf cliffs on untuned configs, scale-only numerical edge cases.
The other cost which will make adoption an uphill battle is that all the optimisation needs to be redone for all hardware. RoPE runs unchanged on a TPU, an AMD card, an Apple M GPU, an inference ASIC, because it sits outside the attention kernel and can be applied identically on any of them. A fused bias has to be ported and re-tuned per backend, most of which reach attention through a vendor library (cuDNN, ROCm, XLA) that supports only the bias patterns the vendor chose, usually none, or ALiBi at best. This risks making adoption slow which would be a shame for a concrete deviation from the prevailing wisdom. Some of this is temporary of course; some, however, is structural. Kernels mature, they get upstreamed, and in a year if there’s demand the fused-bias path may be a supported flag in the mainline libraries. But the unavoidable cost is that the bias in the logits always requires the kernel to do strictly more communication (I/O) and computation than no bias.
The NoPE at large distances is uncontroversial, but combining this with a fully learnt local window representation that strictly increases the capacity of the model is significant. This allows the model to distinguish the local structure of the sequence, while recognising that removing inductive biases from the model and allowing it to find those patterns from sufficient data will win out. The bet isn’t just that removing the inductive biases of RoPE is worth it, but that embarking down a new path which changes the fundamental way tokens relate to each other is worth the cost of years of some of the most optimised kernel engineering. It is completely natural that individual queries benefit from unique positional information based on both the token and the context. Inkling offers this up, and we will definitely be conducting our own ablations.