TimesFM-3

The setup #

• Model: timesfm-3-1400m

(PyTorch, via the official HF repo) - Horizon: 512 steps ahead

• Channel: univariate target only, no covariates
• Frequency token: day

This worked fine on every series I threw at it until I bumped the horizon from 128 to 512. That's when the long-tail blowups started.

The error I dug into #

I caught it in a validation pass:

ValueError: The expanded size of the tensors must be the same, got 513 in the layer after the second attention block.

Not super helpful, but it pointed me at the positional encoding path. I traced it:

1. TimesFM-3 pads internally to align context + horizon.

2. With a 512-step horizon, the internal tensor hits 2048 + 512 + 1 = 2561.

3. The sinusoidal PE table is hardcoded to 2049 entries in the checkpoint.

That's the root. The model was pretrained with a max context of 2048 + 1, and the PE weights are literal lookup buffers — not extrapolatable. When the horizon pushes total length past that, attention starts reading garbage positions, which is why the forecast looked "confident but wrong" rather than throwing immediately.

What I actually tried #

First attempt: I tried slicing the horizon into overlapping 128-step chunks and stitching. That smoothed the spikes but added latency and introduced boundary artifacts every few days.

Second attempt: I patched the PE buffer at load time by extending and interpolating:

import torch.nn.functional as F

old_pe = model.pe.pe  # shape: [1, 2049, d_model]
new_len = 3072
pos = torch.arange(new_len).unsqueeze(1)
dim = torch.arange(old_pe.shape[-1]).unsqueeze(0)
div_term = 1.0 / (10000 ** (2 * (dim // 2) / old_pe.shape[-1]))

sin_pe = torch.zeros(1, new_len, old_pe.shape[-1])
sin_pe[:, :, 0::2] = torch.sin(pos * div_term)
sin_pe[:, :, 1::2] = torch.cos(pos * div_term)

model.pe.pe = torch.nn.Parameter(sin_pe)
model.pe.register_buffer('pe', sin_pe)

This works for inference. Forecasts are reasonable and the spikes are gone. Downside: it's not grounded in the training distribution, so I'm watching for distribution-shift drift in the tails.

The real fix (still looking) #

I think the architectural issue is that TimesFM-3 didn't bake in RoPE or ALiBa-style position handling, which would generalize past the training horizon. Google's blog mentions RoPE is in the works for the next checkpoint, but for now I'm stuck patching buffers.

If anyone's running 512+ horizons on TimesFM-3, are you chunking or extending PE at runtime? Curious whether the official team has a recommended path here, because the model is great when it doesn't hallucinate supply-chain chaos.

Next Nvidia just dropped $3. →

these AI tool field notes, with plenty of directly applicable cases.