# Lessons from building a Multi-Horizon Crypto Prediction System with Transformers

> Source: <https://dev.to/eventhorizon-ia/lessons-from-building-a-multi-horizon-crypto-prediction-system-with-transformers-1cio>
> Published: 2026-07-11 14:24:31+00:00

I started this project in January 2026 with a simple question: can a Transformer trained on technical indicators predict Bitcoin's direction better than a coin toss?

By June, the system had processed over 87,000 training generations, survived three major architectural bugs, and taught me more about machine learning than any course could.

This is not a "how to get rich with AI" post. This is the engineering story.

The system uses a hybrid Transformer + BiLSTM architecture:

The entire network is custom-built in `PyTorch`

. No `AutoML`

. No third-party trading libraries. Every tensor, every gradient, every loss function — implemented from scratch.

Most crypto prediction systems show backtest results. Backtests lie.

I built a verification pipeline that timestamps every prediction. When the target horizon elapses, the system compares the prediction against the real Binance price and logs the result. No cherry-picking. No deleted trades. The audit panel is public.

This commitment to transparency became the project's core value. But it also exposed a bug that almost made me quit.

I was getting 96% accuracy on 5-minute predictions. I celebrated. Then I looked at the raw data:

```
[0, 0, 0, 0, 0, 0, 0, 0, ... 300 zeros in a row ...]
[1, 1, 1, 1, 1, 1, 1, 1, ... 200 ones in a row ...]
```

No model produces results like this. The problem was temporal dependence: I was recording a "trade" every 2 seconds, then verifying all ~900 snapshots against the same market event. The model wasn't predicting price direction — it was just reflecting whether the market went up or down in that window.

I fixed this with statistical subsampling:

```
_FATOR_SUBSAMPLE = {
    5: 1,       # 5s: record every tick
    60: 5,      # 1min: record every 5th tick
    300: 25,    # 5min: record every 25th tick
    1800: 150,  # 30min: record every 150th tick (5 min)
    3600: 300,  # 1h: record every 300th tick (10 min)
}
```

Accuracy dropped from 96% to ~52%. It hurt. But now every trade in the history represents an independent decision by the model.

If you predict a 1-hour horizon, you should only receive the reward 1 hour later — not a second sooner. But my early implementation allowed future price information to leak into the training signal.

The fix was a **Delayed Reward Buffer**. Every prediction is stored with its timestamp. The reward is computed and applied only when the target horizon actually elapses. The model never sees the future during training.

For weeks, my Transformer and LSTM weren't actually learning. The gradient from the loss function was stopping at the prediction heads — it never reached the backbone.

I had to recompute the full forward pass during gradient computation, ensuring the error signal propagated through the entire network. My computer fans immediately spun up louder. The backbone was finally learning.

After fixing these bugs and training on 365 days of Binance data with a T4 GPU: