Show HN: JevPertus – Jev-style option scoring on Apertus

A simple, lightweight implementation of the Jev model on top of the Apertus LLM. JevPertus combines an Apertus text backbone, LoRA adapters, and a small pointer head to assign probabilities to a question's answer options.

It supports multiple-choice questions, ordered rating scales, and true/false statements. Predictions come from scoring the supplied options in a single backbone pass.

The implementation is based on the Blogpost from Archerhume and the Kev repository applied to the Apertus LLM.

The simple idea is to encode a "State", a "Question", the "Options" and a final "Decision" into a single sequence. This looks somthing like this:

<STATE_TOKEN> A Sandwich is defined as a food item consisting of two pieces of bread with a filling in between.
<QUESTION_TOKEN> Is a hot dog a sandwich?
<OPTION_START_TOKEN> Yes: a hot dog is a sandwich.<OPTION_END_TOKEN>
<OPTION_START_TOKEN> No: a hot dog is not a sandwich.<OPTION_END_TOKEN>
<OPTION_START_TOKEN> It depends on the culture.<OPTION_END_TOKEN>
<DECISION_TOKEN>

We can then feed this sequence into an LLM and extract the final hidden states for each of the <OPTION_END_TOKEN> tokens and the <DECISION_TOKEN> token. This gives us a representation of each option and the decision point, which we can then use to score the options and make a final decision.

This is done by passing the final hidden states through a small pointer head, which outputs a probability distribution over the options. For example, if we say <OPTION_END_TOKEN>), and <DECISION_TOKEN>), we can compute the scores for each option using the pointer head as follows:

First, project the decision state into a query vector shared by all options:

For option 1 ("Yes"), project its hidden state into a key vector and compute its score:

For option 2 ("No"), use the same key projection and query:

For option 3 ("It depends on the culture"), repeat the calculation:

Here,

Use Python 3.11 or newer. From the repository root:

python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt

For GPU use, install a PyTorch build compatible with your CUDA environment.

The default backbone is swiss-ai/Apertus-v1.5-8B. The downloads model files from Hugging Face when they are not cached; it also accepts a local checkpoint directory. Configure Hugging Face credentials if your chosen checkpoint requires authentication.

The full backbone must fit on the selected device alongside activations and training state.

The training script reads data/train.jsonl and evaluates on data/test.jsonl. Each line is a JSON object containing a shared state and a mapping of question IDs to labeled questions:

{
  "state": "You live in Zurich.",
  "questions": {
    "q1": {
      "type": "choice",
      "instructions": "Which country do you live in?",
      "criteria": {
        "A": "Switzerland",
        "B": "France"
      },
      "label": "A"
    },
    "q2": {
      "type": "noul",
      "instructions": "You live in Switzerland.",
      "label": true
    },
    "q3": {
      "type": "score",
      "instructions": "How certain are you?",
      "criteria": [
        "Uncertain",
        "Certain"
      ],
      "label": 1
    }
  }
}

Every training and evaluation question needs a label. The converts choice keys and boolean labels into zero-based indices. For inference, supply an individual question with its own state and omit the label.

An example train script can be found in train_jevpertus.py. To configure training, set the following environment variables:

Training writes per-step loss and per-epoch evaluation metrics to loss_history.jsonl, and saves a checkpoint after each epoch:

runs/jevpertus/
├── loss_history.jsonl
├── epoch_1/
│   ├── adapter_config.json
│   ├── jev_config.json
│   └── jev.pt
└── epoch_2/
    ├── adapter_config.json
    ├── jev_config.json
    └── jev.pt

Checkpoints contain the LoRA adapter weights, pointer-head weights, and configuration. Base-model weights and optimizer state are not saved. a checkpoint requires access to its base model. Choose a new OUTPUT_DIR for each run to preserve previous metrics and checkpoints.

To see how JevPertus performs on a few example questions, run inference_jevpertus.py. The script loads a checkpoint and prints predictions for three sample questions.

The examples cover all three question types. Choice and score questions print a probability for each option; noul questions print the probability of true.

Accuracy (%) on the full test splits; Global-MMLU averages English, German, French, and Italian accuracies, while other rows weight each question equally. Both JevPertus runs use checkpoints after two training epochs and zero-shot pointer-head scoring: runs/jevpertus-v1.5-8B/epoch_2 for swiss-ai/Apertus-v1.5-8B and runs/jevpertus-8B-Instruct-2509/epoch_2 for swiss-ai/Apertus-8B-Instruct-2509 (V1). These are scores for the trained Jev model, not the unmodified Apertus backbone.

Paper scores are for Apertus-8B-Instruct (v1), from Table 17 (MMLU and Global-MMLU) and Table 21 (ARC Challenge Chat).

python evaluate_benchmarks.py \
  --checkpoint runs/jevpertus-v1.5-8B/epoch_2 \
  --benchmarks mmlu mmlu-pro arc-challenge global-mmlu \
  --languages en de fr it \
  --device cuda:0 --batch-size 1

Results default to the checkpoint's evals/ subfolder, here runs/jevpertus-v1.5-8B/epoch_2/evals/

JevPertus builds on the Apertus backbone. The banner is inspired by the Apertus wordmark, with custom JevPertus lettering.

• Jan-Lucas Uslu - JevPertus.

• Jared Palmer and the Kev contributors - source of the data included in this repository.

• Code: licensed under theMIT License .

• Data: the datasets indata/ come fromKev and are licensed underApache License 2.0 . Upstream copyright: 2026 Jared Palmer.