{"slug": "show-hn-jevpertus-jev-style-option-scoring-on-apertus", "title": "Show HN: JevPertus – Jev-style option scoring on Apertus", "summary": "A developer released JevPertus, an open-source implementation that adds LoRA adapters and a small pointer head to the swiss-ai/Apertus-v1.5-8B backbone to assign probabilities to a question's answer options in a single forward pass. JevPertus supports multiple-choice questions, ordered rating scales, and true/false statements, encoding a state, question, options, and decision token into one sequence and scoring each option's final hidden state against the decision state. The project requires Python 3.11 or newer and trains on data/train.jsonl with evaluation on data/test.jsonl, with the full backbone needing to fit on the selected device alongside activations and training state.", "body_md": "A simple, lightweight implementation of the Jev model on top of the Apertus LLM. JevPertus combines an Apertus text backbone, LoRA adapters, and a small pointer head to assign probabilities to a question's answer options.\n\nIt supports multiple-choice questions, ordered rating scales, and true/false statements. Predictions come from scoring the supplied options in a single backbone pass.\n\nThe implementation is based on the Blogpost from [Archerhume](https://archerhume.com/posts/jevs-architecture-unmasked) and the [Kev repository](https://github.com/Jaluus/JevPertus/blob/main/...) applied to the [Apertus LLM](https://huggingface.co/swiss-ai/Apertus-v1.5-8B).\n\nThe simple idea is to encode a \"State\", a \"Question\", the \"Options\" and a final \"Decision\" into a single sequence. This looks somthing like this:\n\n```\n<STATE_TOKEN> A Sandwich is defined as a food item consisting of two pieces of bread with a filling in between.\n<QUESTION_TOKEN> Is a hot dog a sandwich?\n<OPTION_START_TOKEN> Yes: a hot dog is a sandwich.<OPTION_END_TOKEN>\n<OPTION_START_TOKEN> No: a hot dog is not a sandwich.<OPTION_END_TOKEN>\n<OPTION_START_TOKEN> It depends on the culture.<OPTION_END_TOKEN>\n<DECISION_TOKEN>\n```\n\nWe can then feed this sequence into an LLM and extract the final hidden states for each of the `<OPTION_END_TOKEN>` tokens and the `<DECISION_TOKEN>` token.\nThis gives us a representation of each option and the decision point, which we can then use to score the options and make a final decision.\n\nThis is done by passing the final hidden states through a small pointer head, which outputs a probability distribution over the options. For example, if we say `<OPTION_END_TOKEN>`), and `<DECISION_TOKEN>`), we can compute the scores for each option using the pointer head as follows:\n\nFirst, project the decision state into a query vector shared by all options:\n\nFor option 1 (\"Yes\"), project its hidden state into a key vector and compute its score:\n\nFor option 2 (\"No\"), use the same key projection and query:\n\nFor option 3 (\"It depends on the culture\"), repeat the calculation:\n\nHere, \n\nUse Python 3.11 or newer. From the repository root:\n\n```\npython -m venv .venv\nsource .venv/bin/activate\npython -m pip install -r requirements.txt\n```\n\nFor GPU use, install a PyTorch build compatible with your CUDA environment.\n\nThe default backbone is `swiss-ai/Apertus-v1.5-8B`. The loader downloads model files from Hugging Face when they are not cached; it also accepts a local checkpoint directory. Configure Hugging Face credentials if your chosen checkpoint requires authentication.\n\nThe full backbone must fit on the selected device alongside activations and training state.\n\nThe training script reads `data/train.jsonl` and evaluates on `data/test.jsonl`. Each line is a JSON object containing a shared `state` and a mapping of question IDs to labeled questions:\n\n```\n{\n  \"state\": \"You live in Zurich.\",\n  \"questions\": {\n    \"q1\": {\n      \"type\": \"choice\",\n      \"instructions\": \"Which country do you live in?\",\n      \"criteria\": {\n        \"A\": \"Switzerland\",\n        \"B\": \"France\"\n      },\n      \"label\": \"A\"\n    },\n    \"q2\": {\n      \"type\": \"noul\",\n      \"instructions\": \"You live in Switzerland.\",\n      \"label\": true\n    },\n    \"q3\": {\n      \"type\": \"score\",\n      \"instructions\": \"How certain are you?\",\n      \"criteria\": [\n        \"Uncertain\",\n        \"Certain\"\n      ],\n      \"label\": 1\n    }\n  }\n}\n```\n\n| Type | `criteria` | Label in JSONL | \n|---|---|---|\n| `choice` | Mapping of answer keys to descriptions, in option order | Answer key such as `\"A\"` , or a zero-based option index | \n| `score` | Ordered list of rating descriptions | Zero-based option index | \n| `noul` | Optional mapping with `\"false\"` and`\"true\"` descriptions | Boolean, or `0` for false and`1` for true | \n\nEvery training and evaluation question needs a label. The loader converts choice keys and boolean labels into zero-based indices. For inference, supply an individual question with its own `state` and omit the label.\n\nAn example train script can be found in `train_jevpertus.py`.\nTo configure training, set the following environment variables:\n\n| Setting | Default | \n|---|---|\n| `BASE_MODEL` | `swiss-ai/Apertus-v1.5-8B` | \n| `DATA_DIR` | `data` | \n| `DEVICE` | `cuda:0` | \n| `EPOCHS` | `2` | \n| `BATCH_SIZE` | `1` | \n| `LORA_RANK` | `16` | \n| `LEARNING_RATE` | `5e-5` | \n| `OUTPUT_DIR` | `runs/jevpertus-v1.5-8B` | \n\nTraining writes per-step loss and per-epoch evaluation metrics to `loss_history.jsonl`, and saves a checkpoint after each epoch:\n\n```\nruns/jevpertus/\n├── loss_history.jsonl\n├── epoch_1/\n│   ├── adapter_config.json\n│   ├── jev_config.json\n│   └── jev.pt\n└── epoch_2/\n    ├── adapter_config.json\n    ├── jev_config.json\n    └── jev.pt\n```\n\nCheckpoints contain the LoRA adapter weights, pointer-head weights, and configuration. Base-model weights and optimizer state are not saved. Loading a checkpoint requires access to its base model. Choose a new `OUTPUT_DIR` for each run to preserve previous metrics and checkpoints.\n\nTo see how JevPertus performs on a few example questions, run `inference_jevpertus.py`. The script loads a checkpoint and prints predictions for three sample questions.\n\nThe examples cover all three question types. Choice and score questions print a probability for each option; `noul` questions print the probability of true.\n\nAccuracy (%) on the full test splits; Global-MMLU averages English, German, French,\nand Italian accuracies, while other rows weight each question equally. Both JevPertus\nruns use checkpoints after two training epochs and zero-shot pointer-head scoring:\n`runs/jevpertus-v1.5-8B/epoch_2` for `swiss-ai/Apertus-v1.5-8B` and\n`runs/jevpertus-8B-Instruct-2509/epoch_2` for `swiss-ai/Apertus-8B-Instruct-2509` (V1).\nThese are scores for the trained Jev model, not the unmodified Apertus backbone.\n\n| Benchmark | Questions | Jev + Apertus v1.5-8B | Jev + Apertus 8B-Instruct-2509 | Apertus v1.5-8B Instruct (original) | Apertus 8B-Instruct-2509 (paper) | \n|---|---|---|---|---|---|\n| MMLU | 14,042 | 50.51 | 54.20 | Coming Soon | 60.9 | \n| MMLU-Pro | 12,032 | 27.90 | 25.85 | Coming Soon | - | \n| ARC-Challenge | 1,172 | 73.63 | 74.32 | Coming Soon | 77.6 | \n| Global-MMLU (language average) | 56,168 | 47.14 | 51.01 | Coming Soon | 55.7 | \n\nPaper scores are for **Apertus-8B-Instruct (v1)**, from\n[Table 17](https://arxiv.org/html/2509.14233v2#S5.T17) (MMLU and Global-MMLU)\nand [Table 21](https://arxiv.org/html/2509.14233v2#S5.T21) (ARC Challenge Chat).\n\n```\npython evaluate_benchmarks.py \\\n  --checkpoint runs/jevpertus-v1.5-8B/epoch_2 \\\n  --benchmarks mmlu mmlu-pro arc-challenge global-mmlu \\\n  --languages en de fr it \\\n  --device cuda:0 --batch-size 1\n```\n\nResults default to the checkpoint's `evals/` subfolder, here\n`runs/jevpertus-v1.5-8B/epoch_2/evals/`\n\n| File | Purpose | \n|---|---|\n| `dataloader.py` | JSONL loading, question encoding, padding, and batching | \n| `modeling/apertus/apertus.py` | Apertus text architecture | \n| `modeling/apertus/loading.py` | Loading compatible original Apertus and V1.5 text checkpoints | \n| `modeling/pointerhead.py` | Option-scoring head | \n| `modeling/jev.py` | Backbone/head composition, LoRA setup, and checkpoint handling | \n| `train_jevpertus.py` | Training and evaluation entry point | \n| `inference_jevpertus.py` | Checkpoint loading and example predictions | \n\nJevPertus builds on the Apertus backbone. The banner is inspired by the Apertus wordmark, with custom JevPertus lettering.\n\n- [Jan-Lucas Uslu](https://github.com/Jaluus) - JevPertus.\n- [Jared Palmer and the Kev contributors](https://github.com/jaredpalmer/kev) - source of the data included in this repository.\n\n- **Code:** licensed under the[MIT License](https://opensource.org/license/mit) .\n- **Data:** the datasets in`data/` come from[Kev](https://github.com/jaredpalmer/kev) and are licensed under[Apache License 2.0](https://github.com/jaredpalmer/kev/blob/main/LICENSE) . Upstream copyright: 2026 Jared Palmer.", "url": "https://wpnews.pro/news/show-hn-jevpertus-jev-style-option-scoring-on-apertus", "canonical_source": "https://github.com/Jaluus/JevPertus", "published_at": "2026-09-25 12:48:48+00:00", "updated_at": "2026-09-25 13:00:10.753397+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-tools", "machine-learning"], "entities": ["JevPertus", "Apertus", "swiss-ai/Apertus-v1.5-8B", "Archerhume", "Hugging Face", "LoRA", "PyTorch", "Jaluus"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-jevpertus-jev-style-option-scoring-on-apertus", "markdown": "https://wpnews.pro/news/show-hn-jevpertus-jev-style-option-scoring-on-apertus.md", "text": "https://wpnews.pro/news/show-hn-jevpertus-jev-style-option-scoring-on-apertus.txt", "jsonld": "https://wpnews.pro/news/show-hn-jevpertus-jev-style-option-scoring-on-apertus.jsonld"}}