# Longtail-SWE: Nebius Benchmark for 13 Underrepresented Coding Languages

> Source: <https://explainx.ai/blog/longtail-swe-nebius-13-language-coding-agent-benchmark-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog>
> Published: 2026-11-11 00:00:00+00:00

**Nebius published Longtail-SWE on November 6, 2026 (arXiv [2611.10304](https://arxiv.org/abs/2611.10304?ref=explainx))** with immediate artifacts on **Hugging Face**: **[nebius/Longtail-SWE](https://huggingface.co/datasets/nebius/Longtail-SWE?ref=explainx)**, **[nebius/Longtail-SWE-trajectories](https://huggingface.co/datasets/nebius/Longtail-SWE-trajectories?ref=explainx)**, and a **[Harbor](https://hub.harborframework.com/datasets/longtail-swe/longtail-swe?ref=explainx)** task bundle. The benchmark fills a gap mainstream **SWE-bench** forks leave open: **repository-level repair** in languages whose communities rarely get agent eval seats.

## TL;DR

| Item | Detail | 
|---|---|
| Tasks | **202** issue-resolution instances | 
| Languages | **13** long-tail ecosystems | 
| Source | Real **GitHub issues and PRs** | 
| Pipeline | **Solver → judge → repairer** on tests and prompts | 
| Artifacts | Datasets, envs, tooling, **agent trajectories** | 
| Extra | Easier task subset for **smaller models** | 

## Why another SWE benchmark exists

Leaderboard-centric benchmarks concentrate on **Python and JavaScript** monorepos. When constructors filter **ambiguous issue text** or **flaky tests**, they often delete the only public tasks for **Clojure**, **Erlang**, or **OCaml** ecosystems entirely. Longtail-SWE argues that discards **real engineering difficulty** and makes **polyglot agents** look better than they are.

The construction insight is **repair instead of reject**. A **solver** attempts each candidate task. A **judge** reads the solver trajectory plus test output and proposes **test and description fixes** that:

- Stop rejecting **valid alternative implementations**
- Stop accepting **incorrect solutions**
- Keep tasks **hard** while preserving original intent

That mirrors human maintainer work more than one-shot SWE-bench importers.

Issue card linked to a patch illustrating Longtail-SWE repository-level agent tasks

*Image: [Nebius Longtail-SWE project](https://longtail-swe.com/?ref=explainx)*

Human hand and gripper sharing a beam for polyglot coding agent evaluation

## Language coverage

The thirteen languages are not decorative tags—they change **build graphs**, **package managers**, and **test runners** your agent must orchestrate:

| Language cluster | Why agents stumble | 
|---|---|
| **BEAM family** (Elixir, Erlang) | Supervision trees, dynamic modules | 
| **FP web** (Clojure, Haskell, OCaml, F#) | Rich type errors and macro layers | 
| **Mobile / JVM** (Kotlin, Scala, Swift) | Gradle or Xcode-sized environments | 
| **Stats / DSL** (R, Julia, Lua) | Domain libraries with sparse LLM priors | 

If your product claims **“any language”** support, a single Python SWE-bench score is insufficient evidence.

## Artifacts you can run this week

**Dataset:** [nebius/Longtail-SWE](https://huggingface.co/datasets/nebius/Longtail-SWE?ref=explainx)

**Trajectories:** [nebius/Longtail-SWE-trajectories](https://huggingface.co/datasets/nebius/Longtail-SWE-trajectories?ref=explainx)

**Project site:** [longtail-swe.com](https://longtail-swe.com/?ref=explainx)

**GitHub:** [github.com/SWE-rebench/LongTail-SWE](https://github.com/SWE-rebench/LongTail-SWE?ref=explainx)

**Harbor:** `longtail-swe/longtail-swe` on the Harbor framework hub

Harbor matters because teams already running **[agent eval playbooks](https://explainx.ai/blog/amazon-bedrock-agentcore-evaluation-playbook-deepeval-autoeval-guide-2026)** or custom **CI sandboxes** can slot Longtail-SWE beside **SWE-bench Verified** without rewriting orchestration.

### Suggested first run

1. Pull Harbor task definitions and inspect **one language** you actually ship (for example**Kotlin** ).
2. Run a **single-task smoke** with your default coding agent—log**tool calls** ,**patch size** , and**test iterations** .
3. Compare against a **Python SWE-bench** task of similar star-count repo to see**token and time inflation** .
4. Only then schedule a **full 202** pass; container cold starts multiply by language toolchains.

## Evaluation design in the paper

Authors test **nine model configurations**, including frontier models, and analyze:

- **Task solve rate** (with language breakdowns in the paper body)
- **Consistency across repeated attempts** —important for noisy agents
- **Resource usage** —wall clock and spend proxies

They also release **easier companion tasks** so **smaller open models** are not blocked from participation. That lowers the barrier for teams fine-tuning **9B–32B** coders who still need **non-Python** signal.

Community comments on the Hugging Face paper page highlight the **judge repairing tests that rejected valid implementations**—a failure mode anyone who has maintained OSS will recognize.

## How Longtail-SWE fits the 2026 eval stack

| Benchmark family | What it stresses | 
|---|---|
| [WorkflowArena](https://explainx.ai/blog/workflowarena-multi-agent-orchestration-benchmark-einsia-2026) | Multi-agent orchestration | 
| [Remote Labor Index](https://explainx.ai/blog/remote-labor-index-fable-5-2-automation-rate-scale-cais-2026) | Economics of automation | 
| Longtail-SWE | **Polyglot repo repair** | 

Pair Longtail-SWE with **internal tickets** from your customers’ least common language. If the agent fails Longtail-SWE **OCaml** tasks but passes **Python**, your marketing should say so.

## Limitations to plan around

- **Construction cost:** judge-repairer loops use LLM labor; tasks may inherit judge blind spots.
- **Star counts:** LongTail-SWE’s GitHub mirror showed modest stars at launch—expect rough edges in docs.
- **License mix:** repos retain their upstream licenses; commercial eval hosts must comply per task.
- **Not a security bench:** passing tests does not imply safe diffs—run separate policy checks if agents touch prod configs.

## Per-language notes for harness engineers

The paper’s value is **per-ecosystem friction**, not a single solve-rate meme. When you schedule eval batches, expect:

**Clojure and Scala (JVM cousins):** Dependency inference agents often hallucinate **Leiningen** versus **deps.edn** versus **sbt** paths. Long tasks reward agents that read **project.clj** before editing.

**Elixir and Erlang (BEAM):** Tests may assume **supervision** semantics. Patches that compile but break **OTP** patterns will fail in ways Python agents rarely see.

**Haskell, OCaml, and F#:** Type errors are long. Allocate higher **turn budgets** or your agent will stop mid-derivation even when close.

**Kotlin and Swift:** Mobile toolchains drag **Gradle** or **Xcode** weight into containers. Cold-start minutes dominate cost more than LLM tokens.

**Julia and R:** Numeric stacks pull heavy native deps. Mirror Nebius **executable environments** exactly—do not slim Dockerfiles for speed.

**Lua and Dart:** Smaller communities mean **sparser** LLM priors. Expect more tool calls to read docs inside the repo.

**Harbor operators:** Import `longtail-swe/longtail-swe` and pin **one language** per nightly job so failures are attributable. Mixing thirteen toolchains in one CI matrix obscures regressions.

## Judge–repairer failure modes (what reviewers should watch)

The pipeline fixes **bad tests**, not **bad agents**. Watch for:

- **Overfitted tests** that accept one hacky patch pattern
- **Over-repaired descriptions** that leak solution hints
- **Judge variance** if the judge model drifts between construction batches

Nebius mitigates by keeping **human-readable trajectories** public. Audit a handful before trusting leaderboard numbers in vendor decks.

## SWE-bench family positioning (conceptual, not ranked)

| Bench style | Strength | Longtail gap it leaves | 
|---|---|---|
| Python-centric SWE-bench forks | Volume and community replicas | Drops rare languages when filters bite | 
| Polyglot ports | Wider language tags | Often still Python-majority issues | 
| Longtail-SWE | Retention via repair | Smaller N, higher setup cost | 

If your COO asks “are we SOTA?”, answer with **language coverage**, not one percentage.

## Cost model for a full 202-task pass

Budget **container minutes** more than **tokens**:

- **202** environments × average**15–40** minutes cold start (varies by language)
- **Agent turns** × model price
- **Storage** for trajectory logs if you archive for compliance

Start with **20 stratified tasks** (two per language) before burning a full cluster weekend.

## Security note for autonomous eval

Longtail-SWE runs **arbitrary project test suites** inside containers. Treat harness hosts like **CI**: network egress limits, no production secrets, and pinned base images. Our [AI usage fragile codebase](https://explainx.ai/blog/ai-usage-fragile-codebase-agentic-coding-hn-2026) thread shows why unreviewed agent patches compound tech debt even when tests pass.

## Integrating Longtail-SWE with Claude Code, Codex, and custom harnesses

Most teams do not run Harbor natively on day one. A practical bridge:

1. Export **one task JSON** from the dataset and craft a**DESIGN.md** -style prompt with repo layout, issue text, and test command—patterns from[Claude Code ownership guides](https://explainx.ai/blog/claude-fable-5-2-code-ownership-prompting-guide-2026) .
2. Run the agent in a **fresh container** per attempt; Longtail-SWE assumes**executable** roots, not partial checkouts.
3. Capture **trajectory format** compatible with Nebius releases so you can compare against published baselines later.
4. If the agent proposes **test changes** , flag that as a**judge-repairer** scenario—human reviewers should decide whether the agent cheated by weakening assertions.

For [eBay blocking third-party agents](https://explainx.ai/blog/ebay-blocks-third-party-ai-agents-browsing-shopping-2026), remember benchmarks are **offline** repos—you are not scraping commerce sites. Keep network egress off in eval sandboxes.

## What a good internal scorecard looks like

| Metric | Definition | 
|---|---|
| **Solve rate** | Tests green without human edit | 
| **Turn count** | Tool calls until patch or abort | 
| **Revert rate** | Tasks where agent broke unrelated modules | 
| **Language delta** | Worst language vs median | 
| **Cost per solve** | GPU container minutes + model spend | 

Report **language delta** to leadership; it exposes “Python-only SOTA” marketing lies.

## Paper metadata and citation

- **arXiv:**[2611.10304](https://arxiv.org/abs/2611.10304?ref=explainx)
- **Authors:** Vladimir Dobrovolskii, Ibragim Badertdinov, Alexander Golubev (Nebius)
- **Submitted to HF daily papers:** November 10, 2026
- **Community upvotes on HF:** trending modestly—signal for researchers, not mass virality

Cite the **dataset version** hash when publishing internal numbers; Nebius may add tasks in v2.

## FAQ for engineering managers

**Will this replace our private eval set?**

No. Use Longtail-SWE for **coverage gaps** in public languages; keep customer tickets for revenue-linked skills.

**Can we fine-tune on trajectories?**

Technically yes, ethically thorny. Read licenses on upstream repos before training on logged patches.

**Does Nebius benefit commercially?**

Nebius sells GPU cloud. A harder public bench nudges buyers toward serious infra—disclose that bias when interpreting their baselines.

**How often should we re-run?**

Quarterly, or whenever you bump **frontier model** versions. Agent regressions hide between releases.

## Nebius and the eval transparency trend

Cloud GPU vendors sponsoring benchmarks is not new, but **releasing trajectories** is. Trajectories let outsiders detect **judge gaming** and **tool abuse**—aligned with [Hugging Face security tradeoff](https://explainx.ai/blog/hugging-face-open-source-ai-security-tradeoff-solaiman-2026) debates about open eval data.

If your roadmap includes **non-English** issue trackers, Longtail-SWE still uses **English issue text** from GitHub—the languages are code, not locale. Plan separate evals for multilingual product copy. Archive raw logs under your internal experiment id so you can compare against Nebius trajectory releases later.

## Related reading

*Dataset URLs and paper id are accurate as of November 11, 2026.*