LongTail-SWE
Can coding agents fix real issues in languages missing from standard benchmarks?
Vladimir Dobrovolskii · Ibragim Badertdinov · Alexander Golubev
Nebius
About the benchmark
LongTail-SWE evaluates coding agents on 202 real-world software engineering tasks across 13 underrepresented programming languages. Each task comes from a GitHub issue and a corresponding pull request, with an executable environment and tests. We review and repair task descriptions and tests to make evaluation more reliable.
Leaderboard
| # | Model | Context | |||||
|---|---|---|---|---|---|---|---|
| 1 | Opus 5.5high | 1M | 81.5 ±1.2 | 89.1 | 73.8 | 4.5M | 53 |
| 2 | GPT 6.1 Solhigh | 1M | 78.4 ±0.9 | 82.2 | 73.3 | 1.8M | 27 |
| 3 | GPT 5.6 Solhigh | 1M | 75.7 ±1.3 | 87.1 | 63.4 | 2.8M | 34 |
| 4 | GPT 5.6 Lunaxhigh | 1M | 71.5 ±0.4 | 84.2 | 56.9 | 13.0M | 77 |
| 5 | Kimi K3max | 262K | 63.7 ±2.4 | 76.2 | 48.5 | 16.8M | 151 |
| 6 | DeepSeek V4 Flashhigh | 1M | 56.1 ±1.5 | 71.8 | 39.1 | 26.5M | 200 |
| 7 | GLM-5.3-Flashmax | 262K | 48.2 ±0.9 | 69.8 | 23.8 | 38.3M | 314 |
| 8 | Qwen 3.8 27Bxhigh | 262K | 42.7 ±0.9 | 62.4 | 20.8 | 23.1M | 214 |
| 9 | Qwen 3.6 27Bthinking | 262K | 16.7 ±0.8 | 23.8 | 9.4 | 7.2M | 113 |
Nine model configurations, all in the same SWE-agent-like scaffold, three attempts per task. Pass@1 is the mean over all attempts, ± run-level SEM. Pass@3 counts tasks solved at least once. All-3 counts tasks solved every time. Tokens and steps are means per attempt. These are the nine configurations reported in the paper; the public results ledger currently covers eight. Qwen 3.8 27B was also used for difficulty selection.
By language
| Language | N | Opus 5.5high | GPT 6.1 Solhigh | GPT 5.6 Solhigh | GPT 5.6 Lunaxhigh | Kimi K3max | DeepSeek V4 Flashhigh | GLM-5.3-Flashmax | Qwen 3.8 27Bxhigh | Qwen 3.6 27Bthinking |
|---|---|---|---|---|---|---|---|---|---|---|
| Clojure | 13 | 82.1 | 74.4 | 51.3 | 46.2 | 64.1 | 35.9 | 30.8 | 38.5 | 10.3 |
| Dart | 14 | 71.4 | 61.9 | 81.0 | 81.0 | 66.7 | 61.9 | 40.5 | 38.1 | 11.9 |
| Elixir | 23 | 75.4 | 81.2 | 72.5 | 65.2 | 53.6 | 34.8 | 31.9 | 30.4 | 8.7 |
| Erlang | 12 | 100.0 | 100.0 | 86.1 | 83.3 | 83.3 | 75.0 | 50.0 | 58.3 | 25.0 |
| F# | 16 | 66.7 | 62.5 | 81.3 | 68.8 | 47.9 | 41.7 | 43.8 | 29.2 | 10.4 |
| Haskell | 14 | 92.9 | 95.2 | 83.3 | 83.3 | 76.2 | 78.6 | 66.7 | 52.4 | 26.2 |
| Julia | 27 | 86.4 | 93.8 | 85.2 | 77.8 | 76.5 | 58.0 | 59.3 | 38.3 | 16.0 |
| Kotlin | 12 | 77.8 | 63.9 | 72.2 | 61.1 | 50.0 | 66.7 | 58.3 | 47.2 | 22.2 |
| Lua | 13 | 89.7 | 89.7 | 89.7 | 74.4 | 66.7 | 66.7 | 59.0 | 51.3 | 17.9 |
| OCaml | 13 | 84.6 | 76.9 | 64.1 | 76.9 | 56.4 | 53.8 | 43.6 | 28.2 | 5.1 |
| R | 14 | 78.6 | 78.6 | 76.2 | 69.0 | 81.0 | 69.0 | 57.1 | 69.0 | 42.9 |
| Scala | 18 | 87.0 | 66.7 | 66.7 | 70.4 | 53.7 | 50.0 | 37.0 | 44.4 | 14.8 |
| Swift | 13 | 69.2 | 64.1 | 69.2 | 69.2 | 51.3 | 56.4 | 53.8 | 46.2 | 12.8 |
| Overall | 202 | 81.5 | 78.4 | 75.7 | 71.5 | 63.7 | 56.1 | 48.2 | 42.7 | 16.7 |
Pass@1 (%) by language, averaged over three attempts per task. N is the number of tasks.
How tasks are built
Merged pull requests linked to a single issue, from 13 languages. 2,640 are clear or repairable and pass screening.
An agent installs the project, runs and audits the tests. Each task gets its own Docker image.
A solver attempts the task. Three judges review it. A repairer fixes the tests and the description.
Tasks solved by Qwen 3.8 27B in all three runs are generally dropped. After validation, 35 tasks are excluded.
FAQ
How do I evaluate my model?
Each task ships as a Docker image with dependencies installed, and the tasks are also available in Harbor format. Run your agent on each task, then score the patch with the evaluator from the repository. An attempt counts as solved only if all fail-to-pass and pass-to-pass tests pass. We report three attempts per task; the agent prompts are in the repository.
How do you know the judges are right?
Two engineers independently reviewed 65 tasks from different pipeline stages and agreed on a label for each: 51 had defects and 14 did not. The three-judge protocol identified 49 of the 51 defective tasks (96.1% recall, 87.5% precision). It also flagged 7 of the 14 acceptable tasks for repair.
What does repair change?
Tests and an appended addendum to the description. The reference patch and initially audited issue text stay fixed during repair. The initial audit may redact implementation hints. Repair removes checks that reject valid alternative solutions and adds checks for requirements the tests missed. On 63 tasks whose description did not change, the same Opus 5.5 patches score 29.1% Pass@1 on the original tests and 67.2% on the repaired ones.
What about contamination?
Repositories in the collection were required to have been updated on or after January 1, 2026. This does not mean that every issue or pull request was created after that date. Tasks come from public issues and pull requests, so overlap with training data cannot be ruled out, and model training cutoffs are often undisclosed.
Are there plans to release data like this for training?
LongTail-SWE is an evaluation benchmark, with an easier companion set of 296 tasks. We are working on a larger release for training. In the meantime, see the SWE-rebench V2.1 RL tasks on Harbor and the SWE-rebench V2 dataset for training and evaluation. The repository provides task data, prompts and scripts for building environments and scoring patches.
Why these 13 languages?
Each language has an active open-source ecosystem with little or no coverage in current issue-resolution benchmarks. Together, they cover functional, JVM, BEAM, scientific and mobile stacks.
How does it relate to SWE-rebench?
LongTail-SWE builds on the SWE-rebench V2 pipeline. It adds screening for repairable candidates, an install-and-audit agent and the solver, judge and repairer loop.
Citation
@misc{dobrovolskii2026longtailswe,
title = {Longtail-SWE: High-Quality SWE Tasks for
Underrepresented Languages},
author = {Dobrovolskii, Vladimir and Badertdinov, Ibragim
and Golubev, Alexander},
year = {2026},
eprint = {2611.10304},
archivePrefix = {arXiv},
primaryClass = {cs.SE},
url = {https://arxiv.org/abs/2611.10304}
}
From the team behind SWE-rebench, Nebius · Code MIT · Data CC BY 4.0