GPT 6.1 Sol seems to do worse on long-tail languages.

A Hacker News commenter using the handle srcreigh reported that GPT 6.1 Sol repeatedly made basic mistakes on a Prolog project, as if the model did not grasp symbolic resolution, and then tagged the programming languages in all 712 GPT 6.1 Sol battles logged by the OpenCode agent-duel battle system using a tool called Jev to test whether the model performs worse outside common programming languages. The tagging drew on requests and available conversation context without showing Jev the outcomes, and the commenter presented the 712-battle dataset as the basis for examining long-tail language performance.

srcreigh https://news.ycombinator.com/user?id=srcreigh wrote in a Hacker News comment https://news.ycombinator.com/item?id=50489509 : The ideal language is the one the agent knows how to use. I used prolog for a project and GPT 6.1 Sol repeatedly made dumb mistakes as if it doesn’t understand the symbolic resolution capabilities. That made me curious: does Sol do worse when it’s working outside the usual programming languages? We have some data to look at We’re working on a battle system for OpenCode https://github.com/bottomless/agent-duel , which lets users compare models on their actual tasks. I used Jev https://openrouter.ai/docs/guides/community/jev to tag the programming languages in all 712 Sol battles, using the requests and available conversation context without showing it the outcomes.