{"slug": "llm-handling-of-programming-language-differences", "title": "LLM handling of programming language differences", "summary": "A study of Python source generated by multiple LLMs, published in 2025, found it was not possible to reliably predict human versus LLM authorship, according to an analysis of how LLMs handle programming language differences. The analysis, drawing on CodeQL variant analysis of 100 C and Java projects, reports that C and Java functions diverge in size distribution for the most common 1-to-10-line range but follow a very similar power law beyond roughly 50 lines. When asked to convert a C function containing the unbracketed expression `! x == y` to Fortran, both Grok 4.6 and ChatGPT 5.5 noted the differing operator precedence and inserted the appropriate parentheses.", "body_md": "[Home](https://shape-of-code.com/)>\n\n[Uncategorized](https://shape-of-code.com/category/uncategorized/)> LLM handling of programming language differences\n\n## LLM handling of programming language differences\n\nLLMs are trained on the publicly available source code, with no consideration given to the programming language used.  There is an implicit assumption that training on code written in *X* and *Y*, rather than just *Y*, produces an LLM that does a better job of generating code written in *Y*.\n\nTo what extent will the characteristics of the training data from language *X* source ‘leak’ into source code generated for language *Y*?\n\nLLM source generation is a very new field and significant improvements are still being made to models and agent harnesses. This article lists some issues, and checks how a few models handle them.\n\nIt would be great to have lots of LLM generated code to measure, and there are a few collections of LLM generated code.  However, reliably distinguishing human from LLM generated code is an open problem.  A 2025 study of [Python source generated by multiple LLMs](https://arxiv.org/html/2409.01382v2) found that it was not possible to reliably predict human/LLM authorship (which suggests that LLM source datasets collected using authorship prediction are likely to be unreliable).  The characteristics of a [dataset of individual functions](https://arxiv.org/html/2609.12708) are unlikely to have the same distribution as a dataset of complete programs.\n\nSome language usage behaviours are the result of the widespread adoption of particular conventions, and from the program correctness perspective these differences are harmless (e.g., [source indentation](https://shape-of-code.com/2026/09/20/source-line-length-before-coding-agents/)), while others are semantic differences that could produce different output.\n\nWhile patterns caused by coding conventions might not change\n\nprogram output, they can change its performance characteristics (e.g., larger functions can alter cache occupancy), and some of these patterns are input to mathematical models of program evolution, e.g., distribution of method size (in LOC) is a factor in modelling the [percentage of methods containing no reported faults](https://shape-of-code.com/2025/09/07/percentage-of-methods-containing-no-reported-faults/).\n\nThe plot below shows the percentage of C and Java functions/methods containing a given number of lines (data recently extracted using CodeQL’s variant analysis of 100 C and Java projects; [code+data](http://www.shape-of-code.com/code-data/C-Java-LOC.tgz)):\n\nThe large number of 1-line Java methods is assumed to be due to the widespread use of getters/setters. For the most common function/method sizes (i.e. 1 to 10 lines), have C and Java very different distributions, but follow a very similar power law for more than around 50 lines. Will LLM generate code contain this C/Java size difference, or will it have a completely different distribution (coding agents do seem to generate more code)?\n\nPre-object-oriented algorithmic languages can be classified into two families, those influenced by Fortran and those influenced by C.\n\nIn Fortran arrays are 1-based, while in C they are 0-based. The base-value for arrays is common knowledge, which LLMs handle. The more obscure differences, which rarely occur in source, are more likely to be overlooked.\n\nFortran and C have differences in operator precedence.  The significant difference is the relative precedence of the unary not operator, and the binary equality/relational operators.  The expression: `! x == y` is equivalent to `.not. (x == y)` in Fortran, and equivalent to: `(!x) == y` in C.  When asked to convert a C function containing the unbracketed form of the expression to Fortran, both [Grok 4.6](https://x.com/i/grok?conversation=2103498777218732427) and [ChatGPT 5.5](https://chatgpt.com/share/6ab85889-fd28-83eb-9667-e93afee4e072) note the different operator precedence and insert the appropriate parenthesis.\n\nBoth C and Python are popular languages with huge quantities of source publicly available.  The token sequence: `x < y == y`, is parsed as: `(x < y) == y` in C, but in Python it is parsed as: `(x < y) and (y == y)`.  Both [Grok](https://x.com/i/grok?conversation=2103498777218732427) and [ChatGPT](https://chatgpt.com/share/6ab85889-fd28-83eb-9667-e93afee4e072)  note the different operator precedence and insert the appropriate parenthesis.\n\nThere are a variety of [differences between C and C++](http://www.knosof.co.uk/cbook/cbook.html).  However, source containing occurrences of all but two of them will produce a compile time error.\n\nThe following difference is known in compiler writer circles, and both [Grok](https://x.com/i/grok/share/08618a5abd3e4fae90d26061f72f4603) and [ChatGPT](https://chatgpt.com/share/6ab92b57-e978-83eb-86ab-822b7746e758) note the different possible behaviors:\n\n| \n\n```\ntemplate_name < a , b > - 5\n             // equivalent to (template_name < a , b >) - 5)\nnon_template_name < a , b > - 5\n             // equivalent to (non_template_name < a) , (b > - 5)\n```\n\n | \n\nThe C/C++ difference that is much more likely to be encountered is the expression `sizeof('a')`, where C promotes the character `'a'` to an `int` (which commonly has size 4; on DSP processors `char` often occupies the same number of bits as an `int`, giving both of them a size of 1), while C++ `'a'` has type `char` (which is defined to have size 1).\n\nThere are sometimes behavior differences, for the same source, between versions of the same language specification.  For instance, the following somewhat obscure [difference between the C90, C99, and C11](https://shape-of-code.com/2018/01/02/was-a-c90-c99-or-c11-compiler-used/) language standards:\n\n| \n\n```\n#include <stdio.h>\n \n#define M(U) sizeof(U\"s\"[0])\n \nint main(void)\n{\n    switch(M(\"\")*2 //**/ 2\n                          )\n       {\n       case 1: printf(\"C90\\n\"); break;\n       case 2: printf(\"C99\\n\"); break;\n       case 8: printf(\"C11\\n\"); break;\n       }\n \n}\n```\n\n | \n\nWhen asked what this program outputs, [ChatGPT only considers C11](https://chatgpt.com/share/6ab7d867-43dc-83eb-88e6-e7232f19d45e), and 'corrects' itself after I pointed out the C90 lexical behavior.  [Grok](https://x.com/i/grok/share/85f5f0f75ddd47cd930ca9ab95310d1f) only considers the latest standard, and when asked about producing other outputs, runs gcc with various `-std` options to find the other cases.\n\nBoth LLMs used (Grok 4.6 and ChatGPT 5.5) handled all the language differences I tried. Will cheaper to use coding agents, using LLMs containing an order of magnitude fewer parameters, be as effective at handling obscure differences? We will have to wait and see.\n\n[Uncategorized](https://shape-of-code.com/category/uncategorized/)\n\n[automatic generation](https://shape-of-code.com/tag/automatic-generation/),\n\n[C](https://shape-of-code.com/tag/c/),\n\n[differences](https://shape-of-code.com/tag/differences/),\n\n[Fortran](https://shape-of-code.com/tag/fortran/),\n\n[Java](https://shape-of-code.com/tag/java/),\n\n[LLM](https://shape-of-code.com/tag/llm/),\n\n[obscure](https://shape-of-code.com/tag/obscure/),\n\n[Python](https://shape-of-code.com/tag/python/),\n\n[Source language](https://shape-of-code.com/tag/source-language/)", "url": "https://wpnews.pro/news/llm-handling-of-programming-language-differences", "canonical_source": "https://shape-of-code.com/2026/09/27/llm-handling-of-programming-language-differences/", "published_at": "2026-09-27 21:55:49+00:00", "updated_at": "2026-09-27 21:59:18.760277+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "developer-tools"], "entities": ["Grok 4.6", "ChatGPT 5.5", "CodeQL", "Fortran", "C", "Java", "Python"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/llm-handling-of-programming-language-differences", "markdown": "https://wpnews.pro/news/llm-handling-of-programming-language-differences.md", "text": "https://wpnews.pro/news/llm-handling-of-programming-language-differences.txt", "jsonld": "https://wpnews.pro/news/llm-handling-of-programming-language-differences.jsonld"}}