LLM handling of programming language differences A study of Python source generated by multiple LLMs, published in 2025, found it was not possible to reliably predict human versus LLM authorship, according to an analysis of how LLMs handle programming language differences. The analysis, drawing on CodeQL variant analysis of 100 C and Java projects, reports that C and Java functions diverge in size distribution for the most common 1-to-10-line range but follow a very similar power law beyond roughly 50 lines. When asked to convert a C function containing the unbracketed expression `! x == y` to Fortran, both Grok 4.6 and ChatGPT 5.5 noted the differing operator precedence and inserted the appropriate parentheses. Home https://shape-of-code.com/ Uncategorized https://shape-of-code.com/category/uncategorized/ LLM handling of programming language differences LLM handling of programming language differences LLMs are trained on the publicly available source code, with no consideration given to the programming language used. There is an implicit assumption that training on code written in X and Y , rather than just Y , produces an LLM that does a better job of generating code written in Y . To what extent will the characteristics of the training data from language X source ‘leak’ into source code generated for language Y ? LLM source generation is a very new field and significant improvements are still being made to models and agent harnesses. This article lists some issues, and checks how a few models handle them. It would be great to have lots of LLM generated code to measure, and there are a few collections of LLM generated code. However, reliably distinguishing human from LLM generated code is an open problem. A 2025 study of Python source generated by multiple LLMs https://arxiv.org/html/2409.01382v2 found that it was not possible to reliably predict human/LLM authorship which suggests that LLM source datasets collected using authorship prediction are likely to be unreliable . The characteristics of a dataset of individual functions https://arxiv.org/html/2609.12708 are unlikely to have the same distribution as a dataset of complete programs. Some language usage behaviours are the result of the widespread adoption of particular conventions, and from the program correctness perspective these differences are harmless e.g., source indentation https://shape-of-code.com/2026/09/20/source-line-length-before-coding-agents/ , while others are semantic differences that could produce different output. While patterns caused by coding conventions might not change program output, they can change its performance characteristics e.g., larger functions can alter cache occupancy , and some of these patterns are input to mathematical models of program evolution, e.g., distribution of method size in LOC is a factor in modelling the percentage of methods containing no reported faults https://shape-of-code.com/2025/09/07/percentage-of-methods-containing-no-reported-faults/ . The plot below shows the percentage of C and Java functions/methods containing a given number of lines data recently extracted using CodeQL’s variant analysis of 100 C and Java projects; code+data http://www.shape-of-code.com/code-data/C-Java-LOC.tgz : The large number of 1-line Java methods is assumed to be due to the widespread use of getters/setters. For the most common function/method sizes i.e. 1 to 10 lines , have C and Java very different distributions, but follow a very similar power law for more than around 50 lines. Will LLM generate code contain this C/Java size difference, or will it have a completely different distribution coding agents do seem to generate more code ? Pre-object-oriented algorithmic languages can be classified into two families, those influenced by Fortran and those influenced by C. In Fortran arrays are 1-based, while in C they are 0-based. The base-value for arrays is common knowledge, which LLMs handle. The more obscure differences, which rarely occur in source, are more likely to be overlooked. Fortran and C have differences in operator precedence. The significant difference is the relative precedence of the unary not operator, and the binary equality/relational operators. The expression: x == y is equivalent to .not. x == y in Fortran, and equivalent to: x == y in C. When asked to convert a C function containing the unbracketed form of the expression to Fortran, both Grok 4.6 https://x.com/i/grok?conversation=2103498777218732427 and ChatGPT 5.5 https://chatgpt.com/share/6ab85889-fd28-83eb-9667-e93afee4e072 note the different operator precedence and insert the appropriate parenthesis. Both C and Python are popular languages with huge quantities of source publicly available. The token sequence: x < y == y , is parsed as: x < y == y in C, but in Python it is parsed as: x < y and y == y . Both Grok https://x.com/i/grok?conversation=2103498777218732427 and ChatGPT https://chatgpt.com/share/6ab85889-fd28-83eb-9667-e93afee4e072 note the different operator precedence and insert the appropriate parenthesis. There are a variety of differences between C and C++ http://www.knosof.co.uk/cbook/cbook.html . However, source containing occurrences of all but two of them will produce a compile time error. The following difference is known in compiler writer circles, and both Grok https://x.com/i/grok/share/08618a5abd3e4fae90d26061f72f4603 and ChatGPT https://chatgpt.com/share/6ab92b57-e978-83eb-86ab-822b7746e758 note the different possible behaviors: | template name < a , b - 5 // equivalent to template name < a , b - 5 non template name < a , b - 5 // equivalent to non template name < a , b - 5 | The C/C++ difference that is much more likely to be encountered is the expression sizeof 'a' , where C promotes the character 'a' to an int which commonly has size 4; on DSP processors char often occupies the same number of bits as an int , giving both of them a size of 1 , while C++ 'a' has type char which is defined to have size 1 . There are sometimes behavior differences, for the same source, between versions of the same language specification. For instance, the following somewhat obscure difference between the C90, C99, and C11 https://shape-of-code.com/2018/01/02/was-a-c90-c99-or-c11-compiler-used/ language standards: | include