[Home](https://shape-of-code.com/)>
[Uncategorized](https://shape-of-code.com/category/uncategorized/)> Flagging poor algorithm choice: LLMs next role
Flagging poor algorithm choice: LLMs next role #
Current use of LLMs in software engineering has focused on using them as proficient coding assistants or having them write programs from specifications.
Algorithm selection is an important issue that has yet to attract much attention within the software engineering LLM ecosystem. While much of a program’s functionality is mundane, and the accuracy/performance of the algorithm chosen is not critical, there is always some functionality where algorithm choice can have a big impact on accuracy/performance.
Existing code repositories provide the training data for both the mundane and the critical functionality.
In the same way that LLMs generate code that contains mistakes, LLMs can use an inappropriate/suboptimal algorithm, i.e., the algorithm does not do what is required, or does it inaccurately/slowly.
The flip side of generating code containing mistakes, is that LLMs can flag mistakes in existing code. A large number of serious coding issues are now being found by LLMs. I suspect that some of these were being flagged by traditional static analysis tools, along with a huge number of non-serious issues. However, LLMs appear to do a better job of separating out the really important issues from the run-of-the-mill.
The ability to find coding mistakes is not surprising. LLMs are creatures of their training data, and there are lots of examples of coding mistakes, along with discussions in fault reports, online posts, and the large number of developer interactions that the major vendors now have.
Fixing coding mistakes in existing code will soon be an infrequent activity. Fixing poor choice of algorithms is the next step up the value chain.
Despite its importance, algorithm selection is something of a niche topic (yes, there are lots of books discussing the mathematics behind algorithms and their implementation), with the common default of using what everybody else uses (which is often a good strategy). There is also the issue that choice of algorithm can be context dependent. For instance, a simple algorithm with quadratic runtime performance may be selected because the number of items processed is expected to be low. As program usage grows this initial assumption may evolve to a high number of items being processed, with quadratic performance becoming a bottleneck. I am not aware of a large source of examples of poor/good algorithm selection.
One domain where algorithm choice can have a huge impact is the numerical solution of the equations used in scientific and engineering applications. How good are LLMs at flagging the use of suboptimal/inappropriate algorithms in this domain?
Pre-LLM, figuring out which algorithm a function implemented, if any, was a very hard problem. The numbers tool was written to pattern match any real literals appearing in the source against a database of known formula and associated constants. The assumption being that any matching formula was the one implemented by the function containing the literals. Maintaining the numbers/formula database was a huge task, and the project died, apart from as a tool to flag possibly inaccurate literal use.
Based on a small sample (some ChatGPT responses), LLMs appear to do a good job of matching a cluster of floating literals to the corresponding formula (in all the cases I tried, they searched the web). All the scientific source I have is readily available on the web, apart from the Climategate code, which only contains simple data analysis and plotting code.
Recent papers appear to still be using the solution via a prompt approach (e.g., The orbit has an eccentricity of 0.6, and at a specific moment, the mean anomaly is 0.6 radians… find the eccentric anomaly in radians), code generation via a specification, or investigating low level improvements to existing code (e.g., detecting instabilities and stabilizing numerical expressions).