We Cannot 100% Trust AI-Generated Output: A Concrete Example

A developer found that Claude, an AI assistant from Anthropic, incorrectly flagged zero-length bugs in the WinReg C++ library's string getters, failing to account for the RegGetValueW API's null-termination guarantee. Claude later corrected itself after the developer pointed out the error, highlighting that AI-generated code cannot be trusted 100% and may contain hidden bugs or wrong assumptions.

Recently I asked Claude to review the code of my WinReg C++ library https://github.com/GiovanniDicanio/WinReg which is a C++ high-level wrapper around the low-level C-interface Windows Registry API . As a result of its analysis, Claude reported that there were zero-length bugs in my code, in particular Claude stated that zero-length REG SZ/REG EXPAND SZ values crash the GetStringValue, GetExpandStringValue, TryGetStringValue and TryGetExpandStringValue methods of the RegKey class. In particular, Claude noted that I correctly guarded against dataSize == 0 in the binary-returning getters like RegKey::GetBinaryValue , but the string getters do not have such guard; they unconditionally do: result.resize dataSize / sizeof wchar t - 1 ; Claude specified that REG SZ and REG EXPAND SZ values can be legitimately stored with cbData == 0 zero bytes, no NUL at all; different from an empty string made by a single NUL ‘\0’ . If you substitute zero for the dataSize variable in the above statement, you end up with: result.resize SIZE MAX ; which would throw a std::length error exception. Claude proposed to fix the above code using the same edge-case check logic I had already implemented in the binary getters to guard against the zero-length case: if dataSize == 0 { result.clear ; } else { result.resize dataSize / sizeof wchar t - 1 ; } My WinReg library is quite battle-tested, and there were bugs related to some edge cases that I had already fixed, so I was curious, and tried writing a zero-length string value in the registry, and read it back with my existing code. And I noted that at least in Windows 11 where I tested my code the dataSize == 0 condition was not hit at run-time. That is because in my C++ code I invoke the RegGetValue W API, which by contract guarantees to return a NUL-terminated string, even if the string stored in the registry doesn’t have a NUL-terminator. This is not the case for older APIs like RegQueryValueEx https://learn.microsoft.com/en-us/windows/win32/api/winreg/nf-winreg-regqueryvalueexw . I replied to Claude pointing that out, and Claude corrected itself: You’re right, and thanks for the pushback — I should have accounted for RegGetValueW ‘s null-termination guarantee before flagging that as a bug. Those AI tools can be very powerful, but the key takeway here is that we should not forget that they are just tools , and we should not trust AI-generated output and code 100%, because bugs and wrong assumptions can be hidden in that AI code, too.