I Asked LLMs and Fuzzers to Find Compiler Bugs. They Barely Overlapped.
Earlier this year, for a university security course, I compared two ways of finding bugs in Nim's std/asynchttpserver : a libFuzzer harness and three LLMs reviewing the source. The fuzzer found one real crash, the request line GET / HTTP/ , and the fix was merged as Nim PR #25793 . The models reported 23 findings. None of them was that crash, and two of the three had read the function and called it fine. The conclusion back then was easy: models produce hypotheses, fuzzers produce evidence. I have now repeated the comparison on a much larger target: Nimony, the experimental new Nim compiler. I keep two public bug trackers for it. One holds what LLM coding agents found, the other what seven fuzzers found: nimony-bugs-llms : 202 issues nimony-bugs-fuzzers : 83 issues, 66 from fuzzers and 17 from small probes I wrote while triaging Nimony is a work in progress, so the counts describe the two methods, not the compiler. And the result is not the one I got at university. H...
