I Asked LLMs and Fuzzers to Find Compiler Bugs. They Barely Overlapped.

Earlier this year, for a university security course, I compared two ways of finding bugs in Nim's std/asynchttpserver: a libFuzzer harness and three LLMs reviewing the source. The fuzzer found one real crash, the request line GET / HTTP/, and the fix was merged as Nim PR #25793. The models reported 23 findings. None of them was that crash, and two of the three had read the function and called it fine.

The conclusion back then was easy: models produce hypotheses, fuzzers produce evidence.

I have now repeated the comparison on a much larger target: Nimony, the experimental new Nim compiler. I keep two public bug trackers for it. One holds what LLM coding agents found, the other what seven fuzzers found:

Nimony is a work in progress, so the counts describe the two methods, not the compiler. And the result is not the one I got at university.

How the Two Methods Work

The agents work the way a developer would. Only 32 of their reports record an origin, and 26 of those came from porting real programs to Nimony using only its standard library. The rest read to me like audits of one part of the compiler.

Each fuzzer generates random valid programs and needs an oracle, something that knows the right answer: the C backend for the other backends, a build without --opt:speed for the optimizer, a small reference interpreter for closures and exceptions, destructor counts for memory, a run-time check for every claim the prover accepted, GNU as for the nifasm assembler. Each ran for about 45 minutes.

What Changed

The agents' reports are no longer hypotheses. All 202 come with a repro. One was closed as invalid, 10 are duplicates of issues in the fuzzer tracker, and 21 are not compiler bugs: enhancement requests, design questions and tooling reports. That leaves 170 distinct bugs, 43 of them crashes and 24 miscompiles.

That is a long way from 23 unconfirmed findings and no crash. My reading is that the agents now run what they claim, and a finding without a program that fails does not get filed.

LLM agentsFuzzers
Distinct bugs17066
Silent (wrong result, no error)28%45%
Rejects valid code or bad diagnostic46%15%
Median repro5 lines10 lines
Not a bug, duplicate or invalid320

The fuzzers are noisy too, but earlier: 1,610 assembler signatures came down to 8 issues, and that triage happens before anything is filed. The agents' noise reaches the tracker as wish lists and duplicates.

They Barely Overlap

Only 10 bugs are in both trackers, and a fuzzer found just 4 of those. The other 6 came from my hand-written probes. Counting shared root causes as well, 18 of the agents' 202 issues have a counterpart in the fuzzer tracker.

The split follows the compiler's stages. Distinct bugs, agents against fuzzers:

  • Semantic checking: 88 against 7
  • Standard library: 27 against 1
  • Prover: 14 against 9
  • Closure lowering: 0 against 15
  • ARC/ORC, exception lowering, optimizer and inliner: 9 against 6
  • C code generator, driver and tooling: 15 against 1
  • Native backend (x86-64 and arm64): 8 against 8
  • LLVM backend: 0 against 4
  • JavaScript and WebAssembly backends: 4 against 4
  • The nifasm assembler: 5 against 11

The reason is the oracle. A fuzzer can only judge what its oracle can judge. It has no oracle for HTTP framing, a missing API or a confusing error message. The agents' oracle is a specification or a real program.

Agents Go Wide

JSON objects are unordered, but fromJson reads object fields by position (llms #95):

import std/[json, syncio]

type
  Point = object
    x: int
    y: int

var t = parseJson("""{"y": 2, "x": 1}""")
let p = fromJson(t, Point)
echo "x=", p.x, " y=", p.y

It prints x=2 y=1, with no error. Expected: x=1 y=2.

std/httpserver serves a request with Transfer-Encoding: gzip, chunked as one with an empty body, then parses the chunked body as a second request (llms #100). That is the request smuggling pattern, and the oracle is RFC 9112.

Fuzzers Go Deep

The fuzzers write programs nobody writes by hand. The exception fuzzer produced this one (fuzzers #49):

import std/syncio

proc t() =
  try:
    try:
      return
    finally:
      echo "inner finally"
  finally:
    return

t()

It prints inner finally twice. Expected: once.

The prover fuzzer found a false proof (fuzzers #14):

{.feature: "staticContracts".}
import std/syncio

proc find(s: seq[int]; x: int): int =
  var idx = -1
  for k in 0 ..< s.len:
    if s[k] == x:
      idx = k
      break
  {.assert: idx >= 0.}      # false when x is not in s
  result = s[idx]           # index "proven": no run-time check is left

echo find(@[10, 20, 30], 20)
echo find(@[10, 20, 30], 99)

It compiles and prints 20 and 0. The second call reads s[-1]. Expected: cannot prove assertion: 0 <= idx and cannot prove contract: i < s.len and 0 <= i.

One more, without code: ORC never collects a node whose last use stores it into its own field. Valgrind does not report the leak. The program's own destructor count does (fuzzers #46).

Better Together

Some of the best results came from combining the two. When fixes for bugs in the fuzzer tracker were in review, agents tested them and found 5 remaining gaps. My probes, written while triaging fuzzer output, added 16 silent bugs that no fuzzer reached.

This is not a controlled experiment. The trackers are not independent: 24 of the agents' issues cite fuzzer issues. I have no cost comparison either: writing and triaging the fuzzers is not counted, and I have no cost figures for the agents.

Conclusion

At university the answer was "fuzzers find bugs, models find hypotheses." On a compiler, with agents that file a failing program with every report, they find different bugs. Fuzzers go deep where an oracle exists. Agents go wide wherever there is a spec to read or a program to port. Using only one of them would have missed most of what the other found.

Both trackers are public. If you work on a compiler or a library, I would like to hear whether your split looks the same.

Comments

Post a Comment

Popular Posts