The code I never had to write

Metaprogramming is the practice of writing programs that inspect, generate, or transform other programs. In Nim, that usually means taking a small piece of source code and turning it at compile time into ordinary, statically typed Nim.

That transformation is what has kept me interested in metaprogramming over the years. I enjoy finding a small description that captures what the programmer means, then letting the compiler expand it into the larger amount of mechanical code the program needs. The generated code is not necessarily clever. Often it is exactly the tedious code I would have written by hand, and am pleased never to write again.

Three of my libraries show how my use of this idea has developed: protocoled, an early macro-based interface experiment; smartcli, which generates a typed command-line parser from help text; and dokime, which validates SQL against SQLite and generates typed query code.

protocoled: generating an interface

I wrote protocoled in the Nim 2 era, when macros were the natural tool for this work. A Nim macro receives syntax trees as NimNode values and returns a new tree for the compiler to insert in their place.

Protocoled uses a macro to turn this compact declaration:

protocol PExpr:
  proc eval(e): int

  impl PLiteral:
    var x: int
    proc eval(e): int = e.x
    proc newLit(x: int): PLiteral =
      result = PLiteral(x: x)

into code roughly like this:

type
  PExpr = ref object of RootObj
    evalImpl: proc(e: PExpr): int {.nimcall.}

  PLiteral = ref object of PExpr
    x: int

proc eval(e: PExpr): int =
  assert e.evalImpl != nil
  e.evalImpl(e)

proc evalPLiteral(e: PExpr): int =
  let e = PLiteral(e)
  result = e.x

proc newLit(x: int): PLiteral =
  result = PLiteral(x: x)
  result.evalImpl = evalPLiteral

None of this code is individually difficult, which is precisely why I do not want to maintain it. The short form records the interesting fact (that PLiteral implements eval) and the macro handles the ceremony.

This early design has an obvious constraint: implementations must live inside the protocol block and follow its construction convention. The types also belong to a single inheritance-based protocol hierarchy. It is a closed little DSL, but the result still amazes me: with one macro, we effectively gave interfaces to a language that did not have them.

From macros to compiler plugins

Nimony takes a different approach. Its primary metaprogramming mechanism is the compiler plugin: a separate Nim program compiled to a native executable. The compiler writes the relevant program fragment in an intermediate format, the plugin transforms it, and the compiler inserts the result back into the program.

I find this execution model especially useful for substantial generators. A plugin has a defined compiler representation as its boundary, is compiled and cached independently, and can use normal libraries and C interfaces. It can emit hygienic names and references, attach diagnostics to the original source, and return code for semantic checking.

smartcli: expanding help text into a parser

I've always found command-line parsing libraries cumbersome. A small CLI can require a builder API, option declarations, parser configuration, result types, validation callbacks.

What I love about smartcli is that the help text already contains almost everything. The commands, arguments, flags, choices, and descriptions are all there. Instead of describing the same interface through a library API, smartcli turns the help text itself into the program:

let options = cliapp"""Greeter v0.1

Commands:
  greet INPUT  Greet from a file
  version      Display the version

Arguments:
  INPUT  Input file

Options:
  --mode=fast|slow  Output mode
  -v, --verbose     Enable verbose output
  -h, --help        Show help"""

That one expression expands into something resembling:

type
  CliCommand = enum
    cmdNone, cmdGreet, cmdVersion

  CliMode = enum
    cliModeNone, cliModeFast, cliModeSlow

  CliOptions = object
    input: string
    command: CliCommand
    mode: CliMode
    verbose: bool

proc parseCli(): CliOptions =
  var parser = initOptParser()
  var argumentSlot = 0

  while true:
    next(parser)
    case parser.kind
    of cmdArgument:
      case argumentSlot
      of 0:
        case parser.key
        of "greet": result.command = cmdGreet
        of "version": result.command = cmdVersion
        else: cliUnexpectedArgument(spec, parser.key)
      of 1:
        result.input = parser.key
      else:
        cliUnexpectedArgument(spec, parser.key)
      inc argumentSlot
    of cmdLongOption, cmdShortOption:
      # Generated cases for help, mode, and verbose.
      discard
    of cmdEnd:
      break

  # Generated checks for missing arguments and version handling.

let options = parseCli()

The real expansion also handles short and long options, validates enum choices, and generates the command-specific argument and error paths. As the help text grows, so does the parser; the source at the call site does not become a second, handwritten parser.

I particularly like that the generated result is not a stringly typed map. options.verbose is a bool, options.mode is an enum, and options.command is another enum. The plugin performs the repetitive translation once during compilation, leaving direct branches and normal types in the executable.

dokime: expanding SQL into typed FFI code

Dokime applies the same idea across a language boundary. A query starts as an ordinary SQL string:

let user = query(db, "SELECT id, name FROM users WHERE id = ?", userId)

echo user.name

During compilation, the plugin prepares that SQL against a development database through SQLite's C API. SQLite resolves the table and columns; dokime reads the parameter count, column names, declared types, and nullability. From that metadata it can generate code along these lines:

block:
  var stmt = prepareStmt(db, "SELECT id, name FROM users WHERE id = ?", 39)
  bindParam(stmt, 1, userId)

  var row: tuple[id: int64, name: string]
  let stepCode = stepStmtCode(stmt)
  if stepReturnedRow(stepCode):
    row = (
      d: columnInt64(stmt, 0),
      name: columnString(stmt, 1)
    )

  let finalizeCode = finalizeStmtCode(stmt)
  checkStepCode(stepCode)
  checkFinalizeCode(finalizeCode)
  requireStepRow(stepCode)
  row

The small query call generates the repetitive prepare-bind-step-decode lifecycle and, more importantly, a result type that matches the selected columns. Nullability is preserved too: a nullable TEXT column becomes Opt[string], while the same metadata can shape an optional row or a stream of rows.

This is the project where compiler plugins became most exciting to me. The plugin is native code, so it can ask SQLite itself what the statement means rather than trying to reproduce SQL validation in Nim. A misspelled table or column becomes a compile error. The metadata returned by SQLite then shapes the Nim program, so user.name is checked like any other field access.

Dokime also generates the unpleasant parts of dynamic optional clauses. Given:

for user in rows(db, """
SELECT id, name FROM users WHERE 1 = 1
  [AND age >= ?]
  [AND name = ?]
""", minAge, name):
  echo user.name

the plugin validates all four possible queries at compile time. The convenience of dynamic filtering does not leave an unchecked query path.

What I would like to explore next

Lately I have been inspired by the Ada/SPARK ecosystem and its vision of high-assurance software built from specifications, contracts, and formal verification. I find the idea of moving from a high-level design toward an implementation whose important properties can be checked, not just tested, exciting.

At some point, I would like to explore what metaprogramming could contribute to a framework that helps bridge that gap, including for embedded systems and microcontroller targets. This is an aspiration rather than a roadmap, but it is the direction that currently makes me most curious about what might come next.