The Derivation Abstraction Has a Price

nix-result-checks started as a result monad for build-time checks. This post is about the other half: eval-time testing. Pure Nix logic with no derivation involved, and the mistake I made trying to fit it into the same shape as build checks.

nix-result-checks went through a breaking 2.0.0 rewrite in June because of one thing I got wrong in 1.0: pure logic was paying for a wholly unnecessary store round-trip.

Two Kinds of Tests§

Nix code splits into two things you’d want to test differently.

Some of it is pure: option values, attribute names, string transforms, the shape of a generated config. None of it touches the filesystem or a builder. Evaluate the expression, compare it to what you expected. Done.

Some of it isn’t: build success, generated file contents, exit codes. That needs a real derivation, a real build, real output to inspect. Build it, run it, compare it to what you expected, done.

Eval Tests as Derivations§

nix-result-checks 1.0’s mkEval produces a derivation. It runs lib.debug.runTests at evaluation time and wraps it in a result derivation to reuse the pipeline. One report generator, one shape.

The uniformity made sense but the mechanism was expensive. Wrapping an eval verdict in a derivation means Nix instantiates and realizes a store path for a fact we already knew. Instantiation, a build step, a $out — all of it spent on something that needed none of it.

Abandoning runTests§

Getting any speedup at all meant restructuring how a test suite is shaped, not just dropping the derivation wrapper. By 2.0, mkEval no longer calls runTests as one batch over a whole attrset.¹ The actual comparison is still runTests’s own “expr vs expected” model, just invoked on each entry instead of the attrset. runTests maps over its attrset with mapAttrs, which is lazy per-key on its own, but then flattens the result through attrValues and concatLists into a single list. Building that list forces every test’s comparison, because list construction depends on all of them. The whole suite collapses into one thunk with no independently-forceable test inside it. mkEntries stops one step earlier, at the mapAttrs, and returns that attrset directly. That leaves each test as its own leaf for nix-eval-jobs to dispatch a worker on.

Measuring the Waste§

With tests shaped that way, how much better does it actually get? I benchmarked four ways of running a suite of pure eval tests:

• nix eval, sequential, the naive runTests-in-a-derivation path,
• nix-eval-jobs, sequential,
• nix-eval-jobs, parallel (4 and 8 workers),
• fastest, a nix-eval-jobs wrapper using --force-recurse and jq to flatten results.

At N=50 tests, parallel nix-eval-jobs ran about 5× faster than the sequential runTests-in-a-derivation approach — 5 seconds against 25. The gap widens as N grows. That is a cost multiplying per-test, it’s the sum of all the store operations. fastest roughly tracked nej-par-8 at every N tested.

The Selector Hack§

nix-eval-jobs walks an attrset and surfaces a leaf only when it looks like a derivation. Feed it a plain nested attrset of ints instead, and it walks the whole thing and emits nothing. nrc calls it with:²

nix-eval-jobs --option eval-cache false --force-recurse --workers N --select select.nix --apply 'drv: drv.result' --no-instantiate .#resultChecks.<system>.evalChecks

select.nix runs once, on the whole evalChecks tree. It walks the evalChecks, wrapping each individual entry in its own stub derivation. This allows one job per test instead of collapsing to one job for the whole tree. --apply then lifts each entry back out into the extraValue field of its NDJSON line; --no-instantiate keeps the stub from ever being realized in the store.

When nix-eval-jobs isn’t on PATH, the whole tree is fetched in one piece instead — sequential, but dependency-free: nix eval --json in flake mode, nix-instantiate --eval --strict --json in file mode.

Where Parallelism Actually Stops§

Parallelizing eval tests only helps if the benefit survives past the benchmark. Bake the eval results into a report derivation and every build of that report forces sequential evaluation: you pay for the parallel run once, then pay again, sequentially, to bake the result into an artifact.

The 2.0.0 fix separates the two kinds of results structurally: { report; evalChecks; }. report is the derivation half, build checks only, built in parallel by the ordinary Nix daemon.³ evalChecks is a lazy attrs tree, keyed by check then test, forced directly by whatever consumes it — nrc, or nix-eval-jobs underneath. Nothing in the eval half depends on a report derivation existing first. The parallelism boundary rests at the evaluator, and the data shape has to reflect that.

The Data Type Is the Contract§

Once eval results stopped being derivation outputs, the actual interface between nix-result-checks and everything consuming it turned out to be one value: { report :: drv; evalChecks :: { check -> test -> entry }; }.

mkReport and mkEvalChecks produce the two halves from one check set, the same shape resultChecks.checks already accepted. In flake mode, that value is namespaced under resultChecks.<system>. The flake-parts module produces it there automatically, or you write the same flake output by hand:

{
    # mkReport and mkEvalChecks from the overlay.
    report = mkReport checks;
    evalChecks = mkEvalChecks checks;
}

In file mode, nrc --file skips the namespace entirely. The file’s top-level value just is that same { report; evalChecks; } shape. Two entry points, one contract.

An earlier API surface got cut for the same reason this contract exists. reportGenerator made the report format pluggable — but nrc and the flake check gate both parse that format directly, so changing it without updating both would break whichever one you missed. Cutting reportGenerator fixes the format, and both consumers can rely on that.

What Collapsed Once Eval Results Were Values§

Snapshotting. Snapshotting an eval result used to mean routing it back through the derivation pipeline. As a value, we only need deep equality against a pinned attrset: mkEntries produces the entries, another eval test asserts they match. The existing eval-test format already covers it.

Orchestration. Ordering checks so one only ran if another passed used to lean on nativeBuildInputs, because that was the only lever when results lived in the store. With results as ordinary values, gating one check on another’s outcome is an if expression. I considered a mkGate helper and skipped it: the policy varies too much (all-must-pass vs. any-must-pass, skip- vs. fail-on-failure) for one helper to cover well. A conditional chosen per call site is perfectly fine.

Artifacts: The Store Is Still the Fixture Cache§

Build-side, the store is still the fixture cache: a check that produces something expensive writes it once, and downstream checks reference the artifact path. That doubles as dependency ordering.

One subtlety: $out is always touched by a derivation, whether or not the builder wrote anything meaningful to it. A skipped or silently-failed producer still yields some output, indistinguishable at the filesystem level from a real empty result. The convention is that a downstream consumer guards on the producer’s exit code before trusting the artifact. It’s opt-in per consumer rather than enforced automatically, on purpose: some snapshots are meant to consume a failed check’s output deliberately. A forced always-on guard would break those.⁴