The first time the trader stayed up for an hour, nobody believed it. The second time, somebody asked whether the supervisor was masking a crash. The third time, when the BEAM had run continuously for six hours across all five paper accounts and the log was a clean march of warmup completions, the right question came into view: what changed?
The honest answer was: a backend. The interesting answer was: a property nobody had asked for.
The crash, briefly
The story, in the version other posts have already told, was that
a C++ Vulkan compute backend — spirit, vendored under
nx_vulkan/native/ — crashed within three minutes
of every restart. The crash was reproducible. The crash was
visible if you knew where to look. The crash was a pointer
ownership bug at the FFI boundary, a Rust resource handle pointing
into a C++ allocator’s freed block, surfacing as
:badarg from Nx.Vulkan.Native.byte_size/1
the instant somebody tried to inspect a tensor.
The replacement, vulkano, was a pure-Rust crate that
modeled the Vulkan API in safe Rust with Arc<Buffer>
as the ownership primitive. The migration took a weekend to spike,
two weeks to land as a sibling backend, and on the evening of May
20 it became the default. The crash didn’t come back.
That is the engineering story, and it has been told. What follows is the part that took two more weeks to surface and is now, in early June, the part that matters.
The spike that wasn’t supposed to work this well
The migration began with a hypothesis and a request for refutation. The hypothesis: a Rust crate using vulkano can load the SPV file that spirit already cached on disk, dispatch it, and produce byte-identical output. The request: please show me where this breaks before I commit to porting twenty-four ops.
The spike was a standalone cargo binary under
nx_vulkan/spike/vulkano_synth/. It read the seven
SPV files that R3 had left cached under
~/.exmc/gpu_node/spv/synth_*.spv, one of them
1.7 MB and produced by the chain shader codegen against the
regime model. It dispatched each with the same f64 input fixture
that the regime model used for its log-probability tests. The
output was compared to the C++ backend’s output via
diff on the binary tensor dump.
On FreeBSD 15.0, on a 2013 Mac Pro with a GeForce GT 650M Mac
Edition, the spike built in 3:18. On super-io
— the Linux dev box with an RTX 3060 Ti — it built in
0:30. The first run on FreeBSD returned zero from
diff. Every q_chain, every p_chain, every gradient
component, every log-probability bit-for-bit identical to the C++
backend on the same input.
This was not what the spike was for. A spike is the right answer
to a hypothesis, and the hypothesis was that the rewrite was
feasible. The answer that came back was: it is feasible, and also
the output is already correct, and also the build system worked
on the first try, on the OS that the existing backend had never
been built on. The honest reaction was suspicion. The author spent
an hour writing additional fixtures, trying to find the place
where vulkano’s default queue handling or its
Arc-based memory model would diverge from the C++
backend’s explicit synchronization. There wasn’t one.
The reason, in retrospect, is the part that mattered. The SPV file is the source of truth. Both backends are SPIR-V dispatchers; neither does any numerical work that isn’t in the shader. What the C++ backend did, that vulkano now does differently, is manage the memory and the descriptor pools and the pipeline cache and the queue submission. None of which the f64 logp result cared about. The bytes that came back were the bytes the shader wrote.
From spike to NIF to backend
The spike became a Rustler crate, nx_vulkan_vulkano, a
sibling of the existing nx_vulkan_native crate that
wrapped spirit. The Rustler boundary stayed familiar — the
same Binary in, Term out idiom that the
old crate used — but the resource type was now
Arc<VulkanoBuffer> instead of a raw pointer
masquerading as one. The byte_size/1 NIF, in this
version, asks the buffer for its size by way of the Arc, and the
Arc tells it. The Arc cannot tell a lie of the kind that
byte_size received in May, because the only way for
the buffer to be deallocated is for every Arc to drop, which is
exactly what the Rustler resource handle prevents.
This is the bug class that disappears structurally. The C++ backend could be made to not crash — it had been, twice — but the absence of the crash was a property of how the allocator was being driven, which depended on which ops were running in which order under which dispatch pattern. The Arc version is not crashing because nothing in the type system permits it to. There is no clever scheduling, no careful ordering, no ref-count handshake between languages. There is a Rust object that owns the buffer and a Rustler resource that holds the Rust object, and the resource’s lifetime is managed by the BEAM’s GC. The buffer dies when the BEAM forgets about it. The buffer cannot die otherwise.
Twenty-four ops landed across five stages: element-wise binary, unary, reductions, shape and movement ops, and matmul. Twenty-four ops is the minimum set the regime model needed to round-trip a log-probability and a gradient. It was not exhaustive coverage of Nx’s op set, and it was not meant to be. The scope was defined by the model.
The descriptor pool that ate the GPU
The first cross-host race told a different story. Super-io was faster than mac-247 at the small matmul, as expected, but it was also slower — consistently slower — by exactly one shape: a small workload where dispatch overhead dominated compute. mac-247, the 2013 Mac Pro with the GT 650M, returned a 0.75 ms mean per dispatch on the small matmul against super-io’s 1.14 ms. FreeBSD’s NVIDIA path, on a card the company had stopped shipping in 2014, was outperforming Linux on a 2021 desktop card.
The honest answer to why is that super-io is a desktop. It runs a window manager, a browser, a Wayland compositor, sometimes a video call. The Linux NVIDIA driver does its level best to multiplex compute and graphics work onto the same card and the multiplexing costs something. mac-247 runs nothing on its GPU but the spike. Compute-bound workloads put super-io ahead immediately and kept it there — at 1024×1024 matmul, twelve times faster; at 2048×2048, twenty-two times faster. But at the smallest shape, where the wall clock was four-fifths Vulkan dispatch overhead and one-fifth actual GPU work, the unloaded card won.
The race results were not the bug. The bug was that mac-247 crashed at around iteration 5000 of the multi-stage race bench, the same way it always had: a vulkano panic claiming the descriptor pool was exhausted. The pool advertised 32 sets per pool. The bench had dispatched many fewer than 32 distinct descriptor sets at a time. The math did not add up.
Reading the vulkano source code took an evening. The relevant
constructor is StandardDescriptorSetAllocator::new,
which receives a StandardDescriptorSetAllocatorCreateInfo
with a default set_count: 32. What the documentation
does not announce, and what the source code makes plain, is that
the allocator keys its pools by PipelineLayout
identity. A new PipelineLayout — not a new
descriptor set, a new layout — gets its own pool. A
pool holds 32 sets and then refuses to allocate more.
The compute pipeline was being rebuilt per dispatch. Per dispatch meant per layout. Per layout meant per pool. Per pool meant the 32-slot recycling never kicked in, because no two calls ever shared a pool to recycle from. mac-247 hit the wall first because its driver path had less headroom; super-io would have hit it eventually, just farther along.
The fix was a pipeline cache keyed by
(spv_path, op_code). Same SPV hash on a cache hit
means same layout identity means same pool means recycling works.
The cache itself was a HashMap behind a
Mutex, and the lookup dropped the steady-state
per-dispatch cost from about three milliseconds to whatever a
HashMap lookup costs on a modern CPU, which is to say,
unmeasurable.
That fix landed on May 23. mac-247 ran the race to completion in
94.5 seconds; super-io ran it in 57.8.
Both finished, both produced clean results, both held their
descriptor pools at a polite single-digit count throughout.
The autograd that wasn’t there
The migration plan had budgeted a week for backward-pass coverage. The reasoning was that the C++ backend had explicit backward callbacks for the differentiable ops, and the new backend would need them too. The week went unused.
What broke the assumption was reading Nx.Defn.grad.
The function operates on the IR graph at compile time. It
transforms the forward graph into a graph that also computes
gradients, expressed in the same forward ops the model already
used. The backend never sees a “backward op.” The
backend sees the same forward ops it was already implementing,
in different positions in the graph, with different shapes. If
the backend covers the forward op set, it covers the gradient.
This was tested by running a complete Axon training step on the
new backend: a Dense layer with sigmoid activation, another Dense,
a mean-squared-error loss, and a gradient computation through the
whole stack. The backend executed it. The gradient of the first
Dense’s kernel matched the BinaryBackend reference to a
relative difference of 8.6e-8, which is roughly the
noise floor of f32 arithmetic and exactly what one expects when
the same computation runs through two different float-handling
paths.
The relevant observation is not that autograd is easy. The relevant observation is that autograd, when the framework decides to express it as a graph transformation, becomes a property of the framework rather than a property of the backend. The backend that worked best was the one that knew the least.
What the trial saw
The first multi-day trial ran across five paper-trading accounts on Alpaca, fifty instruments in regime-model rotation, with checkpoints to disk every update and a hot-reload worker watching the instruments file. It ran without crashing. This is, in 2026, still the most boring sentence one can write about a production system, and also the one that took the longest to write.
The performance numbers tell the engineering story. The end-to-end batched-dispatch path, after Task #154 landed, hit 0.55 ms per instance at N=64 on super-io and 2.96 ms on mac-248. The trial-shape round — 268 instruments through the full coordinator path — completed in 147 ms on super-io, 794 ms on mac-248, 818 ms on mac-247. All under a second. A 2013 Mac Pro on FreeBSD, dispatching Vulkan compute through a Rust NIF, was inside the latency budget for a trading update.
The interesting numbers were the ones nobody had asked for.
The cross-host determinism that fell out
The post-validation pass came out of an audit, not a feature. Somebody asked: if we run the posteriordb regression suite — 33 representative Bayesian models with Stan reference draws — on both Macs through the BinaryBackend fallback, do they give the same answer? The expected answer was “close enough,” because that is what one says about floating-point comparisons across different hardware.
The actual answer was that mac-247 and mac-248 produced bit-identical step sizes, bit-identical divergence counts, and bit-identical posterior means across all 33 models. 28 PASS / 1 FAIL / 4 CRASH on both hosts, with the four crashes failing on the same four models in the same way. The FAIL was the same model, with the same divergence count of 8, the same step size of 0.0039, the same mean error of 0.379, and the same standard deviation ratio of 0.998. The hosts ran different x86 generations on different CPU silicon. They produced identical floating-point streams.
This is not something the BinaryBackend documentation promises.
The Erlang VM’s :rand module produces a
deterministic sequence given a seed, and the BinaryBackend
implements IEEE 754 arithmetic, and IEEE 754 is bit-deterministic
when the rounding mode and the operation order are fixed. The
operation order is fixed by the IR, which is the same on both
hosts. The rounding mode is the default. None of this was
designed to be cross-host deterministic. It happens to be.
The reason is structural. The sampler thread its random state
through :rand, which is deterministic. The tree
builder calls into Nx.Defn.Evaluator, which runs the
forward IR through BinaryBackend ops in the order the IR
specifies. The leapfrog is a closed-form arithmetic sequence
— no parallelism, no rescheduling, no fused multiply-add
that might or might not be available depending on the CPU. The
output of the sampler is a function only of the seed, the IR, and
the floating-point arithmetic, and all three are equal across the
two hosts.
The post-fix run on June 2, after the BinaryBackend
ArithmeticError rescue landed in commit
d8372a3df, took mac-248 to 31 PASS / 2 FAIL / 0
CRASH. The four prior crashes had been a real numerical bug
— a residuals/sigma divide when sigma had
underflowed to exact zero, which on EXLA quietly produces IEEE
infinity and on BinaryBackend raises through Complex.divide.
The fix was a try/rescue at the two leapfrog entry
points, converting the arithmetic exception into a divergent step.
Three of the four prior crashes now pass. The fourth completes
with high mean error, which is what a 200-sample warmup will give
you on a stiff model regardless of whether the platform crashes.
The remaining FAIL is the same kilpisjarvi noise as before.
The thing this run proved, beyond the engineering fix, is that
the bit-determinism property survived. The new code path — a
try/rescue over the JIT-compiled call — produces
the same trajectories on both hosts. The fix did not introduce a
host-dependent control flow. It introduced a host-independent one,
in a place where the platform’s exception semantics
suddenly mattered.
What this means, what it doesn’t
The migration is now behind us. The trial is running. The posteriordb regression is green at 31/33 across two hosts, with zero crashes. The bit-determinism is real, if accidental. The descriptor pool no longer fills up. The autograd works because nothing taught it not to.
What this does not mean: that we are done. Scholar — the elixir-side scikit-learn analogue — has not been tested against the vulkano backend. The R4 cutover that the original plan called for never ran end-to-end; what ran was an incremental migration that turned R4 into a series of smaller checkpoints, none of which crashed loudly enough to be called R4. The matmul path is f32-only. There is no persistent buffer pool yet, which means every compute submission allocates and frees, which means the descriptor pool fix saves us from the allocator pressure that was visible but not from the pressure that is invisible. The roadmap from here is straightforward: cover Scholar, add an f64 matmul, build the buffer pool. None of these are interesting problems. All of them are necessary.
The accident, though, is worth holding onto. The migration was sold internally as a fix for the crash, and as a fix for the crash it succeeded. Nobody promised determinism. Nobody promised that two different x86 boxes running two different NVIDIA drivers under two different versions of FreeBSD would produce bit-identical posterior samples on a regression suite written for Stan reference draws. That property fell out of the code as a side effect of decisions made for other reasons: use the BEAM’s random number generator because Nx’s is slow on the BinaryBackend; use the BinaryBackend because EXLA does not build on FreeBSD; use a sampler that threads random state explicitly because it’s the only way to debug the thing.
A property that falls out is more durable than a property that was designed in. Designed properties get broken by the next refactor. Properties that fall out get broken too, but they get broken by something noticeable, because they were never on the checklist to begin with.
Coda
The right way to think about the migration, two weeks on, is that the backend that worked best was the one that knew the least. The backend doesn’t know about autograd; the framework transforms the graph above it. The backend doesn’t know about random numbers; the BEAM threads them deterministically. The backend doesn’t know about cross-host determinism; the deterministic property is a property of the arithmetic it performs, not a property the backend asserts. The backend just runs a graph and returns the bytes.
Twenty-four ops, three weeks of work, four shipped blog posts before this one, and the punchline is that we wrote less code, not more, and got more properties, not fewer. The 2013 Mac Pro is running probabilistic inference at FreeBSD on a card that was old when the model was written. The trades clear. The supervisor tree, asked to come up after a reboot, comes up. The BEAM, on day sixteen of continuous operation, is running.
The next thing that breaks will be something we did not plan for. That is the only stable property of a working system: that the next bug is in the part we stopped looking at.
Code: commits 0fbc9e8f8 (the initial long-form),
41bf34419 (four-backend matmul comparison),
b4d08f612 (cross-host re-sync), 0273c7f81
(Scholar smoke), d8372a3df (the
ArithmeticError rescue), ad3e634f5 (the
TODO that survived the reboot). Logs and per-model timings:
research/posteriordb/SYNTH_COVERAGE_FINDINGS.md.
Benchmarks: benchmark/results/path_scaling_race_*.csv.
None of these are advertised guarantees. They are what the run
produced, on the hosts named in the post, on the dates
recorded.