Two GPUs, Two FreeBSD Boxes, One Erlang Cluster

Two 2013 Mac Pros sit on a shelf. Both run FreeBSD 15.0. Both have decade-old NVIDIA Kepler GPUs — a GT 750M with 2 GB of VRAM on mac-248, a GT 650M with 1 GB on mac-247. They are connected by a $5 ethernet cable to the same LAN switch. Neither GPU has run graphics work in years. Neither is what a modern compute team would specify. Both are compute hardware, and this post is about the demo that proves it.

The demo runs from mac-248’s iex prompt. Two lines of Elixir connect the nodes and dispatch a NUTS sampling job to both GPUs in parallel — one local, one across the wire. The output, verbatim:

=== DUAL-GPU MCMC DEMO ===
gpu1: NVIDIA GeForce GT 750M @ mac-248
gpu2: NVIDIA GeForce GT 650M @ mac-247
GT 750M: 934ms, 200 samples
GT 650M: 1278ms, 200 samples
Combined 400 samples: mean=-0.081
=== TWO GPUs x TWO FreeBSD x ONE ERLANG CLUSTER ===

Four hundred NUTS samples of Normal(0, 1). Two independent chains on two GPUs on two machines. Combined posterior mean of −0.081, against the expected 0.0. Wall time 1.3 seconds. No Docker, no gRPC, no service mesh, no Kubernetes, no protobuf, no message queue, no orchestrator. What follows is the architecture that made those two lines the whole transport layer, and the compute story sitting underneath them.

The transport was already there

Erlang distribution is a 1986 protocol. It ships with the BEAM. It handles TCP connection setup, wire-format serialisation of any term the language can express, authentication via a shared cookie, and result return by ordinary function-call semantics. It has been production infrastructure for four decades in telecom switches, and it works because it was designed for the exact shape of problem a compute cluster has: many machines, one logical program, transient failures that should not take the whole thing down.

The security model is the cookie. Same cookie means same cluster; different cookie means invisible. On a LAN switch behind a home firewall, this is enough. The command to bring a node into the cluster looks like this:

elixir --name gpu1@192.168.0.248 --cookie zed_gpu_demo -S mix
elixir --name gpu2@192.168.0.247 --cookie zed_gpu_demo -S mix

And the entire cluster-formation call is two lines of Elixir:

Node.connect(:"gpu2@192.168.0.247")
:rpc.call(:"gpu2@192.168.0.247", Exmc.NUTS.Sampler, :sample, [ir, %{}, opts])

That is not a simplification. That is the transport layer for the entire multi-GPU dispatch story. The BEAM handles the TCP socket, the term serialisation, the module resolution on the remote node, the exception marshalling on the return path, and the timeout semantics. A gRPC service for the same problem would be several hundred lines of Protobuf schema, generated stubs, a server implementation, a client implementation, an authentication scheme, and an error-mapping convention. Erlang distribution replaces every one of those with a function call that already knows how to cross a machine boundary.

The compute was already there

Each GPU runs a fused leapfrog chain shader — one SPIR-V compute shader that performs K=32 consecutive NUTS leapfrog steps in a single dispatch, with the closed-form Normal gradient baked into the shader body. No autodiff at runtime. No graph compilation. No JIT warmup. The shader was written once in GLSL, compiled to SPIR-V by glslangValidator on FreeBSD, and vendored as a .spv binary in the nx_vulkan repo. The same binary loads on both Keplers at runtime through the Vulkan driver, and the Vulkan driver hands it to the GPU without asking whether the driver version is identical, the CPU vendor matches, or the operating system agrees.

This is what SPIR-V was designed to be, and it is one of the underappreciated design choices of the Vulkan spec: the compute unit is an intermediate representation, not a source file. Two different GPU generations on two different driver builds accept the same .spv and produce the same numeric result, because the driver is a compiler and the shader is a program the compiler already knows how to lower.

The performance shape follows from what the shader saves you. The unfused path dispatches every leapfrog op through a separate Vulkan compute submission, and each submission pays a fence wait in the driver. On the FreeBSD/Kepler path a fence wait runs about 406 µs; over ~384 fence waits per iteration, the unfused path lands at 283 ms per iteration. Fusing 32 leapfrog steps into a single dispatch reduces the fence-wait count by roughly two orders of magnitude and drops the wall clock to 3.3 ms per iteration — an 86.7× speedup with no change in numerical output. The GPU does the same work; the driver is asked to synchronise about a hundredth as often. That is the entire optimisation.

The insight was where to build the IR

Nx tensors backed by Vulkan GPU memory cannot be serialised across Erlang distribution. The BEAM does not know how to copy a buffer that lives on a foreign device. Sending a Vulkano.Buffer reference across the wire produces a pointer that means one thing on the sender and nothing at all on the receiver.

The naive approach — build the IR on mac-248, ship it to mac-247, execute there — is therefore wrong at the layer where Nx meets the wire, because the IR contains tensor references and the tensor references are local pointers. The right approach is to build the IR on the remote node. The Erlang closure gets serialised — just code, no GPU state — and evaluated on mac-247’s BEAM, where the IR construction and the sampling both happen against local Vulkan buffers, and the only thing sent back is the numeric result:

:rpc.call(:"gpu2@192.168.0.247", :erlang, :apply, [fn ->
  ir = Exmc.Builder.new_ir()
       |> Exmc.Builder.rv("x", Exmc.Dist.Normal,
            %{mu: Nx.tensor(0.0), sigma: Nx.tensor(1.0)})
  {trace, _} = Exmc.NUTS.Sampler.sample(ir, %{}, opts)
  [{_, samples}] = Enum.to_list(trace)
  Nx.to_flat_list(samples)  # return plain floats, not GPU tensors
end, []])

The closure crosses the wire. The GPU buffers do not. The result is a list of floats, which the BEAM knows how to serialise because floats are terms and terms are what the protocol was built for. The pattern is not a workaround. It is the correct answer to the question of where to run compute in a distributed system: as close to the data as you can, with only the summary crossing the wire.

What the numbers said

The per-GPU wall time was 934 ms for the GT 750M on mac-248 and 1,278 ms for the GT 650M on mac-247 for 200 warmup plus 200 sampling iterations. That is 2.3 ms and 3.2 ms per iteration respectively. The GT 650M runs about 37% slower, which is consistent with its lower clock speed on the same number of CUDA cores and a marginally older PCIe generation. On a modern card the same workload runs in about a third of the time; the point of the numbers is not the absolute performance but that 1.3 seconds of wall clock is inside the latency budget for almost every real dispatch pattern — a trading update, a sensor-fusion callback, a batch-inference request — on hardware that would otherwise be in landfill.

The per-fence Vulkan driver latency on the FreeBSD/Kepler path breaks down as follows:

For contrast, the same measurement path on an RTX 3060 Ti under Linux, on a substantially newer driver family, comes back at 1,287 µs per dispatch. The FreeBSD NVIDIA path is 3.1× faster on the fence wait — not on the compute, which the RTX obviously wins by a wide margin at any workload big enough to matter, but on the dispatch overhead. The attribution belongs to the driver, not the shader; the shader is byte-identical. A leaner userland with less to multiplex against appears to complete its fence signalling faster than a Linux driver that is also servicing a Wayland compositor and a browser and a video call. Whether that generalises is a longer conversation than this post; what generalises here is that the measurement is possible at all, because the fence-wait was instrumented with DTrace probes on the actual driver code path rather than inferred from wall-clock deltas.

The fused chain shader turns those 422 µs per dispatch into 3.3 ms per iteration by amortising ~384 sequential leapfrog dispatches into ~4. That is where the 86.7× speedup lives. The compute is unchanged. What changed is how often the CPU asks the GPU to synchronise.

What composes

Four properties fall out of the demo that the demo was not built to prove, and they are worth naming because none of them required any custom infrastructure. The transport is Erlang’s. The compute is Vulkan’s. The rest is the fifty or so lines of Elixir that stitch them together.

The first property is that BEAM distribution is a GPU dispatch fabric. :rpc.call against a named remote node, with a cookie for authentication and a closure for the payload, is the entire cross-machine dispatch story for a compute cluster whose workload fits inside a function call. There is no service mesh in this architecture because there is no service; there is a function.

The second is that FreeBSD is a real compute platform, in the sense that the demo runs. It runs on hardware that was mid-range in 2013 and would be considered surplus today. It runs against a driver that receives fewer commits per year than the Linux equivalent, and it produces a lower per-fence latency than that Linux equivalent. It is not the platform anyone would specify for a new project. It is the platform this project ran on, because that is where the hardware was, and it worked.

The third is that surplus hardware is compute hardware. Two Mac Pros from 2013, destined for recycling, run a distributed Bayesian inference cluster. Each GPU has 384 CUDA-equivalent cores. Each machine has enough VRAM to hold the state a NUTS sampler needs for a small model. The economic story of GPU compute has been dominated for a decade by the assumption that compute means the latest silicon; the honest engineering answer is that compute means whatever silicon the workload fits on, and a lot of workloads fit on Kepler.

The fourth is that the architecture composes. The same chain shader binary loads on both GPUs. The same eXMC sampler runs on both. The same Erlang distribution protocol carries the call. Adding a third Mac Pro with a Kepler card to the cluster is not a code change; it is one elixir --name gpu3@… invocation and one line added to the dispatch map. Adding a fourth is another. The transport is a directed graph, and the compute nodes are vertices; the code does not know how many vertices there are, and does not need to.

The stack, once

For the record, the layers involved:

Elixir (eXMC NUTS sampler)
  |  :rpc.call (Erlang distribution, TCP, cookies)
Nx.Vulkan.Backend (Elixir, Rustler NIF)
  |  extern "C" FFI
vulkano (Rust)
  |  Vulkan API
NVIDIA driver (FreeBSD nvidia-driver-470)
  |  PCIe
GPU (Kepler, 2013)

Seven layers. The same SPIR-V binary sits at the bottom of both stacks. The same Elixir sits at the top. Between them, the only difference across the two hosts is the IP address in the Node.connect call. Every other atom — module name, function name, tensor shape, seed value, sampler configuration — is bit-identical on both sides. That is the property that made the demo compose: the two machines are not running similar code, they are running the same code, in two places, coordinated by a protocol that predates most of the industry’s current tooling by three decades.

Coda

The demo was not built to prove that Erlang distribution is a GPU dispatch fabric. It was built to check whether two GPUs under one supervision tree would give the correct posterior. The answer is that they do; the posterior mean of −0.081 is well within Monte Carlo noise of the true zero, and the two chains mix into the same distribution. That the transport turned out to be adequate for the job, that the SPIR-V binary turned out to load unmodified on both drivers, and that the architecture turned out to accept a third node without a code change — these were consequences of decisions made for other reasons.

The reasons were: use the BEAM because the sampler is written in Elixir; use Vulkan because CUDA does not exist on FreeBSD; use FreeBSD because the hardware was already there. None of these are design principles. All of them are constraints. What composes is what falls out when the constraints happen to align with a workable architecture, and the workable architecture, in this case, is one that treats the network as a function call and the GPU as an accelerator that speaks a portable intermediate representation.

Two GPUs. Two FreeBSD boxes. One Erlang cluster. Four hundred NUTS samples in 1.3 seconds. mean = −0.081. Built with parts that cost less than lunch, on an OS that nobody uses, with a GPU API that nobody chose, connected by a protocol from 1986. It works.