Three Shapes of Vulkano Speedup

What 309 benchmarks across two machines tell you about a Vulkan tensor backend — and why “GPU is faster” is usually the wrong question.

The setup

VulkanoBackend is the pure-Rust Nx backend that nx_vulkan added in the Mission II work. It implements every Nx callback it can on the GPU via Vulkan compute shaders, with a host-fallback path through Nx.BinaryBackend for anything not yet ported. Once it landed, the obvious question was: how much faster, exactly, and where?

The bench script examples/vulkano_ops_bench.exs is the answer. Every callback the backend implements, swept across ~10 shapes per op, compared against BinaryBackend at the same shape. Written to bench_results/<hostname>_<date>.csv so cross-machine diffs are a csvkit join away.

Run on two machines:

On paper, super-io is about 12× the compute of mac-248. The bench result is more interesting than that.

Three curves

Plot vulkano-speedup-vs-BinaryBackend against input size for any op and you get one of three shapes.

Shape A — compute-bound. Matmul. Big elementwise binary and unary ops. Speedup scales nearly linearly with the work. From 1.4× at small sizes to ~10,000× at dot {256×256} on super-io. The GPU was made for this.

Shape B — dispatch-bound. Reductions (sum, reduce_max, reduce_min). Speedup is a hump, not a curve. At small sizes the GPU loses — dispatch overhead dwarfs the work. The hump peaks around 256k elements (7.14× for reduce_max on super-io), then falls back toward parity at 1M as BinaryBackend cache effects play out and the GPU starts saturating memory bandwidth instead of arithmetic. sum on mac-248 never wins significantly: 1.07× at 1M is the best it can do.

Shape C — host-fallback. pad, put_slice, indexed_put, indexed_add, broadcast, concatenate, gather, take. Speedup is always under one. These are host-fallback callbacks: the “compute” runs on BinaryBackend either way. Vulkano just pays for the round trip — backend_transfer → op → from_binary. The transfer is the entire cost, and it grows with the tensor.

The hardware twist

What makes the bench interesting isn’t the headline matmul number. It’s that mac-248 — a thirteen-year-old GT 750M on FreeBSD — beats super-io on dispatch-bound ops up to roughly 16k elements:

The 3060 Ti has ~12× the raw compute. The work fits in its L1 cache ten times over. And yet for a 1024-element vector add it loses by a factor of 1.5.

The cause is the per-dispatch tax: vkCreateBuffer + descriptor set binding + command buffer record + queue submit + fence wait. On Linux with the modern driver and a Wayland desktop alongside, that sequence eats hundreds of microseconds. On FreeBSD with no display server competing for the queue and a simpler driver path, it’s faster — by enough to outweigh the GPU compute disadvantage when the compute is tiny. The same effect was noted on mac-247 last year in the matmul characterisation. The bench reproduces it across the full op set.

What the dispatch tax buys you

The cost of any GPU op on the FreeBSD path is roughly constant — a few tens of microseconds per dispatch, regardless of how much work follows. Once the work crosses that threshold, you’re back on the compute curve where the 3060 Ti dominates.

The crossover happens between 65k and 256k for binary ops. Below it, you’re dispatch-bound and mac-248 wins on driver overhead. Above it, you’re compute-bound and super-io wins on raw FLOPs.

What this means for routing

The sampler-host class is the loud finding. Every op there is ~5-50× slower on vulkano than on BinaryBackend. Including indexed_put and put_slice — the two ops the NUTS leapfrog loop calls dozens of times per step.

In the production eXMC trial those tensors live on VulkanoBackend because we made it the global default at boot. Every leapfrog step pays the host-fallback round trip. The total impact is muted — those ops are tiny relative to the synth-shader dispatch that dominates wall time — but it’s worth knowing. A future routing policy could keep small-shape state tensors on BinaryBackend even when the default backend is vulkano, and reserve the GPU storage class for the buffers that actually feed shaders.

concatenate is the bench’s most embarrassing row, by the way. 0.02× at 65k. BinaryBackend’s concatenate is essentially <<a::binary, b::binary>> — free. Vulkano’s host-fallback path downloads two GPU tensors, concatenates them on the host, uploads the result. The GPU never touches the data. The only thing vulkano contributes is the round trip.

The takeaway

Three curves, two machines, 309 measurements. The numbers don’t say “use the GPU.” They say use the right backend at the right size, and pay attention to which side of the dispatch tax your workload actually lives on.

The conventional reading of “use the GPU for everything” is the sales pitch. The bench is the truth: there is a regime — small, host-routed, latency-sensitive — where a 2013 Mac Pro on FreeBSD will out-dispatch a 2021 RTX 3060 Ti on Linux. Filing that under things I now have data for.

Raw data: bench_results/super-io_2026-05-22.csv, bench_results/free-macpro-nvidia_2026-05-22.csv. Script: examples/vulkano_ops_bench.exs. Run it yourself.