From 0.02× to 34×: How One Op Got Native

A concatenate kernel, three benches, and a lesson about what you’re actually measuring.

The bench that lied

The first version of the operator-coverage bench measured 24 backend ops the way you’d measure them on a slide: a baseline BinaryBackend call, and a VulkanoBackend call with everything downloaded back to the host at the end. The ratio column told a consistent story. Most ops landed somewhere between 0.5× and 2×. Concatenate, alone among them, sat at 0.02×. At 1,048,576 elements it was 0.002× — the GPU path was five hundred times slower than the CPU path it was supposed to be accelerating.

The number was approximately correct. The bench was approximately wrong.

The host-fallback path for concatenate, when invoked by an op-only bench, looks like this: upload N input tensors to the GPU (because the bench fixture puts them there), download N input tensors back to the host, run BinaryBackend’s concatenate, upload the result, then — because the bench wants to read the answer back — download it again. Five buffer copies, one of which is the actual work. The “GPU was 500× slower” was 500× slower at copying memory it didn’t need to copy.

This is the bench-instrumentation cliché in its purest form: the column we were ranking on did not measure the thing we cared about. The fix was not a faster kernel. It was a better bench.

What the consumer actually pays

The replacement bench has five columns:

The number that matters is D/B: is the GPU path faster than the host path, when both are doing realistic IO? The original bench measured something closer to C/A, which is interesting to a kernel author and misleading to anyone else.

For concatenate, the D/E ratio told the secondary story. Of the three buffer copies the original bench was paying, one was gratuitous. The result lived on the GPU only because the host-fallback callback had uploaded it there at the end of the function. Then the bench downloaded it. The next op — whatever it would have been, in a real program — would have to download it again anyway, because BinaryBackend cannot consume a Vulkan buffer.

The fix to that part was not architectural. It was a return statement.

Tier 1: stop uploading back

The host-fallback callbacks in VulkanoBackend all shared a pattern: download inputs, call Nx.<op> on them, upload the result back to the GPU, put it in out.data. The upload was a courtesy. No caller had asked for it.

We added one helper:

defp host_result(out, %T{data: %BinaryBackend{}} = result) do
  %{out | data: result.data}
end

And changed every host-fallback callback’s last line from upload(result) to host_result(out, result). Five-line diff per op, eight ops touched. The result tensor now lives on BinaryBackend; the Nx defn tracer is happy because BinaryBackend is one of the backends it knows how to compose with; the next op runs on whichever backend it wants.

The benchmark immediately changed character. D/E for concatenate dropped to 0.79 at 16k elements and stayed below 1 across the size range — the upload-back was real, measurable, and gone. D/B stayed above 1 (the GPU path was still slower than BinaryBackend for an op BinaryBackend can do in two memcpys), but the gap had narrowed substantially.

This was not a speedup. It was the removal of a tax we were collecting from ourselves.

Tier 2: the op itself goes native

The Tier 1 path still ran the concatenate on the CPU. For an outer-axis concat of buffers that were already GPU-resident, this was the wrong default. vkCmdCopyBuffer exists. It does exactly what an outer-axis concatenate is: copy N source ranges into one destination buffer at sequential offsets. No shader. No dispatch. Memory transfer at PCIe speeds, on the GPU’s side of the bus.

The NIF is uncomplicated:

#[rustler::nif(schedule = "DirtyIo")]
fn concat_buffers<'a>(env: Env<'a>, inputs: Vec<ResourceArc<VulkanoTensor>>)
    -> NifResult<Term<'a>> {
    let total_bytes: u64 = inputs.iter().map(|t| t.n_bytes).sum();
    let dst = alloc_buffer(allocator, total_bytes as usize,
        BufferUsage::STORAGE_BUFFER
        | BufferUsage::TRANSFER_SRC
        | BufferUsage::TRANSFER_DST)?;
    let mut cmd = AutoCommandBufferBuilder::primary(...)?;
    let mut offset: u64 = 0;
    for input in &inputs {
        let dst_slice = dst.clone().slice(offset..offset + input.n_bytes);
        cmd.copy_buffer(CopyBufferInfo::buffers(input.buf.clone(), dst_slice))?;
        offset += input.n_bytes;
    }
    submit_and_wait(cmd)?;
    let tensor = VulkanoTensor { buf: dst, n_bytes: total_bytes };
    Ok((atoms::ok(), ResourceArc::new(tensor)).encode(env))
}

The Elixir side gates on whether all inputs are already on the Vulkan backend:

def concatenate(out, tensors, axis) do
  cond do
    axis == 0 and all_vulkano?(tensors) ->
      concat_vulkano(out, tensors)
    true ->
      bins = Enum.map(tensors, &Nx.backend_transfer(&1, BinaryBackend))
      host_result(out, Nx.concatenate(bins, axis: axis))
  end
end

Mixed-backend concats — and concats along an inner axis — keep falling back to BinaryBackend, with the Tier 1 contract still applying. Pure-GPU outer-axis concats are now a single command buffer. The result stays on the GPU because, and this is the second-order win, no host code ever sees the bytes.

The op-only column moved by an order of magnitude:

The 65× number at 1M is, in fairness, mostly an indictment of the prior path: the Tier 1 fast path was BinaryBackend.concatenate routed through a downloaded Vulkano buffer, which was not, charitably, optimised for the case where the buffer was already where the answer needed to be. The Tier 2 path is closer to what concatenate should always have been.

D/B is what actually matters. On super-io (RTX 3060 Ti, Linux), the GPU path now beats the host path at every size:

Sub-1.0 D/B at every size. The GPU concatenate plus the host read-back the consumer pays is faster than the host concatenate plus the same read-back. This is the regime the original bench thought we were in. We were not in it. Now we are.

The cross-host inversion

The other host is a 2012 Mac Pro running FreeBSD 15 with a GeForce GT 650M. It is not a serious GPU. Its op-only timings are 1–4× slower than the 3060 Ti’s. Its D/B ratios are not:

At {16k}, the fourteen-year-old mobile GPU is closer to parity with its own host than the current-generation discrete card is. Same story as the three-shapes bench from a week ago: FreeBSD’s NVIDIA driver path has measurably lower dispatch overhead than Linux’s when Linux is also running a desktop, and at small problem sizes that overhead is a larger fraction of the total. The compute-bound side of the curve is where super-io wins by an order of magnitude. The dispatch-bound side is where the older, slower hardware on the leaner OS edges ahead. Of itself, mostly — the cross-host C column still has the 3060 Ti winning everywhere — but D/B is a per-host comparison, and the FreeBSD path is winning that comparison more decisively on its own machine.

It is the same crossover the matmul bench surfaced in the previous post, surfacing again in a different op. Hardware ranking depends on the size. OS ranking, less expectedly, also depends on the size.

What the benches taught us

The original bench was correct about concatenate. It was 0.02× BinaryBackend, at the specific thing it measured: an op that uploaded its inputs, downloaded them, ran on CPU, uploaded the result, and then watched the bench harness download it again. That number is real. It is also irrelevant to anyone who isn’t writing an op-only benchmark in isolation.

The interesting question turned out to be the one we hadn’t asked: what does the consumer of this op actually pay? The answer changed everything. Tier 1 — stop the gratuitous upload-back — was a five-line patch per op. Tier 2 — the op goes native — was eighty lines of Rust and an all_vulkano? predicate. Together they moved concatenate from “the worst op in the suite” to “the GPU path is now actually faster,” on both a current-generation discrete card and a 2012 mobile GPU.

The kernel author’s bench was right about the kernel. The system’s bench was right about the system. They were measuring different things, and the gap between them — the cost the kernel imposed on the system that the kernel never saw — was where all the speedup was hiding.

Raw data (consumer-aware schema, 2026-05-24): bench_results/super-io_consumer-aware_2026-05-24.csv, bench_results/mac247-gt650m_consumer-aware_2026-05-24.csv. Script: examples/vulkano_consumer_bench.exs. Tier 2 NIF: commit 672d32f. Run it yourself.