There is a particular kind of humiliation available to anyone who writes tests for a living, and it is not the humiliation of a failing suite. A failing suite is a colleague. It is disagreeable and it is often wrong about the reason, but it is on your side. The humiliation I have in mind is subtler and takes longer to arrive: it is the discovery that a test which has been passing cheerfully for some weeks was never in a position to do anything else.
Here is the specimen. A GPU shader produces a buffer of double-precision floats — a Markov chain trajectory, as it happens, though the content is beside the point. One wishes to assert that none of them is a NaN, that being the characteristic way a numerical kernel announces it has gone wrong without troubling anyone. The idiomatic Elixir is short enough to be beyond suspicion:
for <<v::float-64-little <- bin>>, do: assert v == vA NaN is the only value not equal to itself. The assertion is therefore exact, total, and free. It is also incapable of failing. Not unlikely to fail: incapable. Erlang’s binary comprehension, presented with eight bytes that decode to a NaN, does not raise and does not yield the NaN. It declines the segment and moves on. The offending value is removed from the list before the assertion is permitted to see it, whereupon every survivor passes, as survivors tend to.
<<0,0,0,0,0,0,240,127>> (+inf) -> []
<<1,0,0,0,0,0,240,127>> (NaN) -> []
<<0,0,0,0,0,0,248,63>> (1.5) -> [1.5]I want to be fair to the language, which is behaving to specification: a
comprehension is a filter, and a segment that cannot be decoded as an Erlang
float is a segment that does not match. A direct pattern match on the same bytes
raises a MatchError perfectly properly. But the comprehension is what
one writes for a buffer, because a buffer is a sequence, and the failure mode is
therefore aimed with some precision at the person doing the right thing.
The remedy is mechanical and I record it chiefly so that nobody has to lose
the afternoon I lost: the decoder now counts. It asserts that it extracted
byte_size(bin) / 8 values, and a NaN, having deleted itself from the
list, is convicted by the shortfall.
The general disease
The interesting thing is not the Erlang trivia. It is that this was the
third instrument I had found in the same week that reported confidently
while measuring nothing, and I have written about the other two elsewhere. A
verification script called file to confirm an artifact’s
architecture; file was not installed; the line printed a shell error
between two lines that succeeded, and the architecture was never checked at all.
A batching test dispatched the single-instance code path twice and asserted the
two results were equal — a proposition true of every conceivable
implementation, correct or otherwise, and true most emphatically of a batching
feature that had not been written.
All three looked precisely like passing tests. That is the whole of the difficulty. A test suite is an instrument, and no instrument that has never been deflected can be said to have been calibrated. One would not accept a smoke alarm on the grounds that it has not gone off.
So: every property needs a null arm. Having asserted the thing, arrange for it to be false and confirm the assertion notices. If you cannot construct the input that breaks it, you do not yet know what your test detects, and the honest position is that it probably detects nothing.
The guard that had never fired
Consider next a class of code that is almost designed to accumulate this defect: the refusal path. This particular library’s native layer has seven of them — mismatched vector lengths, a zero step count, a push block of zero bytes or of more than the hardware permits, a dimension exceeding the shader’s workgroup allocation. Each returns an error tuple. Not one of them had ever been observed to return anything, because no test had ever asked.
A guard that has never been seen to fire is indistinguishable from a guard
that cannot fire. This is not a philosophical point, and I can produce the
corpse. The dimension limit — d > 256 — was not
enforced for several months. Past that width the shader wrote an undefined tail
and, worse, computed its log-density by summing only the first 256 elements. It
returned a number. The number was a posterior distribution over the wrong model.
Nothing raised, nothing was slow, no test failed, and the only reason no
inference was corrupted is that an unrelated constraint elsewhere in the system
happened to hold the dimension near thirteen. The bug was harmless by
coincidence, and the coincidence had an expiry date, which arrived when a
downstream author removed the unrelated constraint for unrelated reasons.
The seven guards now have seven tests, each exercising both sides of its boundary — 256 accepted and 257 refused, zero bytes refused and 128 accepted and 129 refused. They cost nothing: every one of them returns before the GPU is so much as opened. They are the cheapest tests in the repository and they are the only evidence that the refusals exist.
Why one cannot simply check the answer
The reader with a background in numerical work will have been waiting some paragraphs to ask why I do not simply compute the right answer on the host and compare. It is the obvious move and I want to explain why it was the wrong one, because the reasoning generalises beyond this repository.
Six probability densities are implemented as GPU shaders. One of them — the normal — has a bit-exact host reference, which was written first and works admirably. The other five had nothing: they compiled, they dispatched, they produced no NaN, and that was the sum of what was known about them.
To supply the missing references means re-deriving five densities and their gradients in Elixir. Two objections, of which the second is fatal. The first is that Student’s t requires a log-gamma function, which Erlang’s standard library does not provide, so one must hand-roll a Lanczos or Stirling approximation — a second implementation, written by the same person on the same afternoon, and therefore liable to be wrong in the same direction as the first. Agreement between two of one’s own mistakes is a well-known phenomenon and is not evidence.
The second objection is architectural. SPIR-V has no double-precision
transcendentals whatever — no exp, no log, no
pow — so every one of those five shaders casts down through
single precision to compute them and back up again. A host comparison is
therefore not a test of the shader. It is an assertion that Erlang’s
:math.exp agrees with this particular vendor’s fast-math to
within a fudge factor, on four different drivers spanning a 2012 Kepler, a Tegra
X1, and an Ampere. One would be tuning a tolerance until the fleet went green,
which is not testing but negotiation.
The shader is examined against itself
What one does instead is ask the code a question to which it must know the answer without being told what the answer is. Each shader produces two outputs: a log-density and its gradient. If the second is genuinely the derivative of the first, then a central finite difference of the log-density must reproduce the gradient. Perturb one coordinate up, perturb it down, subtract, divide by twice the step.
Nothing about the model is restated. No density is re-derived. The shader is convicted or acquitted on its own internal consistency, and the test works identically for all six families and any family added later. There is a further economy, which pleased me more than it should have: a normalising constant differentiates to zero, so precisely the part requiring log-gamma — the part that made the host reference impractical — falls out of the comparison and needs no implementation at all.
The step size, naturally, had to be measured rather than assumed, and it taught me something. Maximum discrepancy at four dimensions:
family h = 1e-3 h = 1e-4
normal_f64 7.3e-14 1.6e-12
cauchy_f64 4.7e-05 5.2e-04
exponential_f64 9.0e-05 9.7e-04
halfnormal_f64 2.7e-05 3.1e-04
studentt_f64 1.4e-04 4.1e-04
weibull_f64 1.7e-04 2.2e-03The smaller step is uniformly worse, by roughly a factor of ten, which is the opposite of what the textbook error term suggests and exactly what one should expect on reflection. The single-precision detour puts about a part in ten million of noise on the log-density; a central difference divides that noise by twice the step, so halving the step doubles the noise while shrinking a truncation term that was already negligible. The optimum is governed by the instrument, not by the mathematics.
And note the first row. The normal family sits at 1e-13 rather
than 1e-5 because its shader contains no single-precision detour
anywhere — every constant is computed on the host in full double precision.
That is an eight-order-of-magnitude confirmation, arrived at sideways, of why it
is the only one of the six for which a bit-exact host reference was ever
coherent. I did not design the experiment to establish that. It fell out, which
is the most one can ask of an experiment.
What one asserts when one cannot assert the value
The remaining properties all take the same form, and it is worth naming because it is the form that survives a heterogeneous fleet. One does not compare a number to an expectation. One compares two paths that must agree by construction, and one demands that they agree to the bit.
Instance i of a batched dispatch must be identical to that chain dispatched alone, because the batched shader is the single shader with its indices offset. The first n steps of a seven-step run must be identical to an n-step run, because each step depends only on its predecessors. A dispatch of one instance must be identical to the unbatched path, there being one workgroup either way. None of these requires knowing what a leapfrog integrator ought to produce; all of them hold on every vendor’s driver regardless of what liberties it takes with the arithmetic; and none of them can be satisfied by a plausible wrong answer, which is the entire hazard on a GPU. A shader that reads one parameter out of another’s bytes does not crash. It returns a number, and the number looks like a number.
The prefix property turns out to carry weight I did not anticipate when I wrote it. The consumer of this library batches chains of unequal length by dispatching them all at the greatest length and slicing each caller back to what it asked for — sound if and only if a longer run’s prefix is exactly a shorter run’s output. It is, and it is now pinned, and a future edit to the step loop can no longer break their arithmetic in silence.
A confession about generated inputs
I should admit that this is property testing without a property-testing library. The shapes are swept — dimension at 1, 2, 7, 19, 41 and 256, step count at 1, 2, 4, 7 and 11, instance count at 1 through 4, each axis at its boundaries with the others held in the middle — but the interior values are literals in the source rather than draws from a generator.
The orthodoxy would object, and on a different codebase I would agree with it. The defence is specific. Every other assertion in this suite compares exact bytes, and a fleet member reproduces a failure in minutes rather than seconds; a seed-replay step between “the Jetson is red” and “I can see it” is a step that nobody performs at two in the morning, and a suite whose failures are not directly reproducible is a suite that gets ignored. What a generator buys is adversarial search, which wants a shrinker and a corpus. What this code needed was coverage of a small, entirely enumerable space, which fifteen chosen shapes deliver and which can be read off the page. I added no dependency. I may be wrong about this and have written down why, which is the best one can do with a judgement call.
The moral, such as it is
Nobody is taught to distrust a passing test. The entire apparatus of the profession — the green tick, the badge, the merge gate — is arranged to convert a passing test into a permission and then to stop thinking. A red suite is investigated for hours; a green one is investigated never. This is precisely backwards with respect to where the losses are, because a wrong test that fails is corrected within the day, and a wrong test that passes is load- bearing masonry for as long as the building stands.
Fourteen tests were added. Perhaps four of them will ever catch anything. But seven refusal paths are now known to refuse, five densities that had never been checked against anything are checked against themselves, a decoder that could not detect a NaN now counts, and a boundary that silently produced posteriors over the wrong model is enforced on both sides. The corresponding number of bugs found on the day was one, in my own test helper, which is about the ratio one should expect and rather more than I deserved.
Arrange for your test to fail. If you cannot, you have not written a test. You have written a decoration, and it will decorate a wrong answer quite as willingly as a right one.