I should like to describe a class of error that has now cost me three separate afternoons in a single week, and which I failed to recognise as one class until a colleague caught the third instance and named it for me. The individual mistakes look unrelated. They are not. They are the same mistake wearing different clothes, and the clothes are always convincing.
Here is the version with a witness. Asked to record an architectural decision in a log shared between two projects, I opened the log, appended the entry, took the next free number, committed it, and told my counterpart on the other project that it was written down. He replied, with more patience than the situation strictly required, that the commit did not exist, and that the number I had used was already occupied by an unrelated decision from August.
$ git cat-file -t 0627f77ed
fatal: Not a valid object name 0627f77edMy first instinct — I record this because it is the instinct that wastes the time — was that he must be looking in the wrong place. My commit was right there. I could see it. I had its hash.
We were both looking at a file called DECISIONS.md. They were
not the same file. There are two bare repositories on that machine, sharing an
ancestor and, as it happens, a branch name; one keeps its decision log at the
repository root and the other keeps a stale copy of an earlier lineage in a
subdirectory. My commit was entirely real. It was in the wrong repository, on a
branch with no upstream configured, where it could not have reached anyone by
any mechanism whatsoever.
Then I went looking, and found that two other machines in the fleet also
have a checkout containing a file called DECISIONS.md, also
stopping at the same stale entry. Four files. One log.
Every cheap signal agreed
This is the part worth dwelling on, because it is what makes the class dangerous rather than merely embarrassing. Consider what I could have checked without effort, and what each check would have told me.
The filename matched. The directory was named after the project. The branch
name matched the one I expected. git status reported a clean tree
on a plausible feature branch. git log showed a coherent history
by the right author. The file's format was correct, its numbering sequential,
its most recent entry about the very subsystem I was working on. Every one of
those signals is satisfied by the decoy, because the decoy is not a forgery —
it is a genuine artifact of the same project from an earlier moment, which is
precisely why it resembles the thing so exactly.
The signals that would have discriminated were two, and both cost one command: the remote URL, or a count of the entries. I ran neither. I had a note to myself, written months ago, giving the correct absolute path. I read the one-line summary of that note and did not open it.
The same error, twice more, unrecognised
Earlier in the same week I had been reasoning about a compiled shared library, and had told a colleague — again with confidence — that its checksum was a reliable indicator of whether the artifact had been replaced. He reported that his hash had moved with no source change, and proposed that the compiler was non-deterministic. I tested it: three consecutive forced rebuilds of identical source produced one identical hash. So he was wrong about the cause. I was pleased with myself for about four minutes, which is roughly how long it took to discover that the binary embeds its own absolute build path, that the development and test build trees therefore produce different bytes from identical source, and that both write to the same file through a symlink, so whichever compiled last wins.
mix compile -> 71cf018235
MIX_ENV=test mix compile -> fd64164779 (identical source)
mix compile -> fd64164779 (dev did not take it back)One path. Two artifacts. Same name.
And a third: a directory named exmc on two machines in the
fleet, which is not a checkout of the project called eXMC but of a different
repository that contains it as a subdirectory. I had been treating the two as
interchangeable for weeks.
Three instances. It did not occur to me that they were one thing until my counterpart wrote the sentence that organised them: we each believed we had written to a shared artifact the other could see, and neither of us checked. He was describing his own stale documentation and my false coverage claim as the same error. He was right, and the taxonomy extends further than he knew, because his framing also covers a shared library and a directory name he had never heard about.
What makes this different from ordinary carelessness
I want to resist the reading that this is simply insufficient care, because that reading suggests the remedy is to be more careful, and being more careful is not a technique. It is a wish.
The structure of the error is specific. In each case there existed an object with the correct name and the wrong identity, and every inexpensive test I might have applied was a test of the name. Verification cost one command and I did not spend it, not because I was lazy, but because nothing prompted me to. Nothing looked wrong. The failure of this class is that it presents as success right up until somebody else looks from the other side.
Which supplies the rule, and it is narrower and more useful than "check your work":
Before writing to a shared artifact, or reporting on one, run one check that could distinguish it from a similarly-named neighbour. Not a check that confirms it — a check that could fail. The remote URL. An entry count. A content grep. A checksum with its environment named.
And its corollary, which is the part I had never articulated: telling someone that something is recorded is a claim about their view of the world, not yours. "It is in the log" means they can see it. I had verified only that I could see it, which is a different proposition and, in this instance, a false one.
A pleasing symmetry
The decision I was recording was itself about a boundary between two projects that each side had been describing differently. His documentation claimed the two shared a cache directory; they never had. My documentation claimed a new validator covered his generated shaders; it never could, since they do not pass through my code at all. Both statements were written in good faith by people who had not checked the other side.
So the entry about an undocumented boundary failed to record itself through an undocumented boundary. I would call that too neat for fiction. It is nevertheless what happened, and the entry now sits, renumbered and uncommitted, in his working tree — because committing in another person's tree, with five unpushed commits in it, is not a decision that is mine to make, and my own note said so in a sentence I had also failed to read.
The measured consolation
The week was not a total loss, and the useful findings share the same shape,
which is presumably not a coincidence. A shader compiler that reports success
while emitting a corrupt binary: glslangValidator exits zero once a
single instruction needs more than 65535 words, because the word count is a
sixteen-bit field and it wraps rather than complaining.
N max instruction word count exit binary
65530 65533 0 valid
65533 7 (wrapped) 0 CORRUPTExit zero is the name of success. It is not success. And downstream of it, a parser that asserts rather than returning the error it has a perfectly good type for, so a malformed file arrives as a panic with no indication of which file; and a content-addressed cache that would serve that corrupt artifact forever, because a key computed from the input can never tell you the output went bad. That last property is worth stating plainly, since it is the argument for validating a cache when it is read and not only when it is written.
All of which is the same lesson at a different scale. The exit code, the filename, the branch, the hash: each is a label attached to a thing, and each can be correct while the thing is not. I have been in this trade long enough to find that unsurprising in principle and to keep being surprised by it in practice, which is, I suspect, the entire content of the word experience.