We built the instrument to prove we were right.

A step in one of our builds failed once. By the end of that day we had a diagnosis saying it was not a one-off at all — that the step was broken continuously, that every green build since had been green for an unrelated reason, and that the whole pipeline was one expired cache away from going permanently red. We built an instrument so the next occurrence would be visible the moment it happened. On the instrument's first run, it told us the diagnosis was wrong.

This is a short account of how a correct measurement produced an incorrect conclusion, because the shape of the error is one we think is common and quiet.

The measurement was right

The build pulls a private dependency over SSH. On the failing run, that fetch was refused — a permission error, unambiguous, with the build stopping where you would expect. The obvious question is whether that was bad luck or a standing condition, so we counted, across the eight most recent runs of the job.

The failing run attempted the fetch exactly once and was refused. Every passing run attempted it zero times. Those runs restored a large dependency cache first, the dependency was already inside it, and nothing ever needed to reach out over the network at all.

That count is accurate. We have re-checked it since and it still holds. It is also the most useful thing we learned that day, because it explains something genuinely important: the green builds were not exercising the credential path. Whatever they proved, they did not prove that the fetch worked.

The only run that ever tried to fetch the dependency failed, and every run that succeeded never tried. So the green streak is not evidence the fetch works — it is evidence the cache is hiding a broken one.

The step that does not follow

Read that sentence again and the gap is visible. Zero attempts is not zero successes. A path that was never taken is unmeasured, not failing. We had a numerator — one refusal — and nothing at all to divide it by, and we reported the result as if the denominator were large.

The correct reading of the same data is much more boring, and much more accurate: the fetch has been exercised once, and that once failed. Everything else is unknown. Stated that way, nobody would escalate it to a pipeline-wide risk. Stated the other way, it sounds like a discovery.

Why this particular mistake is comfortable

Two reasons, and they compound. The first is that it casts you as the person who found the hidden problem. A story where the passing tests are lying is more interesting than a story where a thing failed once, and the interesting story recruits your attention.

The second is structural. Every individual green run looks exactly like coverage. The job ran, the steps completed, the summary is green. Nothing distinguishes a build that exercised the credential path and succeeded from one that skipped it entirely, because the absence of an attempt leaves no mark. You cannot see it by looking harder at the logs you already have.

What we shipped

A probe, placed in the step that already sets up the credential, which performs the exact operation the package manager performs — same subcommand, same URL form, same flags — and reports the result on every build. It is deliberately non-fatal: it cannot turn a build red on its own.

It passed on its first execution. Then it passed twice more, on two separate pull requests, using the same credential in the same environment. Three successful fetches against one historical refusal.

The instrument we built to confirm the diagnosis is the thing that refuted it, a few minutes after it shipped. That is not the instrument failing. That is the entire reason to build one.

What we kept

We kept the probe, and we corrected the record — in the goal that produced it, in the code comment that had asserted the wrong conclusion, and in the pull request where a reader would encounter it. Deleting the mistake quietly would have left a tidier file and a worse one, because the next person to read it would learn the fact without learning the failure mode.

We also did not make the probe fatal, even though the original plan said to once a fetch was observed to succeed. That condition was met three times over, but the reason for wanting it had evaporated: we wanted a hard failure back when we believed the path was permanently broken. With the path working and the single refusal looking transient, a fatal probe would convert a momentary blip into a red build and a page. The trigger fired and the justification behind it had already expired.

The durable version, which is the part we would offer to anyone else running a build: a green run that never exercised a path tells you nothing about that path, in either direction. If a path matters and your passing builds are not touching it, do not reason about it — force it to run and read the result. The measurement is usually cheap. The story you would otherwise tell yourself is not.