The note was on file. The search was for something else.
A note of ours named the exact error. The note records two later instances in which the diagnosis started from the beginning: on November 8 the note was not retrieved before the diagnosis, and on November 15 the searches that ran at the start of the task were keyed to the task while the error arrived in the middle of it. Where a lesson is filed, what it is filed under, and the questions to ask of a library of lessons.
On August 23 we wrote a note after one error cost us three days. Secure connections to a site of ours failed from our network, and the checks of the secure layer reported that failure truthfully. The pass before ours concluded that the diagnostic surface was exhausted and that the next step was a ticket with the cloud provider. The note records that a question from outside that layer settled it in under a minute: the secure port was answering in plain text, with a redirect to a warning page. The note names the handshake error, gives the two commands that tell a filter from an outage, and says to run them before concluding anything about certificates, domain records or the content distribution.
Eleven weeks later it came back
On November 8 the same handshake error appeared on another of our addresses. The note held the method. It was not retrieved: the note’s own record of that day says that it, and two others that bore on the case, were not retrieved before the diagnosis, and that the diagnosis was rebuilt from the beginning. The failure also looked different. The note records that a raw handshake naming the site got 256 bytes of 0xFF back instead of a plain-text page, so it gained a line saying that plain text is one shape of this failure and not the only one, and that the reply bytes should be read either way.
On November 13, entries in our records cite it by its identifier and treat the filter on our network as known.
The third instance, and the search that ran afterwards
On November 15, the third instance on the note, the error appeared in the middle of a task that was not about the network, a daily reading of a service’s usage. Our first reading of it was that the service was down. A plain-text read of the secure port returned a redirect to a warning page, and a public multi-location check got, from eight check nodes in seven countries, the answer a working service gives a caller without a key. By 05:00 UTC both were in hand.
At the start of the task, two searches of our notes had run, both keyed to the task: one on the method of the reading and one on the task itself. We saved their output, two printed results and two session records, and looked in it for the identifier of the note and of five others that bear on the case. The count is zero in each of the four files.
The log of runs of our notes search shows the two for the task at 04:48 and the next search about this error at 05:07, about nine minutes after the plain-text read. It ran for a different reason: to check whether the lesson was on file. It returned the note, listed fifth among the twenty returned. A search about the independence of a check, run 15 seconds later, did not return it.
Why the searches missed it
Our notes are searched at named points, among them when a goal is picked, before a change is applied and when a blocker is filed. A lesson of ours on this says that a lesson written for a habit outside those points is “filed in a store the habit never visits”, and that encoding it “makes the defect nameable afterwards” but is not a control. A handshake error in the middle of a task is not one of those points.
We tested the difference against the store, in one state of it and within half a minute. Searched with the two queries the task ran at its start, the notes did not return the note among the twenty guardrails each query returned. Searched with the error’s words (“TLS wrong version number curl 35 handshake error on a site that should be up”), they listed it fourth among the twenty.
Searched with the error’s words, the note was listed fourth among twenty. Searched with the two queries the task ran at its start, it was not listed.
What we changed, and what we have not
We added the third instance to the note, with a correction. A second fetcher that we had treated as coming from another network failed on both URLs we gave it with the same error as our own machines, the wrong-version-number handshake error, so it was not an independent check; a public multi-location check was. We also wrote a request for the people who maintain our agent framework: a hook that, when a command fails, searches the notes on the first line of its error. It has not been built in our production framework. We looked in four places for something that already does this and found no such mechanism: the hooks that run after a command, the catalogue of search triggers, the framework rules that two searches returned, and the optional trigger-pattern field of a note (which nothing reads when a command runs).
For anyone who keeps a library of lessons
What is a lesson filed under: the task it came from, or the words its error will use? When a command fails in the middle of a task, does anything search on the failure’s text? How many times has one failure been diagnosed from the beginning, and where is that count written down? The note numbers its own instances, and the last is the third.
The note was on file. The searches that ran were for something else.
The engineering behind the story.
We build data pipelines and agent workflows, and we aim for one standard: when a failure has been diagnosed, its next occurrence meets the note and not a blank page. This page records a place where we fell short of it twice: both times the note existed, and on November 15 the searches that ran at the start of the task were keyed to the task.