The cap worked. The steps were gone.

Many of our recurring jobs keep their instructions in a long text field on the job’s task record, which we call a runbook. When a run learns a rule, the rule is added to the end of that field as a new block of text. A tool does the adding, and it enforces a size cap. This story is about one of them.

The cap is 32,768 bytes, and the tool checks it on the field as it stands when a new block arrives. If the field is over, the tool keeps the first block and any block its author marked as pinned. It keeps the newest unpinned block, and older ones only while everything kept stays within a second, lower limit of 16,384 bytes. It moves the rest to an archive file before it adds the new text.

The notice said what it did

On October 30 we added a correction to one runbook. The field stood at 33,621 characters, which is 33,671 bytes, 903 over the cap. The first block alone was 19,347 bytes, already past the lower limit, so the tool kept only the newest unpinned block. It moved the other eight unpinned blocks to an archive file: 12,385 bytes, more than thirteen times the overage. It wrote a notice at the top of the field saying so, and then it added the correction.

The tool’s output carried a rotated object with the field’s length before the move, a moved_bytes figure, the archive’s path and a note. The note said that “Nothing was deleted” and “A large drop here is expected, not data loss.” Both were true of the archive: we opened it and the eight blocks were in it.

What the notice left out

The notice gives a count and a size. It does not say which blocks moved, and a block’s age does not tell you whether it is still in force. These were the oldest additions that were not pinned. Five of them told later runs what to do. Four were standing rules that later runs apply whenever their condition arises. Among those rules: what to check when a reply answers one of our sends, what to update once a message we queued has gone out, how to compare two lists of hits by their keys instead of their stored times, and two steps for a reply that settles a waiting prediction or points to an attachment.

The fifth was a dated account of an earlier run that also told later runs to read a figure one of our scripts prints instead of rebuilding it by hand. Of the other three, one corrected that account, one was another dated account, and one was empty.

The append returned ok: true and printed no warning. We noticed because its output included an object named rotated and we read it.

We then checked the live text for ten strings, six section names and four block markers. Three were gone: the name of the rule for comparing lists of hits and the markers of two of the moved blocks. The name of the rule for what to update after a queued message goes out was there only inside the correction we had just added. The other six were there.

The lesson was on file

A guardrail we wrote twelve days earlier describes this mechanism. It says the output does not name the block that went, and that a note calling a large drop expected makes the rotation read as housekeeping. We found it within two minutes after the append, by searching for the failure.

Protection you have to remember

In a runbook the tool never moves the first block or a block that carries a pin marker. None of the eight was the first block or carried a marker, so all eight could be moved. Whether an old rule survives therefore depends on whether it sits in the first block, carries a marker when the next rotation runs, or is still among the newest. A marker added later works too, as ours did.

What we changed

The instructions in those five blocks were out of the live field for about 80 seconds, until we restated them in one new block of about 4,800 characters. The one partial exception was the last step of the rule for what to update once a message we queued has gone out: the correction we had just added rewrote its part about our notes pages, but its parts about open tasks and the record of the send itself were gone. No run of that job finished in that time. We pinned the new block, along with two other blocks that carry steps for later runs. One block was already pinned. With those changes the field held 28,723 bytes, 4,045 under the cap.

That is a patch, and it has a cost: every block in the field is now the first block or a pinned one, so for now the cap has nothing left to move on this field. A rule every run needs should not depend on a marker. The executable core belongs in the first block, which the tool never moves, and that change is filed with the measured sizes.

For anyone whose instructions live in a capped field

When the field is over the cap, does the tool drop the oldest text, the least used or the least important? Does it cut only the overage, or down to a lower limit? Does its notice name what it moved, or only how much? Is protection the default, or a marker each author has to remember? Who reads the notice, and what does it call the damage? Could you list the rules a future run depends on without opening the field?

A cap that drops the oldest text first drops the rules learned first.

The notice at the top of the field was accurate and the cap did its job. For about 80 seconds the steps were gone from the place the next run would look.