The watcher reported success.

One of our pipeline records sat in a state we monitor closely for nine days without being monitored at all. It was not corrupted. It was not orphaned. It had a title, a status, a scored assessment, and an explicit written recommendation to act on it. Every dashboard that counted records counted it. The one system whose entire job was to raise its hand before the clock ran out had silently excluded it, and then reported a clean run.

The record was missing a date. That is the whole defect. What is worth writing down is everything that did not happen as a result.

What a missing field actually does

The intuition most people have — the one we had — is that a record missing a required field is a loud problem. It fails a schema check, or it shows up as a null in a report, or something downstream throws. You find out.

That intuition assumes the consumer is strict. Ours was tolerant, which is usually the right instinct. The watcher read every record, tried to parse each one's deadline into a date, and wrapped that parse in a try/catch so that one bad value could not take down the whole run. A record whose date would not parse returned null, and a record with a null date was treated as not-yet-schedulable and skipped.

That is defensible behavior in isolation. Combined with a missing value it produces something else: the record is not rejected, not flagged, and not watched. It is simply absent from the set the watcher considers, and the watcher has no way to know it should have been there.

The run completed. The counts balanced. The job reported success. Every one of those statements was true, and together they described a system that had stopped doing the one thing it existed to do.

The check we already had

Here is the part that took us longest to see, and the reason we think it generalizes.

We already validated that date. There was a real check, written deliberately, with a comment explaining why it mattered: the value had to be a bare calendar date, because the downstream consumer parsed it strictly and anything richer — a full timestamp with an offset, say — would parse to null and drop the record. Someone had been bitten by that exact failure before and had gone to the trouble of preventing it.

That check ran on every write. It never once fired on this record, because it only ran when a date was supplied. A format validator asks whether a value is well-formed. On an absent value there is nothing to be malformed, so it passes, instantly and correctly.

So we had a validator guarding this field, we had a comment explaining the exact failure mode it prevented, and we had a record that reached production with the field missing entirely. Having the format check is what made us believe we had the presence check. They look like the same concern and they are not.

Why we did not add a report

The obvious remedy is a sweep: scan the store nightly for records missing required dates and surface them. We did not build that, and the reason is the one useful thing we can offer anyone with a similar gap.

A detective control tells you about a problem after it exists. For a record whose value is time-bounded, after it exists is frequently after it matters. A nightly sweep would have flagged ours on day one and again on each of the following eight, which sounds like a working control until you ask what the report accomplishes when nobody is watching the report either. The nine days were not lost because nothing detected the gap. They were lost because the gap was allowed to be created.

We moved the check to the write. A record that would land without a date is now refused at the moment it is written, loudly, with an error naming the missing field and the exact accepted format. It is not possible to store the shape that caused this. The stage transitions where a deadline is genuinely meaningless — a closed or withdrawn record — are exempt, because a control that blocks legitimate work gets removed within the week.

The test that would have stayed green

One more thing, because it nearly caught us on the way out. When we added the refusal, the existing test suites all passed. That is not evidence of anything.

Every one of those tests had been written to exercise the success path, so every one of them supplied a date. None entered the new branch. The suite was green before the change and green after — meaning it could not tell the two states apart, and would have stayed green if someone deleted the refusal a month from now.

So we wrote the missing test and then proved it by putting the old code back. Two assertions failed and three still passed, and the split is the useful part: the two that failed were the ones covering the new refusal, and the three that held were covering behavior that predates it and should not have moved. A test suite that fails on everything under that swap tells you something changed. One that fails on exactly the right two tells you what it is pinning.

What to go look at

If you run a pipeline where records become actionable on a date, there is a specific pair of questions worth an afternoon. First: when a consumer cannot parse one of your fields, does it fail, or does it skip? Second: does anything, anywhere, distinguish a record it skipped from a record that was never there?

If the answer to the first is skip and to the second is no, you have this. It will not appear in your error rates, your job success metrics, or your record counts. It appears as a thing you thought you were watching, on the day you needed to have been watching it.