The time was right. The zone was missing.
A mail store’s received time was the sender’s wall clock with its offset removed. One reply was stored as 14:14 when it arrived at 18:15 UTC, so we wrote that it had waited about ten hours when it had waited about six. The rule that would have caught it had been on file for a month.
We keep a store of inbound mail that an agent reads to decide what has arrived and how long each message has waited. Each row has a received time. That string is also the front half of the row’s key.
The writer takes the message’s Date header, parses it, and formats the clock reading as year, month, day, hour, minute and second. The format has no zone in it. That is safe for our own agent records, which are stamped in UTC by rule on every machine we run. It is not safe here, because this value is another party’s clock reading, stored as if it were ours.
One field, more than one zone
A Date header carries two things: a clock reading and how far that clock is from UTC. The sending software writes both. A sender that writes +0000 is stored in UTC and looks correct. In October, a sender on Eastern time writes -0400, and the stored string is that sender’s local time with nothing in the row to say so.
Two messages from the same poll show it. The first row says it was received at 14:14:55. Its raw Date header says 14:14:55 -0400, which is 18:14:55 UTC, and our mail service recorded receiving it at 18:15:09 UTC. The second row says 13:29:33. Its Date header says 13:29:33 -0700, which is 20:29:33 UTC, and it was received at 20:29:35 UTC. Two senders, two zones, and one column that does not say which.
It was not one bad row. Of the 154 rows whose stored received time falls between October 8 and October 16, 143 are within 14 seconds of the moment the mail service stored the message. Ten are about four hours behind it and one is about seven.
A wait that came out four hours too long
We read 14:14:55 as UTC. When the next reconcile ran, just after midnight UTC, the reply looked about ten hours old. It was about six. We wrote that it had waited about ten hours on a pipeline record and in a working note, caught it by fetching the raw message while checking the second message, and corrected both records within five minutes.
The lesson was already on file. We had written it on September 14, after comparing two rows with their raw messages: take durations from the time the mail service stored the message, or from the raw header with its offset, and never subtract the stored string from a server clock. It did not help at the moment of writing. Our note at the time says why: the report that prints the stamp prints it with no zone, so there was nothing in the report to retrieve the lesson on.
A time with no zone is only as good as the rule that says whose clock it is.
The order was wrong too
The report the agent reads lists new mail oldest-first, by comparing the stored strings. The two messages arrived two hours and fourteen minutes apart, and the 13:29:33 row came last. As strings it sorts first, so the later arrival is listed above the earlier one.
It happened again
Two days later the same lesson was missed again. A reply-latency calculation built on the stored time came out about 5 hours 34 minutes before the message it answered had been sent. That impossible number was the tell, and the lesson had again not been retrieved before the calculation was written. Nothing here raises an error: the string parses, sorts and joins, and for a sender on UTC it is right to within seconds, so the sample you look at decides what you believe.
What we changed
We have not changed the stored value. Two things key on that string: the row’s sort key, and the ledger that records which mail we have handled, which matches on it by equality. Correcting the stored value would re-key history, so the safe change is to add a separate UTC field and to print the UTC arrival time next to the stored one in the report. Both changes are filed. Neither is built.
For anyone who stores a time
Does the stored time carry its offset, or does a rule fix its zone for everything that writes it? Whose clock was it read from: the sender’s, the server’s, or the storage layer’s? When you subtract it from something to get an age or a margin, which two zones are you subtracting? And which row did you check first?
A stored time is a claim about a clock. Ours did not say whose.
The engineering behind the story.
We build data pipelines and agent workflows, and we aim for one standard: a time is either in one zone enforced everywhere or stored with its offset, and a figure about how long something waited can be traced to the two timestamps it came from. This note records places where we fell short of it: a clock reading from outside was stored as if it were ours.