Incident Digest

1. A piece that had passed review was silently rolled back, and nobody noticed for six and a half hours

What happened: On 16 August, a Chinese-language deep dive that had already passed quality review existed only in a working directory, never committed. Someone tidying that directory silently reverted it to an older version. For 6.7 hours nothing flagged it. When the site rebuilt that afternoon, it shipped the old version.

What actually caused it: Several jobs work in the same directory at once, and every command the version-control tool offers — stash, restore, switch — acts on the entire tree, not on "the lines I just changed." One "put my changes aside for a moment" command that afternoon swept up 230 files. Exactly one was recovered.

The rule we now hold ourselves to: (1) anything that passes quality review is committed on the spot — no finished piece may live only in a working directory; (2) before shipping, a machine reconciles "the file about to go out" against "the version that passed review," and stops the send if they differ; (3) sending only your own hunks now goes through one official pinned-revision procedure, which structurally cannot pick up someone else's work.

2. The live service lost two features for three and a half hours while monitoring showed green

What happened: The same day, two features were simply absent from the live service for 3.5 hours — a reader survey could not write back, and one public endpoint did not exist at all. Drift monitoring reported "in sync" the whole time. Zero alerts.

What actually caused it: The monitor asked "did the source change?" The source genuinely had not. What had been swapped out was the artefact built from that source. Measuring only the input and never the output is a structural blind spot.

The rule we now hold ourselves to: the monitor's question changed from "did the input change?" to "is what I last produced still where I put it?" If the artefact is gone, it goes red — whether or not the input moved.

3. Running a front-end build in a working directory put it live five minutes later

What happened: That same evening, two builds made purely for testing went live for real. No one pressed publish, nothing was committed, and none of the 19 quality checks ran.

What actually caused it: The build artefact was the deployment source. There was no shipping gate in between, so "making it" and "deciding to ship it" were the same act — which means any slip of the hand is a release.

The rule we now hold ourselves to: a shipping-provenance check now stands in the way. If the version stamp does not match the actual contents, or the working directory is dirty, nothing is swapped in. Making and shipping are two separate acts from now on.

4. An alarm that rang punctually every noon had no recipient

What happened: The internal management brief failed to arrive on 16 of these 30 days. A machine reconciled delivery at 12:00 every day and logged the miss every day — and nobody was on the other end of that log.

What actually caused it: The reconciliation ran, the record was written, but no role was ever named as the recipient of that record. The alarm was ringing into an empty room. Which is why, on the day we compiled this list, we still could not say whether those 16 days were a breakage or a deliberate pause — nobody had ever attributed them day by day.

The rule we now hold ourselves to: everything that gets blocked, filtered, or logged must have a named destination and a named recipient. A record with no recipient does not count as handled. Compiling this list was itself the first product of that rule — listing your incidents is how you find the bells that have been ringing with nobody there to answer.

5. A safeguard that was built halfway, with a green light saying it was done

What happened: After an earlier "this chart looks tampered with" incident, we built a safeguard: a legitimate revision must declare in the data who changed it. The safeguard's own checks were green, and the field really was there in production — but the interface never displayed it. So the same problem happened a second time, and a second time the only reading available to the user was "the data was tampered with."

What actually caused it: acceptance stopped at the data layer instead of at the user's eyes. A green light measures what we did, not what the reader saw.

The rule we now hold ourselves to: a safeguard's acceptance criterion now always ends at "the user can see it." Present in the data but absent from the screen counts as unfinished.

What we have not fixed

This month had four main threads — whole-tree rollbacks, shipping without checks, two jobs colliding on one file, and commits being blocked by someone else's work in progress. All four had a defence land the same day. But three of them sealed the symptom, not the cause:

• the mutual-exclusion lock was fixed on the claim side; the send side is still open;
• collisions on send were routed around by giving every work item its own tree, but the check itself still measures the wrong object;
• the family of files most exposed to rollback has been catalogued, and the audit is still running — the third occurrence happened on the same day.

Every one of the 17 open incidents has been seen, filed, and assigned a work item. The status is "not yet converged," not "unknown." We write this out rather than reporting only the 21 fixed ones because "symptom fixed" and "cannot recur" are two different things, and anyone considering this system needs to know that we can tell them apart.

What held all of this up

Worth saying last: the 43 incidents above are the residue left after several layers of automatic defence.

Over the same period, lock contention on commit was absorbed by automatic retry more than a thousand times; over seven hundred same-type signals were chain-muted so that one root cause could not flood the diagnostic queue; more than a hundred genuine overlaps between two pieces of work were parked intact in a quarantine area for a human to review rather than force-merged; six times the system hit a service quota ceiling and stopped dispatching or switched accounts instead of burning out mid-task; and 13 drafts that did not meet the bar were stopped before they were sent.

A system's reliability is not that it does not fail. It is whether something catches it when it does, and whether someone writes it down when nothing does. This digest is the evidence for the second half.

Limits of these numbers (part of disclosing honestly)

1. "A couple of thousand failure signals" is a signal count, not an incident count; one root cause can emit hundreds. The 43 items here are incidents after manual consolidation. The two numbers cannot be derived from each other.

2. The self-healing classification log is only complete after 6 August; earlier figures are reconstructed from other records and may be understated.

3. Dates for lesson files are taken from the text inside them, not from when they were committed —on 16 August a batch of older lessons was committed late, and counting by commit date would record all of them as that day.

4. "Fixed" means there is a work item, a fix, and a regression test. It does not mean "cannot recur." The section above lists three where the symptom is sealed and the cause is not.