Fifty-nine thousand scheduled runs across forty-four commands and three weeks, and the failure count is zero.

success   52,013
running    7,107
failed          0

Nobody believes forty-four cron jobs ran fifty-two thousand times without a single failure. The question is what the dashboard is actually reporting, and it matters because the pattern is common in home-grown job telemetry: the zero was not a measurement. It was a code path that could not be reached.

What that column can and cannot say

The recorder writes a row when a task starts, marked running, and updates it when the task finishes — to success, or to failed if the framework reports a failure.

That design has an implicit consequence. running is not a state the task reports; it is the absence of a completion record. A task that finished is a task that got back to update its own row. Anything that prevents that update — a crash, a killed container, a deploy, a hook that never fires — leaves the row looking indefinitely like a task still in flight. The row is indistinguishable from work in progress, forever.

So the table does not show three outcomes. It shows two outcomes and a pile of runs that are permanently ambiguous.

The reason the failures could not be written

In this application fourteen scheduled events are declared with runInBackground(). Laravel runs those in a child process and then executes their after and onFailure hooks in a separate schedule:finish process.

The recorder had handed the open row's id forward in a process-local variable. In that second process the variable is empty, so the completion hook hit its "no open run" guard and returned without writing anything. Both hooks — the one for success and the one for failure — die at the same guard.

That is why the failure count was zero rather than low. For fourteen of the tasks, including the one that runs every minute and the one that takes the nightly backup, the dashboard was structurally incapable of reporting a failure. Not unlikely to. Unable to.

production, before the fix:
    running   128,955
    success   106,930
    failed          0

More rows stuck than completed. A dashboard reading "last run: success" was reading the last row that happened to belong to a foreground task.

Did the fix work?

The repair was to pass the open row's id through a store both processes can see — the shared cache — rather than through process memory. In concept that is a one-line change. Whether it worked is a measurable question, and the reason for this guide.

Splitting the table at the date the fix shipped:

                runs     still running    share
before        10,873           5,884      54.1%
after         48,249           1,223       2.5%

From more than half to one in forty. That is a real repair, and "we fixed the telemetry" is exactly the kind of claim that needs checking rather than assuming. Such fixes go unverified for years, because the tool you would check with is the thing that was broken.

What the remaining 2.5% is

The remaining 2.5% are not failed tasks. They are 1,223 runs whose completion was never recorded, and from this table we cannot tell which of them failed, which were interrupted, and which finished fine while the handoff missed.

posts:publish-scheduled   851
inbox:escalate-stale      170
inbox:flush-digests        64
email:routing:cache        57
errors:monitor             57

The distribution is a clue rather than a verdict: the worst offender runs every minute, so it has the most opportunities to be interrupted by a deploy or a restart, which is what you would expect if most of these are interruptions rather than failures.

These stuck rows matter because nothing ever removes them. The pruning job that ages out old rows deliberately exempts anything marked running, on the correct reasoning that an in-flight task's open row must not be deleted underneath it. So a stuck row is never closed and never deleted. The set only grows, and every one of them is a small permanent lie about a task that is not running.

The fix for that is a sweep that closes out runs older than any plausible duration for their command. It does not exist here yet, and it is raised as its own piece of work rather than quietly bolted onto a guide.

What to check on your own scheduler

1. count rows by status
   → is the failure count implausibly round?
2. count rows with no completion timestamp
   → the same number by a different route; if it is large, look at why
3. group those by command
   → background tasks and every-minute tasks will lead
4. find the oldest row still marked running
   → anything older than the longest task is not running

Step four is the one that turns a suspicion into a fact. A row that started two weeks ago and is still "in progress" is not in progress. Ours went back fifteen days.

How this was measured, and what is not ours

59,120 rows across 44 commands and three weeks of production, read directly from the table rather than from the dashboard that reports on it. Every figure is a count.

The diagnosis is not new work. The background-process mechanism was found and fixed by earlier work on this codebase and is documented in the recorder's own source, including the pre-fix production numbers quoted above. What this adds is the verification — whether the repair held, three weeks and forty-eight thousand runs later — and the observation that the residue accumulates because nothing is allowed to delete it.

A stuck row is not a failed task, and this table cannot distinguish them. Treating the 1,223 as failures would repeat the original error in the opposite direction: inventing certainty from a record that does not carry it.

This is one framework's scheduling model. The specific trap — completion hooks running in a different process from the hooks that started the work — is a property of how background scheduling is implemented here. The question to ask of your own system is whether your job recorder can express a failure at all, and the cheapest way to find out is to look for a zero that has no business being one.