A green health check implies a working system. That is an inference, and it is only ever as good as the check itself.

To test ours, we stopped each of the stack's dependencies in turn — database, cache, search, queue worker — and for each outage recorded what the health check said, what the site did, and what a reader would actually have seen. Two of the four outages left every indicator green. One produced the opposite problem, which turns out to be more interesting.

What the check covers

Ours is an ordinary readiness probe, and a reasonable one. It opens a database connection, confirms the log directory is writable, and pings Redis:

{
  "status": "ok",
  "checks": {
    "database": "ok",
    "storage":  "ok",
    "redis":    "ok"
  }
}

Three dependencies. The stack has six. Search and the queue worker are not in the list — not because anyone decided they did not matter, but because the list was written once and the stack grew afterwards. That is the ordinary way this happens.

The matrix

stopped        health   homepage   search links   jobs left
───────────────────────────────────────────────────────────
(nothing)         200        200            12          0
database          503        200             0          1
redis             503        500             0          —
meilisearch       200        200             0          0
queue-worker      200        200            12          1

"Search links" counts links to real content on the results page, with the site's own furniture excluded. Redis has no job count because reading the queue depth needs Redis.

Read the last two rows first. Search died and the health check said 200. The queue worker died and the health check said 200. In both cases the homepage was fine, the status code was fine, and a monitor polling that endpoint would have reported an unbroken green line through the whole outage.

The search outage looks exactly like an empty search

The search outage is the one worth a closer look. With the search engine stopped, the search page did not error. It returned HTTP 200 and rendered its ordinary empty state:

engine up:    "31 results", 12 links to content
engine down:  "No results", 0 links to content

The four links still on the page were the tool-category links that appear in the furniture of every page, working and broken alike — which is worth knowing if you are tempted to detect this by counting links.

A user typing a query got the same page they would have got for a genuinely unmatched term. They had no way to tell, the page had no way to tell them, and nothing in the system recorded that anything had happened. A dependency outage was rendered as an ordinary negative result. This is the most expensive kind of silence: the feature is not down, it is quietly answering wrong.

The queue is worse, because nothing is even wrong yet

With the worker stopped, a dispatched job simply stayed in the queue. Nothing failed. No page changed. The health check was green because it only checks three things, and all three were fine.

The consequences are all downstream and delayed: emails never sent, images never fetched, the search index going stale. By the time somebody notices, the symptom is a missing side effect hours old, and the queue depth that would have explained it in one number was never recorded.

A worker that has stopped consuming is not a subtle condition. It is one integer, available at any moment, that nothing was asking for.

The database outage inverts the problem

Now the row that surprised us. With the database stopped the health check correctly returned 503 — and the homepage returned 200.

Not a working homepage. The response was 85,587 bytes against a normal 160,490, and it contained none of its sixteen article links. Header, navigation, footer, and a hole where the site goes.

The readiness probe and the uptime monitor disagreed, and both were right about different things. The probe said the machine was not fit to serve. The monitor, watching for a success code, saw a success code. If your alerting rests on status codes — and a great deal of alerting does — this outage is invisible in exactly the place people look first.

And the ordering is not what you would guess

Losing the database left the homepage answering 200. Losing Redis returned 500 outright.

Most people would rank the database as the more critical of the two. The order follows from configuration — where sessions and cache live — rather than from any general sense of importance. The framework can render a page with no data; it cannot render one with nowhere to put a session. Which dependency takes your site down is a fact about your configuration, not about the dependency, and the only way to know your ordering is to stop things and look.

What to do about it

The fix is not a longer list of pings, because a ping tells you a port is open. Check the thing you actually need:

search    run a known query, assert it returns results
queue     read the queue depth and the worker's last heartbeat
cache     write a key and read it back
database  run a query against a table you depend on

Then separate two ideas that get conflated into one endpoint. Readiness answers "should traffic come here" and belongs to the load balancer; it should fail closed and fail fast. Health answers "is anything degraded" and belongs to your alerting; it can report a partial state without pulling a node out of rotation. Our search outage is a health problem and not a readiness one — the node was perfectly able to serve, it just could not answer questions — and an endpoint with one boolean has nowhere to say that.

How this was measured, and what it does not cover

One Laravel stack in Docker — app, MySQL 8, Redis, Meilisearch, a queue worker and a scheduler — on a development machine. Each service was stopped with docker compose stop, probed, and restarted; the harness restores every service on any exit path. Every figure is an observation from that run, and the stack was verified back to baseline afterwards (health 200, 12 search links, empty queue).

These are one stack's results, not a universal rule. Which dependency takes your homepage down depends on where your sessions live, what your cache holds, and how much your pages tolerate missing data. The method transfers; the matrix does not.

Stopping a container is a clean failure. Real outages are frequently worse: a database that accepts connections but answers slowly, a search engine returning stale results, a worker that is running but wedged. Those are harder to detect and this drill does not simulate them. It covers the clean failure, which is the one currently going unnoticed.

We did not test recovery. Every service came back and the site returned to baseline, but the time from restart to genuinely serving was not measured here, and the gap between a green probe and a working site deserves its own clock.

The homepage that returned 200 with nothing in it is not only an outage symptom. It is a shape a page can have at any time, and finding those is a separate exercise.