A health check answers only the question it was written for. If that isn’t the question the application needs answered, the check can be green while the application fails on its first connection.
Here is a fictional example. A containerized web application depends on a database container. The database’s health check runs a command inside the database container that asks the server whether it is ready, and it reports healthy. The application starts, runs its migrations, and fails immediately with “connection refused”.
| What was checked | Where it ran | Result |
|---|---|---|
| Database server answers a local command | Inside the database container | Healthy |
| Application opens a network connection to the database | From the application container | Refused |
Both results can be true at the same time. Many database images start a temporary server during first-time initialization, one that accepts only local connections, and then restart it to accept network connections. A check that talks to the local server can pass during that first phase, while the application, which connects over the network, has no server to reach. The startup order looks correct. The database is “ready”, just not for the connection the application actually makes.
When this happens, it’s tempting to blame the migrations, since that’s the step that failed. The logs say otherwise, if you read them in order. The migration tool never ran a statement; it couldn’t open a connection. That distinction tells you where to look: the path between the containers, not the migration files.
So the useful sequence is:
- Collect the logs from both containers, with timestamps, before changing anything.
- Find the first failed connection and what the database was doing at that moment.
- Compare the path the health check uses with the path the application uses (a local socket versus a network address, for example).
- Change the check to use the application’s path, such as a TCP connection to the database’s network address, or have the application retry its connection for a short, bounded time.
The first fix makes the signal truthful, and the second makes the application tolerant of a slow start. They solve different problems, and the second doesn’t excuse a misleading check. Neither should be bundled into an unrelated feature change, because then a startup problem and a feature end up in the same review.
This is a diagnosis that fits the symptoms, not a proof that every connection-refused failure is a startup race. Reproduce it in your own environment before you trust it.
A health check is a claim about the system. Just make sure it’s checking the thing the next service depends on.
—jhunterj