Skip to content

fix: drop the Alloy healthcheck that could only ever fail - #96

Merged
cnmaia merged 1 commit into
mainfrom
fix/alloy-healthcheck
Sep 25, 2026
Merged

cnmaia merged 1 commit into
mainfrom
fix/alloy-healthcheck

Conversation

@cnmaia

@cnmaia cnmaia commented Sep 25, 2026

Copy link
Copy Markdown
Contributor

Found while validating the deploy: datamap_alloy had been unhealthy for twenty hours.

exec: "wget": executable file not found in $PATH

The image carries a shell but no HTTP client, so the check was running a command that does not exist. It never had a chance to pass.

It was working the whole time

alloy      http://datamap_alloy:12345/metrics        up
gatekeeper http://datamap_gatekeeper:9095/metrics    up
gatekeeper http://datamap_gatekeeper_b:9095/metrics  up
loki       http://datamap_loki:3100/metrics          up
prometheus http://localhost:9090/metrics             up

And Loki holds the logs of all eleven datamap_* containers, and none of the three neighbouring projects' — so the discovery, the filter and the shipping are all fine.

Why I missed it

Loki had exactly this problem and I fixed it when its image turned out to be distroless. I did not check Alloy the same way: the local run only ever showed health: starting before I moved on to the next thing, so the check never had time to fail in front of me.

The lesson is the narrow one — health: starting is not healthy, and waiting for the difference is the whole point of looking.

What replaces it

Nothing new. The Prometheus scrape was already the real signal, and TargetDown was already the alert — which is why a container reporting unhealthy for a day changed nothing about whether the logs arrived.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EQda9NZvkbStEeNU54Tqgh

It reported unhealthy for twenty hours while working perfectly: Prometheus
scraped it the whole time and Loki holds the logs of all eleven datamap_
containers. The image carries a shell but no http client, so the check was
running a command that does not exist.

Loki had the same problem and was fixed when its image turned out to be
distroless; Alloy was not checked the same way, because the local test only
ever saw it as `health: starting`.

The scrape is the signal, and TargetDown is the alert.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQda9NZvkbStEeNU54Tqgh
@cnmaia
cnmaia requested a review from andrenmaia as a code owner September 25, 2026 20:34
@cnmaia
cnmaia merged commit d9fb0e7 into main Sep 25, 2026
3 checks passed
@cnmaia
cnmaia deleted the fix/alloy-healthcheck branch September 25, 2026 20:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant