feat(watch): post-cutover monitoring that flags broken redirects and hands back the fix

[itsdylon]

Closes claims #2 (post-cutover traffic and 404 monitoring with email alerts) and #3 (auto flag and fix).

Exporting a redirect file is a prediction. It gets deployed by a human into a stack Redirx never sees, and every common failure is invisible from our side at export time: the rules never shipped, a CDN rule shadowed them, a trailing-slash normaliser turned one hop into three, someone shipped 302. A watch asks the live site what it actually does with each approved old URL, ranks the failures by the traffic they cost, and hands back a corrective file.

The 026 traffic baseline was built for this — its own migration note says "monitoring and recovery reporting depend on it existing."

A real bug the unit tests could not see

Probing real sites during development caught this, and it would have made the feature useless:

http://github.com and http://google.com were both reported as redirect loops.

Loop detection reused normalize_for_compare, which deliberately ignores scheme and www. so an HSTS hop is not flagged as a wrong target on every URL of a site. Turned on loop detection, that same rule says http://x and https://x are the same URL — so every site doing http→https or bare→www, i.e. most of the web, came back broken.

The fix separates the two questions. normalize_for_compare answers "same page?"; visit_identity answers "same request?" and keeps scheme, host and trailing slash. A genuine /a → /a/ → /a oscillation is still caught. Both directions are regression-tested, at the unit level and against a live server.

Probing

• Redirects followed one hop at a time rather than via allow_redirects — the diagnosis lives in the shape of the chain: hop count, permanent vs temporary, and where it actually landed.
• Every hop re-validated against the SSRF rules. Following a redirect means letting another server choose our next URL, and Location: http://169.254.169.254/ is a thing a server is allowed to say.
• HEAD first, retried as GET on the statuses that mean "this server dislikes HEAD" rather than "this URL is broken".
• Own rate-limiter namespace, paced slower than discovery. Unlike a crawl this recurs forever: a rate a site tolerates once is one it will be asked to tolerate every few hours for months.

Reporting, without becoming noise

Issues are current state keyed by URL, not an event log. A redirect broken for a week is one row and one email, not seven of each. alerted_at gates re-reporting; a different failure at the same URL reopens it, because that is news. Transient types wait for a second consecutive sighting — one failed request is a blip.

Closing an issue requires having re-probed that URL. Absence from a sweep's findings is not evidence of repair: the per-sweep cap leaves a large site's tail unchecked, and treating that as fixed would silently close real breakage on exactly the sites that most need monitoring.

The fix half

Every proposed target resolves to something already justified — the approved mapping, or the destination the site itself proved reachable:

flagged	proposed
missing / 404 / wrong target	the approved mapping
extra hops	where the chain already ends, collapsed to one
302	same target, issued as a 301
loop / outage / 5xx	nothing

wrong_target proposes the approved mapping, not the URL the probe observed — proposing where it actually landed would ratify the bug. Rows with no actionable target are omitted rather than emitted blank, since a half-filled rule replaces a broken redirect with a differently-broken one.

fix_rows() shapes issues as old_url/new_url so redirect_export renders the correction in all eight existing formats. A fix the customer cannot deploy the way they deployed the original is not a fix.

Where it runs

The worker, not the API. Probing a few thousand URLs takes tens of minutes and the API is a single sync gunicorn worker where that would block every other request. One sweep at a time, running alongside jobs rather than taking a job slot, since a sweep is almost entirely idle time waiting on the rate limiter. A long lease instead of a lease-extension loop: nothing is waiting on the result, so a stale lease costs monitoring latency rather than a user-visible delay.

Agent surface

POST/GET /api/v1/migrations/{id}/watch and GET .../watch/fixes. The zero-touch story from #31 did not actually end at export — an agent that deploys a redirect file had no way to learn whether the deploy worked, and only the live site can answer that. checked is returned separately from the issue count so an agent polling right after starting a watch can distinguish "clean" from "not looked yet".

Tests

96 new backend, 5 frontend. The classifier's are mostly cases that must not be flagged — a monitor that cries wolf gets muted, and then it reports nothing. Hop following, HEAD fallback, canonical hops and loop termination are tested against a real local aiohttp server, with the SSRF guard asserted separately since it is what makes that server unreachable in production.

Backend 436 tests, frontend 285. Pre-existing backend failure count unchanged (3 failures, 15 errors — all the known local SUPABASE_KEY mismatch).

Migration 029 applied.

Not in this PR

• No plan gating. Pricing V3 (docs: outline the pricing V3 paywall move before building it #29) is still an open outline and Watch is priced there at ~$29/mo; this ships ungated so it can be used and measured first, and the gate is a decision rather than a missing line of code.
• Content fetching during a sweep still does not consult robots.txt, consistent with Deep Match today.

🤖 Generated with Claude Code