fix(site-update): Make site update recovery resilient and recoverable

[balamurali27]

Problem

When a site update (Pull/Migrate onto a new bench) fails, recovery could end up Fatal with no automatic path back and no clear signal to the user:

• Recovery migrate queries on large sites could exceed the database server's max_statement_time and get killed mid-query, turning a recoverable failure into a fatal one.
• A migrate recovery that failed mid table-restore had no automatic follow-up — the site was left Broken even though its tables could still be restored from the backup. Re-running the full recovery doesn't work: the agent's move_site is not idempotent (the site has already been moved back), so it fails at "Move Site".
• An update run with backups skipped that failed gave a generic notification, leaving the user unsure it needs manual intervention and how to do it.

Changes

max_statement_time bump before a recovery migrate (and revert after)

• Before a recovery migrate on a large database (over LARGE_DATABASE_SIZE, 2 GB), bump max_statement_time by an hour (dynamic MariaDB variable, no restart), recorded as a comment on the Site Update. Smaller databases finish well within the timeout and are skipped.
• The pre-bump value is stashed on the Site Update (previous_max_statement_time) and restored once recovery finishes — at the recover job's terminal state, or, when the fallback table restore runs, at that job's callback. Without this the value ratcheted up an hour on every recovery.

One-shot Restore Site Tables fallback when a migrate recovery fails — but only when safe

Press makes one automatic attempt to bring the site back up, only when all hold:

• it was a Recover Failed Site Migrate;
• its "Move Site" step succeeded (so the site is back on the source bench — otherwise restoring tables would target the wrong bench); and
• it failed due to a transient DB error (MySQL server has gone away / Lost connection to MySQL server, detected from the job output/traceback and step output). Other failures are genuine problems left Fatal for manual attention.

Since the failed recovery has already moved the site back, only the table restore is left undone, so Press re-issues just that job (linked on the Site Update as a comment for traceability).

On a successful fallback Restore Site Tables, the update stays Fatal

The site becomes Active again and the fatal update's cause of failure is marked resolved (set_cause_of_failure_is_resolved, which also clears the site's fatal_site_update). The update itself still failed and is recorded as such, rather than being silently flipped to Recovered.

Actionable notification for skipped-backups failures

Replaces the generic message with one telling the user the site can't be recovered automatically, linking to the SSH docs to fix it manually. (Skipped-backups failures go straight to Fatal — no recovery, no table restore, since there's no backup.)

Notes

Site Usage stores sizes in MB, not bytes (despite unitless Int fields) — so the 2 GB threshold is 2048, not 2048**3. Documented in a new Site Usage README.

While here, all agent-job failure notification messages were flattened to single lines via implicit string concatenation — the fix dialog renders them with whitespace-pre-wrap, so source indentation and mid-paragraph line breaks were showing as literal whitespace.

Docs

• docs/code/site-update/index.md — the recovery flow, status lifecycle, the transient-only restore fallback, the max_statement_time bump/revert, and the constants.
• press/press/doctype/site_update/README.md — Fatal-state behavior with the transient-error fallback.
• press/press/doctype/site_usage/README.md — the MB-not-bytes gotcha.
• Removed the redundant top-level guide-to-testing.md in favour of docs/code/testing/.

Tests

bench --site <site> run-tests --app press --module press.press.doctype.site_update.test_site_update — passing, including:

• a transient-error migrate recovery triggers Restore Site Tables and, on success, leaves the update Fatal with its cause of failure resolved and the site Active
• a transient-error recovery followed by a failed table restore goes Fatal with fatal_site_update set
• a non-transient recovery failure does not restore tables (stays Fatal)
• a recovery that fails before "Move Site" does not restore tables
• a successful recovery does not restore tables (stays Recovered)
• a skipped-backups update failure goes straight to Fatal with no recovery/restore, and surfaces the actionable SSH notification
• the recovery-migrate max_statement_time bump is stashed and restored to its pre-bump value
• increase_max_statement_time bumps the DB server variable by an hour; Site.database_size returns the latest Site Usage value (in MB)

Also verified end-to-end on a local Vagrant Frappe Cloud: real Update Site Migrate → Recover Failed Site Migrate (killed mid-restore) → Restore Site Tables success, ending Fatal with cause resolved and the site Active.

🤖 Generated with Claude Code

[greptile-apps]

Confidence Score: 3/5

The recovery fallback logic is well-tested and correctly gated, but a new Delivery Failure mapping in process_restore_tables_job_update clears database_name when the restore never ran, and several edge cases in the max_statement_time bump/restore path remain open from prior review rounds.

The Delivery Failure branch introduced here reaches frappe.db.set_value("Site", job.site, "database_name", None) even when the restore job never touched the agent, corrupting the site's database-name record while leaving the actual database intact. Combined with unresolved concerns from earlier iterations around the max_statement_time zero-value sentinel and synchronous Ansible failures propagating out of trigger_recovery_job, the recovery path has multiple failure modes that leave sites broken with no automatic path back.

press/press/doctype/site/site.py (process_restore_tables_job_update Delivery Failure branch and the max_statement_time zero-value guard) and press/press/doctype/site_update/site_update.py (bump_max_statement_time_before_recovery exception propagation)

Important Files Changed

Filename	Overview
press/press/doctype/site_update/site_update.py	Core recovery orchestration; adds max_statement_time bump/restore, the one-shot Restore Site Tables fallback, and transient-error detection — several edge cases remain from prior review rounds
press/press/doctype/site/site.py	Adds database_size property, increase_max_statement_time, set_max_statement_time, and updates process_restore_tables_job_update to handle fatal_update; zero-value and exception-propagation edge cases from prior rounds remain open
press/press/doctype/agent_job/agent_job_notifications.py	Adds skipped-backups actionable notification and flattens multi-line message strings; logic is correct, i18n gap noted in prior thread
press/press/doctype/site_update/test_site_update.py	Comprehensive new tests covering all major recovery scenarios including transient-error fallback, Move Site guard, non-transient no-op, skipped-backups notification, and max_statement_time bump/restore
press/press/doctype/site_update/site_update.json	Adds previous_max_statement_time Int field with no_copy and read_only; field definition is correct

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    UF[Update Job Fails] --> SB{Backups skipped?}
    SB -- Yes --> FT[Fatal + SSH notification]
    SB -- No --> TR[trigger_recovery_job]
    TR --> bump{Large DB AND deploy_type=Migrate?}
    bump -- Yes --> BMP[bump max_statement_time store previous_max_statement_time]
    bump -- No --> RJOB
    BMP --> RJOB[Recover Failed Site job]
    RJOB --> RES{Recovery result}
    RES -- Success --> REC[Recovered restore max_statement_time]
    RES -- Failure --> FAT[Fatal site Broken fatal_site_update set]
    FAT --> COND{Recover Failed Site Migrate + Move Site=Success + transient DB error?}
    COND -- No --> LEAVE[Leave Fatal restore max_statement_time]
    COND -- Yes --> RST[Restore Site Tables job keep max_statement_time elevated]
    RST --> RSTRES{Restore result}
    RSTRES -- Success --> ACT[Site Active update stays Fatal cause_of_failure_is_resolved restore max_statement_time]
    RSTRES -- Failure --> BRK[Site Broken fatal_site_update remains restore max_statement_time]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    UF[Update Job Fails] --> SB{Backups skipped?}
    SB -- Yes --> FT[Fatal + SSH notification]
    SB -- No --> TR[trigger_recovery_job]
    TR --> bump{Large DB AND deploy_type=Migrate?}
    bump -- Yes --> BMP[bump max_statement_time store previous_max_statement_time]
    bump -- No --> RJOB
    BMP --> RJOB[Recover Failed Site job]
    RJOB --> RES{Recovery result}
    RES -- Success --> REC[Recovered restore max_statement_time]
    RES -- Failure --> FAT[Fatal site Broken fatal_site_update set]
    FAT --> COND{Recover Failed Site Migrate + Move Site=Success + transient DB error?}
    COND -- No --> LEAVE[Leave Fatal restore max_statement_time]
    COND -- Yes --> RST[Restore Site Tables job keep max_statement_time elevated]
    RST --> RSTRES{Restore result}
    RSTRES -- Success --> ACT[Site Active update stays Fatal cause_of_failure_is_resolved restore max_statement_time]
    RSTRES -- Failure --> BRK[Site Broken fatal_site_update remains restore max_statement_time]

Loading

Prompt To Fix All With AI

Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
press/press/doctype/site/site.py:5028
**`database_name` cleared on `"Delivery Failure"` even though nothing ran**

For `"Delivery Failure"` the `Restore Site Tables` job was never delivered to the agent — the database was never touched. The else branch still runs `frappe.db.set_value("Site", job.site, "database_name", None)`, which wipes the site's database name even though the database exists and is intact. The site is now `Broken` and trying to reconnect to the database would fail because the name record is gone. For `"Failure"` this clearing is arguably intentional (partial restore, unknown state); for `"Delivery Failure"` it introduces corruption that wouldn't have existed before this PR added the mapping entry.

_{Reviews (13): Last reviewed commit: "fix(site): Handle Delivery Failure in re..." | Re-trigger Greptile}

[codecov-commenter]

Codecov Report

❌ Patch coverage is 94.71545% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 50.91%. Comparing base (b56a7c7) to head (5c6fc7c).
⚠️ Report is 24 commits behind head on develop.

Files with missing lines	Patch %	Lines
...press/doctype/agent_job/agent_job_notifications.py	60.00%	8 Missing ⚠️
press/press/doctype/site/site.py	91.17%	3 Missing ⚠️
press/press/doctype/site_update/site_update.py	97.05%	1 Missing ⚠️
...ress/press/doctype/site_update/test_site_update.py	99.36%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           develop    #6729       +/-   ##
============================================
- Coverage    62.94%   50.91%   -12.03%     
============================================
  Files          117      995      +878     
  Lines        18110    84143    +66033     
  Branches       527      527               
============================================
+ Hits         11399    42843    +31444     
- Misses        6678    41267    +34589     
  Partials        33       33               

Flag	Coverage Δ
dashboard	62.94% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[greptile-apps]

database_size silently returns 0 when no Site Usage row exists yet, causing the > LARGE_DATABASE_SIZE guard in trigger_recovery_job to be False for any site without usage history. A large site whose usage was never recorded will have max_statement_time skipped, re-exposing the exact timeout risk this feature aims to prevent. Returning None and handling the sentinel explicitly in the caller makes the "no data" case visible rather than silently treating it as "small".

Suggested change

	return (
	frappe.db.get_value("Site Usage", {"site": self.name}, "database", order_by="creation desc") or 0
	)
	return (
	frappe.db.get_value("Site Usage", {"site": self.name}, "database", order_by="creation desc") or None
	)

Prompt To Fix With AI

This is a comment left during a code review.
Path: press/press/doctype/site/site.py
Line: 1184-1186

Comment:
`database_size` silently returns `0` when no Site Usage row exists yet, causing the `> LARGE_DATABASE_SIZE` guard in `trigger_recovery_job` to be `False` for any site without usage history. A large site whose usage was never recorded will have `max_statement_time` skipped, re-exposing the exact timeout risk this feature aims to prevent. Returning `None` and handling the sentinel explicitly in the caller makes the "no data" case visible rather than silently treating it as "small".

```suggestion
		return (
			frappe.db.get_value("Site Usage", {"site": self.name}, "database", order_by="creation desc") or None
		)
```

How can I resolve this? If you propose a fix, please make it concise.

[balamurali27]

Note for reviewers — review the net diff, not the commit history.

The early commits of this PR (roughly 7ff506d9…b5365565) were over-ambitious and no longer reflect the final design — please don't review them commit-by-commit.

They built a full retry mechanism around recovery: up to MAX_RECOVERY_RETRIES (3) recover-job retries on transient DB errors, a statement-timeout retry loop around restore tables, flipping the update to Recovered after a restore, etc. That machinery was later torn back out in favour of a minimal one-shot fallback: when a migrate recovery fails after moving the site back due to a transient DB error, re-issue just Restore Site Tables once; everything else stays Fatal. The max_statement_time bump survived (now also reverted after recovery), and the threshold moved to 2 GB.

So most of those starting commits are effectively superseded. The parts that carried through unchanged are the tests (the scaffolding/helpers in test_site_update.py) and the SSH/skipped-backups notification. The current behaviour is best read from the PR description and the final state of the files, not the intermediate commits.

[greptile-apps]

Unconditional fatal_site_update clear replaced by error-prone conditional

The old code always ran frappe.db.set_value("Site", job.site, "fatal_site_update", None) after reset_previous_status. Now the clear depends on set_cause_of_failure_is_resolved() completing successfully. If frappe.get_doc("Site Update", fatal_update) raises (e.g. the SiteUpdate was deleted) or restore_max_statement_time() raises (e.g. the database server is unreachable), the site will be left Active (status already persisted by reset_previous_status → self.save()) with fatal_site_update still set and cause_of_failure_is_resolved still 0. Adding a try/finally that calls frappe.db.set_value("Site", job.site, "fatal_site_update", None) — mirroring the old unconditional clear — would prevent this inconsistency.

Prompt To Fix With AI

This is a comment left during a code review.
Path: press/press/doctype/site/site.py
Line: 5029-5038

Comment:
**Unconditional `fatal_site_update` clear replaced by error-prone conditional**

The old code always ran `frappe.db.set_value("Site", job.site, "fatal_site_update", None)` after `reset_previous_status`. Now the clear depends on `set_cause_of_failure_is_resolved()` completing successfully. If `frappe.get_doc("Site Update", fatal_update)` raises (e.g. the SiteUpdate was deleted) or `restore_max_statement_time()` raises (e.g. the database server is unreachable), the site will be left `Active` (status already persisted by `reset_previous_status` → `self.save()`) with `fatal_site_update` still set and `cause_of_failure_is_resolved` still `0`. Adding a `try/finally` that calls `frappe.db.set_value("Site", job.site, "fatal_site_update", None)` — mirroring the old unconditional clear — would prevent this inconsistency.

How can I resolve this? If you propose a fix, please make it concise.

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[greptile-apps]

restore_max_statement_time is silently a no-op when the DB server's max_statement_time was already 0 (MariaDB's "no limit" sentinel). increase_max_statement_time calls int(float(current_timeout)) if current_timeout else DEFAULT_MAX_STATEMENT_TIME, so a stored value of "0" yields old_timeout = 0, which is then stored as previous_max_statement_time. The guard if not self.previous_max_statement_time: treats 0 as falsy and returns early without restoring, permanently leaving the DB server at 3600 s after the recovery. Change the clear value to None and guard with is None so the "never bumped" and "bumped from 0" cases are distinguishable.

Suggested change

	def restore_max_statement_time(self) -> None:
	# No-op unless a recovery migrate bumped it (see bump_max_statement_time_before_recovery).
	if not self.previous_max_statement_time:
	return
	frappe.get_doc("Site", self.site).set_max_statement_time(self.previous_max_statement_time)
	self.db_set("previous_max_statement_time", 0)
	def restore_max_statement_time(self) -> None:
	# No-op unless a recovery migrate bumped it (see bump_max_statement_time_before_recovery).
	if self.previous_max_statement_time is None:
	return
	frappe.get_doc("Site", self.site).set_max_statement_time(self.previous_max_statement_time)
	self.db_set("previous_max_statement_time", None)

Prompt To Fix With AI

This is a comment left during a code review.
Path: press/press/doctype/site_update/site_update.py
Line: 561-566

Comment:
`restore_max_statement_time` is silently a no-op when the DB server's `max_statement_time` was already `0` (MariaDB's "no limit" sentinel). `increase_max_statement_time` calls `int(float(current_timeout)) if current_timeout else DEFAULT_MAX_STATEMENT_TIME`, so a stored value of `"0"` yields `old_timeout = 0`, which is then stored as `previous_max_statement_time`. The guard `if not self.previous_max_statement_time:` treats `0` as falsy and returns early without restoring, permanently leaving the DB server at 3600 s after the recovery. Change the clear value to `None` and guard with `is None` so the "never bumped" and "bumped from 0" cases are distinguishable.

```suggestion
	def restore_max_statement_time(self) -> None:
		# No-op unless a recovery migrate bumped it (see bump_max_statement_time_before_recovery).
		if self.previous_max_statement_time is None:
			return
		frappe.get_doc("Site", self.site).set_max_statement_time(self.previous_max_statement_time)
		self.db_set("previous_max_statement_time", None)
```

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

Timeout-bump failure silently blocks all recovery

bump_max_statement_time_before_recovery calls set_max_statement_time → add_or_update_mariadb_variable(update_variables_synchronously=True), which runs Ansible synchronously. If the database server is temporarily unreachable or Ansible fails (which is plausible exactly when an update has just failed), the exception propagates out of trigger_recovery_job before the recovery AgentJob is ever created. The site is left Broken with no recovery path and no user-visible signal — the opposite of what this PR aims to fix. Wrap the bump in a try/except so a bump failure is logged but doesn't abort recovery.

Prompt To Fix With AI

This is a comment left during a code review.
Path: press/press/doctype/site_update/site_update.py
Line: 631

Comment:
**Timeout-bump failure silently blocks all recovery**

`bump_max_statement_time_before_recovery` calls `set_max_statement_time` → `add_or_update_mariadb_variable(update_variables_synchronously=True)`, which runs Ansible synchronously. If the database server is temporarily unreachable or Ansible fails (which is plausible exactly when an update has just failed), the exception propagates out of `trigger_recovery_job` before the recovery `AgentJob` is ever created. The site is left `Broken` with no recovery path and no user-visible signal — the opposite of what this PR aims to fix. Wrap the bump in a `try/except` so a bump failure is logged but doesn't abort recovery.

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

database_name cleared on "Delivery Failure" even though nothing ran

For "Delivery Failure" the Restore Site Tables job was never delivered to the agent — the database was never touched. The else branch still runs frappe.db.set_value("Site", job.site, "database_name", None), which wipes the site's database name even though the database exists and is intact. The site is now Broken and trying to reconnect to the database would fail because the name record is gone. For "Failure" this clearing is arguably intentional (partial restore, unknown state); for "Delivery Failure" it introduces corruption that wouldn't have existed before this PR added the mapping entry.

Prompt To Fix With AI

This is a comment left during a code review.
Path: press/press/doctype/site/site.py
Line: 5028

Comment:
**`database_name` cleared on `"Delivery Failure"` even though nothing ran**

For `"Delivery Failure"` the `Restore Site Tables` job was never delivered to the agent — the database was never touched. The else branch still runs `frappe.db.set_value("Site", job.site, "database_name", None)`, which wipes the site's database name even though the database exists and is intact. The site is now `Broken` and trying to reconnect to the database would fail because the name record is gone. For `"Failure"` this clearing is arguably intentional (partial restore, unknown state); for `"Delivery Failure"` it introduces corruption that wouldn't have existed before this PR added the mapping entry.

How can I resolve this? If you propose a fix, please make it concise.