CBG-5496: Mark background process state as error on synchronous start failure

[RIT3shSapata]

CBG-5496

Fixes a bug where passing invalid collection names to POST /db/_resync?action=start left the background process permanently stuck reporting running, even though it had actually failed. Since the process was never marked as errored, subsequent _resync calls (start or otherwise) kept getting rejected with "already in progress" until the underlying heartbeat/status state was manually cleared.

• BackgroundManager now transitions to the Error state when Process.Init fails synchronously (e.g. invalid collections), instead of being left stuck on the Running state that markStart had already set before Init ran.
• BackgroundManager also transitions to Error if the initial cluster status persist fails right after Init succeeds - by that point Process.Run is already executing in its own goroutine, so without this the caller would see an error while the process kept running unbeknownst to them.
• Previously, Stop() had no concept of an Error state to recover from — markStop treated Error the same as any other terminal state (Completed/Stopped) and returned early as a no-op, so a process stuck in Error had no path back to Stopped at all; only a fresh Start() (which markStart does allow from Error) could move it anywhere.
• Added StopFromErrorState, and Stop() now routes into it whenever the process is in the Error state. It exists because the normal Running → Stopping → Stopped progression only happens after Process.Run returns (handled in the wrapper goroutine in start()) — but several error paths (like Init failing) mean Run is never invoked in the first place, so that progression never has a chance to fire. StopFromErrorState fills that gap by explicitly moving the process straight to Stopped, updating the cluster-aware status doc so other nodes see the change, and cleaning up the heartbeat doc so a subsequent Start() isn't blocked.
• Behavior change: because Error never resolved to Stopped on its own (that transition normally only happens after Process.Run returns, which never happens if Init failed), a resync that ends up in the Error state now must be explicitly stopped via _resync?action=stop before a new _resync?action=start will succeed. Previously this same restriction existed too, just as an unintended side effect of the stuck state rather than a designed recovery path.
• rest/api.go's resync stop handler now captures the status before calling Stop() and resets the database-level DBResyncing state back to DBOffline if that prior status was Error, since the normal reset (a deferred call inside ResyncManagerDCP.Run) never fires when Run was never invoked.
• Added unit test coverage in db/background_mgr_test.go:
  • Init failing across all three BackgroundManager modes (local, single-node cluster-aware, multi-node cluster-aware), followed by Stop.
  • The initial status persist failing after Process.Run has already been launched.

Pre-review checklist

• Removed debug logging (fmt.Print, log.Print, ...)
• Logging sensitive data? Make sure it's tagged (e.g. base.UD(docID), base.MD(dbName))
• Updated relevant information in the API specifications (such as endpoint descriptions, schemas, ...) in docs/api

Dependencies (if applicable)

• Link upstream PRs
• Update Go module dependencies when merged

Integration Tests

• https://jenkins.sgwdev.com/job/SyncGatewayIntegration/942/

[Copilot]

Pull request overview

This PR fixes a resync background-process lifecycle bug where synchronous start failures (e.g. invalid collection filters) could leave the resync process/state stuck as “running”, blocking subsequent resync operations until manual cleanup. It extends BackgroundManager error-state handling and adds test coverage for the affected failure modes.

Changes:

• Mark BackgroundManager as Error when Process.Init fails synchronously, and also when initial cluster-status persistence fails after Run has already started.
• Add an Error-recovery stop path (StopFromErrorState) so _resync?action=stop can reliably clear errored processes.
• Add unit/integration tests covering init failures and cluster-status persistence failures for resync/background manager.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
db/background_mgr.go	Updates start/stop state transitions to correctly enter Error on synchronous init/persist failures and adds stop recovery from Error.
rest/api.go	Adjusts _resync?action=stop handler to reset DB state when stopping an error-state resync that never reached Run().
db/background_mgr_test.go	Adds unit tests for init-error behavior across manager modes and for persist-failure-after-run-start behavior.
rest/adminapitest/resync_test.go	Adds admin API test verifying _resync behavior when started with invalid collection names and subsequent stop/restart flow.

[torcolvin]

I think something like this might be true - I think you can write a test for this scenario.

I propose changing this to something like:

		// markStart already set state to Running (and, for single-node mode, registered a heartbeat)
		// before Init ran, but the process never actually started - Process.Run is never invoked in this
		// path. Reset back to the uninitialized status (as if Start had never been called) and report the
		// failure directly to the caller, rather than leaving the manager in the Error state where it
		// would need an explicit Stop to become startable again.
		b.Terminate()
		b.clearStatus()
		if b.mode() == backgroundManagerModeSingleNode {
			_ = b.clusterAwareOptions.metadataStore.Delete(ctx, b.clusterAwareOptions.HeartbeatDocID())
		}

where:

// clearStatus resets the in-memory status back to its uninitialized zero value, e.g. to undo an aborted start attempt.
func (b *BackgroundManager[O]) clearStatus() {
	b.statusLock.Lock()
	defer b.statusLock.Unlock()
	b.status = BackgroundManagerStatus{}
}

The idea comes from the comment below where I don't think we want to require stop to be called or change any of hte start code. I think with this change, all of the other changed code around Stop and api.go can be dropped.

[torcolvin]

This comment raised the idea that the original idea I had was wrong for how to fix this, I do not like this being the way thing works.

I think a better way of things working would be.

1. /db/_resync?action=start with invalid collections -> (optimally this would return 400, but this can be out of scope, I'm not sure that all situations warrant returning 400 if Init fails, we could pull this into a separate PR)
2. /db/_resync?action=start with invalid collections runs

It would be non intuitive that you have to stop a resync that was never running in the first place.