fix(watchdog): address PR review on metrics and devnet stack

docs/watchdog/README.md

@@ -168,9 +168,11 @@ then `head.json` is atomically replaced to point at it.
 
 `init` stores the operator-provided bootstrap CM snapshot into this layout. `tick`
 requires both `config.json` and `head.json`; it never bootstraps from env.
-`CARTESI_WATCHDOG_BLOCKCHAIN_HTTP_ENDPOINT` is intentionally read at tick time, not persisted in
-`config.json`, so operators can rotate RPC endpoints without rewriting watchdog
-state.
+`CARTESI_WATCHDOG_BLOCKCHAIN_HTTP_ENDPOINT` is not persisted in `config.json`, so
+operators can rotate RPC endpoints without rewriting watchdog state. It is
+required at `tick` for L1 reads, and optionally present at `init` when
+auto-detecting `CARTESI_WATCHDOG_BLOCKCHAIN_ID` via `eth_chainId` (prefer setting
+the chain id explicitly).
 
 - `CARTESI_WATCHDOG_CM_SNAPSHOT_DIR`
 - `CARTESI_WATCHDOG_CM_SNAPSHOT_SAFE_BLOCK`
@@ -202,9 +204,9 @@ host scheduling should provide the same non-overlap guarantee. Each tick:
 
 Runtime knobs:
 
-- `CARTESI_WATCHDOG_BLOCKCHAIN_HTTP_ENDPOINT`: current L1 JSON-RPC endpoint for tick.
+- `CARTESI_WATCHDOG_BLOCKCHAIN_HTTP_ENDPOINT`: current L1 JSON-RPC endpoint for tick (and optional at `init` for chain-id auto-detect).
 - `CARTESI_WATCHDOG_SEQUENCER_URL`: optional tick-time override of the URL persisted at `init` (useful when ephemeral ports change).
-- `CARTESI_WATCHDOG_BLOCKCHAIN_ID`: optional chain id label persisted at `init` for `status.prom`.
+- `CARTESI_WATCHDOG_BLOCKCHAIN_ID`: optional chain id label persisted at `init` for `status.prom` (prefer explicit; tick never queries `eth_chainId`).
 - `CARTESI_WATCHDOG_METRICS_FILE`: optional override for the Prometheus textfile path (default `$CARTESI_WATCHDOG_STATE_DIR/status.prom`).
 - `CARTESI_WATCHDOG_RETRY_ATTEMPTS`: bounded retry attempts per run, default `3`.
 - `CARTESI_WATCHDOG_RETRY_DELAY_SEC`: delay between retry attempts, default `5`.
@@ -230,18 +232,19 @@ Exit codes map to `state` only (`0→ok`, `1→warning`, `2→failed`); we do no
 export a separate exit-code or last-tick gauge — Prometheus scrape/push already
 carries a sample timestamp.
 
-Set `CARTESI_WATCHDOG_BLOCKCHAIN_ID` at `init` for the `chain` label. At `tick`,
-the watchdog also accepts that env var or queries `eth_chainId` from
-`CARTESI_WATCHDOG_BLOCKCHAIN_HTTP_ENDPOINT` when unset (defaults to `unknown`
-only when the RPC is unavailable). Golden fixtures: [`tests/fixtures/watchdog_status_ok.prom`](../../tests/fixtures/watchdog_status_ok.prom),
+Set `CARTESI_WATCHDOG_BLOCKCHAIN_ID` at `init` for the `chain` label. If unset,
+`init` queries `eth_chainId` from `CARTESI_WATCHDOG_BLOCKCHAIN_HTTP_ENDPOINT` and
+persists the result. At `tick`, the env var overrides a missing persisted value;
+the exit path never blocks on RPC (defaults to `unknown` only when neither source
+is set). Golden fixtures: [`tests/fixtures/watchdog_status_ok.prom`](../../tests/fixtures/watchdog_status_ok.prom),
 [`tests/fixtures/watchdog_status_failed.prom`](../../tests/fixtures/watchdog_status_failed.prom).
 
 Example after a clean tick:
 
 ```prometheus
-cartesi_watchdog_status{chain="11155111",app_address="0x4CE...",state="ok"} 1
-cartesi_watchdog_status{chain="11155111",app_address="0x4CE...",state="warning"} 0
-cartesi_watchdog_status{chain="11155111",app_address="0x4CE...",state="failed"} 0
+cartesi_watchdog_status{app_address="0x4CE...",chain="11155111",state="ok"} 1
+cartesi_watchdog_status{app_address="0x4CE...",chain="11155111",state="warning"} 0
+cartesi_watchdog_status{app_address="0x4CE...",chain="11155111",state="failed"} 0
 ```
 
 Example Prometheus alert (pull or push gateway — operator choice):

docs/watchdog/getting-started.md

@@ -174,8 +174,8 @@ Full operator runbook: **[`operator-deployment.md`](operator-deployment.md)**.
 | Variable | Required | Description |
 |----------|----------|-------------|
 | `CARTESI_WATCHDOG_SEQUENCER_URL` | yes (also tick override) | e.g. `http://127.0.0.1:54321`; tick-time env overrides persisted URL |
-| `CARTESI_WATCHDOG_BLOCKCHAIN_HTTP_ENDPOINT` | tick | Current L1 JSON-RPC; not persisted by `init` |
-| `CARTESI_WATCHDOG_BLOCKCHAIN_ID` | init / tick | Chain id label for `status.prom`; also read at tick from env or `eth_chainId` when unset |
+| `CARTESI_WATCHDOG_BLOCKCHAIN_HTTP_ENDPOINT` | tick (also init for chain auto-detect) | Current L1 JSON-RPC; not persisted by `init`. Optional at `init` only when auto-detecting `BLOCKCHAIN_ID` via `eth_chainId` |
+| `CARTESI_WATCHDOG_BLOCKCHAIN_ID` | init / tick | Chain id label for `status.prom`. Prefer setting at `init` (persisted). Tick may override from env; never queries `eth_chainId` |
 | `CARTESI_WATCHDOG_METRICS_FILE` | tick | Optional override for Prometheus textfile path |
 | `CARTESI_WATCHDOG_CONTRACTS_INPUT_BOX_ADDRESS` | yes | InputBox contract |
 | `CARTESI_WATCHDOG_APP_ADDRESS` | yes | Rollup application contract |
@@ -196,7 +196,7 @@ See `watchdog/config.lua` for the full list.
 | `lua.h: No such file or directory` when building lcurl | Install `liblua5.4-dev` (Debian/WSL), or set `LUA_INC` to your Lua headers directory (Homebrew/nix) before `just watchdog-lua-deps` |
 | `lcurl` / `cURL` not found at runtime | Run `just watchdog-lua-deps`, set `CARTESI_WATCHDOG_LUA_DEPS=.deps/lua` |
 | Connection errors on `tick` | Sequencer not reachable at `CARTESI_WATCHDOG_SEQUENCER_URL` (stack stopped or wrong port); error includes the URL |
-| `chain="unknown"` in `status.prom` | Export `CARTESI_WATCHDOG_BLOCKCHAIN_ID` before `init`, or rely on tick-time `eth_chainId` from the L1 RPC |
+| `chain="unknown"` in `status.prom` | Export `CARTESI_WATCHDOG_BLOCKCHAIN_ID` before `init` (reliable), or also export `CARTESI_WATCHDOG_BLOCKCHAIN_HTTP_ENDPOINT` at `init` so auto-detect can query `eth_chainId`. Tick never queries RPC for the chain label |
 | `cartesi Lua module is required` | Install Cartesi Machine; use nix/direnv shell; ensure `cartesi-machine` on `PATH` |
 | `inspect endpoint not implemented` | Rebuild CM image: `just canonical-build-machine-image` |
 | CM inspect ~27 bytes / JSON in error | Stale image (old JSON inspect); rebuild: `just canonical-build-machine-image` |

docs/watchdog/operator-deployment.md

@@ -151,13 +151,13 @@ Today `WalletApp::default()` / `WalletConfig::sepolia()` align with Sepolia stag
 | Variable | Where it comes from |
 |----------|---------------------|
 | `CARTESI_WATCHDOG_SEQUENCER_URL` | Ops: internal HTTP base (see network diagram) |
-| `CARTESI_WATCHDOG_BLOCKCHAIN_HTTP_ENDPOINT` | Ops: current chain RPC for `tick` (archive for historical `getLogs`; not persisted by `init`) |
+| `CARTESI_WATCHDOG_BLOCKCHAIN_HTTP_ENDPOINT` | Ops: current chain RPC for `tick` (archive for historical `getLogs`; not persisted by `init`). Also needed at `init` if auto-detecting `BLOCKCHAIN_ID` via `eth_chainId` |
 | `CARTESI_WATCHDOG_APP_ADDRESS` | This rollup’s Cartesi **application** contract |
 | `CARTESI_WATCHDOG_CONTRACTS_INPUT_BOX_ADDRESS` | InputBox on that L1 ([Cartesi deployed contracts](https://docs.cartesi.io/cartesi-rollups/2.0/deployment/self-hosted.md)) |
 | `CARTESI_WATCHDOG_STATE_DIR` | Persistent volume on watchdog host |
 | `CARTESI_WATCHDOG_CM_SNAPSHOT_DIR` | Bootstrap CM snapshot (`init` only) |
 | `CARTESI_WATCHDOG_CM_SNAPSHOT_SAFE_BLOCK` | L1 block that bootstrap snapshot represents (= finalized `inclusion_block` at bootstrap) |
-| `CARTESI_WATCHDOG_BLOCKCHAIN_ID` | Chain id label for `status.prom` metrics (optional; defaults to `unknown`) |
+| `CARTESI_WATCHDOG_BLOCKCHAIN_ID` | Chain id label for `status.prom` metrics (prefer set at `init`; optional auto-detect via `eth_chainId` when L1 endpoint is present at `init`) |
 | `CARTESI_WATCHDOG_METRICS_FILE` | Override path for the Prometheus textfile written by each `tick` |
 | `CARTESI_WATCHDOG_LUA_DEPS` | `.deps/lua` |
 
@@ -174,8 +174,10 @@ Pick one:
 3. **Replay from genesis** (only for new rollups / low block height — slow).
 
 Run `init` once to store the bootstrap CM snapshot into the watchdog state
-layout. `init` does not need `CARTESI_WATCHDOG_BLOCKCHAIN_HTTP_ENDPOINT`; the RPC URL is read by
-each `tick` so it can rotate without editing state:
+layout. The L1 RPC URL is not persisted — each `tick` reads
+`CARTESI_WATCHDOG_BLOCKCHAIN_HTTP_ENDPOINT` so it can rotate without editing
+state. If `CARTESI_WATCHDOG_BLOCKCHAIN_ID` is unset at `init`, auto-detect also
+needs that endpoint present then (prefer setting the chain id explicitly):
 
 ```bash
 sequencer-watchdog init
@@ -194,17 +196,18 @@ side. Gauges:
 Exit mapping is `0→ok`, `1→warning`, `2→failed` (no separate exit-code /
 last-tick gauges — Prom already timestamps samples).
 
-Set `CARTESI_WATCHDOG_BLOCKCHAIN_ID` at `init` so `chain` is labeled. At `tick`,
-the env var overrides a missing persisted value; otherwise the watchdog queries
-`eth_chainId` from `CARTESI_WATCHDOG_BLOCKCHAIN_HTTP_ENDPOINT` (falls back to
-`unknown` only when the RPC is unavailable).
+Set `CARTESI_WATCHDOG_BLOCKCHAIN_ID` at `init` so `chain` is labeled. If unset,
+`init` queries `eth_chainId` from `CARTESI_WATCHDOG_BLOCKCHAIN_HTTP_ENDPOINT` and
+persists the result. At `tick`, the env var overrides a missing persisted value;
+the exit path never blocks on RPC (falls back to `unknown` only when neither
+source is set).
 
 Example `status.prom` after a successful tick:
 
 ```prometheus
-cartesi_watchdog_status{chain="11155111",app_address="0x4CE...",state="ok"} 1
-cartesi_watchdog_status{chain="11155111",app_address="0x4CE...",state="warning"} 0
-cartesi_watchdog_status{chain="11155111",app_address="0x4CE...",state="failed"} 0
+cartesi_watchdog_status{app_address="0x4CE...",chain="11155111",state="ok"} 1
+cartesi_watchdog_status{app_address="0x4CE...",chain="11155111",state="warning"} 0
+cartesi_watchdog_status{app_address="0x4CE...",chain="11155111",state="failed"} 0
 ```
 
 On divergence (exit `2`), `state="failed"` is `1` and

tests/e2e/src/bin/devnet_stack.rs

@@ -5,15 +5,21 @@
 //!
 //! Prints `CARTESI_WATCHDOG_*` exports, then blocks until Ctrl+C or a child exits.
 
+use std::collections::VecDeque;
+use std::fs::File;
+use std::io::{BufRead, BufReader};
+use std::path::Path;
 use std::process::ExitStatus;
 use std::time::Duration;
 
 use rollups_harness::{
-    DEVNET_CHAIN_ID, HarnessResult, ManagedSequencer, devnet_sequencer_config_no_faketime, paths,
+    DEVNET_CHAIN_ID, HarnessResult, ManagedSequencer, StackChildExit,
+    devnet_sequencer_config_no_faketime, paths,
 };
 
 const CHILD_POLL: Duration = Duration::from_secs(5);
 const HEARTBEAT_EVERY: u32 = 12; // 12 * 5s = 60s
+const LOG_TAIL_LINES: usize = 40;
 
 #[tokio::main]
 async fn main() -> HarnessResult<()> {
@@ -92,7 +98,9 @@ async fn main() -> HarnessResult<()> {
     eprintln!("a new Anvil history still needs a fresh state dir + init.");
     eprintln!();
     eprintln!("Leave this terminal running. Ctrl+C stops Anvil + sequencer.");
-    eprintln!("If a child exits on its own, this process prints status + log paths and exits.");
+    eprintln!(
+        "If Anvil or the sequencer exits on its own, this process prints status + log paths and exits."
+    );
     eprintln!(
         "[devnet-stack] ready at {}; waiting for Ctrl+C or child exit",
         runtime.endpoint()
@@ -106,13 +114,17 @@ async fn main() -> HarnessResult<()> {
                 eprintln!("Shutting down Anvil + sequencer...");
                 return runtime.shutdown().await;
             }
-            observed = runtime.observe_for(CHILD_POLL) => {
+            observed = runtime.observe_stack_for(CHILD_POLL) => {
                 match observed? {
-                    Some(status) => {
-                        report_child_exit(&runtime, status);
+                    Some((which, status)) => {
+                        report_child_exit(&runtime, which, status);
                         let _ = runtime.shutdown().await;
+                        let label = match which {
+                            StackChildExit::Sequencer => "sequencer",
+                            StackChildExit::Anvil => "anvil",
+                        };
                         return Err(std::io::Error::other(format!(
-                            "sequencer exited unexpectedly ({status}); see logs above"
+                            "{label} exited unexpectedly ({status}); see logs above"
                         ))
                         .into());
                     }
@@ -132,26 +144,22 @@ async fn main() -> HarnessResult<()> {
     }
 }
 
-fn report_child_exit(runtime: &ManagedSequencer, status: ExitStatus) {
-    let log_path = runtime.log_path();
+fn report_child_exit(runtime: &ManagedSequencer, which: StackChildExit, status: ExitStatus) {
+    let (label, log_label, log_path) = match which {
+        StackChildExit::Sequencer => ("sequencer", "Sequencer log", runtime.log_path()),
+        StackChildExit::Anvil => ("anvil", "Anvil log", runtime.anvil_log_path()),
+    };
     eprintln!();
-    eprintln!("=== sequencer exited unexpectedly ({status}) ===");
-    eprintln!("Sequencer log: {}", log_path.display());
+    eprintln!("=== {label} exited unexpectedly ({status}) ===");
+    eprintln!("{log_label}: {}", log_path.display());
     eprintln!(
         "Other logs under: {}/devnet-stack-*",
         paths::resolve_from_workspace(std::path::Path::new("tests/e2e/results")).display()
     );
     eprintln!("Tail last lines:");
-    match std::fs::read_to_string(log_path) {
-        Ok(contents) => {
-            for line in contents
-                .lines()
-                .rev()
-                .take(40)
-                .collect::<Vec<_>>()
-                .into_iter()
-                .rev()
-            {
+    match tail_last_lines(log_path, LOG_TAIL_LINES) {
+        Ok(lines) => {
+            for line in lines {
                 eprintln!("  {line}");
             }
         }
@@ -161,3 +169,17 @@ fn report_child_exit(runtime: &ManagedSequencer, status: ExitStatus) {
     }
     eprintln!();
 }
+
+fn tail_last_lines(path: &Path, max_lines: usize) -> std::io::Result<Vec<String>> {
+    let file = File::open(path)?;
+    let reader = BufReader::new(file);
+    let mut lines = VecDeque::with_capacity(max_lines);
+    for line in reader.lines() {
+        let line = line?;
+        if lines.len() == max_lines {
+            lines.pop_front();
+        }
+        lines.push_back(line);
+    }
+    Ok(lines.into_iter().collect())
+}

tests/harness/src/lib.rs

@@ -18,7 +18,8 @@ pub use rollups::{DEVNET_CHAIN_ID, DevnetRollupsStack};
 pub use sequencer::{
     BatchCounts, DEFAULT_DEVNET_SEQUENCER_BIN, DEFAULT_TEST_LOGS_DIR, ManagedSequencer,
     ManagedSequencerConfig, RecoveryCheckpoint, RecoverySetupParams, RespawnAttemptOutcome,
-    RespawnPolicy, default_devnet_sequencer_config, devnet_sequencer_config_no_faketime,
+    RespawnPolicy, StackChildExit, default_devnet_sequencer_config,
+    devnet_sequencer_config_no_faketime,
 };
 pub use wallet::{
     TestSigner, WalletL1Client, WalletL2Client, address_from_signing_key, sign_user_op_hex,

tests/harness/src/rollups.rs

@@ -2,7 +2,7 @@
 // SPDX-License-Identifier: Apache-2.0 (see LICENSE)
 
 use std::fs::{self, OpenOptions};
-use std::path::Path;
+use std::path::{Path, PathBuf};
 use std::process::Stdio;
 use std::time::Duration;
 
@@ -106,11 +106,20 @@ impl DevnetRollupsStack {
     pub async fn shutdown(self) -> HarnessResult<()> {
         self.anvil.shutdown().await
     }
+
+    pub fn try_wait_anvil(&mut self) -> std::io::Result<Option<std::process::ExitStatus>> {
+        self.anvil.try_wait()
+    }
+
+    pub fn anvil_log_path(&self) -> &Path {
+        self.anvil.log_path()
+    }
 }
 struct ManagedAnvil {
     child: Child,
     shutdown_timeout: Duration,
     endpoint: String,
+    log_path: PathBuf,
     input_box_address: Address,
     application_factory_address: Address,
     erc20_portal_address: Address,
@@ -186,12 +195,17 @@ impl ManagedAnvil {
             child,
             shutdown_timeout: DEFAULT_ANVIL_SHUTDOWN_TIMEOUT,
             endpoint,
+            log_path,
             input_box_address,
             application_factory_address,
             erc20_portal_address,
         })
     }
 
+    fn log_path(&self) -> &Path {
+        self.log_path.as_path()
+    }
+
     async fn shutdown(mut self) -> HarnessResult<()> {
         send_graceful_terminate(&mut self.child).await;
         match tokio::time::timeout(self.shutdown_timeout, self.child.wait()).await {
@@ -207,6 +221,10 @@ impl ManagedAnvil {
         }
     }
 
+    fn try_wait(&mut self) -> std::io::Result<Option<std::process::ExitStatus>> {
+        self.child.try_wait()
+    }
+
     async fn mine_blocks(&self, block_count: u64) -> HarnessResult<()> {
         // L1 block-time coupling: each mined block advances the L1
         // timestamp by `SECONDS_PER_BLOCK` (Ethereum mainnet parity, also

tests/harness/src/sequencer.rs

@@ -116,6 +116,13 @@ pub struct RespawnPolicy {
     pub advance_per_retry: Option<Duration>,
 }
 
+/// Which child in the devnet stack exited during [`ManagedSequencer::observe_stack_for`].
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum StackChildExit {
+    Sequencer,
+    Anvil,
+}
+
 pub struct ManagedSequencer {
     rollups: DevnetRollupsStack,
     child: Child,
@@ -707,6 +714,10 @@ impl ManagedSequencer {
         self.log_path.as_path()
     }
 
+    pub fn anvil_log_path(&self) -> &Path {
+        self.rollups.anvil_log_path()
+    }
+
     pub fn data_dir(&self) -> &Path {
         self.data_dir_path.as_path()
     }
@@ -829,6 +840,38 @@ impl ManagedSequencer {
         }
     }
 
+    /// Like [`Self::observe_for`], but also watches the Anvil child. Returns
+    /// which process exited when one dies inside `grace`.
+    pub async fn observe_stack_for(
+        &mut self,
+        grace: Duration,
+    ) -> HarnessResult<Option<(StackChildExit, std::process::ExitStatus)>> {
+        let deadline = tokio::time::Instant::now() + grace;
+        loop {
+            if let Some(status) = self
+                .child
+                .try_wait()
+                .map_err(|err| io_other(format!("sequencer.try_wait(): {err}")))?
+            {
+                return Ok(Some((StackChildExit::Sequencer, status)));
+            }
+            if let Some(status) = self
+                .rollups
+                .try_wait_anvil()
+                .map_err(|err| io_other(format!("anvil.try_wait(): {err}")))?
+            {
+                return Ok(Some((StackChildExit::Anvil, status)));
+            }
+
+            let now = tokio::time::Instant::now();
+            if now >= deadline {
+                return Ok(None);
+            }
+            let remaining = deadline - now;
+            tokio::time::sleep(std::cmp::min(remaining, Duration::from_millis(200))).await;
+        }
+    }
+
     /// Wait for the sequencer process to exit on its own. Returns the
     /// process's exit status. Times out after `timeout` to avoid hanging
     /// tests when the process refuses to exit.