Skip to content

Commit b60c028

Browse files
fix(runner): keep logs from a timed-out run and gate on the E2E suites
nextest now writes its output to a file in the guest, and the runner copies nextest.log and junit.xml out after every run, so a run that times out or loses SSH still leaves a log. A failed JUnit copy no longer turns a test failure into an error. The runner exits with 1 when a test failed and 2 when the run could not complete, and annotates the GitHub Actions run with which it was. The E2E suites lane is now part of the Core E2E result. Its disk cache key also covers tmachine's cache versions in setup.rs and install.rs. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
1 parent d6f93fa commit b60c028

6 files changed

Lines changed: 134 additions & 57 deletions

File tree

‎.agents/skills/watch-github-actions/SKILL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -127,7 +127,7 @@ gh run list --json databaseId,status,headBranch,url --jq '.[] | {id: .databaseId
127127

128128
For tmachine Kubernetes conformance failures, inspect the `tmachine-diagnostics-*` artifact. The K3s report includes Agent Sandbox controller state, leader leases, events, and current and previous controller logs. The conformance playbook checks reconciliation after guest boot with a temporary scheduling-gated Sandbox probe before running tests. Distinguish a failed controller probe from a failed conformance scenario; gateway connectivity and controller Deployment availability do not prove reconciliation works.
129129

130-
For the `e2e-suites` lane (`.github/workflows/e2e-suites.yml`), the last lines of the job log give the runner's result: PASS, FAIL after the end of the nextest log (which names the failed tests), or ERROR with the step that broke. Then inspect the `e2e-results-<family>-<package>-<environment>-<installer>` artifact, which holds `junit.xml`, `nextest.log` and, after a failure, the guest's `diagnostics/`.
130+
For the `e2e-suites` lane (`.github/workflows/e2e-suites.yml`), the last lines of the job log give the runner's result: PASS, FAIL (exit 1) after the end of the nextest log (which names the failed tests), or ERROR (exit 2) with the step that broke. FAIL and ERROR also appear as annotations on the run's summary page. Then inspect the `e2e-results-<family>-<package>-<environment>-<installer>` artifact, which holds `junit.xml`, `nextest.log` and, after a failure, the guest's `diagnostics/`.
131131

132132
`Pre-release` temporarily omits Codex Security from its `Security Scan` job.
133133
`Release` uses the passing prerelease qualification result. Other scanners and

‎.github/workflows/branch-e2e.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -304,8 +304,7 @@ jobs:
304304
]
305305
306306
# Run the testkit suites in tests/suites/ on tmachine. The environments and
307-
# suites are listed in e2e-suites.yml. Advisory: not part of
308-
# core-e2e-result.needs.
307+
# suites are listed in e2e-suites.yml.
309308
e2e-suites:
310309
needs: [pr_metadata, prepare-integration]
311310
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
@@ -505,6 +504,7 @@ jobs:
505504
needs:
506505
- pr_metadata
507506
- conformance-integration
507+
- e2e-suites
508508
- feature-specific-integration
509509
- driver-specific-integration
510510
- docker-e2e

‎.github/workflows/e2e-suites.yml‎

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -71,12 +71,13 @@ jobs:
7171
uses: actions/cache@caa296126883cff596d87d8935842f9db880ef25 # v5.1.0
7272
with:
7373
path: ~/.cache/tmachine
74-
# tmachine keys its disk layers on a hash of the Ansible tree.
75-
# actions/cache doesn't save after an exact key hit, so a fixed key
76-
# would rebuild the changed layers on every run. Key on the playbooks
77-
# with no restore-keys fallback: restoring an older entry would carry
78-
# its stale disks into the new one, which then only grows.
79-
key: tmachine-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('tests/tmachine/ansible/**', 'tests/tmachine/images.nix') }}
74+
# tmachine keys its disk layers on a hash of the Ansible tree and
75+
# the cache versions in setup.rs and install.rs. actions/cache
76+
# doesn't save after an exact key hit, so a key that misses one of
77+
# those would rebuild the changed layers on every run. No
78+
# restore-keys fallback: restoring an older entry would carry its
79+
# stale disks into the new one, which then only grows.
80+
key: tmachine-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('tests/tmachine/ansible/**', 'tests/tmachine/images.nix', 'tests/tmachine/src/qemu/setup.rs', 'tests/tmachine/src/qemu/install.rs') }}
8081

8182
- name: Download integration inputs
8283
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
@@ -102,7 +103,10 @@ jobs:
102103
--results artifacts/test-results
103104
104105
# The results directory holds the JUnit report (junit.xml), the nextest
105-
# log, and the guest's diagnostics when the run failed.
106+
# log, and the guest's diagnostics when the run failed. The log and the
107+
# report are collected even when the run timed out.
108+
# The runner exits with 1 when a test failed and 2 when the run could
109+
# not complete.
106110
- name: Upload results
107111
if: always()
108112
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7

‎TESTING.md‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -172,9 +172,11 @@ nix run .#openshell-test-runner -- \
172172
It needs KVM, and tmachine only boots guests of the host's architecture. The
173173
runner points the tests at the installed CLI and the gateway the installer
174174
registered (`--gateway`, default `openshell`). A run passes if nextest
175-
succeeds; nextest fails a run with no tests. The runner writes `junit.xml`,
176-
`nextest.log` and, after a failure, the guest's diagnostics to
177-
`artifacts/test-results/` (`--results`).
175+
succeeds; nextest fails a run with no tests. The runner exits with 1 when a
176+
test failed and 2 when the run could not complete, for example because the
177+
environment didn't come up. It writes `junit.xml`, `nextest.log` and, after a
178+
failure, the guest's diagnostics to `artifacts/test-results/` (`--results`),
179+
even when the run timed out.
178180

179181
`--filter <filterset>` narrows the tests with a nextest filterset, and
180182
`--keep` leaves the VM running until you press Enter. Two runs on one host
@@ -187,8 +189,8 @@ out of every environment until it's fixed.
187189

188190
**CI.** `.github/workflows/e2e-suites.yml` lists the environments and suites as
189191
an explicit matrix and runs the runner once per row. Branch E2E calls it when
190-
the `test:e2e` label is set. It is advisory, so a failure doesn't block
191-
merging. To run a suite in another environment on CI, add a row.
192+
the `test:e2e` label is set, and a failure fails the Core E2E result. To run
193+
a suite in another environment on CI, add a row.
192194

193195
### Python E2E (`e2e/python/`)
194196

‎tests/runner/src/execute.rs‎

Lines changed: 90 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111
1212
use std::fmt;
1313
use std::path::Path;
14-
use std::process::{ExitStatus, Output};
14+
use std::process::{ExitCode, ExitStatus, Output};
1515
use std::time::Duration;
1616

1717
use anyhow::{Context, Result, ensure};
@@ -21,6 +21,8 @@ use tmachine::config::Config;
2121
use crate::bundle::{Bundle, PROFILE, READY_BINARY};
2222

2323
const GUEST_DIR: &str = "/var/tmp/openshell-tests";
24+
/// The nextest log's name, in the guest directory and in the results.
25+
const NEXTEST_LOG: &str = "nextest.log";
2426
/// How long the gateway has to become healthy after the environment is up.
2527
// Matches the installers' own wait: an HA k3s gateway booted from a cached
2628
// disk can take minutes to report healthy.
@@ -40,6 +42,12 @@ const NEXTEST_NO_TESTS_RUN: i32 = 4;
4042
/// ends its output with a summary that names the failed tests.
4143
const LOG_TAIL_LINES: usize = 40;
4244

45+
/// The runner's exit code when a test failed.
46+
const TEST_FAILED_EXIT: u8 = 1;
47+
/// The runner's exit code when the run could not complete. clap also exits
48+
/// with 2 on a usage error.
49+
pub const ERROR_EXIT: u8 = 2;
50+
4351
/// How a run ended.
4452
#[derive(Debug, PartialEq, Eq)]
4553
pub enum Status {
@@ -55,6 +63,16 @@ impl Status {
5563
!matches!(self, Self::Passed)
5664
}
5765

66+
/// The runner's exit code: 1 when a test failed and 2 when the run could
67+
/// not complete, so CI can tell a test failure from a broken environment.
68+
pub fn exit_code(&self) -> ExitCode {
69+
match self {
70+
Self::Passed => ExitCode::SUCCESS,
71+
Self::Failed => ExitCode::from(TEST_FAILED_EXIT),
72+
Self::Error(_) => ExitCode::from(ERROR_EXIT),
73+
}
74+
}
75+
5876
/// The status of a nextest run that exited with `status`. nextest fails
5977
/// a run that finds no tests.
6078
fn of_nextest(status: ExitStatus, log: &Path) -> Self {
@@ -117,10 +135,14 @@ pub async fn run(config: &Config, run: &Run<'_>) -> Status {
117135
Ok(status) => status,
118136
Err(error) => Status::Error(format!("{error:#}")),
119137
};
120-
if status.is_failure()
121-
&& let Err(error) = collect_diagnostics(&session, &run.results.join("diagnostics")).await
122-
{
123-
eprintln!("==> {label}: collecting diagnostics failed: {error:#}");
138+
if let Err(error) = collect_results(&session, run.results).await {
139+
eprintln!("==> {label}: collecting test results failed: {error:#}");
140+
}
141+
if status.is_failure() {
142+
print_log_tail(&run.results.join(NEXTEST_LOG));
143+
if let Err(error) = collect_diagnostics(&session, &run.results.join("diagnostics")).await {
144+
eprintln!("==> {label}: collecting diagnostics failed: {error:#}");
145+
}
124146
}
125147

126148
if run.keep {
@@ -175,7 +197,7 @@ async fn provision(config: &Config, environment: &str, installer: &str) -> Resul
175197
/// can't pass for this one's.
176198
fn prepare_results(results: &Path) -> Result<()> {
177199
std::fs::create_dir_all(results).with_context(|| format!("create {}", results.display()))?;
178-
for stale in ["junit.xml", "nextest.log"] {
200+
for stale in ["junit.xml", NEXTEST_LOG] {
179201
let path = results.join(stale);
180202
if path.exists() {
181203
std::fs::remove_file(&path).with_context(|| format!("remove {}", path.display()))?;
@@ -190,8 +212,6 @@ fn prepare_results(results: &Path) -> Result<()> {
190212
}
191213

192214
async fn run_tests(session: &Session, run: &Run<'_>) -> Result<Status> {
193-
let results = run.results;
194-
195215
let openshell_bin = checked(
196216
"find the installed openshell CLI",
197217
session.exec(&["sh", "-c", "command -v openshell"]).await?,
@@ -268,33 +288,62 @@ async fn run_tests(session: &Session, run: &Run<'_>) -> Result<Status> {
268288
if let Some(filter) = run.filter {
269289
nextest.extend(["-E", filter]);
270290
}
271-
let output = session.exec_timeout(&nextest, RUN_TIMEOUT).await?;
272-
let mut log = output.stdout.clone();
273-
log.extend_from_slice(&output.stderr);
274-
let log_path = results.join("nextest.log");
275-
std::fs::write(&log_path, &log)?;
291+
// nextest writes its output to a file in the guest, so a run that times
292+
// out or loses SSH still leaves a log for `collect_results`.
293+
let log = format!("{guest}/{NEXTEST_LOG}");
294+
let mut command = vec!["sh", "-c", r#"exec "$@" >"$0" 2>&1"#, &log];
295+
command.extend(nextest);
296+
let output = session.exec_timeout(&command, RUN_TIMEOUT).await?;
297+
Ok(Status::of_nextest(
298+
output.status,
299+
&run.results.join(NEXTEST_LOG),
300+
))
301+
}
276302

277-
// The JUnit report is uploaded with the results; the status comes from
278-
// nextest's exit code.
279-
let found = checked(
280-
"find JUnit report",
281-
session.exec(&["find", guest, "-name", "junit.xml"]).await?,
282-
)?;
283-
match found.lines().next() {
284-
Some(path) => session.copy_out(path, &results.join("junit.xml")).await?,
285-
None => eprintln!("==> nextest wrote no JUnit report"),
303+
/// Copy the nextest log and the JUnit report out of the guest, whether or not
304+
/// the run finished. Either may be missing, for example when the gateway never
305+
/// became ready.
306+
async fn collect_results(session: &Session, results: &Path) -> Result<()> {
307+
let found = session
308+
.exec(&[
309+
"find",
310+
GUEST_DIR,
311+
"-type",
312+
"f",
313+
"(",
314+
"-name",
315+
NEXTEST_LOG,
316+
"-o",
317+
"-name",
318+
"junit.xml",
319+
")",
320+
])
321+
.await?;
322+
let mut copied = Vec::new();
323+
for path in String::from_utf8_lossy(&found.stdout).lines() {
324+
let Some(name) = Path::new(path).file_name() else {
325+
continue;
326+
};
327+
if !copied.contains(&name) {
328+
session.copy_out(path, &results.join(name)).await?;
329+
copied.push(name);
330+
}
286331
}
332+
Ok(())
333+
}
287334

288-
let status = Status::of_nextest(output.status, &log_path);
289-
if status.is_failure() {
290-
let log = String::from_utf8_lossy(&log);
291-
let lines: Vec<_> = log.lines().collect();
292-
eprintln!("==> the end of {}:", log_path.display());
293-
for line in &lines[lines.len().saturating_sub(LOG_TAIL_LINES)..] {
294-
eprintln!("{line}");
295-
}
335+
/// Print the end of the nextest log, which names the failed tests, or the
336+
/// test that was running when the run stopped.
337+
fn print_log_tail(log: &Path) {
338+
let Ok(contents) = std::fs::read(log) else {
339+
return;
340+
};
341+
let contents = String::from_utf8_lossy(&contents);
342+
let lines: Vec<_> = contents.lines().collect();
343+
eprintln!("==> the end of {}:", log.display());
344+
for line in &lines[lines.len().saturating_sub(LOG_TAIL_LINES)..] {
345+
eprintln!("{line}");
296346
}
297-
Ok(status)
298347
}
299348

300349
/// Write the OpenShell gateway's journal to `dir/gateway.log`.
@@ -346,4 +395,14 @@ mod tests {
346395
);
347396
assert!(matches!(status(1), Status::Error(_)));
348397
}
398+
399+
#[test]
400+
fn exit_code_separates_test_failures_from_errors() {
401+
assert_eq!(Status::Passed.exit_code(), ExitCode::SUCCESS);
402+
assert_eq!(Status::Failed.exit_code(), ExitCode::from(1));
403+
assert_eq!(
404+
Status::Error("broken".to_string()).exit_code(),
405+
ExitCode::from(2)
406+
);
407+
}
349408
}

‎tests/runner/src/main.rs‎

Lines changed: 23 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ use std::process::ExitCode;
1313
use anyhow::{Context, Result, ensure};
1414
use clap::Parser;
1515

16-
use crate::execute::Run;
16+
use crate::execute::{Run, Status};
1717

1818
mod bundle;
1919
mod execute;
@@ -63,7 +63,8 @@ async fn main() -> ExitCode {
6363
Ok(Ok(code)) => code,
6464
Ok(Err(error)) => {
6565
eprintln!("error: {error:#}");
66-
ExitCode::FAILURE
66+
annotate("E2E suites: runner error", &format!("{error:#}"));
67+
ExitCode::from(execute::ERROR_EXIT)
6768
}
6869
}
6970
}
@@ -114,16 +115,27 @@ async fn run(cli: Cli) -> Result<ExitCode> {
114115
)
115116
.await;
116117

117-
eprintln!(
118-
"\n{} / {} / {}: {status}",
119-
cli.environment, cli.installer, cli.family
120-
);
118+
let label = format!("{} / {} / {}", cli.environment, cli.installer, cli.family);
119+
eprintln!("\n{label}: {status}");
121120
eprintln!("Results: {}", results.display());
122-
Ok(if status.is_failure() {
123-
ExitCode::FAILURE
124-
} else {
125-
ExitCode::SUCCESS
126-
})
121+
match &status {
122+
Status::Passed => {}
123+
Status::Failed => annotate("E2E suites: test failure", &format!("{label}: {status}")),
124+
Status::Error(_) => annotate("E2E suites: runner error", &format!("{label}: {status}")),
125+
}
126+
Ok(status.exit_code())
127+
}
128+
129+
/// On GitHub Actions, show `message` as an error on the run's summary page.
130+
fn annotate(title: &str, message: &str) {
131+
if std::env::var_os("GITHUB_ACTIONS").is_none() {
132+
return;
133+
}
134+
let message = message
135+
.replace('%', "%25")
136+
.replace('\r', "%0D")
137+
.replace('\n', "%0A");
138+
println!("::error title={title}::{message}");
127139
}
128140

129141
/// Load the tmachine configuration and enter `tests/tmachine/`, which its

0 commit comments

Comments
 (0)