Skip to content

Commit 46ce709

Browse files
test(conformance): port the CLI, lifecycle, file-transfer and persistent-workspace suites to testkit
Add the lifecycle, cli, persistent_workspace and file_transfer test files to the sandbox crate. The lifecycle, persistent-workspace and file-transfer tests are adapted from #4629. persistent_workspace_survives_stop_start stays in conformance: every conformant driver keeps the workspace across stop and start, and the one that can't yet (MXC) isn't conformant until it can. The policy-advisor suite needs no port: it is a filter over the existing conformance tests that no workflow runs. The OCI image, Keycloak and Podman userns suites are ported separately, in the features and drivers families. The old conformance/cli crate stays until the release lanes move to plans. Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
1 parent 78813fc commit 46ce709

7 files changed

Lines changed: 837 additions & 5 deletions

File tree

‎tests/suites/conformance/Cargo.lock‎

Lines changed: 3 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎tests/suites/conformance/sandbox/Cargo.toml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,3 +13,6 @@ publish = false
1313

1414
[dependencies]
1515
openshell-testkit = { path = "../../../testkit/testkit" }
16+
serde_json = "1"
17+
tempfile = "3"
18+
tokio = { version = "1", features = ["time"] }

‎tests/suites/conformance/sandbox/README.md‎

Lines changed: 13 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -8,10 +8,17 @@ the file with `_` replaced by `-` (`file_transfer.rs` is `file-transfer`).
88
| Suite | Covers |
99
|---|---|
1010
| `smoke` | Create a sandbox, run `echo` in it, delete it |
11+
| `lifecycle` | SDK: long-running and run-to-completion sandboxes, the default command, restart of a stopped sandbox, deleting a stopped sandbox |
12+
| `cli` | The same lifecycle through the `openshell` CLI: status, create, get, list, exec, stop, start, delete |
13+
| `persistent-workspace` | Files in `/sandbox` survive stop and start |
14+
| `file-transfer` | `sandbox upload` and `download`: round trips, `.gitignore` filtering, workspace-escape and option-looking names |
1115

12-
The suite is small on purpose. It proves that a plan can run a real test through
13-
the runner on CI. The older suites move to testkit by adding suites here (or a
14-
new family under `tests/suites/`) and listing them in a plan.
16+
`smoke` is the smallest end-to-end check that a plan can run a real test through
17+
the runner on CI. The other suites are ports of the old `conformance/cli`
18+
scenarios and of Evan Lezar's suites in NVIDIA/OpenShell#4629 (credited in each
19+
file). `file-transfer` runs `git` on the machine that runs the tests, so a test
20+
environment needs it installed (`tests/tmachine/ansible/playbooks/test-tools.yaml`
21+
does that for the tmachine environments).
1522

1623
## Running it
1724

@@ -47,8 +54,9 @@ leave driver behavior alone.
4754
published docs.
4855
- Assert `phase` and `exit_code`. Don't assert condition reasons.
4956
- Check that something did not happen by reading state, not by sleeping.
50-
- Every environment must pass every test here. If a behavior is left to the
51-
driver, the test belongs in a `features` or `drivers` suite instead.
57+
- Don't test behavior that only some drivers have. A test that can't pass on
58+
every environment isn't conformance; put it in `tests/suites/features` or
59+
`tests/suites/drivers`.
5260
- Use the CLI for upload, download, connect and forward. Rely on its exit code
5361
and `--output json`.
5462
- Add a testkit helper instead of calling raw gRPC from a test.
Lines changed: 170 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,170 @@
1+
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
// SPDX-License-Identifier: Apache-2.0
3+
4+
//! Sandbox lifecycle through the `openshell` CLI.
5+
//!
6+
//! Ported from the `smoke` and `sandbox-lifecycle` scenarios of the old
7+
//! `conformance/cli` suite. The other lifecycle tests act through the SDK; these
8+
//! check that the CLI commands for create, get, list, exec, stop, start, and
9+
//! delete work against the same gateway.
10+
11+
use serde_json::Value;
12+
use testkit::Context;
13+
14+
const PAGE_SIZE: &str = "1000";
15+
/// More pages than any test gateway should have; stops a repeated token from
16+
/// looping forever.
17+
const MAX_PAGES: usize = 1000;
18+
19+
/// `openshell status` succeeds, and a sandbox created with the CLI is
20+
/// retrievable and Ready, appears in `sandbox list`, runs `sandbox exec`, and is
21+
/// gone from `sandbox list` after `sandbox delete`.
22+
#[testkit::test]
23+
async fn cli_create_exec_list_delete(cx: &mut Context) -> testkit::Result {
24+
cx.cli(["status"]).await?.success()?;
25+
26+
let name = cx.name("cs");
27+
cx.track(&name);
28+
cx.cli(["sandbox", "create", "--name", &name, "--detach"])
29+
.await?
30+
.success()?;
31+
32+
let sandbox = cx.adopt(&name).await?;
33+
let got: Value = cx
34+
.cli(["sandbox", "get", &name, "--output", "json"])
35+
.await?
36+
.success()?
37+
.json()?;
38+
testkit::ensure!(
39+
got["name"] == name.as_str(),
40+
"sandbox get returned {:?}; expected {name:?}",
41+
got["name"]
42+
);
43+
testkit::ensure!(
44+
got["phase"] == "Ready",
45+
"sandbox {name} is in phase {}; expected Ready",
46+
got["phase"]
47+
);
48+
49+
let listed = find_listed(cx, &name).await?;
50+
testkit::ensure!(listed.is_some(), "sandbox {name} is not in sandbox list");
51+
52+
let marker = format!("openshell-e2e-{}", cx.run_id());
53+
let exec = cx
54+
.cli([
55+
"sandbox", "exec", "--name", &name, "--no-tty", "--", "echo", &marker,
56+
])
57+
.await?
58+
.success()?;
59+
testkit::ensure!(
60+
exec.stdout == format!("{marker}\n"),
61+
"sandbox exec stdout was {:?}; expected {:?}",
62+
exec.stdout,
63+
format!("{marker}\n")
64+
);
65+
66+
cx.cli(["sandbox", "delete", &name]).await?.success()?;
67+
wait_unlisted(cx, &name).await?;
68+
cx.expect_absent(&sandbox).await
69+
}
70+
71+
/// A sandbox created with the CLI can be stopped, rejects `sandbox exec` while
72+
/// stopped, starts again, and can be stopped and deleted.
73+
#[testkit::test]
74+
async fn cli_stop_start_delete(cx: &mut Context) -> testkit::Result {
75+
let name = cx.name("cl");
76+
cx.track(&name);
77+
cx.cli([
78+
"sandbox",
79+
"create",
80+
"--name",
81+
&name,
82+
"--detach",
83+
"--no-tty",
84+
"--",
85+
"sh",
86+
"-c",
87+
"exec sleep infinity",
88+
])
89+
.await?
90+
.success()?;
91+
let sandbox = cx.adopt(&name).await?;
92+
sandbox.wait_ready().await?;
93+
94+
cx.cli(["sandbox", "stop", &name]).await?.success()?;
95+
sandbox
96+
.wait_phase(testkit::sdk::SandboxPhase::Stopped, testkit::STOP_TIMEOUT)
97+
.await?;
98+
cx.cli(["sandbox", "exec", "--name", &name, "--no-tty", "--", "true"])
99+
.await?
100+
.failure()?;
101+
102+
cx.cli(["sandbox", "start", &name]).await?.success()?;
103+
sandbox.wait_ready().await?;
104+
cx.cli(["sandbox", "stop", &name]).await?.success()?;
105+
sandbox
106+
.wait_phase(testkit::sdk::SandboxPhase::Stopped, testkit::STOP_TIMEOUT)
107+
.await?;
108+
109+
cx.cli(["sandbox", "delete", &name]).await?.success()?;
110+
wait_unlisted(cx, &name).await?;
111+
cx.expect_absent(&sandbox).await
112+
}
113+
114+
/// Wait until `sandbox list` no longer shows the sandbox.
115+
async fn wait_unlisted(cx: &Context, name: &str) -> testkit::Result {
116+
testkit::poll_until(
117+
&format!("sandbox {name} to leave sandbox list"),
118+
testkit::DELETE_TIMEOUT,
119+
std::time::Duration::from_secs(1),
120+
async || {
121+
Ok(match find_listed(cx, name).await? {
122+
None => testkit::Poll::Ready(()),
123+
Some(phase) => testkit::Poll::Pending(format!("still listed in phase {phase}")),
124+
})
125+
},
126+
)
127+
.await
128+
}
129+
130+
/// The phase of the sandbox in `sandbox list`, if it is listed. Follows every
131+
/// page.
132+
async fn find_listed(cx: &Context, name: &str) -> testkit::Result<Option<String>> {
133+
let mut token = String::new();
134+
for _ in 0..MAX_PAGES {
135+
let page: Value = cx
136+
.cli([
137+
"sandbox",
138+
"list",
139+
"--page-size",
140+
PAGE_SIZE,
141+
"--page-token",
142+
&token,
143+
"--output",
144+
"json",
145+
])
146+
.await?
147+
.success()?
148+
.json()?;
149+
if let Some(entry) = page["sandboxes"]
150+
.as_array()
151+
.into_iter()
152+
.flatten()
153+
.find(|entry| entry["name"] == name)
154+
{
155+
return Ok(Some(
156+
entry["phase"].as_str().unwrap_or_default().to_string(),
157+
));
158+
}
159+
token = page["next_page_token"]
160+
.as_str()
161+
.unwrap_or_default()
162+
.to_string();
163+
if token.is_empty() {
164+
return Ok(None);
165+
}
166+
}
167+
Err(testkit::Error::msg(format!(
168+
"sandbox list has more than {MAX_PAGES} pages"
169+
)))
170+
}

0 commit comments

Comments
 (0)