Skip to content

Commit f595dd0

Browse files
refactor(testkit): find the gateway the way the CLI does
Replace the target file (OPENSHELL_TEST_TARGET) with the CLI's own gateway selection: OPENSHELL_GATEWAY_ENDPOINT, then OPENSHELL_GATEWAY, then the active gateway. A registered gateway's endpoint, auth mode, and mTLS material come from its CLI config directory. cx.cli runs OPENSHELL_BIN against the same gateway and workspace as the SDK. Suites now run unchanged under e2e/with-docker-gateway.sh and e2e/with-podman-gateway.sh, and environments need no extra file. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
1 parent cada6df commit f595dd0

11 files changed

Lines changed: 427 additions & 450 deletions

File tree

‎TESTING.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -133,8 +133,9 @@ driver configuration boundary.
133133
### testkit (`tests/testkit/`)
134134

135135
`testkit` is the support library for SDK-driven end-to-end suites. Tests are
136-
`#[testkit::test]` functions with a required doc comment. They run against the
137-
gateway described by the target file named in `OPENSHELL_TEST_TARGET`. See
136+
`#[testkit::test]` functions with a required doc comment. They find the
137+
gateway the same way the CLI does, through `OPENSHELL_GATEWAY`,
138+
`OPENSHELL_GATEWAY_ENDPOINT`, or the active gateway. See
138139
[tests/testkit/README.md](tests/testkit/README.md) for how to write and run
139140
tests.
140141

‎tests/testkit/Cargo.lock‎

Lines changed: 18 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎tests/testkit/README.md‎

Lines changed: 21 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ This is a separate Cargo workspace with its own lockfile:
77

88
| Crate | Purpose |
99
|---|---|
10-
| `testkit` | Reads the target file, connects to the gateway, creates and cleans up sandboxes, runs the CLI. Also builds `testkit-ready`. |
10+
| `testkit` | Finds and connects to the gateway, creates and cleans up sandboxes, runs the CLI. Also builds `testkit-ready`. |
1111
| `testkit-macros` | The `#[testkit::test]` attribute. |
1212

1313
## Writing a test
@@ -51,35 +51,36 @@ test creates, such as providers, in the test itself.
5151

5252
## Running against a gateway
5353

54-
Tests read the system under test from the target file named by
55-
`OPENSHELL_TEST_TARGET`. Relative paths resolve against the file's directory.
56-
57-
```yaml
58-
gateway:
59-
endpoint: https://127.0.0.1:17670 # http:// for plaintext
60-
tls: # optional; cert and key together select mTLS
61-
ca: ca.crt
62-
client_cert: client.crt
63-
client_key: client.key
64-
auth: # optional
65-
bearer_token_file: token
66-
cli: # needed by tests that call cx.cli
67-
binary: /usr/bin/openshell # default: `openshell` on PATH
68-
gateway: local # default: the CLI's active gateway
69-
workspace: default
70-
```
54+
Tests find the gateway the same way the `openshell` CLI does:
55+
56+
1. `OPENSHELL_GATEWAY_ENDPOINT`: a gateway URL. Plaintext, or TLS without a
57+
client certificate.
58+
2. `OPENSHELL_GATEWAY`: the name of a gateway registered with the CLI.
59+
3. The CLI's active gateway, set by `openshell gateway select`.
60+
61+
A registered gateway's endpoint and auth mode come from
62+
`$XDG_CONFIG_HOME/openshell/gateways/<name>/metadata.json`. For mTLS, the CA
63+
and client certificate come from its `mtls/` directory. testkit supports
64+
plaintext and mTLS gateways. `OPENSHELL_WORKSPACE` selects the workspace
65+
(default `default`), and `OPENSHELL_BIN` selects the CLI that `cx.cli` runs
66+
(default `openshell` on PATH). The CLI always targets the same gateway and
67+
workspace as the SDK. Each test logs the gateway it used.
7168

7269
Wait for the gateway, then run a suite. Run tests that share a gateway one at
7370
a time:
7471

7572
```shell
76-
export OPENSHELL_TEST_TARGET=$PWD/target.yaml
73+
export OPENSHELL_GATEWAY=<name>
7774
cargo run --manifest-path tests/testkit/Cargo.toml --bin testkit-ready -- 300
7875
cargo nextest run --manifest-path <suite>/Cargo.toml --test-threads 1
7976
```
8077

8178
`testkit-ready [TIMEOUT_SECS]` exits non-zero if the gateway is not healthy in
82-
time or the target's CLI does not report it connected.
79+
time or the CLI does not report it connected.
80+
81+
`e2e/with-docker-gateway.sh` and `e2e/with-podman-gateway.sh` start an
82+
ephemeral gateway and export `OPENSHELL_GATEWAY` and `OPENSHELL_BIN`, so a
83+
suite runs under them unchanged.
8384

8485
## Checks
8586

‎tests/testkit/testkit/Cargo.toml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,12 +15,13 @@ publish = false
1515
name = "testkit"
1616

1717
[dependencies]
18+
openshell-bootstrap = { path = "../../../crates/openshell-bootstrap" }
19+
openshell-core = { path = "../../../crates/openshell-core", default-features = false }
1820
openshell-sdk = { path = "../../../crates/openshell-sdk" }
1921
openshell-testkit-macros = { path = "../testkit-macros" }
2022
futures = "0.3"
2123
rand = "0.9"
22-
serde = { version = "1", features = ["derive"] }
24+
serde = "1"
2325
serde_json = "1"
24-
serde_yml = { package = "noyalib", version = "0.0.53", default-features = false, features = ["std", "compat-serde-yaml"] }
2526
tokio = { version = "1.43", features = ["macros", "process", "rt", "time"] }
2627
tonic = { version = "0.14", features = ["tls-native-roots"] }

‎tests/testkit/testkit/src/bin/testkit-ready.rs‎

Lines changed: 10 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,17 @@
11
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22
// SPDX-License-Identifier: Apache-2.0
33

4-
//! Wait until the target's gateway is ready for tests.
4+
//! Wait until the gateway is ready for tests.
55
//!
6-
//! Usage: `testkit-ready [TIMEOUT_SECS]` (default 60), with the target file
7-
//! named by `OPENSHELL_TEST_TARGET`. Run it once per environment before the
8-
//! test binaries; exits non-zero with the last error if the gateway is not
9-
//! healthy in time, or if the environment's CLI does not reach it.
6+
//! Usage: `testkit-ready [TIMEOUT_SECS]` (default 60), with the gateway
7+
//! selected as for tests. Run it once per environment before the test
8+
//! binaries; exits non-zero with the last error if the gateway is not healthy
9+
//! in time, or if the CLI does not reach it.
1010
1111
use std::process::ExitCode;
1212
use std::time::Duration;
1313

14-
use testkit::Target;
14+
use testkit::{Cli, Gateway};
1515

1616
const DEFAULT_TIMEOUT_SECS: u64 = 60;
1717

@@ -26,9 +26,10 @@ async fn main() -> ExitCode {
2626
}
2727
};
2828
let result = async {
29-
let target = Target::from_env()?;
30-
let health = target.wait_ready(Duration::from_secs(timeout)).await?;
31-
target.check_cli().await?;
29+
let gateway = Gateway::from_env()?;
30+
eprintln!("testkit-ready: {}", gateway.describe());
31+
let health = gateway.wait_ready(Duration::from_secs(timeout)).await?;
32+
Cli::from_env(&gateway).check_connected().await?;
3233
testkit::Result::Ok(health)
3334
}
3435
.await;

‎tests/testkit/testkit/src/cli.rs‎

Lines changed: 56 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22
// SPDX-License-Identifier: Apache-2.0
33

4-
//! Driving the `openshell` CLI the environment installed.
4+
//! Driving the installed `openshell` CLI.
55
//!
66
//! Some promised behavior exists only in the CLI: file upload and download,
77
//! interactive connect, and port forwarding. Tests act through the CLI for
@@ -16,16 +16,18 @@ use std::time::Duration;
1616
use serde::de::DeserializeOwned;
1717

1818
use crate::error::{Error, Result, ResultExt};
19-
use crate::target::CliTarget;
19+
use crate::gateway::{ENDPOINT_ENV, GATEWAY_ENV, Gateway, WORKSPACE_ENV};
2020

2121
/// How long one CLI invocation may take.
2222
pub const CLI_TIMEOUT: Duration = Duration::from_mins(5);
23+
/// The CLI binary to run. Defaults to `openshell` on PATH.
24+
pub const BIN_ENV: &str = "OPENSHELL_BIN";
2325

24-
/// Runs the target's CLI against the target's gateway.
26+
/// Runs the CLI against the gateway and workspace the tests use.
2527
#[derive(Clone, Debug)]
2628
pub struct Cli {
2729
binary: PathBuf,
28-
gateway: Option<String>,
30+
gateway: Gateway,
2931
}
3032

3133
/// The buffered result of one CLI invocation.
@@ -38,13 +40,31 @@ pub struct CliOutput {
3840
}
3941

4042
impl Cli {
41-
pub(crate) fn new(target: &CliTarget, binary: PathBuf) -> Self {
43+
/// The CLI named by [`BIN_ENV`], pointed at `gateway`.
44+
pub fn from_env(gateway: &Gateway) -> Self {
45+
let binary = std::env::var_os(BIN_ENV)
46+
.filter(|value| !value.is_empty())
47+
.map_or_else(|| PathBuf::from("openshell"), PathBuf::from);
4248
Self {
4349
binary,
44-
gateway: target.gateway.clone(),
50+
gateway: gateway.clone(),
4551
}
4652
}
4753

54+
/// Check that the CLI reaches the gateway: `openshell status` must report
55+
/// it connected.
56+
pub async fn check_connected(&self) -> Result {
57+
let output = self.run(["status", "--output", "json"]).await?.success()?;
58+
let status: serde_json::Value = output.json()?;
59+
crate::ensure!(
60+
status["status"] == "connected",
61+
"`{}` reports {}",
62+
output.command,
63+
status
64+
);
65+
Ok(())
66+
}
67+
4868
/// Run the CLI with `args` and buffer its output. A non-zero exit is
4969
/// returned, not raised; use [`CliOutput::success`] to require success.
5070
pub async fn run<I, S>(&self, args: I) -> Result<CliOutput>
@@ -75,14 +95,15 @@ impl Cli {
7595
process
7696
.args(&args)
7797
.env("NO_COLOR", "1")
78-
// Select the target's gateway, never one from the caller's
79-
// environment.
80-
.env_remove("OPENSHELL_GATEWAY_ENDPOINT")
98+
.env(WORKSPACE_ENV, &self.gateway.workspace)
8199
.stdin(Stdio::null())
82100
.kill_on_drop(true);
83-
match &self.gateway {
84-
Some(gateway) => process.env("OPENSHELL_GATEWAY", gateway),
85-
None => process.env_remove("OPENSHELL_GATEWAY"),
101+
// Select exactly the gateway the SDK connected to.
102+
match &self.gateway.name {
103+
Some(name) => process.env(GATEWAY_ENV, name).env_remove(ENDPOINT_ENV),
104+
None => process
105+
.env(ENDPOINT_ENV, &self.gateway.endpoint)
106+
.env_remove(GATEWAY_ENV),
86107
};
87108
if let Some(dir) = dir {
88109
process.current_dir(dir);
@@ -144,22 +165,41 @@ mod tests {
144165
fn cli(binary: &str) -> Cli {
145166
Cli {
146167
binary: PathBuf::from(binary),
147-
gateway: Some("tmachine".to_string()),
168+
gateway: Gateway::named_for_test("tmachine"),
148169
}
149170
}
150171

151172
#[tokio::test]
152-
async fn runs_with_the_target_gateway_selected() {
173+
async fn runs_with_the_gateway_selected() {
153174
let output = cli("sh")
154-
.run(["-c", "echo \"$OPENSHELL_GATEWAY\"; exit 3"])
175+
.run([
176+
"-c",
177+
"echo \"$OPENSHELL_GATEWAY|${OPENSHELL_GATEWAY_ENDPOINT-unset}|$OPENSHELL_WORKSPACE\"; exit 3",
178+
])
155179
.await
156180
.unwrap();
157-
assert_eq!(output.stdout, "tmachine\n");
181+
assert_eq!(output.stdout, "tmachine|unset|default\n");
158182
assert_eq!(output.exit_code, 3);
159183
let err = output.success().unwrap_err();
160184
assert!(err.to_string().contains("exited with 3"), "{err}");
161185
}
162186

187+
#[tokio::test]
188+
async fn runs_with_a_raw_endpoint_selected() {
189+
let cli = Cli {
190+
binary: PathBuf::from("sh"),
191+
gateway: Gateway::endpoint_for_test("http://127.0.0.1:1"),
192+
};
193+
let output = cli
194+
.run([
195+
"-c",
196+
"echo \"${OPENSHELL_GATEWAY-unset}|$OPENSHELL_GATEWAY_ENDPOINT\"",
197+
])
198+
.await
199+
.unwrap();
200+
assert_eq!(output.stdout, "unset|http://127.0.0.1:1\n");
201+
}
202+
163203
#[tokio::test]
164204
async fn parses_json_output() {
165205
let output = cli("sh")

‎tests/testkit/testkit/src/context.rs‎

Lines changed: 4 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -34,13 +34,13 @@ type Registry = Arc<Mutex<BTreeSet<String>>>;
3434
pub struct Context {
3535
client: WorkspaceScopedClient,
3636
grpc: Grpc,
37-
cli: Result<Cli>,
37+
cli: Cli,
3838
run_id: String,
3939
sandboxes: Registry,
4040
}
4141

4242
impl Context {
43-
pub(crate) fn new(client: &OpenShellClient, workspace: &str, cli: Result<Cli>) -> Self {
43+
pub(crate) fn new(client: &OpenShellClient, workspace: &str, cli: Cli) -> Self {
4444
Self {
4545
client: client.workspace(workspace),
4646
grpc: Grpc {
@@ -60,7 +60,7 @@ impl Context {
6060
I: IntoIterator<Item = S>,
6161
S: AsRef<std::ffi::OsStr>,
6262
{
63-
self.cli_driver()?.run(args).await
63+
self.cli.run(args).await
6464
}
6565

6666
/// Like [`Context::cli`], with `dir` as the working directory.
@@ -69,13 +69,7 @@ impl Context {
6969
I: IntoIterator<Item = S>,
7070
S: AsRef<std::ffi::OsStr>,
7171
{
72-
self.cli_driver()?.run_in(Some(dir), args).await
73-
}
74-
75-
fn cli_driver(&self) -> Result<&Cli> {
76-
self.cli
77-
.as_ref()
78-
.map_err(|error| Error::msg(error.to_string()))
72+
self.cli.run_in(Some(dir), args).await
7973
}
8074

8175
/// The ID unique to this test run, used in resource names.

0 commit comments

Comments
 (0)