Repository navigation
test(conformance): handle gateways whose sandboxes can't reach the test machine #5409
Description
User Story
I run openshell-conformance, or the Kubernetes E2E with OPENSHELL_E2E_KUBERNETES_FEATURES=e2e,e2e-kubernetes, against a remote Kubernetes or OpenShift gateway. Its sandboxes can't reach the machine running the tests. I want conformance to tell me that this environment requirement isn't met, instead of failing tests that say nothing about whether my gateway meets the contract.
Description
The credential_gating conformance tests (#5318) start servers on the machine that runs the tests (tests/support/host-server) and reach them from sandboxes as host.openshell.internal. When sandboxes can't reach that machine, for example on a remote cluster, these tests fail.
Before #5318, the same scenarios lived in e2e/rust behind the e2e-host-gateway feature, so OPENSHELL_E2E_KUBERNETES_FEATURES=e2e,e2e-kubernetes excluded them on a remote cluster. openshell-conformance has no feature selection, so it now runs them unconditionally. The compatibility review on #5318 flagged this, and the review accepted it as a tracked follow-up.
Options:
- Keep an opt-in. Gate the host-dependent conformance tests behind a declared environment capability, such as an env var or a
openshell-conformanceflag, ase2e-host-gatewaydid. - Move them to a feature suite. Keep conformance free of tests that need the test machine, and run
credential_gatingfromtests/suites/features/on lanes that provide reachability. - Check reachability up front. Before the host-dependent tests run, start a host server and have a sandbox connect to it through
host.openshell.internal. If that fails, skip those tests with a clear message that names the missing requirement.
Context
tests/suites/conformance/src/credential_gating.rs,tests/support/host-server/.TESTING.md(remote cluster section) andtests/suites/conformance/README.mdstate the requirement.e2e/rust/e2e-kubernetes.sh:OPENSHELL_E2E_KUBERNETES_FEATURES,e2e-host-gateway.tests/suites/features/host-gatewaychecks the same reachability on its own.- Other tests ported later (test(e2e): decide which e2e/rust tests to port and in what order #5297) that use the test machine as an upstream will have the same requirement.
Definition of Done
- One option is chosen, with the reasoning recorded here.
-
openshell-conformanceagainst a gateway whose sandboxes can't reach the test machine doesn't report the host-dependent tests as failures of the gateway, and its output names the missing requirement. - Lanes that provide reachability still run the host-dependent tests.
-
TESTING.mdandtests/suites/conformance/README.mddescribe how to run conformance against a remote cluster.
Activity
elezar commentedon Nov 10, 2026
Suggested framing for this issue: the question is where the host-dependent credential-gating tests belong, not only how conformance behaves on remote clusters.
TESTING.mdsays conformance tests run on every compute driver, and that behaviour only some drivers have belongs intests/suites/features. Four of the six credential-gating tests make a sandbox connect to a server on the test machine throughhost.openshell.internal:profile_rest_endpoint_keeps_body_placeholder,profile_websocket_endpoint_denies_binary_frames,policy_binding_rest_body_keeps_placeholder_without_rewriteandpolicy_binding_rest_body_rewrite_sends_secret_upstream. That is the behaviourfeatures/host-gatewayalready covers as a feature.Proposal:
host-gateway, gated and skipped the same way, so a gateway on a remote cluster passes conformance again.policy_binding_requires_inspection_or_opt_inandpolicy_binding_updates_provider_environment_livein conformance. Neither connects to the host server. It is started only for a free port. Consider a fixed port and the base image there, which would also drop the Python image dependency from conformance.create_sandbox,sandbox_with,run_python,install_gating_provider, the recorders) live in the conformance crate, so the move needs them intestkitor a support crate.check-conformance-program.shand the JUnit expectations for hard-coded test lists, and drop the README sentence saying the program rules out remote Kubernetes clusters.elezar commentedon Nov 10, 2026
One more point for the split:
policy_binding_requires_inspection_or_opt_in, which stays in conformance, asserts on the gateway's error wording ("credentialed endpoint", "L4-only", and "tls:" with "skip"). The wording checks come from the #5264 tests.TESTING.mdsays conformance tests assert on the public contract and not on messages.If that wording is in the published docs or the
proto/comments, the assertions are fine. If it isn't, assert only theFailedPreconditioncode, which the test already checks, or document the wording first.