feat: add shared smoke test framework to mcp_common.testing

[markballew]

Problem

Each MCP repo needs read-only smoke tests that verify CLI and MCP tools work against live backends. Currently:

• mcp_common.testing provides mcp_client, assert_tool_exists, assert_tool_success for in-process MCP testing
• pyproject.toml across all MCP repos declares smoke / integration / e2e markers but no shared test patterns exist for smoke tests
• CI workflows in netbox-mcp and redfish-mcp have ad-hoc "smoke" steps (import check, --help, pytest) but nothing reusable
• Every MCP repo reimplements .env loading, CLI subprocess helpers, credential skip logic, and MCP client initialization

The awx-mcp repo now has a reference implementation (tests/test_smoke.py, PR #25) that exercises both CLI and MCP paths. The reusable patterns should be extracted to mcp_common.testing.

What to Extract

1. .env loader for test contexts

from mcp_common.testing import load_dotenv_for_tests

load_dotenv_for_tests()  # searches .env up the directory tree

Simple helper that loads .env without requiring python-dotenv — just key=value parsing with comment/quote stripping. Every MCP smoke test needs this since secrets live in .env locally.

2. CLI subprocess runner

from mcp_common.testing import cli_runner, cli_json

# Run any CLI command, auto-discovers binary or falls back to `uv run`
result = cli_runner("awx-cli", "ping")
assert result.returncode == 0

# Run with --json and return parsed output
data = cli_json("netbox-cli", "search", "dcim.device", "--query", "gpu001")
assert data["count"] > 0

Features:

• Auto-discovers CLI binary on PATH, falls back to uv run <name>
• Configurable timeout (default 30s)
• Sets cwd to repo root
• cli_json variant parses JSON output

3. Credential-gated skip decorator

from mcp_common.testing import require_env

@require_env("AWX_HOST", "AWX_TOKEN")
class TestAWXSmoke:
    ...

@require_env("NETBOX_URL", "NETBOX_TOKEN")
class TestNetBoxSmoke:
    ...

Replaces the boilerplate pytestmark = [pytest.mark.skipif(not os.getenv(...), reason=...)] pattern that every MCP repo duplicates.

4. MCP server initializer for smoke tests

from mcp_common.testing import smoke_mcp_client

@pytest.fixture
async def client():
    # Auto-discovers the FastMCP server and initializes backend client from Settings
    async for c in smoke_mcp_client("awx_mcp.server", "mcp", "awx"):
        yield c

Handles the common pattern of:

1. Import the server module
2. Check if the backend client global is None
3. Initialize it from the project's Settings class
4. Wrap in mcp_client()

5. Pytest marker registration

Add smoke to the standard marker set in mcp_common docs/template so mcp-template and all downstream repos get it.

Reference Implementation

See vhspace/awx-mcp PR #25, specifically tests/test_smoke.py:

• _load_dotenv() — .env loading
• _awx_cli() / _cli_json() — CLI subprocess helpers
• pytestmark skip logic — credential gating
• TestMCPSmoke.client fixture — MCP server initialization

Acceptance Criteria

• mcp_common.testing exports: load_dotenv_for_tests, cli_runner, cli_json, require_env, smoke_mcp_client
• Works without additional dependencies (no python-dotenv needed)
• Documented in mcp_common.testing module docstring
• mcp-template updated with example smoke test using the new helpers
• Existing mcp_client, assert_tool_exists, assert_tool_success remain unchanged

Why This Matters

There are 11+ downstream MCP repos. Each will need smoke tests. Without shared helpers, each will reimplement .env loading, CLI subprocess wrappers, and credential skip logic slightly differently. The awx-mcp implementation took ~180 lines — with shared helpers it would be ~50 lines of actual test logic.

[markballew]

Agent Runtime Invocation Patterns (from awx-mcp PR #27 testing)

Tested all 4 agent runtimes for layer-2 skill smoke tests. Here's what works and what doesn't — this should inform the mcp_common.testing framework.

Invocation Reference

Agent	CLI Command	Non-Interactive Flag	Task/Prompt Flag	Status
Cursor	N/A (IDE only)	N/A	N/A	Structural tests only (validate .cursor-plugin/ sync, frontmatter)
Claude Code	claude	--print --dangerously-skip-permissions	-p "prompt"	Works reliably in subprocess
OpenCode	opencode	run subcommand	positional arg after run	Hangs — blocks on stdout in non-TTY subprocess
OpenHands	openhands (often a wrapper script)	--headless --exit-without-confirmation	-t "prompt"	Fails — .openhands/mcp.json tries to start MCP server which crashes

Proposed mcp_common.testing Helpers

from mcp_common.testing import AgentRunner, run_skill_smoke

# Discover available agents and run a skill test across all of them
results = run_skill_smoke(
    skill_path="skills/awx-automation/SKILL.md",
    prompt="Read the skill, run awx-cli ping, report results as JSON",
    agents=["claude", "opencode", "openhands"],  # skips unavailable
    timeout=180,
)

# Or use the low-level runner
runner = AgentRunner.discover()  # finds what's on PATH
for agent in runner.available:
    result = runner.run(agent, prompt="...", timeout=120)

Key Design Decisions for the Framework

1. Agent discovery should be shutil.which() based, with env var overrides (OPENHANDS_BIN, etc.) for non-standard paths
2. Timeout handling is critical — OpenCode hangs forever without one. Use subprocess.run(timeout=...) with proper cleanup
3. Output extraction — agents wrap output in prose/markdown. Need a robust extract_json(text) that tries fenced blocks, inline patterns, then raw parse
4. Skip on known failures — OpenHands MCP connection errors, OpenCode TTY issues should skip gracefully, not fail
5. Cursor is structural only — can't invoke from CLI, but can validate .cursor-plugin/ files are in sync with source

Known Issues to Document

Agent	Issue	Workaround
OpenCode	Hangs in non-TTY subprocess (no output)	120s timeout + skip on timeout
OpenHands	.openhands/mcp.json auto-starts MCP server which may crash	Skip on McpError: Connection closed
OpenHands	Wrapper scripts source .env with set -euo pipefail, broken lines crash the wrapper	Use binary directly, not shell function
Claude Code	Reads cached marketplace skill, not repo source	Test reports stale cache as finding

Reference implementation: https://github.com/vhspace/awx-mcp/pull/27