feat(image-test): run LISA test suites locally via 'image test'

[liunan-ms]

Summary

Previously LISA-type test suites were rejected by azldev image test and only served as metadata for external orchestration. This PR adds a local runner that generates a LISA runbook from a suite's configured test cases, boots the image in a QEMU VM, and runs the tests.

Changes

• Extend LisaConfig with a framework git source, test-cases, pip pre-install/extras, and extra-args; validate that the framework ref is a full commit SHA and that test-cases is non-empty.
• Generate a qemu-platform runbook that inlines the image path and an ephemeral admin key, with keep_environment=no so VMs are torn down after the run.
• Auto-generate and clean up an ephemeral admin SSH key pair per run.
• Require the image to already be qcow2 (reject other formats) before booting.
• Regenerate CLI docs, JSON schema, and scenario snapshots.

Testing

• mage unit — pass
• mage check all — pass (golangci-lint, editorconfig, staticcheck, licenses, ruff, pyright)
• mage scenarioUpdate — snapshots regenerated

Usage in microsoft/azurelinux#18042

[Copilot]

Pull request overview

This PR enables running LISA-based image test suites locally via azldev image test, expanding the existing test-suite model (previously only pytest was runnable) to support generating a LISA runbook, booting the image via QEMU, and executing LISA against it.

Changes:

• Extend test-suite configuration with an optional [lisa] subtable (LisaConfig) including framework git source + test cases and related options, plus validation of the framework ref and presence of test cases.
• Add a local LISA runner that clones the framework repo at a pinned commit, creates/reuses a per-suite venv, generates a qemu-platform runbook, generates an ephemeral SSH keypair, and invokes LISA.
• Regenerate schema, CLI docs, and scenario snapshots to reflect the new configuration surface.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
schemas/azldev.schema.json	Generated JSON schema updates to include GitSourceConfig and LisaConfig.
scenario/snapshots/TestSnapshotsContainer_config_generate-schema_stdout_1.snap	Snapshot refresh for schema generation output (container mode).
scenario/snapshots/TestSnapshots_config_generate-schema_stdout_1.snap	Snapshot refresh for schema generation output (non-container mode).
internal/projectconfig/testsuite.go	Adds LisaConfig, GitSourceConfig, and new validation logic for LISA suites.
internal/projectconfig/testsuite_test.go	Unit tests for new LISA validation behavior.
internal/projectconfig/loader_test.go	Loader test updated to parse and validate a LISA suite config.
internal/app/azldev/cmds/image/test.go	Routes lisa suites to the new local runner and updates help text.
internal/app/azldev/cmds/image/lisarunner.go	New implementation: clone framework, venv setup, runbook generation, qcow2 enforcement, and LISA invocation.
internal/app/azldev/cmds/image/lisarunbook.go	New runbook YAML model + generator for QEMU platform.
internal/app/azldev/cmds/image/lisarunbook_internal_test.go	Unit tests for runbook YAML generation.
docs/user/reference/cli/azldev_image_test.md	Regenerated CLI docs describing LISA suite behavior.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

[Tonisal-byte]

+1, be sure to give test-suites.md a once over to confirm the new implementation details are correct

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

internal/projectconfig/testsuite.go:147

• Error wrapping in GitSourceConfig.Validate doesn't follow the repo convention of adding a newline before the wrapped error ("context:\n%w"). As written, nested errors become harder to read and inconsistent with surrounding codebase patterns.

	if err := validateCommitSHA(g.Ref); err != nil {
		return fmt.Errorf("%s: %w", context, err)
	}

docs/user/reference/config/test-suites.md:54

• This section says a [test-suites.<name>.lisa] subtable "must" be provided, but the implementation/schema treat it as optional (a type = "lisa" suite may omit lisa, and only suites with lisa config are runnable locally). The doc should match the actual behavior to avoid confusing users migrating existing metadata-only LISA suites.

When `type = "lisa"`, a `[test-suites.<name>.lisa]` subtable must be provided. LISA suites are executed locally by `azldev`: it clones the LISA framework at a pinned commit, creates (or reuses) a Python virtual environment and installs the framework into it, generates a runbook from the configured `test-cases`, and boots the image in a QEMU VM to run those cases. VMs are torn down after the run.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

internal/projectconfig/testsuite.go:147

• Error wrapping in [GitSourceConfig.Validate] is inconsistent with the repo’s established pattern of putting a newline before a wrapped error. This makes multi-layer errors harder to read (and differs from other error wraps in this file, e.g. merge failures).

	if err := validateCommitSHA(g.Ref); err != nil {
		return fmt.Errorf("%s: %w", context, err)
	}

internal/projectconfig/testsuite.go:196

• For type = "lisa", config validation currently allows omitting the [lisa] subtable, but azldev image test will always call RunLisaSuite, which errors when suiteConfig.Lisa is nil. This means invalid configs are accepted at load time and only fail later at runtime. Consider requiring the [lisa] subtable when Type is lisa (and validating framework + non-empty test-cases unconditionally) so users get a clear config error up front and docs/schema stay consistent.

	case TestTypeLisa:
		// The [lisa] subtable is optional; when present it must be internally consistent.
		if t.Lisa != nil {
			frameworkContext := fmt.Sprintf("test suite %#q lisa.framework", t.Name)
			if err := t.Lisa.Framework.Validate(frameworkContext); err != nil {

internal/projectconfig/testsuite.go:118

• The comment for [LisaConfig.PipExtras] says extras are installed as pip install -e ".[...]", but the current runner builds the target as frameworkDir + "[extras]". Either update the comment to match the implementation, or change the implementation to match the documented command format to avoid confusion for maintainers.

	// PipExtras lists pip extras to install from the LISA framework package (e.g., "azure",
	// "legacy"). These are appended to the pip install command as pip install -e ".[extra1,extra2]".
	PipExtras []string `toml:"pip-extras,omitempty" json:"pipExtras,omitempty" jsonschema:"title=Pip extras,description=Pip extras to install from the LISA framework package"`

internal/app/azldev/cmds/image/lisarunner.go:270

• ensureGitRepo caches clones only by the first 12 chars of ref. If the configured git-url changes but the ref stays the same (or shares the same prefix), azldev may silently reuse a checkout from a different repository. Including the URL in the cache key (or otherwise verifying the repo identity) would prevent running tests against the wrong framework source.

	shortSHA := source.Ref[:shortSHALength]
	repoDir := filepath.Join(baseDir, category, shortSHA)

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

docs/user/reference/config/test-suites.md:54

• This section says a [test-suites.<name>.lisa] subtable “must be provided”, but TestSuiteConfig.Validate() explicitly allows type = "lisa" suites with no [lisa] subtable. Update the wording to match the implementation and indicate that the subtable is only required for local execution.

When `type = "lisa"`, a `[test-suites.<name>.lisa]` subtable must be provided. LISA suites are executed locally by `azldev`: it clones the LISA framework at a pinned commit, creates (or reuses) a Python virtual environment and installs the framework into it, generates a runbook from the configured `test-cases`, and boots the image in a QEMU VM to run those cases. VMs are torn down after the run.

internal/app/azldev/cmds/image/lisarunner.go:270

• The framework checkout directory is keyed only by the first 12 chars of the commit SHA. That can accidentally reuse the wrong repository if git-url changes (or two different repos happen to share the same short SHA prefix), which could run tests against an unexpected framework checkout. Include a stable URL-derived component in the cache directory so different git-url values don’t collide.

	shortSHA := source.Ref[:shortSHALength]
	repoDir := filepath.Join(baseDir, category, shortSHA)

docs/user/reference/config/test-suites.md:14

• The table currently marks the lisa subtable as required whenever type = "lisa", but the code allows LISA suites without a [lisa] subtable (metadata-only suites). This doc row should reflect that the subtable is optional, and clarify that it’s required only when you intend to run the suite locally via azldev image test.

This issue also appears on line 54 of the same file.

| Lisa | `lisa` | table | When `type = "lisa"` | LISA-specific configuration (see below) |

internal/app/azldev/cmds/image/lisarunner.go:46

• If a suite is type = "lisa" but omits the optional [lisa] subtable, this error message doesn’t explain what the user needs to add to make local execution work. Consider making the message actionable by pointing to the missing TOML keys/subtable required by the local runner.

	if lisaConfig == nil {
		return fmt.Errorf("test suite %#q is missing lisa configuration", suiteConfig.Name)
	}

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

internal/app/azldev/cmds/image/lisarunner.go:275

• ensureGitRepo caches clones under a directory derived only from the commit SHA prefix. If the configured 'git-url' changes (e.g., switching to a fork) but the ref stays the same, azldev will silently reuse the previously cloned repository from the old URL, which can run the wrong framework code. Key the clone directory by both URL and ref (or verify the existing repo's origin URL and re-clone on mismatch).

	shortSHA := source.Ref[:shortSHALength]
	repoDir := filepath.Join(baseDir, category, shortSHA)

internal/app/azldev/cmds/image/lisarunner.go:241

• The ssh-keygen invocation doesn't wire stdout/stderr. If key generation fails, the user may lose the underlying error details, making failures hard to diagnose. Pipe stderr (and optionally stdout) through so errors are visible (or capture stderr into the returned error).

	keygenCmd := exec.CommandContext(
		env, "ssh-keygen", "-t", "rsa", "-b", "4096", "-f", privateKeyPath, "-N", "", "-q",
	)

	cmd, err := env.Command(keygenCmd)