feat(sources): add opt-in %fedora_upstream_* provenance macros (replaces #285)

[liunan-ms]

Summary

This PR supports azldev emits %fedora_upstream_version and %fedora_upstream_release macros for Fedora upstream components that opt in via a new build.emit-upstream-provenance flag. Values are read from the pristine upstream Fedora spec on disk (before overlays are applied) and the Release tag's %{?dist} token is expanded to the resolved Fedora dist tag (e.g. .fc43). The macros are surfaced through the component's existing generated macros file.

This is a replacement for #285 ("capture upstream version/release provenance and expose %azl_upstream_* macros"). That PR stored the version and release in the per-component lock file and derived the macros at build time from the lock. This PR takes a different, simpler approach.

Closed #285.

Advantages over #285

Aspect	#285 (lockfile-based)	This PR (render-time)
Data flow	Fetch NEVR at component update → store in lockfile → read at render/build	Read NEVR fresh at render/build from the pristine upstream spec that is already on disk
Lockfile schema	+4 fields per component (upstream-name, upstream-epoch, upstream-version, upstream-release)	No schema change
update command	New EVR-capture step, freshness/backfill guard, hasLockedUpstreamVersionRelease gate	Untouched
SourceIdentity plumbing	New struct + ResolveIdentity/ResolveSourceIdentity signature change threaded through Fedora, RPM, and local providers	Untouched
Network at resolution time	Extra clone + git show <commit>:<name>.spec per component during identity resolution	None (the render pipeline already clones the pristine spec)
Failure surface	Clone failure fatal for pinned commits (deterministic identity); best-effort for spec parse	Non-fatal Warn on missing/unparseable spec (opt-in components only)
Scope of rendered-output change	All Fedora upstream components acquire 4 new lock fields	Opt-in: only components that need the macros (currently just grub2 for SBAT) get a .azl.macros file
Provenance freshness	Cached in lock; can become stale if commit pin advances without an update run	Always reflects the pinned upstream commit
Diff size	~18 files, ~650 LOC	8 files (excl. regenerated schema), ~200 net LOC
Macro naming	%azl_upstream_*	%fedora_upstream_* (explicit about the source)

Design

• New option: sources.WithUpstreamProvenance(distTag string) on SourcePreparer. distTag is the resolved %{?dist} expansion (FedoraDistTag(distroName, releaseVer) returns .fc<N> for Fedora, "" otherwise, disabling the feature).
• Hook point: writeMacrosFile (in the pre-overlay window). The pristine spec exists on disk right after FetchComponent, before overlays rewrite it. parseSpecVersionRelease reads Version/Release verbatim via the existing spec.VisitTagsPackage path; only %{?dist} is substituted.
• Opt-in flag: build.emit-upstream-provenance = true in [components.<name>.build]. Fingerprintable — flipping it changes rendered output. Registered with the component history collector.
• Precedence: build.defines wins (macros are layered via setMacroIfAbsent).
• Emission mechanism unchanged: reuses the existing .azl.macros + %{load:%{_sourcedir}/...} + Source9999 path. No second mechanism.
• Gates in order: dist-tag non-empty → opt-in flag → spec.type = upstream. Local, SRPM, and non-Fedora upstream components are skipped, so the flag is a no-op for them.
• Failure mode: best-effort. If the opt-in is set but the spec can't be located or parsed, a WARN is logged and the macros are skipped; render/build proceeds without them.

Consumer example (grub2 SBAT)

An accompanying comp.toml change in the azurelinux repo enables the flag and adds a spec-search-replace overlay:

[components.grub2.build]
emit-upstream-provenance = true

[[components.grub2.overlays]]
description = "Use %fedora_upstream_* macros for the grub.rh SBAT vendor_version (@@VERSION_RELEASE@@)"
type = "spec-search-replace"
regex = '-e "s,@@VERSION_RELEASE@@,%\{version\}-%\{release\},g"'
replacement = '-e "s,@@VERSION_RELEASE@@,%{?fedora_upstream_version}-%{?fedora_upstream_release},g"'

Result: the built sbat.csv's grub.rh line carries the pristine upstream Fedora NEVR (e.g. grub.rh,2,Red Hat,grub2,2.12-45.fc43,...) instead of the AZL build V-R (…azl4).

Testing

• mage unit — passing (7 new tests cover happy path, opt-out, local skip, non-Fedora skip, user-define wins, missing spec best-effort, empty release, and dist-tag mapping).
• mage check all — passing (linting, static analysis, license, schema).
• mage docs regenerated the JSON schema for the new field.
• grub2 build succeeded and the SBAT overlay is verified working using command:

rpm2cpio base/out/rpms/rpm-base/grub2-efi-x64-2.12-46.azl4.x86_64.rpm \
  | cpio -i --to-stdout ./boot/efi/EFI/azurelinux/grubx64.efi 2>/dev/null \
  | objcopy -O binary --only-section=.sbat /dev/stdin /dev/stdout \
  | tr -d '\000'

Both EFI binaries (x64 and ia32) carry identical SBAT data:

sbat,1,SBAT Version,sbat,1,https://github.com/rhboot/shim/blob/main/SBAT.md
grub,5,Free Software Foundation,grub,2.12,https//www.gnu.org/software/grub/
grub.rh,2,Red Hat,grub2,2.12-40.fc43,mailto:secalert@redhat.com

Fix: AB#12798
ADR: AB#28331

Limitations (documented)

• Tag values are read as plain text; only %{?dist} is expanded. A tag built from other macros (e.g. Version: %{majorver}.%{minorver}) is captured with those macros unexpanded.
• Listing fedora_upstream_version / _release in build.undefines does not suppress the injected macros. Use emit-upstream-provenance = false (or omit it) instead.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

[Tonisal-byte]

This forwards the raw [Release] tag rather than the resolved Fedora release. For an upstream spec using [Release: %autorelease], this writes %fedora_upstream_release %autorelease; later expansion occurs against azldev’s overlay/synthetic Git history, not the pristine Fedora dist-git history. The same issue applies to macro-defined [Version]/[Release] values

[liunan-ms]

Thanks @Tonisal-byte ! %autorelease is now resolved eagerly at macros-file-write time, which happens when the upstream .git is still pristine (before overlays and synthetic commits are layered on). azldev shells out to rpmautospec calculate-release --complete-release <spec> in that checkout, so the count comes from Fedora's history.

I validated this end-to-end on fwupd-efi (Release: %autorelease): the built binary's .sbat section now shows:

fwupd-efi.fedora,1,The Fedora Project,fwupd-efi,1.8-1.fc43,https://src.fedoraproject.org/rpms/fwupd-efi

Macro-defined Version/Release (e.g. Version: %{majorver}.%{minorver}) are still emitted unexpanded, but because the macros file is loaded back into the same spec via %{load:...}, RPM expands them lazily at build time against the spec's own definitions, so the consuming spec sees the correct value. The one residual caveat is that lazy expansion resolves against the overlaid spec, but it only happens when an overlay redefined the version macros. I opted not to fully expand these eagerly (e.g. via rpmspec -q) because that'd be a much larger change than the bounded rpmautospec dependency and it's very rare in practice. It's documented as a known limitation.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

internal/app/azldev/core/sources/upstream_provenance.go:211

• rpmautospec calculate-release failures are currently logged without stderr, which makes diagnosing why %fedora_upstream_release was skipped much harder (and diverges from the established pattern elsewhere, e.g. internal/utils/git/git.go:244-256). Capture stderr on the exec.Cmd and include it in the warning when the command fails.

	execCmd := exec.CommandContext(ctx, rpmautospecBinary, "calculate-release", "--complete-release", specPath)
	execCmd.Dir = sourcesDir

	cmd, err := p.cmdFactory.Command(execCmd)
	if err != nil {

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

internal/app/azldev/core/sources/upstream_provenance.go:166

• Autorelease and dist-macro detection here is too narrow: it only recognizes the bare "%autorelease" substring and only expands "%{?dist}". Fedora specs can also use braced/conditional forms like "%{autorelease}" / "%{?autorelease}" (see internal/app/azldev/core/sources/release.go:25), and some specs use "%{dist}". In those cases this code will skip rpmautospec resolution and/or leave the dist macro to expand in the Azure Linux buildroot, producing an incorrect %fedora_upstream_release.

	if !strings.Contains(release, autoreleaseMacro) {
		return strings.ReplaceAll(release, distPlaceholder, p.upstreamDistTag)
	}

	number := p.calculateAutorelease(ctx, componentName, sourcesDir, specPath)

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

internal/app/azldev/core/sources/upstream_provenance.go:165

• When a pristine Release tag contains %autorelease with additional suffix/prefix text (e.g. %autorelease.1%{?dist}), the current implementation returns number + p.upstreamDistTag, which discards the rest of the Release expression and can emit an incorrect %fedora_upstream_release value. Preserve the full Release string by substituting %autorelease with the computed number (and still expanding %{?dist}).

	return number + p.upstreamDistTag

internal/app/azldev/core/sources/upstream_provenance.go:189

• In dry-run mode, cmd.RunAndGetOutput intentionally returns an empty string without executing rpmautospec, but the current parsing treats empty output as an invalid release number and logs a WARN. This produces noisy/incorrect warnings during dry runs. Skip %autorelease resolution entirely when p.dryRunnable.DryRun() is true (debug-level is fine).

	if p.cmdFactory == nil || !p.cmdFactory.CommandInSearchPath(rpmautospecBinary) {
		slog.Warn("Skipping %fedora_upstream_release; rpmautospec is not available to resolve %autorelease",
			"component", componentName)

		return ""
	}

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

internal/app/azldev/core/sources/upstream_provenance.go:158

• resolveUpstreamRelease only detects the bare %autorelease token via strings.Contains(release, "%autorelease"). Elsewhere in this package (release.go) %autorelease detection supports braced/conditional forms like %{autorelease} and %{?autorelease}. With the current logic, those forms will be treated as non-autorelease and could be emitted verbatim into %fedora_upstream_release, causing build-time expansion against the Azure Linux history/macros (the case this feature is trying to avoid). Also, %{dist} (non-conditional) won’t be substituted, so the macro can still pick up the buildroot’s dist tag instead of Fedora’s.

	if !strings.Contains(release, autoreleaseMacro) {
		return strings.ReplaceAll(release, distPlaceholder, p.upstreamDistTag)
	}

docs/user/reference/config/components.md:222

• The example uses %{fedora_upstream_version} / %{fedora_upstream_release} unconditionally, but this feature is best-effort (macros can be skipped if the spec can’t be parsed or rpmautospec is unavailable for %autorelease). Using the conditional form (%{?name}) avoids hard failures when the macros are absent.

%sbat_generate_metadata ... derived from grub2 %{fedora_upstream_version}-%{fedora_upstream_release}

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

internal/app/azldev/core/sources/upstream_provenance.go:172

• The explanation about replacement order is misleading: strings.ReplaceAll does exact substring replacement, so replacing %{dist} cannot partially match %{?dist} and leave a stray %{? behind. This comment could confuse future maintainers about why the order matters.

// replaceDistTag substitutes both the conditional (%{?dist}) and unconditional
// (%{dist}) dist macros in a Release value with the resolved Fedora dist tag.
// The conditional form is replaced first so the %{dist} pass does not match the
// inner {dist} of a %{?dist} token and leave a stray "%{?" behind.

internal/app/azldev/core/sources/upstream_provenance.go:223

• When rpmautospec calculate-release fails, the warning logs only include the Go error. Capturing stderr (like internal/utils/git/git.go:238-256) makes failures diagnosable, especially when rpmautospec emits useful details on stderr.

	// --complete-release prints the release with any prerelease flags but without
	// the dist tag, which is exactly what we combine with the resolved Fedora
	// dist tag.
	execCmd := exec.CommandContext(ctx, rpmautospecBinary, "calculate-release", "--complete-release", specPath)
	execCmd.Dir = sourcesDir

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

internal/app/azldev/core/sources/upstream_provenance.go:207

• calculateAutorelease sets execCmd.Dir = sourcesDir but passes specPath that is already joined with sourcesDir (via findSpecInDir). If sourcesDir/specPath are relative (common for component prepare-sources -o ./...), rpmautospec will be invoked from sourcesDir with an argument like ./out/comp/comp.spec, which resolves to sourcesDir/./out/comp/comp.spec and fails to find the spec. Pass only the spec filename (or filepath.Base(specPath)) when setting Dir.

	execCmd := exec.CommandContext(ctx, rpmautospecBinary, "calculate-release", "--complete-release", specPath)
	execCmd.Dir = sourcesDir

[Copilot]

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

internal/app/azldev/cmds/component/diffsources.go:104

• component diff-sources wires WithUpstreamProvenance(...) but does not pass WithMockProcessor(...). For opted-in Fedora upstream specs using %autorelease, this means %fedora_upstream_release will be skipped here even though build/render can resolve it, so diff output can diverge from build/render behavior.

	preparer, err := sources.NewPreparer(sourceManager, env.FS(), env, env,
		sources.WithUpstreamProvenance(sources.FedoraDistTag(distro.Ref.Name, distro.Version.ReleaseVer)))
	if err != nil {
		return nil, fmt.Errorf("failed to create source preparer:\n%w", err)
	}

internal/app/azldev/core/sources/mockprocessor.go:296

• git config --global --add safe.directory '*' disables Git’s dubious-ownership protection for all paths inside the chroot. Since the bind mount location is known (/tmp/provenance), this can be scoped to just that directory to reduce the blast radius while still avoiding ownership-mismatch failures.

	// safe.directory guards against any residual host/chroot ownership mismatch
	// on the bind-mounted .git, mirroring the render batch path. Wrapped in
	// `sh -c` so the shell operators survive shellquote joining in CmdInChroot.
	shellCmd := "git config --global --add safe.directory '*' && " +
		"rpmautospec calculate-release --complete-release " + shellquote.Join(specInChroot)

internal/app/azldev/cmds/component/preparesources.go:166

• prepare-sources enables upstream provenance via WithUpstreamProvenance(...), but never supplies a MockProcessor (WithMockProcessor(...)). As a result, opted-in Fedora upstream components whose pristine spec uses %autorelease will consistently miss %fedora_upstream_release in this code path (even when WithGitRepo is enabled), diverging from build/render behavior.

	opts = append(opts,
		sources.WithUpstreamProvenance(sources.FedoraDistTag(distro.Ref.Name, distro.Version.ReleaseVer)))

[Tonisal-byte]

Looks good to me!

[Tonisal-byte]

Add some unit tests for CalculateRelease please

[Copilot]

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated no new comments.

Suppressed comments (2)

internal/app/azldev/cmds/component/diffsources.go:102

• component diff-sources enables WithUpstreamProvenance(...) but never passes WithMockProcessor(...). For Fedora upstream specs whose pristine Release: uses rpmautospec %autorelease, this means %fedora_upstream_release can never be resolved in diff output (even when the project has a mock config), diverging from component render/component build behavior and the documented best-effort semantics (skip only when no mock config or rpmautospec fails).

	preparer, err := sources.NewPreparer(sourceManager, env.FS(), env, env,
		sources.WithUpstreamProvenance(sources.FedoraDistTag(distro.Ref.Name, distro.Version.ReleaseVer)))
	if err != nil {

internal/app/azldev/cmds/component/preparesources.go:166

• component prepare-sources wires WithUpstreamProvenance(...) but does not provide WithMockProcessor(...). As a result, opted-in Fedora upstream components whose pristine spec uses rpmautospec %autorelease will always skip emitting %fedora_upstream_release in this command, even when a project mock config exists (unlike component render/component build). Consider creating a mock processor for the command (best-effort when config exists) and passing it via sources.WithMockProcessor(...), with a corresponding cleanup at the command boundary.

	opts = append(opts,
		sources.WithUpstreamProvenance(sources.FedoraDistTag(distro.Ref.Name, distro.Version.ReleaseVer)))