fix: exclude owned Go modules from OS packages

[Eljees]

Summary

• exclude an embedded Go module when a comprehensive distro package owns it
• add a regression case for a distro package and Go module whose versions differ

Why

The overlap relationship already establishes that the distro package owns the embedded Go module. The previous version-prefix check ran before the comprehensive-distro ownership rule, so a legitimate owned module was retained when its module version differed from the distro package version.

This can leave duplicate package representations available for vulnerability matching even though the OS package is the authoritative representation.

Validation

• git diff --check
• go test ./grype/pkg was attempted with golang:1.26.3; dependency downloads failed with upstream TLS handshake timeouts before the test package could run

[spiffcs]

@Eljees thanks for the PR. Can you provide an example of an incorrect grype vulnerability report that existed before this PR and a new report that highlights the fix?

The previous version-prefix check ran before the comprehensive-distro ownership rule, 
so a legitimate owned module was retained when 
--> its module version differed from the distro package version.

This can leave duplicate package representations available for vulnerability matching 
even though the OS package is the authoritative representation.

Let me know if I'm reading the above spec/description correctly.

If we find there a module version that is different from the distro package version then that suggests there were two modules discovered. We would have a package for both the distro owned package (distro package version) AND the other package that had a version mis match.

If you have an example of an SBOM or vulnerability report that this fixes I'd like to look at those cases to check if we SHOULD be eliminating a package here, or if it's correct to have both.

[Eljees]

Thanks @spiffcs — you were right to push on this. I compared before/after on the pinned quality SBOMs with the same imported DB (candidate build vs v0.116.0), and the evidence does not support the change in its current form.

Concrete counter-example from the quality corpus (docker.io/gitlab/gitlab-ce@sha256:04d4219d…, Ubuntu-based): the omnibus gitlab-ce 15.6.1-ce.0 deb owns 1,836 embedded Go modules via ownership-by-file-overlap (gitaly, gitlab-shell, …). With this PR all of their go-module matches are dropped — 877 on this image alone, including CVE-2023-48795 (Terrapin) against golang.org/x/crypto — and 869 of them have no corresponding OS-package match, because Ubuntu's feed doesn't track vulnerabilities in a third-party deb's vendored Go code. Corpus-wide the change removes 1,271 go-module matches and adds none; 1,195 have no distro-level replacement. So these are real false negatives, not duplicate cleanup: distroFeedIsComprehensive tells us about the distro type, not whether the owning package is actually covered by the distro feed.

I'll narrow this to the case the issue is actually about: only exclude a go-module child when it is the parent package's own main module (e.g. the containerd rpm ↔ github.com/containerd/containerd), keeping dependency modules matchable. Converting to draft while I rework it.

[Eljees]

Reworked in 056da3d as described above. The exclusion now only applies when the embedded Go module appears to be the owning package's own main module — the module path basename matches the package name (containerd rpm ↔ github.com/containerd/containerd), plus stdlib when it is owned by the distro's Go toolchain package. Dependency modules embedded in third-party packages stay matchable.

Validation against the pinned quality SBOMs with a shared DB (same binary pair, before = this branch's base):

• gitlab-ce (Ubuntu omnibus deb owning 1,836 embedded Go modules): zero match changes — the previous revision dropped 877 matches there, 869 of them with no distro-level replacement (incl. CVE-2023-48795 on golang.org/x/crypto).
• azurelinux3 test image: only the stdlib go1.22.5 ↔ golang 1.22.5-1.azl3 rpm duplicates are removed (49 matches; 11 have alias-equivalent CVE matches on the rpm, the rest are absent from the Azure Linux feed, which is authoritative for a comprehensive distro).
• Unit tests updated: the containerd case still excludes; a new case asserts a gitlab-ce-style dependency module is kept.

Marking ready for review again — happy to adjust scope further if you'd prefer the stdlib case split out.

[Eljees]

@spiffcs you're reading it right, and the answer is that the original patch was wrong — it produces false negatives, not duplicate cleanup.

I built this branch and its base with the same imported DB and ran both against the pinned quality-gate SBOM corpus. On the gitlab/gitlab-ce image the omnibus gitlab-ce 15.6.1-ce.0 deb owns 1,836 embedded Go modules by file overlap (gitaly, gitlab-shell, ...). The first revision dropped 877 matches on that image alone — 869 of them with no corresponding OS-package match, including CVE-2023-48795 (Terrapin) on golang.org/x/crypto. Corpus-wide it removed 1,271 go-module matches and added none, 1,195 of which had no distro-level replacement.

So the assumption the patch leaned on doesn't hold: distroFeedIsComprehensive tells us about the distro type, not about whether the owning package's vendored code is covered by that feed. Ubuntu doesn't track vulnerabilities in a third-party deb's vendored Go code, so dropping those children loses real matches.

056da3d narrows it to the case the issue is actually about: an embedded Go module is only excluded when it looks like the owning package's own main module — module path basename matches the package name, e.g. the containerd rpm and github.com/containerd/containerd — plus stdlib when it's owned by the distro's Go toolchain package. Dependency modules embedded in a third-party package's binaries stay matchable.

Re-run on the same binary pair:

• gitlab-ce: zero match changes.
• azurelinux3: only the stdlib go1.22.5 / golang 1.22.5-1.azl3 rpm duplicates are removed (49 matches; 11 have alias-equivalent CVE matches on the rpm, the rest are absent from the Azure Linux feed).

Unit tests updated — the containerd case still excludes, and a new case asserts a gitlab-ce-style dependency module is kept. Happy to split the stdlib half out if you'd rather scope it tighter.

[Eljees]

Ping on this one and #3584, #3606 - the oldest has been open since 18 July. They are independent; happy to rebase any of them.

[Eljees]

@spiffcs — I said on #3584 that I'd stop pinging, so this isn't one. It's an offer to close.

Where this stands: you asked for a concrete before/after, and the honest answer was that my original patch was wrong. Measured against the pinned quality-gate SBOM corpus with a shared imported DB, it removed 1,271 go-module matches corpus-wide and added none — 1,195 with no distro-level replacement, including CVE-2023-48795 on golang.org/x/crypto. distroFeedIsComprehensive tells us about the distro type, not whether the owning package's vendored code is covered by that feed, which is where the assumption broke.

056da3d narrows it to the case the issue is actually about: exclude an embedded Go module only when it looks like the owning package's own main module, plus stdlib owned by the distro's Go toolchain package. On the same corpus that's zero change on gitlab-ce and only the azurelinux3 stdlib duplicates.

If that scope still isn't what you want, say so and I'll close it — no explanation needed, and I won't re-open the topic. If it is what you want, it's green and mergeable and I'll rebase whenever suits you.

[noqt]

Reviewing from NOQT at exact head 056da3df17b22cd5fd23b7b09d8a6acbc76d348d.

The rework correctly preserves embedded dependency modules, but the basename heuristic still recreates the same coverage error for third-party OS packages whose own main module matches the package name. comprehensiveDistroOwner knows only the image distro and package type; it has no repository or source provenance for the deb/rpm.

I added a reviewer-only regression with an Ubuntu image, parent deb:vault@1.16.2-1, and child go-module:github.com/hashicorp/vault@v1.16.2. Current main keeps both packages. This head drops the Go module because vault == path.Base(module), even though an Ubuntu feed cannot be assumed to cover an arbitrary third-party vault deb. That can hide the upstream Go findings the child exists to match.

Please retain the main-module child when versions differ unless there is evidence that the owning package is actually covered by the selected distro feed, and add this third-party-package boundary as a regression. If the toolchain/stdlib case has a stronger provenance signal, that can be split or narrowed independently.

The PR's focused overlap suite, affected-package vet, diff hygiene, and clean merge onto current main f6f2ce0a2293e65c97bf5468926cb0c9ddad822f all pass. The reviewer regression passes on current main and fails on this head; that false-negative boundary is the blocker.

[Eljees]

@noqt I reproduced your scenario as a test case on this exact head, and you are right about the mechanics: with deb:vault@1.16.2-1 owning go-module:github.com/hashicorp/vault@v1.16.2 on an Ubuntu image, the module is removed.

Before concluding that is a defect introduced here, I ran the same scenario with a binary child instead of a Go module, on this branch's base (no changes of mine involved):

deb:vault@1.16.2-1  ->  binary:vault@1.16.2      (ubuntu)
actual: []string{"deb:vault@1.16.2-1"}

The binary child is absorbed too, by the pre-existing overlap rule (version-prefix similarity plus the comprehensive-distro check). The boundary you describe — an OS package the distro feed does not actually cover absorbing its child — is a property of the existing mechanism, because the SBOM carries no repository provenance for a deb/rpm; comprehensiveDistroOwner can only see the image's distro and the package type, exactly as you note. This change deliberately mirrors that same trade-off for the one Go-module shape where the version check cannot fire (module versions carry v prefixes and lack distro revisions), rather than inventing a new, stricter rule for Go modules than the one binaries already live under.

On the specific suggestion to retain the main-module child when versions differ: that gate would also retain deb:containerd@1.7.24~ds1-4ubuntu1 vs go-module:github.com/containerd/containerd@1.7.22, since distro revisions and upstream module tags essentially never match textually — which reintroduces the duplicate this PR (and the linked issue) is about. A version gate that is loose enough to catch the containerd case is loose enough to drop your vault case as well; the two are not separable on version evidence alone.

Where the line could actually be drawn correctly is provenance: if syft ever records the package's origin repository, gating the exclusion (both the existing binary rule and this one) on "the owning package really comes from the distro" would fix your scenario and keep the duplicate suppression. That is out of scope for grype's data today.

If the maintainers prefer to be conservative here, the change can be narrowed to the stdlib-owned-by-toolchain case only, which is provenance-safe and still removes the bulk of the observed duplicates (the azurelinux3 stdlib/golang pairs). @spiffcs — happy to go either way; the corpus numbers for both variants are in the earlier comments.

[noqt]

You're right about the current baseline. I reproduced the binary case on clean main f6f2ce0: deb:vault@1.16.2-1 already absorbs binary:vault@1.16.2 through the version-prefix rule.

That changes my framing, not the merge recommendation. The provenance gap is pre-existing; this PR expands that accepted false-negative trade-off to a new package class whose differing versions previously kept it matchable. Existing binary behavior isn't evidence that the expansion is safe.

Your conservative option is the right repair from NOQT's perspective: narrow this PR to the stdlib-owned-by-go/golang toolchain case and leave the main-module basename heuristic out until package-origin provenance exists. I withdraw my suggestion that a version gate can separate containerd from the third-party case—you've shown that it can't.

So the changes-requested verdict remains, but with the repair path narrowed to the stdlib slice you proposed. Thanks for testing the counterexample instead of hand-waving it away.

[Eljees]

Agreed on the binary case being pre-existing behaviour. Since we now agree the stdlib-owned-by-toolchain slice is the provenance-safe subset, I can push a revision narrowed to just that case whenever @spiffcs wants it — the corpus impact of that slice is the azurelinux3 stdlib/golang pairs measured earlier, and it sidesteps the third-party-package boundary entirely. Holding until the maintainers weigh in, since the main-module-basename heuristic is the part still genuinely in dispute and that call is theirs to make.