Skip to content

perf(licenses): prefetch remote licenses concurrently for js lockfiles - #5223

Draft
CAOShurong wants to merge 1 commit into
anchore:mainfrom
CAOShurong:codex/5193-concurrent-enrich
Draft

perf(licenses): prefetch remote licenses concurrently for js lockfiles#5223
CAOShurong wants to merge 1 commit into
anchore:mainfrom
CAOShurong:codex/5193-concurrent-enrich

Conversation

@CAOShurong

Copy link
Copy Markdown
Contributor

Fixes #5193

@CAOShurong

Copy link
Copy Markdown
Contributor Author

Marked as draft while CI runs — will mark ready for review once checks pass. Full description below.


Problem

With --enrich javascript (or search-remote-licenses: true), each package constructor calls getLicenseFromNpmRegistry inline, so a lockfile with N entries makes N strictly sequential HTTP round-trips. For a ~1500-package pnpm-lock.yaml, the reporter measured ~10 minutes, almost all of it waiting on the registry.

The lookups are independent of one another, so there is no reason for them to serialize.

Change

  • Add a shared bounded-concurrency helper, licenses.Prefetch(ctx, keys, n, fn) (worker pool, default 8), which logs rather than propagates individual failures.
  • In the three JS lockfile parsers (package-lock.json, pnpm-lock.yaml, yarn.lock), collect the (name, version) pairs that will need a remote lookup and warm a per-parse in-memory cache with concurrent registry requests before constructing packages.
  • The per-package constructors consult the cache first; on a miss they fall back to the original inline request path, so behavior (including error logging) is unchanged for anything the prefetch could not fetch.
  • Treat HTTP 404 from the npm registry as an explicit error instead of trying to decode an error document as JSON.

Dev-dependency filtering and alias handling are applied before building the pair list, so no extra requests are issued for packages that would be skipped anyway.

Verification

  • go build ./... exit 0; go vet clean on both touched packages
  • go test ./syft/pkg/cataloger/internal/licenses/ -count=1 — all pass (4 new tests: every key attempted, empty-keys no-op, continue-after-error, concurrency high-water-mark ≤ limit)
  • go test ./syft/pkg/cataloger/javascript/ -run "TestNpmLicensePrefetch" -count=1 -v — 4 new tests pass:
    • exactly one request per pair
    • no requests when SearchRemoteLicenses is false
    • concurrency proof: server holds each request open until all N arrive (would time out if serialized)
    • a failing lookup does not prevent later lookups
  • Pre-existing failures in the package suite (Test_JavascriptCataloger, glob tests, yarn dev-dependency tests) reproduce identically on unmodified main via stash-run baseline — environment-dependent, untouched by this change.

@CAOShurong
CAOShurong force-pushed the codex/5193-concurrent-enrich branch from 8c24761 to 2909922 Compare August 24, 2026 11:57
With --enrich javascript (or search-remote-licenses), each package
constructor performed its own npm registry request inline, so a lockfile
with N entries made N strictly sequential HTTP round-trips. For a
1500-package pnpm-lock.yaml this dominated scan time
(see anchore#5193).

Collect the (name, version) pairs during parsing and warm a per-parse
in-memory cache with bounded-concurrency lookups (8 workers) before the
package constructors run; constructors then resolve locally, falling
back to the original inline request when an entry was not prefetched.

Also treat 404 from the registry as an explicit error instead of trying
to decode an error document as JSON.

Fixes anchore#5193

Signed-off-by: Shurong Cao <CAOShurong@users.noreply.github.com>
@CAOShurong
CAOShurong force-pushed the codex/5193-concurrent-enrich branch from 2909922 to bc6719d Compare August 24, 2026 12:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support for concurrent package registry requests when using --enrich

1 participant