Skip to content

Add first-class APS OpenRTB integration - #918

Open
ChristianPavilonis wants to merge 10 commits into
mainfrom
issue-764-aps-openrtb
Open

Add first-class APS OpenRTB integration#918
ChristianPavilonis wants to merge 10 commits into
mainfrom
issue-764-aps-openrtb

Conversation

@ChristianPavilonis

Copy link
Copy Markdown
Collaborator

Summary

  • Replace the legacy APS contextual/TAM request path with a first-class APS OpenRTB provider using /e/pb/bid.
  • Carry minimized, typed APS renderer state through direct auctions, navigation/page-bids, mediation, and GPT/Universal Creative rendering.
  • Add default-off script-creative gating, opaque-origin rendering, nonce-bound messaging, PBS coexistence safeguards, migration docs, and regression coverage.

Changes

File Change
crates/trusted-server-core/src/integrations/aps.rs Builds independent APS OpenRTB requests, validates and reduces responses, creates minimized renderer envelopes, and serves the sandboxed static renderer.
crates/trusted-server-core/src/auction/* Adds typed APS renderer metadata, preserves bid identity through floors and orchestration, and exposes APS winners without adm.
crates/trusted-server-core/src/integrations/prebid.rs Prevents APS from also being sent to Prebid Server for Trusted Server APS cohorts.
crates/trusted-server-core/src/{publisher.rs,creative_opportunities.rs} Carries APS renderer data through navigation/page-bids while retaining legacy slot configuration only for compatibility.
crates/trusted-server-js/lib/src/integrations/aps/render.ts Validates exact renderer descriptors and envelopes, creates opaque sandboxed frames, and enforces fragment-bound nonces and readiness acknowledgements.
crates/trusted-server-js/lib/src/integrations/gpt/index.ts Routes APS winners through the Universal Creative dynamic-renderer bridge without cache fetches, beacons, or native APS display calls.
crates/trusted-server-integration-tests/browser/tests/shared/aps-renderer.spec.ts Verifies CSP sandboxing, nonce and replay protections, same-origin rejection, restrictive-CSP compatibility, and parent-DOM isolation.
trusted-server.example.toml and integration fixtures Adds canonical account_id, the APS OpenRTB endpoint, and default-off allow_script_creatives.
docs/**, CHANGELOG.md, TESTING.md Documents migration, security boundaries, orchestration behavior, rollout gates, and validation coverage.

Closes

Closes #764

Test plan

  • cargo test-fastly && cargo test-axum
  • cargo clippy-fastly && cargo clippy-axum
  • cargo fmt --all -- --check
  • JS tests: cd crates/trusted-server-js/lib && npx vitest run
  • JS format: cd crates/trusted-server-js/lib && npm run format
  • Docs format: cd docs && npm run format
  • WASM build: cargo build --package trusted-server-adapter-fastly --release --target wasm32-wasip1
  • Manual testing via fastly compute serve
  • Other: cargo test-cloudflare, cargo test-spin, all Cloudflare/Spin clippy aliases, 13 cross-adapter parity tests, browser TypeScript compilation, full Next.js/WordPress Playwright suite, TSJS bundle build, and VitePress build

Checklist

  • Changes follow CLAUDE.md conventions
  • No unwrap() in production code — use expect("should ...")
  • Uses project log macros (not println!)
  • New code has tests
  • No secrets or credentials committed

Rollout note

Broad production rollout and production tagtype=script enablement remain gated on controlled APS-account validation and APS account-team confirmation. Script creatives remain disabled by default.

Comment thread crates/trusted-server-js/lib/test/integrations/aps/render.test.ts Fixed
Comment thread crates/trusted-server-js/lib/test/integrations/aps/render.test.ts Fixed
@ChristianPavilonis
ChristianPavilonis marked this pull request as draft July 16, 2026 15:17
@ChristianPavilonis
ChristianPavilonis marked this pull request as ready for review July 17, 2026 02:07

@aram356 aram356 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add APS metadata to result of the auction?

@aram356 aram356 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Replaces the legacy APS TAM/contextual path with a first-class OpenRTB provider, adds a sandboxed opaque-origin creative renderer, a Prebid-adapter rendering route, PBS coexistence safeguards, and an inventory-identity override. The security core is well-hardened and I verified rather than assumed it — allow-same-origin is never combined with allow-scripts (pinned by a regression test), the nonce/envelope contract matches byte-for-byte across Rust and JS, and the two CodeQL alerts were genuinely resolved. CI is green.

That said, I found 7 blocking defects — an auction-wide failure from one malformed bid, two ways APS demand can still reach Prebid Server, two blank-slot / consumed-impression bugs in the new rendering routes, a privacy leak in the identity override, and a silent config break on upgrade. Requesting changes.

A process note: the PR body's change table documents only the first commit. Two later commits ("Add APS inventory identity overrides", "Render APS bids from the trustedServer Prebid adapter") added whole subsystems it omits — worth refreshing before merge.

Most findings are filed as inline comments. The cross-cutting ones are below.

Blocking

🔧 wrench

  • APS exclusion is case-sensitive and leaks via the head-insert path (prebid.rs:1427/1434, prebid.rs:909): Both PBS guards compare name == "aps" exactly, so "APS"/"Aps" in the free-form config.bidders slips through to PBS and is sold twice. Separately, head_inserts serializes config.bidders/client_side_bidders verbatim into window.__tsjs_prebid, so aps still ships to the browser — the safeguard is server-side only, and client_side_bidders is never filtered at all. The durable fix for this and the stored-request finding is to reject aps in [prebid].bidders/client_side_bidders at config-validation time.

Non-blocking

🤔 thinking

  • Full client-controlled Referer path+query now enters the bidstream (formats.rs:93-116): site.page changed from a bare https://{domain} to the full same-origin Referer including query string, so publisher URLs carrying PII (?email=, session tokens, gclid) are forwarded to every SSP — up to 8 KiB of attacker-chosen data (the host check constrains only the authority). This mirrors client-side Prebid so it may be intentional, but for a privacy-preserving edge server it deserves an explicit decision: strip query/fragment by default, or gate behind config.
  • adserver_mock mediation is last-write-wins and the new test enshrines it (adserver_mock.rs:107, :899): build_bid_index keys on (provider, slot, bidder); if APS ever stops reducing to one bid per slot, the mediator restores the losing candidate's renderer against the winning bid's price — wrong creative at wrong price, silently. bid_id now exists and would disambiguate; the mock only echoes crid. At least raise the collision log from debug! to warn!.
  • providers.aps is now write-only and fails silently (creative_opportunities.rs:366,378): the field is a genuine deny_unknown_fields deser-compat shim (removing it hard-fails existing TOML), but to_ad_slot no longer reads it, so an operator who sets aps.slot_id expecting routing gets a no-op with zero diagnostics. Add a load-time warning.

♻️ refactor

  • BidRenderer::aps() is an infallible accessor on a single-variant enum (types.rs:217-225, used at publisher.rs:1957): the moment a second variant lands the natural "fix" is a panic in the fallback arm. Prefer as_aps(&self) -> Option<&ApsRendererV1>. Note the generic bid.bid_id field this PR adds already carries the same value, so publisher.rs could derive hb_adid renderer-agnostically.

📝 note

  • Undocumented /auction wire-contract change: for renderer bids, crid goes from always-present {bidder}-creative to bid.creative_id (absent when the bidder omits crid), id from {bidder}-{slot} to the upstream bid id, and adm is omitted entirely. tsjs absorbs all three, but POST /auction is a documented OpenRTB endpoint; any non-tsjs consumer reading those fields breaks. The CHANGELOG's Breaking section doesn't mention the response shape.
  • Test quality across the new suites: the browser same-origin rejection test (aps-renderer.spec.ts:731-805) is effectively vacuous — a fixed waitForTimeout(100) with no positive control, driven through a hand-rolled harness instead of renderApsCreative, so it would pass if the guard broke. Several sandbox assertions (:464,:555) assert the harness's own input rather than product behavior. And there is no JS-side boundary coverage for the size caps (envelope 256 KiB / base64 349528, account-id 1024, creative-url 4096) nor a regression test for the two blocking rendering-route bugs. (I did confirm the same-origin guard at aps.rs:74 is live, not dead code, via a browser test — so that check is real; the test just doesn't exercise it.)

⛏ nitpick

  • creative_id is the only renderer field without a length cap (aps.rs:682) — bounded by the 2 MiB body cap but inconsistent with account_id/creative_url.
  • dropped_bid_count and drop_reasons don't reconcile (aps.rs:776-799) — price-losers increment the count with no reason entry; the test asserts dropped=1, drop_reasons={}.
  • auction/README.md:120 ASCII box right border is off by one column.
  • trusted-server.example.toml regressed a fictional aps.example.com/e/dtb/bid to a real vendor host; several docs/tests use real *.amazon-adsystem.com endpoints against the project's "example.com only" rule. These are functionally-required vendor endpoints (not customer data or credentials) and amazon-adsystem predates this PR, so noting rather than blocking.

CI Status

  • fmt: PASS
  • clippy (all six adapter targets): PASS
  • rust tests (fastly/axum/cloudflare/spin + parity): PASS
  • js tests (vitest): PASS
  • browser integration + CodeQL: PASS

Comment thread crates/trusted-server-core/src/auction/formats.rs Outdated
Comment thread crates/trusted-server-core/src/auction/formats.rs Outdated
Comment thread crates/trusted-server-core/src/auction/formats.rs Outdated
Comment thread crates/trusted-server-core/src/integrations/prebid.rs Outdated
Comment thread crates/trusted-server-core/src/integrations/prebid.rs Outdated
Comment thread crates/trusted-server-js/lib/src/integrations/prebid/index.ts Outdated
Comment thread crates/trusted-server-js/lib/src/integrations/aps/render.ts
Comment thread crates/trusted-server-js/lib/src/integrations/aps/render.ts Outdated
Comment thread crates/trusted-server-js/lib/src/integrations/aps/render.ts
Comment thread crates/trusted-server-integration-tests/browser/tests/shared/aps-renderer.spec.ts Outdated

@prk-Jr prk-Jr left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

First-class APS OpenRTB integration replacing the legacy contextual/TAM path, with a sandboxed (opaque-origin, nonce-gated, CSP-enforced) renderer that deliberately withholds adm from the client outside that sandbox. The core security design (bid parsing never reads adm, envelope byte-bounds, iframe sandbox without allow-same-origin, identity-checked postMessage, server-side CSP header) is solid and well-tested, including a genuinely adversarial browser test suite (nonce replay, CSP-only isolation with the sandbox attribute omitted, live same-origin attack attempts). The main blocker is that this branch does not currently merge cleanly with main — see Blocking #1, which is also the root cause of the two failing CI checks.

Blocking

🔧 wrench

  • PR does not merge cleanly with main: main's commit 8bb5450dc (#890) added a make_test_bid_with_creative() helper in publisher.rs that builds a Bid{} literal without this PR's new bid_id/creative_id/renderer fields. Rebasing/merging onto current main fails with error[E0063]: missing fields — this is exactly what's failing the cargo fmt and cargo test CI checks shown on this PR (GitHub builds the synthetic merge commit, not the raw branch tip). See inline comment on auction/types.rs.

❓ question

  • Debug mode can leak adm to the browser (crates/trusted-server-core/src/integrations/aps.rs:628): undermines the PR's core claim that adm never reaches the client outside the sandbox. See inline comment.
  • One bad bid can abort the entire /auction response (crates/trusted-server-core/src/auction/formats.rs:313): no multi-slot blast-radius test exists. See inline comment.

Non-blocking

♻️ refactor

  • Bid (types.rs:253) has no #[derive(Default)] despite every field type supporting it — deriving it directly prevents recurrence of Blocking #1 the next time a field is added.
  • aps.rs:1169 — no test exercises the fail-closed startup path (enabled+invalid config → register_providers() returning Err), only ApsConfig::validate() directly.
  • gpt/index.ts:999renderingAdIds grows unbounded on the success path, inconsistent with sibling bounded caches added elsewhere in this PR.

🤔 thinking

  • aps.rs:1047 — context-free parse_response silently discards a valid response if ever called without context; safe today only because all real call sites use parse_response_with_context.
  • aps.rs:380device.language has no explicit byte cap, unlike other forwarded fields in this file.
  • orchestrator.rs:28backend_to_provider's bare 4-tuple is duplicated across 3 sites; same fragility class as Blocking #1.
  • formats.rs:302 — when both bid.creative and bid.renderer are Some, creative silently wins with no log/assert.
  • prebid.rs:1427 — the "aps" bidder exclusion (preventing APS double-serving through Prebid Server) is case-sensitive; pre-existing pattern, worth eq_ignore_ascii_case given the stated goal.
  • publisher.rs:2207 — full client query string is now forwarded into site.page for every provider; unlike aps.rs (8192-byte cap), prebid.rs's equivalent has no bound.
  • render.ts:325 — rapid re-render of the same slot orphans the prior frame's message listener/timeout instead of routing through its own cleanup (self-heals within 10s).
  • TESTING.md:49 — still describes APS as "mocked" / lists "Implement real APS" as a future step, contradicting this PR and the log excerpt this PR itself just updated a few lines above.
  • docs/guide/auction-orchestration.md:623 — config reference table omits inventory_domain/inventory_page_origin, real validated fields documented elsewhere in this PR.

⛏ nitpick

  • aps.rs:851 — a seatbid entry with no bid array is skipped without incrementing drop_reasons, making a no-bid response harder to diagnose.
  • render.test.ts:216 — no test for "correct nonce, wrong event.source" rejection specifically (the guard is implemented correctly at render.ts:351, just not locked in by that exact case).
  • render.ts:388 — the embedded renderer-document string hand-duplicates constants (including the 10000ms timeout) instead of interpolating them.

🌱 seedling

  • Byte-limit constants (MAX_ACCOUNT_ID_BYTES, etc.) exist as 3 independently hardcoded copies across Rust, TS, and the embedded HTML string — worth a single source of truth eventually.
  • creative_opportunities.rs:366 — legacy providers.aps.slot_id is now a silent no-op with no operator-facing warning if left configured.
  • aps-renderer.spec.ts:169 — no browser-level test for cross-ad-unit capability theft (the messageSourceBelongsToAdUnit guard exists and works, just isn't e2e-tested against a mismatched ad unit).

📝 note

  • auction/README.md:364 — this example is stale relative to the new renderer flow (pre-existing drift, not touched by this PR).

👍 praise

  • aps.rs: parse_bid never reads adm; envelope is byte-bounded and fixture-verified byte-for-byte between Rust and TS; CSP header (APS_RENDERER_CSP) is correctly sent server-side with no allow-same-origin.
  • orchestrator.rs: provider_request_context/effective_timeout threading fix, pinned by dispatched_collection_reuses_provider_launch_context.
  • render.ts: validateApsRenderer's exact-key allowlist cross-checking the outer descriptor vs. the decoded OpenRTB payload; the event.source === iframe.contentWindow identity check (the right check for an opaque-origin sandbox).
  • prebid.rs: the trustedServer bidder-expansion path correctly re-strips "aps" even when reintroduced via config.bidders, with a dedicated regression test.
  • aps-renderer.spec.ts: nonce-replay, CSP-only isolation (with the sandbox attribute omitted), and same-origin attack tests are genuinely adversarial — real attempts against a live server, not tautological checks of config strings.

CI Status

  • fmt: FAIL (caused by the merge-conflict issue above, not this branch's own code)
  • rust tests: FAIL (same root cause)
  • clippy (all adapters): PASS
  • cross-adapter parity: PASS
  • browser/integration tests: PASS
  • vitest: PASS
  • CodeQL: PASS

Comment thread crates/trusted-server-core/src/auction/types.rs
Comment thread crates/trusted-server-core/src/integrations/aps.rs Outdated
Comment thread crates/trusted-server-core/src/auction/formats.rs Outdated
Comment thread crates/trusted-server-core/src/integrations/aps.rs
Comment thread crates/trusted-server-js/lib/src/integrations/gpt/index.ts Outdated
Comment thread crates/trusted-server-core/src/integrations/aps.rs
Comment thread crates/trusted-server-js/lib/test/integrations/aps/render.test.ts
Comment thread crates/trusted-server-js/lib/src/integrations/aps/render.ts Outdated
Comment thread crates/trusted-server-core/src/creative_opportunities.rs

@aram356 aram356 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Re-reviewed at 10cd6d3ce. Every finding from the previous review is fixed — I verified all seven blocking items and the non-blocking ones in source, and several were implemented exactly as suggested. CI is green across all 19 checks.

Two new blocking issues, both introduced by work landed since the last review: the fix for the case-sensitive APS exclusion over-corrected into a config hard-fail that can take a service down on upgrade, and the new debug-metadata commit captures upstream response headers without the fail-closed filtering this codebase applies elsewhere. Both have small fixes.

Blocking

🔧 wrench

  • The aps config rejection is an undocumented hard-fail that bricks every request (prebid.rs:346-368): config.validate() runs at settings.rs:257 before the is_enabled() check at line 265, and IntegrationRegistry::new (registry.rs:791) propagates the error with ?. An operator whose published config contains bidders = ["aps", "kargo"] — previously harmless, silently stripped — deploys this WASM and every request fails, including when the prebid integration is disabled entirely. The CHANGELOG Breaking entry says operators must "disable native APS demand" but never states the config will refuse to boot; docs/guide/configuration.md doesn't either. This PR's own tests construct ["kargo", "aps"], so it is a config the codebase expects to exist in the wild. The hard-fail is also redundant with the runtime filter added in the same commit (lines 1571/1578 already strip aps case-insensitively). Either downgrade to warn-and-strip, or keep the error and add a dedicated **Breaking** CHANGELOG line plus a docs migration note — and confirm ts config push rejects it before the runtime does.

Non-blocking

🤔 thinking

  • Skipped winners are still counted as wins in telemetry (formats.rs:341,351): emit_auction_events_best_effort_lazy(... Completed { result }) runs at auction/endpoints.rs:320, before response conversion, and telemetry.rs:697-711 emits is_win = 1 per winning bid. A bid dropped by the new continue is reported as a win in analytics but never delivered, and endpoints.rs:331 still logs the pre-drop count. There's also no machine-readable drop counter — only a log::warn! — which is inconsistent with the drop_reasons metadata this same PR adds on the APS side. Consider counting skips into ResponseExt.orchestrator, or emitting the terminal event after conversion.
  • An all-APS page now reports a Prebid provider error rather than a no-bid (prebid.rs:1598-1604): when every slot is APS-only the imp vector is empty and the existing guard returns Err(TrustedServerError::Prebid), surfacing as BidStatus::Error in provider_details. That's a normal configuration, not a failure, and it will pollute provider error rates and alerting. Suggest an explicit no-bid response for "all imps intentionally dropped", reserving Err for "no valid banner formats".
  • Debug body capture inherits the 2 MiB upstream cap (aps.rs:945-956, 970-978): captured bytes pass through String::from_utf8_lossy (3 bytes per invalid byte) and then JSON escaping, so an adversarial 2 MiB upstream body can inflate the /auction response to several MB, buffered in WASM heap. Per-auction rather than per-bid, so it doesn't accumulate, but a dedicated smaller preview cap mirroring MAX_BID_CREATIVE_DUMP_BYTES would be safer.
  • The Referer privacy fix is only half-applied: /auction sanitizes, but the SSR path builds publisher.page_url at publisher.rs:2018 as scheme://host{request_path_and_query} — full query string and edge host, unfiltered — and passes it onward. Worth routing both producers through one sanitizer. Related: HeaderValue::from_str failing on a non-ASCII URL silently drops Referer with no log.
  • Debug requestbody is reconstructed, not captured (aps.rs:596-605): debug_request() re-runs build_openrtb_request at parse time and presents the result as what was sent. That holds only while the function stays pure. If it ever gains a uuid, timestamp, or any mutation between build and send, debug will silently report a payload APS never received and an operator will chase a fabricated request. Worth a test asserting the debug body byte-matches the captured outbound body, plus a doc comment recording the invariant.
  • One browser assertion passes vacuously (aps-renderer.spec.ts:364-366): expect(result.foreignUniversalCreativeResponse).not.toEqual(expect.objectContaining({ apsRenderer })) succeeds when the promise times out to undefined, and would also succeed if the bridge served a slightly different renderer. The intended contract is "a foreign frame gets nothing" — assert toBeUndefined().

♻️ refactor

  • validateApsRenderer now runs 3–4× per APS bid on the critical path: admission (prebid/index.ts:213), registration (render.ts:210), and the GPT bridge (gpt/index.ts:998). Each pass allocates a TextEncoder, base64-decodes up to 256 KB, and JSON.parses it. The admission call is new. Consider caching the validated descriptor on first success.
  • Four-element anonymous tuple in backend_to_provider (orchestrator.rs:28, 460, 877-880): (String, Instant, Arc<dyn AuctionProvider>, u32) destructured as (provider_name, start_time, _, _) at five call sites, with an un-self-describing u32. A named struct would match the convention the project applies to argument lists.

📝 note

  • Two trade-offs here came from my own suggestions last round — flagging so they get an explicit decision rather than sliding in as side effects. (1) Suffix host matching (formats.rs:110-119) is implemented correctly — the prefix.ends_with('.') check properly rejects evilpublisher.example — but it does mean any subdomain can now determine the site.page every SSP sees; on a deployment with user-content subdomains that is worth a conscious accept. (2) Unconditional set_query(None) is good for privacy but collapses every page of a query-driven site (/index.php?id=123) to a single URL, degrading contextual targeting and per-page reporting. Consider a key allowlist or a config switch, and document the trade-off either way.
  • debug flag discoverability (aps.rs:132-136): the one-line doc comment carries no risk warning, whereas the analogous flags in settings.rs:1897-1923 each have multi-line "never enable in production" rustdoc. An operator auditing the [debug] section also won't find this switch, since it lives under [integrations.aps]. Worth mirroring the stronger wording.

⛏ nitpick

  • attach_debug_metadata is skipped only on the missing_request_context branch (aps.rs:996-1002), contradicting the docs' claim that the exchange is emitted for success, 204, malformed and non-success statuses. Unreachable today, so latent rather than live.
  • The debug requestheaders map is a hardcoded content-type: application/json (aps.rs:604-608) rather than the headers actually sent at line 1042. They agree today and will drift; it also slightly misrepresents the PBS httpcalls shape it mirrors.
  • On failed renderer registration the descriptor is now retained on the bid (prebid/index.ts:546-552). I checked reachability and it is effectively defensive-only — admission already validated the descriptor, adId is Prebid-generated, and validPrebidIdentity accepts GAM-style codes — so this is hygiene, not a live bug.
  • publisher.rs:2139 still uses expect("should serialize typed renderer") while formats.rs:331-342 was hardened to handle the identical serialization failing; worth resolving the asymmetry in one direction.

CI Status

  • fmt: PASS
  • clippy (all six adapter targets): PASS
  • rust tests (fastly/axum/cloudflare/spin + parity): PASS
  • js tests (vitest): PASS
  • browser integration + CodeQL: PASS

Comment thread crates/trusted-server-core/src/integrations/prebid.rs Outdated
Comment thread crates/trusted-server-core/src/integrations/aps.rs
Comment thread crates/trusted-server-core/src/integrations/aps.rs Outdated
Comment thread crates/trusted-server-core/src/auction/formats.rs
Comment thread crates/trusted-server-core/src/auction/formats.rs
Comment thread crates/trusted-server-core/src/integrations/prebid.rs
Comment thread crates/trusted-server-js/lib/src/integrations/prebid/index.ts Outdated
Comment thread crates/trusted-server-integration-tests/browser/tests/shared/aps-renderer.spec.ts Outdated
aram356 added a commit that referenced this pull request Jul 24, 2026
auction/orchestrator.rs:
- Combined imports (http::Request + std::collections::{HashMap, HashSet}).
- Kept main's post-launch backend-name collision defense and resolved-name
  correlation on both parallel and sequential dispatch, while preserving #918's
  per-provider effective_timeout in the backend_to_provider 4-tuple (declarations
  and read sites already expect the 4th element).
- Took main's test provider fields (configured_timeout_ms / predicted_timeouts)
  and its DivergentBackendProvider; updated a #918 test stub to the merged struct
  shape (configured_timeout_ms: 125 to preserve its capped-launch-timeout assertion).

publisher.rs:
- Advertise the configured publisher_domain in the page URL (main's fix; the edge
  Host must not leak into the bid request per the in-code comment) using #918's
  request_path_and_query field (the field the merged MatchedSlotsContext exposes);
  updated a stale test literal accordingly.

Verified: cargo check (axum) clean; orchestrator + build_auction_request tests pass.
ChristianPavilonis added a commit that referenced this pull request Jul 28, 2026
# Conflicts:
#	CHANGELOG.md
#	crates/trusted-server-core/src/auction/README.md
#	crates/trusted-server-core/src/auction/formats.rs
#	crates/trusted-server-core/src/auction/orchestrator.rs
#	crates/trusted-server-core/src/auction/types.rs
#	crates/trusted-server-core/src/creative_opportunities.rs
#	crates/trusted-server-core/src/integrations/adserver_mock.rs
#	crates/trusted-server-core/src/integrations/aps.rs
#	crates/trusted-server-core/src/integrations/mod.rs
#	crates/trusted-server-core/src/integrations/prebid.rs
#	crates/trusted-server-core/src/openrtb.rs
#	crates/trusted-server-core/src/publisher.rs
#	crates/trusted-server-integration-tests/browser/tests/shared/aps-renderer.spec.ts
#	crates/trusted-server-integration-tests/fixtures/configs/trusted-server.integration.toml
#	crates/trusted-server-js/lib/src/core/auction.ts
#	crates/trusted-server-js/lib/src/core/request.ts
#	crates/trusted-server-js/lib/src/core/types.ts
#	crates/trusted-server-js/lib/src/integrations/aps/render.ts
#	crates/trusted-server-js/lib/src/integrations/gpt/index.ts
#	crates/trusted-server-js/lib/src/integrations/prebid/index.ts
#	crates/trusted-server-js/lib/test/core/auction.test.ts
#	crates/trusted-server-js/lib/test/core/request.test.ts
#	crates/trusted-server-js/lib/test/integrations/aps/render.test.ts
#	crates/trusted-server-js/lib/test/integrations/gpt/ad_init.test.ts
#	crates/trusted-server-js/lib/test/integrations/gpt/spa_hook.test.ts
#	crates/trusted-server-js/lib/test/integrations/prebid/index.test.ts
#	docs/guide/auction-orchestration.md
#	docs/guide/configuration.md
#	docs/guide/integrations/aps.md
#	docs/superpowers/plans/2026-07-15-aps-openrtb-first-class-integration.md
#	docs/superpowers/specs/2026-07-15-aps-openrtb-first-class-integration-design.md
#	trusted-server.example.toml
ChristianPavilonis added a commit that referenced this pull request Jul 28, 2026
The requestId APS renderer fallback from #963 is already incorporated in the rebased #918 head. Preserve the newer RC/#918 tree while recording the latest divergent #963 head as merged.
@aram356

aram356 commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Pushed bbaf5735a to this branch to fix the CI failures (all three workflows — Run Tests, Run Format, Integration Tests — were failing at compile on 0cb06efdb).

Root cause: when the APS stack was rebased onto main, the conflict resolution in ece42a022 resolved publisher.rs against its pre-streaming shape (−3,905 lines). That silently reverted the publisher-side integration of #867 (async end-to-end streaming) and the inline SSAT adm path — both merged in main — while streaming_processor.rs kept the full #867 machinery. The orphaned BodyStreamDecoder/BodyStreamEncoder family then tripped -D warnings dead-code errors in every workflow, so no tests ran on the branch. Everything else from #867 (settings, platform trait, fastly adapter hooks, proxy.rs) had survived the rebase intact; the damage was confined to publisher.rs.

The fix is a three-way re-merge of publisher.rs (base: pre-streaming main 5efa34c95, ours: branch tip, theirs: main). Only 3 conflict regions, all in build_bid_map:

  • main's signature (settings, request_origin, include_debug_bid), zero-dimension w/h omission, and always-inline sanitized adm are restored, with the APS renderer field and renderer-aware hb_adid fallback (cache_id → renderer bid_idad_id) reapplied on top
  • the APS bid-map test moves to the new signature; the include_adm flag it exercised is superseded by main's debug_bid design

Also fixed a masked test regression: debug_metadata_preserves_malformed_and_error_responses still asserted retry-after appears in APS debug headers, which the fail-closed allowlist from the security review deliberately redacts. The compile failure had hidden this since the allowlist landed. The test now asserts content-type is preserved and retry-after is redacted.

Verification (local): cargo fmt --check, fastly release wasm build with -D warnings, test-fastly (1745 core tests), test-axum, test-cloudflare, test-spin, cloudflare wasm check with -D warnings, and the parity suite all pass. Worth a live Fastly smoke of the streaming path since publisher.rs is the merge of two large features — the unit/parity suites cover it, but the end-to-end TTFB behavior was the point of #867.

@prk-Jr prk-Jr left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

The APS OpenRTB integration itself is in good shape: the typed BidRenderer contract, the dual Rust-build/JS-revalidate envelope check, the delivery report that stops dropped winners from being counted as wins, and the browser suite with real positive controls all read well, and every finding from the previous rounds that was marked "fixed" verifies as fixed.

The blocker is not in the APS code. crates/trusted-server-core/src/integrations/prebid.rs and crates/trusted-server-core/src/auction/orchestrator.rs are based on a pre-merge revision and lost work that is currently on main. The merge base is bf61fa189 (current main tip), so this is not a stale-branch artifact that a merge will fix — the deletions are in the diff. CI is green because the regression tests that covered the deleted code were deleted with it.

What is gone, versus origin/main:

Deleted at HEAD Source on main Effect
is_unusable_bidder_params + the deterministic expanded/direct merge + 9 tests 2dc55d755 (#899) Fabricated {} bidder params reach PBS; real inline params can be clobbered by HashMap order
Duplicate-provider and mediator-as-provider startup validation + 2 tests orchestrator.rs:200–230 Double-dispatch and double-called mediator are accepted again
Pre-launch and post-launch backend-name collision guards + 4 tests orchestrator.rs:540, :1006 Colliding backend names silently overwrite backend_to_provider
All 11 canonicalize_transport_timeout_ms call sites + 4 tests #847 work Budget-derived timeouts are no longer quantized, so backend names churn per request

Recovering those two files onto the current main versions and then re-applying the APS-specific edits on top is the smallest safe path. Everything else here can land as-is.

Blocking

🔧 wrench

  • Fabricated empty bidder params ship to PBS againis_unusable_bidder_params and its drop pass were deleted; expand_trusted_server_bidders still fabricates {} for configured bidders with no inline params (prebid.rs:1610). Ported main's own test onto this HEAD and it fails.
  • Real inline bidder params can be clobbered nondeterministically — the expanded/direct split that made the merge order-independent is gone; bidder.extend(...) now races the direct insert on slot.bidders iteration order (prebid.rs:1567–1596).
  • Backend-name collisions and duplicate providers are no longer rejected — orchestrator.rs:550 / :976 insert unconditionally; the startup validation at orchestrator.rs:228 and both collision guards are gone.
  • AuctionBidData no longer declares w/h, and its adm doc is now wronggpt/index.ts:1149-1150 still reads both fields (two new tsc errors), and adm is emitted whenever a creative survives sanitize, not only under inject_adm_for_testing.

Non-blocking

🤔 thinking

  • Missing price still aborts the whole /auction response while a missing render source now drops one bid (formats.rs:344). Worth unifying under the new AuctionDeliveryReport.
  • Debug requestheaders is now near-empty — the response allowlist is also applied to the outbound request, so only content-type survives (aps.rs:638). Combined with the still-reconstructed requestbody, the debug block no longer shows what was actually sent.

🌱 seedling

  • The external Prebid bundle is not deployed by the deployer — CI now builds build:prebid-external for the browser suite, but trusted-prebid.js still ships out of band, so the APS registry changes in prebid/index.ts can half-ship in production.
  • The renderer document is served with no cache headers (aps.rs:1216), so every render refetches it.

👍 praise

  • Envelope cross-validation on both sides — the Rust-built envelope and the JS re-validation check every duplicated descriptor field against the decoded bid, and the base64 round-trip closes the non-canonical-encoding gap from the last round.
  • The browser suite proves its negative with a positive control in the same test (aps-renderer.spec.ts:361–372) — the foreign Universal Creative request is asserted undefined next to a full toEqual on the legitimate one, so the rejection assertion cannot pass vacuously.
  • Delivery report plumbed into telemetrydelivered_winner_slots closes the "skipped winner still counted as a win" gap end to end, across /auction, /_ts/page-bids, and both buffered paths.

CI Status

All 19 checks pass on 488f7b108.

  • fmt: PASS
  • clippy (fastly / axum / cloudflare / cloudflare-wasm / spin native / spin wasm): PASS
  • rust tests (fastly, axum, cloudflare, spin, parity, ts CLI): PASS
  • js tests (vitest): PASS
  • integration + browser integration tests: PASS
  • format-typescript / format-docs / CodeQL: PASS

Note that green CI does not cover the four regressions above: the tests that covered the deleted code were deleted in the same commit (ebc0a8070), and there is no tsc --noEmit gate, so the two new type errors are invisible to CI.

// When no inline PBS bidder params exist (e.g. creative-opportunity slots
// whose PBS params live in stored requests), tell PBS to resolve bidder
// config from the stored request keyed by this slot ID.
let storedrequest = bidder.is_empty().then(|| ImpStoredRequest {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 wrench — This rewrite deletes is_unusable_bidder_params and its drop pass, which are on main, so fabricated {} bidder params reach PBS again.

expand_trusted_server_bidders still fabricates Json::Object(empty) for every configured bidder missing from the per-bidder map (prebid.rs:1068). With the drop pass gone, bidder is non-empty, storedrequest stays None, and the imp ships imp.ext.prebid.bidder.<name> = {}. Per the comment main carries on that pass, PBS rejects such an imp, and a single non-2xx collapses into an auction-wide bid wipeout in parse_response_inner.

The merge base is bf61fa189 (current main tip), so this is a deletion in the diff, not a stale branch. Commit ebc0a8070 removed the function together with nine regression tests from 2dc55d755 (#899):

to_openrtb_drops_fabricated_empty_bidder_params
to_openrtb_drops_an_explicitly_empty_bidder
to_openrtb_drops_non_object_bidder_params
to_openrtb_prefers_direct_bidder_params_over_fabricated_empty
to_openrtb_direct_empty_does_not_clobber_trusted_server_params
to_openrtb_keeps_a_fabricated_bidder_that_an_override_populates
to_openrtb_falls_back_to_stored_request_when_all_bidders_are_fabricated_empty
to_openrtb_falls_back_to_stored_request_for_real_params_of_unconfigured_bidder
to_openrtb_uses_stored_request_when_slot_has_no_pbs_bidder_params

I ported the first one verbatim onto this HEAD (every helper it needs still exists) and it fails:

test integrations::prebid::tests::to_openrtb_drops_fabricated_empty_bidder_params ... FAILED
panicked at: should drop a fabricated empty bidder with no inline params or override

Fix: restore prebid.rs from origin/main and re-apply the APS-specific edits (remove_aps_bidders, the eq_ignore_ascii_case("aps") skip, PrebidImpressionDisposition, site.ref = None, bid_id) on top. The APS changes are additive and compose with the drop pass — excluded_aps && bidder.is_empty() just needs to be evaluated after it rather than before.

}
if name == TRUSTED_SERVER_BIDDER {
expanded.extend(expand_trusted_server_bidders(&self.config.bidders, params));
bidder.extend(expand_trusted_server_bidders(&self.config.bidders, params));

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 wrench — Same root cause, second symptom: the deterministic expanded/direct split is gone, so real inline params can be silently replaced by fabricated empties depending on HashMap iteration order.

slot.bidders is a HashMap. For {"trustedServer": {...}, "kargo": {"placementId": "kn1"}}:

  • kargo visited first → bidder.insert("kargo", real), then bidder.extend(expansion) overwrites it with {}, and the real params are lost.
  • trustedServer visited first → the direct insert wins and the real params survive.

So the same config yields different wire bytes across requests in the same process. main collects the two sources separately precisely so the merge is order-independent, and only lets a direct entry win when it is not an unusable value clobbering real params. Restoring the file per the comment above fixes this too.

}
continue;
};
backend_to_provider.insert(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 wrench — This file is also based on a pre-merge revision. Four pieces of orchestrator hardening that are on main are deleted here, and their tests with them.

1. Backend-name collision guards (this line, and the dispatch path at :976). main checks backend_to_provider.contains_key(&backend_name) before request_bids fires the outbound send, plus a post-launch check on the resolved name. Both are gone; the insert is unconditional. When two providers canonicalize to the same backend name the second insert overwrites the first ProviderLaunchState, and since collect does backend_to_provider.remove(&backend_name) exactly once, one provider's response is parsed by the other provider's parser and the remaining handle is abandoned. Deleted tests: parallel_duplicate_backend_name_fails_second_provider_attributably, dispatched_duplicate_backend_name_fails_second_provider_attributably, dispatched_post_launch_resolved_name_collision_fails_second_provider_attributably, dispatched_resolved_backend_name_diverging_from_prediction_still_correlates.

2. Startup validation (orchestrator.rs:228). main rejects a provider listed twice in [auction].providers and a mediator that also appears in providers. Both return Err(...Configuration) blocks are deleted, so a duplicate now double-sends and a mediator-as-provider is called twice per auction. Deleted tests: rejects_duplicate_configured_providers, rejects_mediator_also_listed_as_provider.

3. Transport-timeout canonicalization. canonicalize_transport_timeout_ms has 11 call sites in main's orchestrator and zero at HEAD — git grep finds it only in the trait definition and a test_support doc comment. Budget-derived timeouts are no longer quantized, so the backend name changes with every distinct remaining-budget value, which is the churn #847 fixed. Deleted tests: parallel_launch_applies_canonical_timeout_to_name_and_request, dispatched_collect_applies_canonical_timeout_to_both_paths, synchronous_mediation_applies_canonical_timeout_to_mediator, zero_canonical_timeout_skips_parallel_launch, plus the recording-backend harness (ensure/predict_name/record/recording) they share.

4. test_apply_floor_prices_drops_bids_with_undecoded_price is also gone.

CI is green because the code and its tests were removed in the same commit.

Fix: restore orchestrator.rs from origin/main, then re-apply the APS-driven changes on top — ProviderRequestOutcome handling, ProviderLaunchState (which is a genuine improvement over the bare tuple), parse_state threading, and snapshot_context_request. Note that the mediator timeout change at :312 also drops canonicalization; remaining_ms.min(mediator.timeout_ms()) should go back through canonicalize_transport_timeout_ms for the same backend-name-stability reason.

/** Typed winning-bid renderer capability. */
renderer?: AuctionBidRenderer;
/** Raw creative markup. Only present when `[debug] inject_adm_for_testing = true`. */
adm?: string;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 wrench — Two contract errors in this hunk.

w/h were deleted but are still read. gpt/index.ts:1149-1150 does matchedBid.w ?? fallbackWidth / matchedBid.h ?? fallbackHeight, and the server still emits both (publisher.rs:3203, :3206). npx tsc --noEmit reports two errors that are new to this PR:

src/integrations/gpt/index.ts(1149,30): error TS2339: Property 'w' does not exist on type 'AuctionBidData'.
src/integrations/gpt/index.ts(1150,31): error TS2339: Property 'h' does not exist on type 'AuctionBidData'.

Runtime is unaffected (esbuild strips types, and CI has no tsc gate), but the type now says the field does not exist, so the next person to "fix" the error deletes the winning-creative dimensions and every multi-size slot silently falls back to its first configured format.

The adm doc is now wrong. "Only present when [debug] inject_adm_for_testing = true" — build_bid_map inserts adm whenever a creative survives sanitize/rewrite and is non-empty (publisher.rs:3295); include_debug_bid at :3304 gates only debug_bid. main's doc said the opposite explicitly ("This is NOT gated by inject_adm_for_testing"), and the bridge's local-render path at gpt/index.ts:1152 depends on adm being present without the debug flag.

Fix:

  /** Winning creative width; the bridge sizes the inline render from this. */
  w?: number;
  /** Winning creative height; the bridge sizes the inline render from this. */
  h?: number;
  /**
   * Sanitized winning creative markup for the pbRender bridge. Present whenever
   * the winning bid carried a creative that passed the server-side
   * sanitize/rewrite boundary. NOT gated by `inject_adm_for_testing`.
   */
  adm?: string;

let mut delivery = AuctionDeliveryReport::default();

for (slot_id, bid) in &result.winning_bids {
let price = bid.price.ok_or_else(|| {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 thinking — This is now the only per-bid failure that still aborts the whole multi-slot response. The two cases below it (renderer_extension_serialization_failed, no_render_source) drop one winner and record a reason in AuctionDeliveryReport, which is the right shape; a winner with no decoded price still ?s out and fails every other slot with it.

Worth folding into the same mechanism now that it exists:

let Some(price) = bid.price else {
    log::warn!(
        "Auction {}: skipping winning bid for slot '{}' from '{}' because it has no decoded price",
        auction_request.id, slot_id, bid.bidder
    );
    delivery.record_drop("no_decoded_price");
    continue;
};

Low risk today — both providers require a finite price before constructing a Bid — but it is a mediator bug away from taking down a whole page's demand, and dropped_winner_reasons would make it diagnosable instead of a 500.

if let Some(http_call) = http_call.as_object_mut() {
if let Some(request) = request {
http_call.insert("requestbody".to_string(), json!(request.body));
http_call.insert("requestheaders".to_string(), json!(request.headers));

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 thinking — The response-header allowlist is also applied to the outbound request (Self::debug_headers(outbound_request.headers()) at :1075), so requestheaders can only ever contain content-type — the one header the code sets itself two lines earlier. Fail-closed is right for the response direction, but on the request side it removes all diagnostic value; the outbound headers are ours, not upstream's, so there is nothing to leak.

Combined with the still-reconstructed requestbody (acknowledged as open), the debug block now shows neither the headers nor the body that were actually sent. If the plan is to capture the real request, capturing its headers at the same point would fix both at once.

Comment thread crates/trusted-server-js/lib/src/integrations/aps/render.ts
.header("x-content-type-options", "nosniff")
.header("referrer-policy", "no-referrer")
.header(header::CONTENT_SECURITY_POLICY, APS_RENDERER_CSP)
.body(EdgeBody::from(APS_RENDERER_DOCUMENT))

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🌱 seedling — Not for this PR: this document is a fixed static asset keyed only by its route, but it is served with no cache-control, so every APS render on every page view refetches it from the edge. A long-lived immutable cache header would remove one round trip from the critical render path.

The response is already nosniff + no-referrer + CSP, so caching adds no exposure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add the Amazon APS Prebid Adapter as first class integration

4 participants