ci: Evaluate removing app metrics (startup/size) tests #6890

sentry-cocoa is removing its Collect App Metrics CI job (getsentry/sentry-cocoa#9350). We should check whether the equivalent metrics job in this repo is still worth its cost.

What sentry-cocoa found (Sep 14 – Oct 14)

• 358 runs used 48.3 h of macos-15 time. 83% of that went to the Sauce Labs startup step (50 runs per job). Sauce Labs device time is billed separately.
• It was the last job holding up the Release check in 85% of runs (median delay 5.8 min). Removing it should cut median CI time by about 30%.
• Binary size is still tracked through Sentry Size Analysis (size-analysis.yml). The startup-time gate goes away with no replacement.

What we run here

metrics job in .github/workflows/e2e-v2.yml, using getsentry/action-app-sdk-overhead-metrics:

• Matrix: ios (macos-26-xlarge) / android (ubuntu-24.04) × legacy / new architecture = 4 jobs, capped at max-parallel: 2 because of the Sauce Labs concurrency limit.
• Each job builds performance-tests/TestAppPlain and TestAppSentry, then runs:
  • startupTimeTest: 50 runs. Allowed diff is -3000..150 ms on iOS and -20..150 ms on Android.
  • binarySizeTest: 600–1850 KiB on iOS and 1–6.1 MiB on Android.
• Configs: performance-tests/metrics-ios.yml, performance-tests/metrics-android.yml.
• size-analysis.yml already uploads Android and iOS builds to Sentry Size Analysis.

Questions to answer

1. Cost: runner hours (macOS xlarge), Sauce Labs minutes, and how often metrics is the last job to finish or is the cause of a flaky failure (see Flaky E2E Metrics App Start Time #3413, Sauce Labs dropped Appium 'stable' version, breaking metrics tests #5834).
2. Value: has the startup or size gate caught a real regression recently? The iOS diffMin: -3000 is so loose that it can't catch an improvement, only a slowdown.
3. Coverage after removal: does Size Analysis cover the size checks well enough (thresholds or status checks, not just tracking)? Are we fine with no CI startup-overhead gate, or should startup move to a cheaper or scheduled job (for example, nightly on main only)?

Options

• Use Sentry Size Analysis (size-analysis.yml) for the size check instead of binarySizeTest. Set up thresholds or status checks there so that size regressions still fail or flag PRs, then drop binarySizeTest from both metrics configs.
• Remove the job and performance-tests/ entirely, as sentry-cocoa did.
• Keep only startup, run it on a schedule or on main only, and rely on Size Analysis for size.
• Reduce the cost: fewer runs, one architecture, or skip it on PRs.