[TT-17304] Automated downstream testing for github-actions PRs

[konrad-sol]

What

Adds .github/workflows/test-downstream.yml — a new workflow that automatically tests affected tyk-analytics pipelines whenever a PR is opened or updated in this repo.

Why

tyk-analytics pins all TykTechnologies/github-actions references to specific commit SHAs. Until now there was no automated way to verify that a change to an action here would not break the downstream pipelines that consume it. Every verification had to be done manually.

How it works

PR opened/updated in github-actions
        │
        ▼
Job 1 – detect
  • git diff vs base branch
  • extracts changed action dirs (.github/actions/**) and workflow paths (.github/workflows/*.yml)
  • outputs: changed_paths[], pr_sha
        │
        ▼
Job 2 – downstream  (skips if no .github/ paths changed)
  1. Checkout tyk-analytics
  2. Clean up any leftover temp branch from a previous run (idempotent)
  3. Grep tyk-analytics/.github/workflows/ for references to the changed paths
     → builds the list of affected workflow files only
  4. Create branch  ga-pr-{PR_NUMBER}  in tyk-analytics:
       • Patch every  TykTechnologies/github-actions/…@<sha>  to @<pr_sha>
       • Inject  workflow_dispatch: {}  for any affected workflow that lacks it
         (temp-branch only — never a permanent change)
  5. Trigger each affected workflow via workflow_dispatch API on that branch
  6. Poll until all runs complete (max 30 min)
  7. Comment the results table on this PR
  8. Delete the temp branch (always, even on failure)

Example PR comment

Workflow	Run	Result
release.yml	12345678	✅ success
nightly-e2e-tests.yml	12345679	✅ success

Which workflows get triggered for which changes

Changed path	Triggered in tyk-analytics
.github/actions/tests/test-controller	release.yml, nightly-e2e-tests.yml
.github/actions/tests/env-up + others	release.yml, nightly-e2e-tests.yml
.github/actions/checkout-pr	ci-tests.yml, lint-swagger.yml
.github/actions/gh-logs-analyser	release.yml, ci-tests.yml, and others
.github/workflows/dependency-guard.yml	release.yml, ci-tests.yml, k8s-api-tests.yaml, lint-swagger.yml
.github/workflows/branch-suggestion.yml	intelligent-branch-recomendations.yml

Nightly e2e tests are only triggered when the PR actually touches the test-related actions, keeping expensive runs proportional to the change scope.

Prerequisites

ORG_GITHUB_TOKEN must be available as a secret in this repo. The token needs repo scope on TykTechnologies/tyk-analytics (create/delete branches, trigger workflow dispatch, read run status).

tyk-analytics changes

None. The temp branch approach handles everything — including the workflow_dispatch injection — without leaving any permanent footprint.

[probelabs]

This PR introduces a new GitHub Actions workflow, .github/workflows/test-downstream.yml, to automate testing of downstream consumers. When a pull request in this repository modifies a reusable action or workflow, this new pipeline automatically runs tests against the tyk-analytics repository to ensure no breaking changes are introduced.

Files Changed Analysis

• .github/workflows/test-downstream.yml: A single new file containing the entire workflow logic. It defines two jobs:
  1. detect: Identifies which actions (.github/actions/*) or workflows (.github/workflows/*.yml) have been changed in the PR.
  2. downstream: If changes are detected, this job checks out the tyk-analytics repository, finds workflows that use the modified actions, and creates a temporary branch. In this branch, it patches the workflows to use the PR's commit SHA and triggers them. Finally, it reports the results back to the PR as a comment and cleans up the temporary branch.
• .github/actions/tests/env-up/action.yaml: A trivial one-line comment change, likely added to demonstrate and test the new downstream workflow.

Architecture & Impact Assessment

• What this PR accomplishes: It automates a previously manual verification process, creating a CI feedback loop that prevents changes in this repository from breaking downstream pipelines in tyk-analytics.
• Key technical changes introduced: The workflow leverages git diff to detect changes, grep to find affected downstream files, and sed and a Python script to dynamically patch workflow files in a temporary branch. It uses the gh CLI extensively to interact with the GitHub API for triggering workflows, polling for results, and managing branches. Authentication is handled via a GitHub App token for secure cross-repository access.
• Affected system components: The primary impact is on the CI process of this repository. It also interacts directly with the TykTechnologies/tyk-analytics repository by creating/deleting branches and triggering workflows. It requires GitHub App credentials with permissions to the downstream repository.

Here is a diagram illustrating the workflow's logic:

sequenceDiagram
    participant PR in github-actions
    participant test-downstream.yml
    participant tyk-analytics repo

    PR->>test-downstream.yml: PR opened/updated
    activate test-downstream.yml

    test-downstream.yml->>test-downstream.yml: **Job: detect**<br/>git diff vs base<br/>find changed paths

    test-downstream.yml->>tyk-analytics repo: **Job: downstream**<br/>Checkout repo
    activate tyk-analytics repo
    test-downstream.yml->>tyk-analytics repo: Clean up old temp branch (if any)

    test-downstream.yml->>tyk-analytics repo: Grep for affected workflows
    test-downstream.yml->>tyk-analytics repo: Create new temp branch (ga-pr-133)
    test-downstream.yml->>tyk-analytics repo: Patch workflows:<br/>- Update action SHA to PR SHA<br/>- Inject workflow_dispatch
    test-downstream.yml->>tyk-analytics repo: Push temp branch

    test-downstream.yml->>tyk-analytics repo: Trigger patched workflows via API
    test-downstream.yml->>tyk-analytics repo: Poll for run completion
    deactivate tyk-analytics repo

    test-downstream.yml->>PR: Post results comment
    test-downstream.yml->>tyk-analytics repo: Delete temp branch
    deactivate test-downstream.yml

Loading

Scope Discovery & Context Expansion

This workflow establishes a pattern for downstream testing that could be extended to other repositories that consume these actions. The DOWNSTREAM environment variable is currently hardcoded to TykTechnologies/tyk-analytics, but the design allows for future expansion. The core logic is generic and reusable, assuming downstream consumers pin actions to full commit SHAs. The change is confined to CI/CD automation and has no impact on the functionality of the actions themselves, only on how they are tested before being merged.

Metadata

• Review Effort: 4 / 5
• Primary Label: enhancement

---

Powered by Visor from Probelabs

Last updated: 2026-05-29T12:44:20.809Z | Triggered by: pr_updated | Commit: 4e73fc6

💡 TIP: You can chat with Visor using /visor ask <your question>

[probelabs]

Security Issues (3)

Severity	Location	Issue
🔴 Critical	.github/workflows/test-downstream.yml:110	A command injection vulnerability exists in the "Find affected workflows" step. The `path` variable, derived from file paths in the pull request, is used unsafely in a `grep` command. An attacker could submit a pull request with a maliciously crafted file path (e.g., containing `$(some_command)`) to execute arbitrary commands on the runner. This is highly critical as the runner's environment contains a GitHub App token with write permissions to the downstream repository. 💡 Suggestion The `grep` command should be replaced with a safer alternative that does not interpret shell metacharacters from the `path` variable. Using `find` in combination with `awk` for pattern matching is a safer approach, as `awk` will treat the pattern as a string rather than executable code. Replace the vulnerable grep call: done < <(grep -rl "TykTechnologies/github-actions/${path}@" "$DIR" 2>/dev/null || true) With a safer alternative: done < <(find "$DIR" -type f -exec awk -v p="TykTechnologies/github-actions/${path}@" '$0 ~ p { print FILENAME; nextfile }' {} + 2>/dev/null || true)
🟡 Warning	.github/workflows/test-downstream.yml:132	The workflow automatically triggers downstream builds for every affected workflow. A malicious pull request that modifies a large number of actions could trigger an excessive number of downstream builds. This could lead to resource exhaustion on the downstream repository's runners or hit GitHub API rate limits, causing a denial of service. 💡 Suggestion Implement a limit on the number of workflows that can be triggered. Add a step after "Find affected workflows" to check the count of found workflows and fail the job if it exceeds a reasonable threshold (e.g., 10).
🟡 Warning	.github/workflows/test-downstream.yml:70-75	The workflow uses a GitHub App token that, according to the pull request description, requires `repo` scope. This scope grants broad access to the repository, which violates the principle of least privilege. If the token were to be compromised, the impact would be unnecessarily severe. 💡 Suggestion Review the permissions of the GitHub App. Instead of the broad `repo` scope, grant only the specific permissions required by the workflow, which appear to be `contents: write` (to manage branches) and `actions: write` (to trigger and monitor workflows).

Architecture Issues (2)

Severity	Location	Issue
🟠 Error	.github/workflows/test-downstream.yml:70	The downstream repository `TykTechnologies/tyk-analytics` is hard-coded as an environment variable. This tightly couples the workflow to a single consumer, preventing reuse for other downstream repositories and making the solution less scalable. This approach violates the principle of creating generic, reusable CI/CD components. 💡 Suggestion To improve reusability and extensibility, parameterize the downstream repository. This could be achieved by using a matrix strategy that iterates over a list of repositories defined as a variable, secret, or workflow input. This would allow the same workflow to test multiple downstream consumers without modification.
🟠 Error	.github/workflows/test-downstream.yml:230-260	The `Wait for results` step polls for the completion of downstream workflow runs sequentially. It waits for the first run to complete (for up to 30 minutes) before it starts polling the second run. If multiple workflows are triggered, the total wait time becomes the sum of all individual run times, which is inefficient, slow, and wastes runner resources. 💡 Suggestion Refactor the polling logic to check the status of all triggered runs concurrently. A better approach would be to loop until all runs are complete, checking the status of every active run in each iteration of the loop, with a single `sleep` at the end of the loop. This would make the total wait time equal to the duration of the longest-running downstream workflow, not the sum of all of them, providing much faster feedback.

Performance Issues (2)

Severity	Location	Issue
🟠 Error	.github/workflows/test-downstream.yml:247-276	The script waits for each triggered workflow run to complete sequentially. The outer loop processes one run ID at a time, and an inner loop polls it for up to 30 minutes. This means the total wait time is the sum of the durations of all triggered workflows, instead of the duration of the longest one. For example, if 3 workflows each take 10 minutes, this step will take 30 minutes instead of 10. 💡 Suggestion Refactor the logic to poll all pending runs concurrently. A single polling loop should iterate every 30 seconds. Inside the loop, it should check the status of all runs that have not yet completed. This will ensure the total wait time is determined by the longest-running downstream job.
🟡 Warning	.github/workflows/test-downstream.yml:198-224	The script triggers workflows and waits for their run IDs to appear sequentially within the same loop. For each workflow, it can poll for up to 60 seconds before proceeding to the next. This creates a cumulative delay. For example, with 5 affected workflows, this step alone could take up to 5 minutes before it even starts waiting for the runs to complete. 💡 Suggestion Trigger all workflows first in a tight loop. Afterwards, enter a second, separate polling loop to wait for all their run IDs to appear concurrently. This would make the wait time for run ID discovery concurrent rather than sequential.

---

Powered by Visor from Probelabs

Last updated: 2026-05-29T12:43:45.151Z | Triggered by: pr_updated | Commit: 4e73fc6

💡 TIP: You can chat with Visor using /visor ask <your question>

[github-actions]

❌ Downstream test results — TykTechnologies/tyk-analytics

Tested against commit 4e73fc69 on temp branch ga-pr-133 (now deleted).

Workflow	Run	Result
release.yml	26637859488	⏱️ timed_out
nightly-e2e-tests.yml	26637864511	❌ failure