Add Salt package download benchmark by AkashKumar7902 · Pull Request #12341 · uyuni-project/uyuni

AkashKumar7902 · 2026-07-27T15:22:38Z

What does this PR change?

Adds an opt-in Cucumber benchmark that downloads every binary RPM from one Uyuni software channel on a configured set of Salt minions.

Benchmark flow

Read the channel label and Salt minion IDs from environment variables.
Capture the channel's binary-RPM snapshot through the Uyuni API and confirm that every minion is registered and subscribed.
Find the ready Uyuni server pod.
Prepare the minions with:
- state.apply channels
- grains.item os_family osarch
- pkg.get_repo
- pkg.refresh_db
- pkg.list_repo_pkgs
Clear the RPM payload cache outside the measured interval.

Run one concurrent Salt cmd.run_all workload on all selected minions:

zypper --quiet --non-interactive --no-refresh download \
  --all-matches \
  --repo susemanager:<channel-label> \
  '*'

Inventory the downloaded RPMs, match them to the channel checksums, and confirm that the channel snapshot remained unchanged.
Write a structured JSON result.

Commands follow this path:

testsuite controller
  -> get_target('localhost').run_local
  -> kubectl exec into the Uyuni server pod
  -> salt --list <minions>
  -> Salt minions

Only the package download workload is timed. Setup, cache cleanup, and verification run outside the measured interval. The RPMs are downloaded into each minion's zypper package cache and are not installed.

The benchmark expects an existing channel and existing subscriptions. It does not create channels, subscribe systems, install a Helm release, or configure a StorageClass.

Configuration

Prerequisites

The testsuite controller can reach the Kubernetes cluster with kubectl.
The Uyuni server pod is running and ready in the uyuni namespace.
The selected Salt minions are registered, reachable, and subscribed to the configured channel.
The selected minions are SUSE systems with the same architecture.
The channel contains binary RPM packages and is available on every selected minion.
Initialize the Cobbler API client lazily in cobbler_steps.rb #12240 is present so a run set without Cobbler scenarios does not initialize the Cobbler client during Cucumber startup.

Required inputs

Variable	Description	Example
`UYUNI_BENCH_MINIONS`	JSON array containing the exact Salt minion IDs	`["minion-1.example.test","minion-2.example.test"]`
`UYUNI_BENCH_CHANNEL`	Uyuni software channel label	`uyuni-bench-channel`

The normal testsuite environment must also provide SERVER and its API configuration.

Optional inputs

Variable	Default	Description
`UYUNI_BENCH_TIMEOUT_SECONDS`	`14400`	Timeout for the measured all-minion download workload
`UYUNI_BENCH_STORAGE_CLASS`	unset	StorageClass name recorded as result metadata
`SCENARIO_HARD_LIMIT`	workload timeout + 1800 seconds	Overall Cucumber scenario watchdog override

Control operations use a maximum timeout of 600 seconds. UYUNI_BENCH_STORAGE_CLASS does not select or configure storage.

How to run

Run the dedicated Rake task from a configured testsuite controller:

source /root/.bashrc
cd /root/spacewalk/testsuite

export UYUNI_BENCH_MINIONS='["minion-1.example.test","minion-2.example.test"]'
export UYUNI_BENCH_CHANNEL='uyuni-bench-channel'
export UYUNI_BENCH_TIMEOUT_SECONDS=14400
export UYUNI_BENCH_STORAGE_CLASS='nfs-csi'

bundle exec rake cucumber:kubernetes_package_download_benchmark

If StorageClass metadata is not needed, omit UYUNI_BENCH_STORAGE_CLASS.

Results are written to:

testsuite/results/package-download/<timestamp>-<pid>/result.json

The result contains the measured duration, per-minion command status, package and payload counts, downloaded bytes, channel snapshot digest, and verification errors.

Codespace

GUI diff

No difference.

DONE

Documentation

The configuration and run instructions are included in this PR description.
DONE

Test coverage

Validated with:

Ruby syntax checks for the benchmark steps and Rakefile;
RuboCop for the benchmark step definitions;
focused checks for API snapshot mapping and the Salt preflight sequence;
focused checks for controller target selection, kubectl exec serialization, Salt arguments, zypper arguments, and timeout values;
focused checks for RPM cache verification and result schema;
run-set linkage validation;
git diff --check.
DONE

Links

Issue(s): openSUSE/mentoring#252
Related: #12077
Depends on: #12240
Related Sumaform storage work: uyuni-project/sumaform#2246
Port(s): none

DONE

Changelogs

No changelog needed

Re-run a test

Re-run test "changelog_test"
Re-run test "backend_unittests_pgsql"
Re-run test "java_pgsql_tests"
Re-run test "schema_migration_test_pgsql"
Re-run test "susemanager_unittests"
Re-run test "frontend_checks"
Re-run test "spacecmd_unittests"

Before you merge

Check How to branch and merge properly!

AkashKumar7902 · 2026-08-17T20:19:14Z

@Pablogoliva Review updates are pushed in c8594d3a52 on top of current master:

removed @long_running while keeping an active run-set-specific watchdog;
replaced every custom Open3 execution path with RemoteNode#run on the registered localhost controller target;
preserved shell-safe argv handling, separate stdout/stderr, explicit exit status, and TERM/KILL timeouts;
verified that plain salt is the established server-side CLI while venv-salt-* selection is minion-side; and
corrected the PR description so it no longer claims closed Sumaform Refreshing repositories on .deb servers through Uyuni proxy very slow #2241 supplies UYUNI_BENCH_MINIONS.

Ruby syntax, targeted RuboCop, runner/quoting assertions, and diff checks pass. A new full benchmark run still requires deployed Uyuni infrastructure and minions.

Pablogoliva

Change the way to use system commands.

AkashKumar7902 · 2026-08-18T14:04:05Z

@Pablogoliva Follow-up on the system-command review is complete on current head.

The benchmark now calls the standard RemoteNode#run_local method directly at every controller execution site.
The shared SSH helper now actually forwards the caller's timeout to the SSH channel; a real transport timeout reports 124, while unrelated transport failures remain -1.

I also ran the full mounted-volume pipeline against the external RKE2 environment. The isolated checkout included the declared #12240 dependency; its testsuite directory was bind-mounted into the controller, with kubectl and the RKE2 kubeconfig mounted read-only.

1 scenario passed; 8 / 8 steps passed
measured workload:        264.839248 seconds
successful Salt targets:  2 / 2
verified packages:        6,054 / 6,054
verified payload bytes:   37,105,623,752
missing/mismatched/extra: 0 / 0 / 0
timed_out:                false
uncertain_completion:     false
channel snapshot stable:  yes

The PR description now contains the updated timeout model and validation result. Please re-review the current head when convenient.

Pablogoliva · 2026-08-19T15:23:04Z

+  code = nil
+  exception = nil
+  begin
+    stdout, stderr, code = get_target('localhost').run_local(


Suggested change

stdout, stderr, code = get_target('localhost').run_local(

stdout, stderr, code = get_target('controller').run_local(

Also in the other occurrences.

the existing code already runs on the controller.

plus, the controller is not defined in:
https://github.com/uyuni-project/uyuni/blob/master/testsuite/features/support/constants.rb

Pablogoliva · 2026-08-19T15:23:44Z

+    package_download_benchmark_salt_call(
+      inputs,
+      pod,
+      'cmd.run_all',


What's the matter of this?

It runs a Python inventory command on each minion to collect the paths and sizes of files under /var/cache/zypp/packages. The returned inventory is used for post-download verification.

Order:

Before benchmark
cmd.run_all → clear old RPM cache

During benchmark (timed)
cmd.run_all → run zypper download

After benchmark
cmd.run_all → inventory downloaded RPM files

Verification
Compare inventory with expected channel packages

So line 848 is post-benchmark verification and is not included in the measured duration.

github-actions · 2026-08-25T15:33:17Z

👋 Hello! Thanks for contributing to our project.
Acceptance tests will take some time (approx. 1h), please be patient ☕

You can see the progress at the end of this page and at https://github.com/uyuni-project/uyuni/pull/12341/checks
Once tests finish, if they fail, you can check 👀 the cucumber report. See the link at the output of the action.
You can also check the artifacts section, which contains the logs at https://github.com/uyuni-project/uyuni/pull/12341/checks.

If you are unsure the failing tests are related to your code, you can check the "reference jobs". These are jobs that run on a scheduled time with code from master. If they fail for the same reason as your build, it means the tests or the infrastructure are broken. If they do not fail, but yours do, it means it is related to your code.

Reference tests:

KNOWN ISSUES

Sometimes the build can fail when pulling new jar files from download.opensuse.org . This is a known limitation. Given this happens rarely, when it does, all you need to do is rerun the test. Sorry for the inconvenience.

For more tips on troubleshooting, see the troubleshooting guide.

Happy hacking!

cbosdo · 2026-08-25T14:25:56Z

+    status: workload_errors.empty? ? 'passed' : 'failed',
+    command: zypper,
+    timeout_seconds: timeout,
+    timed_out: workload_error.to_s.include?('exited with 124'),


This exit code is not in zypper's man page with the others… Where does it come from?

it comes from network_utils.rb where we throw this exit code when timeout happens

cbosdo · 2026-08-25T14:26:44Z

@@ -0,0 +1,18 @@
+# Copyright (c) 2026 SUSE LLC


Copyright should be yours: change the SUSE LLC with your full name

cbosdo · 2026-08-26T09:16:31Z

How comes that you add a new run set? Shouldn't this run after the reposync benchmark to reuse the imported repository? In which case it should be in the same run_set.

I though of keeping it separate , will be making changes to accomodate both

cbosdo · 2026-08-26T09:17:04Z

  rescue Timeout::Error
-    puts "SSH operation timed out after #{timeout} seconds."
+    stderr = "SSH operation timed out after #{timeout} seconds."
+    exit_code = 124


Where is this exit_code coming from?

it is a timeout error code which we are synthesizing

cbosdo · 2026-08-26T09:22:10Z

+  inputs = @package_download_inputs
+  api = $api_test
+
+  # 1. Read the current channel state from the Uyuni API.


AI tends to add numbered comments, but this is a pain to maintain in the long run. Remove the numbered list items from them please.

ok, should i just remove the numbers, or the enitre comment ?

updated, removed the numbers

cbosdo · 2026-08-26T09:26:06Z

+  )
+end
+
+Given('a ready server pod is reachable from the benchmark controller') do


This function needs to be shared with the other PR as it's a duplicate. Better base this PR on the other one to share code with it

cbosdo · 2026-08-26T09:26:35Z

+  inputs = @package_download_inputs
+  pod = @package_download_pod
+
+  # 1. Run `state.apply channels` so Uyuni updates the assigned software channel


Again numbered comments…

cbosdo · 2026-08-26T12:15:46Z

+    '--all-matches',
+    '--repo',
+    inputs[:repo_alias],
+    '*'


I wonder if we could have issue with packages conflicting each other in the repo

Pablogoliva · 2026-09-02T14:37:12Z

    Timeout.timeout(timeout) do # Enforce timeout on the entire SSH operation
      Net::SSH.start(host, nil, port: port, verify_host_key: :never, timeout: timeout, keepalive: true, max_pkt_size: buffer_size, config: true) do |ssh|
-        stdout, stderr, exit_code = ssh_exec!(ssh, command, timeout: DEFAULT_TIMEOUT)
+        stdout, stderr, exit_code = ssh_exec!(ssh, command, timeout: timeout)


Leave the previous timeout pls

Snapshot all binary RPMs in the configured Uyuni channel and verify that every target minion is subscribed. Run one concurrent zypper download --all-matches command per minion, record the measured duration, and verify every cached artifact against the frozen channel checksum set. Signed-off-by: Akash Kumar <meakash7902@gmail.com> Signed-off-by: Akash Kumar <91385321+AkashKumar7902@users.noreply.github.com>

Run kubectl and Salt through the testsuite controller target while preserving process timeouts and separated command output. Keep the normal scenario watchdog enabled with a benchmark-specific limit derived from the configured download timeout. Signed-off-by: Akash Kumar <91385321+AkashKumar7902@users.noreply.github.com>

Signed-off-by: Akash Kumar <91385321+AkashKumar7902@users.noreply.github.com>

Stop converting Ruby timeouts into a shell-style exit code and remove result metadata that depends on parsing it. Signed-off-by: Akash Kumar <91385321+AkashKumar7902@users.noreply.github.com>

Move the feature into the kubernetes_storage_benchmark run set so it reuses the channel imported by the reposync benchmark: wait for the channel repodata, subscribe the configured minions through the API, and share the reposync helpers for the server pod, integer inputs and the storage backend label. Drop the network_utils.rb timeout change and require DEFAULT_TIMEOUT to cover the download timeout instead. Replace the inline Python cache inventory with a find command. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Akash Kumar <91385321+AkashKumar7902@users.noreply.github.com>

This was referenced Jul 27, 2026

Support multiple SUSE minions in the test suite uyuni-project/sumaform#2241

Closed

Support external testsuite controller hosts #12340

Closed

github-actions Bot added ci-pipelines testing ruby_rubocop test-framework build-validation-features labels Jul 27, 2026

AkashKumar7902 force-pushed the testsuite/package-download-benchmark branch from 244449d to 68355d2 Compare July 28, 2026 20:47

AkashKumar7902 marked this pull request as ready for review July 29, 2026 15:34

AkashKumar7902 requested a review from a team as a code owner July 29, 2026 15:34

Pablogoliva reviewed Jul 29, 2026

View reviewed changes

Comment thread testsuite/features/build_validation/benchmarks/srv_salt_package_download_benchmark.feature Outdated

Pablogoliva reviewed Jul 29, 2026

View reviewed changes

Comment thread testsuite/features/step_definitions/package_download_benchmark_steps.rb Outdated

Pablogoliva reviewed Jul 29, 2026

View reviewed changes

Comment thread testsuite/features/step_definitions/package_download_benchmark_steps.rb

Pablogoliva reviewed Jul 29, 2026

View reviewed changes

Comment thread testsuite/features/step_definitions/package_download_benchmark_steps.rb

Pablogoliva requested changes Jul 29, 2026

View reviewed changes

Pablogoliva reviewed Jul 30, 2026

View reviewed changes

Comment thread testsuite/features/step_definitions/package_download_benchmark_steps.rb Outdated

AkashKumar7902 force-pushed the testsuite/package-download-benchmark branch from 68355d2 to c8594d3 Compare August 17, 2026 20:13

Pablogoliva reviewed Aug 18, 2026

View reviewed changes

Comment thread testsuite/features/step_definitions/package_download_benchmark_steps.rb Outdated

Pablogoliva requested changes Aug 18, 2026

View reviewed changes

AkashKumar7902 force-pushed the testsuite/package-download-benchmark branch from c8594d3 to 2d693f5 Compare August 18, 2026 13:07

Pablogoliva reviewed Aug 19, 2026

View reviewed changes

AkashKumar7902 mentioned this pull request Aug 24, 2026

Evaluating Cloud-Native Storage: Performance Benchmarking for Uyuni on RKE2 openSUSE/mentoring#252

Open

cbosdo added the gsoc label Aug 25, 2026

AkashKumar7902 mentioned this pull request Aug 26, 2026

Allow contributor copyright notices in RuboCop #12504

Merged

13 tasks

cbosdo reviewed Aug 26, 2026

View reviewed changes

AkashKumar7902 force-pushed the testsuite/package-download-benchmark branch 2 times, most recently from 09c97e8 to 2229bd8 Compare August 28, 2026 09:31

Pablogoliva reviewed Sep 2, 2026

View reviewed changes

AkashKumar7902 mentioned this pull request Sep 2, 2026

Add libvirt example for the external RKE2 benchmark uyuni-project/sumaform#2275

Merged

AkashKumar7902 and others added 17 commits September 20, 2026 20:07

Use RemoteNode timeouts for package benchmark

9013497

Signed-off-by: Akash Kumar <91385321+AkashKumar7902@users.noreply.github.com>

Cap package benchmark control operations

176cc9f

Signed-off-by: Akash Kumar <91385321+AkashKumar7902@users.noreply.github.com>

Run the package download benchmark workflow

ea785d0

Signed-off-by: Akash Kumar <91385321+AkashKumar7902@users.noreply.github.com>

Define the package download benchmark steps

12e19e5

Signed-off-by: Akash Kumar <91385321+AkashKumar7902@users.noreply.github.com>

Describe package record usage

2d0a37e

Signed-off-by: Akash Kumar <91385321+AkashKumar7902@users.noreply.github.com>

Describe benchmark package records

c5d5b61

Signed-off-by: Akash Kumar <91385321+AkashKumar7902@users.noreply.github.com>

Use shared API client for package benchmark

5cc6d09

Signed-off-by: Akash Kumar <91385321+AkashKumar7902@users.noreply.github.com>

Document package benchmark validation

6efd2a2

Signed-off-by: Akash Kumar <91385321+AkashKumar7902@users.noreply.github.com>

Clarify server pod readiness check

65de362

Signed-off-by: Akash Kumar <91385321+AkashKumar7902@users.noreply.github.com>

Document package benchmark preflight stages

3b99288

Signed-off-by: Akash Kumar <91385321+AkashKumar7902@users.noreply.github.com>

Correct benchmark copyright holders

3a5cab4

Signed-off-by: Akash Kumar <91385321+AkashKumar7902@users.noreply.github.com>

Remove numbered benchmark comments

a55bcf5

Signed-off-by: Akash Kumar <91385321+AkashKumar7902@users.noreply.github.com>

Format benchmark comment lists

b9ec66f

Signed-off-by: Akash Kumar <91385321+AkashKumar7902@users.noreply.github.com>

Remove synthesized timeout exit code

ca3e7b5

Stop converting Ruby timeouts into a shell-style exit code and remove result metadata that depends on parsing it. Signed-off-by: Akash Kumar <91385321+AkashKumar7902@users.noreply.github.com>

AkashKumar7902 force-pushed the testsuite/package-download-benchmark branch from 70fb1fb to 2c1ac3c Compare September 20, 2026 14:53

	stdout, stderr, code = get_target('localhost').run_local(
	stdout, stderr, code = get_target('controller').run_local(

Conversation

AkashKumar7902 commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR change?

Benchmark flow

Configuration

Prerequisites

Required inputs

Optional inputs

How to run

Codespace

GUI diff

Documentation

Test coverage

Links

Changelogs

Re-run a test

Before you merge

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AkashKumar7902 commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Pablogoliva left a comment

Choose a reason for hiding this comment

Uh oh!

AkashKumar7902 commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 25, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

AkashKumar7902 commented Jul 27, 2026 •

edited

Loading

AkashKumar7902 commented Aug 17, 2026 •

edited

Loading

AkashKumar7902 commented Aug 18, 2026 •

edited

Loading