Skip to content

SRE-3984 Build: Fixups for OpenSUSE #18944

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
JohnMalmberg wants to merge 1 commit into
base: master
Choose a base branch
from
Open

SRE-3984 Build: Fixups for OpenSUSE #18944

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
101 changes: 77 additions & 24 deletions Jenkinsfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -480,6 +480,36 @@ String sconsArgs() {
return sconsFaultsArgs() + ' ' + params.CI_SCONS_ARGS
}

// This is a temporary workaround for a bug in pipeline-lib proxy handling.
// Because we can not risk failures until we verify that that bug can be
// fixed, we will use this function to remove the bug for each active branch
// Proxy rules:
// IF jenkins system has an HTTPS_PROXY set then it needs to be passed
// to the docker build command. If the system does not have an HTTPS_PROXY
// set then the docker build MUST NOT include an HTTPS_PROXY build argument.
//
// The DAOS_NO_PROXY build argument was put in because the HTTPS_PROXY
// build argument was being passed to the docker build when it should not
// have been instead of fixing the bug in pipeline-lib.
// This function will be removed when the bug is fixed.
// The use of no_proxy/NO_PROXY environment variables to avoid a proxy is
// unsafe because there is no single standard used for for what is legal to
// be in it.
//
// If a local mirror is available via REPOSITORY_URL/ARTIFACTORY_URL it
// should always be used instead of a proxy server for reliability.
// Applications should where possible always use configuration file to
// for any needed proxy or mirror settings instead of environment variables
// which will avoid needing no_proxy issues.
String daosDockerBuildArgs(Map args = [:]) {
String buildArgs = dockerBuildArgs(args)
if (!env.HTTPS_PROXY) {
buildArgs += ' --build-arg HTTPS_PROXY='
buildArgs += ' --build-arg DAOS_NO_PROXY='
}
return buildArgs
}

/**
* Update default commit pragmas based on files modified.
*/
Expand All @@ -503,7 +533,6 @@ pipeline {
TEST_RPMS = cachedCommitPragma(pragma: 'RPM-test', def_val: 'true')
COVFN_DISABLED = cachedCommitPragma(pragma: 'Skip-fnbullseye', def_val: 'true')
REPO_FILE_URL = repoFileUrl(env.REPO_FILE_URL)
HTTPS_PROXY = ''
PYTHON_VERSION = '3.11'
}

Expand Down Expand Up @@ -614,21 +643,27 @@ pipeline {
booleanParam(name: bashName('Functional on EL 9 with Valgrind'),
defaultValue: false,
description: 'Run the Functional on EL 9 with Valgrind stage.')
// >>> TEMP PATCH: shift default functional testing from EL 9 to Leap 15/SLES 15 <<<
// >>> REVERT these 3 defaultValue lines back to EL9=true, Leap15=false, SLES15=false <<<
Comment on lines +646 to +647

@daltonbohning daltonbohning Aug 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use commit pragmas?

@JohnMalmberg JohnMalmberg Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commit pragmas take too long to lookup and figure out the right combination to work, and if you for get to copy them to a new commit, a lot of test time get wasted.

Preferred solution to commit pragmas is an optional file that if present is processed by commit pragmas, but github actions blocks from landing if it is present. Has to be removed by PR user once testing is done.

booleanParam(name: bashName('Functional on EL 9'),
defaultValue: true,
// defaultValue: true,
defaultValue: false,
description: 'Run the Functional on EL 9 stage.')
booleanParam(name: bashName('Functional on Leap 15'),
defaultValue: false,
// defaultValue: false,
defaultValue: true,
description: 'Run the Functional on Leap 15 stage.')
booleanParam(name: bashName('Functional on SLES 15'),
defaultValue: false,
// defaultValue: false,
defaultValue: true,
description: 'Run the Functional on SLES 15 stage.')
booleanParam(name: bashName('Functional on Ubuntu 20.04'),
defaultValue: false,
description: 'Run the Functional on Ubuntu 20.04 stage.')
booleanParam(name: bashName('Fault injection testing'),
defaultValue: true,
description: 'Run the Fault injection testing stage.')
// >>> END TEMP PATCH <<<
booleanParam(name: bashName('Test RPMs on EL 9'),
defaultValue: true,
description: 'Run the Test RPMs on EL 9 stage.')
Expand All @@ -638,26 +673,33 @@ pipeline {
booleanParam(name: bashName('Test Hardware'),
defaultValue: true,
description: 'Run the Test Hardware stage.')
// >>> TEMP PATCH: run all hardware stages on SLES 15.7 for PR coverage <<<
// >>> REVERT these false->true defaultValue lines after SLES/Leap validation <<<
booleanParam(name: bashName('Functional Hardware Medium'),
defaultValue: false,
// defaultValue: false,
defaultValue: true,
description: 'Run the Functional Hardware Medium stage.')
booleanParam(name: bashName('Functional Hardware Medium MD on SSD'),
defaultValue: true,
description: 'Run the Functional Hardware Medium MD on SSD stage.')
booleanParam(name: bashName('Functional Hardware Medium VMD'),
defaultValue: false,
// defaultValue: false,
defaultValue: true,
description: 'Run the Functional Hardware Medium VMD stage.')
booleanParam(name: bashName('Functional Hardware Medium Verbs Provider'),
defaultValue: false,
// defaultValue: false,
defaultValue: true,
description: 'Run the Functional Hardware Medium Verbs Provider stage.')
booleanParam(name: bashName('Functional Hardware Medium Verbs Provider MD on SSD'),
defaultValue: true,
description: 'Run the Functional Hardware Medium Verbs Provider MD on SSD stage.')
booleanParam(name: bashName('Functional Hardware Medium UCX Provider'),
defaultValue: false,
// defaultValue: false,
defaultValue: true,
description: 'Run the Functional Hardware Medium UCX Provider stage.')
booleanParam(name: bashName('Functional Hardware Large'),
defaultValue: false,
// defaultValue: false,
defaultValue: true,
description: 'Run the Functional Hardware Large stage.')
booleanParam(name: bashName('Functional Hardware Large MD on SSD'),
defaultValue: true,
Expand All @@ -668,6 +710,7 @@ pipeline {
booleanParam(name: bashName('Functional Cluster Box Medium Verbs Provider MD on SSD'),
defaultValue: true,
description: 'Run the Functional Cluster Box Verbs Provider test stage')
// >>> END TEMP PATCH <<<
string(name: 'CI_UNIT_VM1_LABEL',
defaultValue: 'ci_vm1',
description: 'Label to use for 1 VM node unit and RPM tests')
Expand Down Expand Up @@ -833,7 +876,7 @@ pipeline {
' --build-arg REPOS="' + prRepos() + '"' +
' --build-arg POINT_RELEASE=.7' +
" --build-arg PYTHON_VERSION=${env.PYTHON_VERSION}" +
" --build-arg DAOS_DEPS_INSTALL=yes"
' --build-arg DAOS_DEPS_INSTALL=yes'
}
}
steps {
Expand Down Expand Up @@ -886,14 +929,14 @@ pipeline {
dockerfile {
filename 'utils/docker/Dockerfile.leap.15'
label 'docker_runner'
additionalBuildArgs dockerBuildArgs(repo_type: 'stable',
parallel_build: true,
deps_build: true) +
additionalBuildArgs daosDockerBuildArgs(repo_type: 'stable',
parallel_build: true,
deps_build: true) +
" -t ${sanitized_JOB_NAME()}-leap15" +
' --target build-ci' +
' --build-arg POINT_RELEASE=.6' +
" --build-arg PYTHON_VERSION=${env.PYTHON_VERSION}" +
" --build-arg DAOS_DEPS_INSTALL=yes"
' --build-arg DAOS_DEPS_INSTALL=yes'
}
}
steps {
Expand Down Expand Up @@ -1271,7 +1314,8 @@ pipeline {
default_tags: startedByTimer() ? 'pr daily_regression' : 'pr',
nvme: 'auto',
job_status: job_status_internal,
image_version: 'el9.7'
// image_version: 'el9.7'
image_version: 'sles15.7'
),
'Functional Hardware Medium MD on SSD': getFunctionalTestStage(
name: 'Functional Hardware Medium MD on SSD',
Expand All @@ -1283,7 +1327,8 @@ pipeline {
default_tags: startedByTimer() ? 'pr daily_regression' : 'pr',
nvme: 'auto_md_on_ssd',
job_status: job_status_internal,
image_version: 'el9.7'
// image_version: 'el9.7'
image_version: 'sles15.7'
),
'Functional Hardware Medium VMD': getFunctionalTestStage(
name: 'Functional Hardware Medium VMD',
Expand All @@ -1296,7 +1341,8 @@ pipeline {
default_tags: startedByTimer() ? 'pr daily_regression' : 'pr',
nvme: 'auto',
job_status: job_status_internal,
image_version: 'el9.7'
// image_version: 'el9.7'
image_version: 'sles15.7'
),
'Functional Hardware Medium Verbs Provider': getFunctionalTestStage(
name: 'Functional Hardware Medium Verbs Provider',
Expand All @@ -1309,7 +1355,8 @@ pipeline {
default_nvme: 'auto',
provider: 'ofi+verbs;ofi_rxm',
job_status: job_status_internal,
image_version: 'el9.7'
// image_version: 'el9.7'
image_version: 'sles15.7'
),
'Functional Hardware Medium Verbs Provider MD on SSD': getFunctionalTestStage(
name: 'Functional Hardware Medium Verbs Provider MD on SSD',
Expand All @@ -1322,7 +1369,8 @@ pipeline {
default_nvme: 'auto_md_on_ssd',
provider: 'ofi+verbs;ofi_rxm',
job_status: job_status_internal,
image_version: 'el9.7'
// image_version: 'el9.7'
image_version: 'sles15.7'
),
'Functional Hardware Medium UCX Provider': getFunctionalTestStage(
name: 'Functional Hardware Medium UCX Provider',
Expand All @@ -1335,7 +1383,8 @@ pipeline {
default_nvme: 'auto',
provider: cachedCommitPragma('Test-provider-ucx', 'ucx+ud_x'),
job_status: job_status_internal,
image_version: 'el9.7'
// image_version: 'el9.7'
image_version: 'sles15.7'
),
'Functional Hardware Large': getFunctionalTestStage(
name: 'Functional Hardware Large',
Expand All @@ -1347,7 +1396,8 @@ pipeline {
default_tags: startedByTimer() ? 'pr daily_regression' : 'pr',
default_nvme: 'auto',
job_status: job_status_internal,
image_version: 'el9.7'
// image_version: 'el9.7'
image_version: 'sles15.7'
),
'Functional Hardware Large MD on SSD': getFunctionalTestStage(
name: 'Functional Hardware Large MD on SSD',
Expand All @@ -1359,7 +1409,8 @@ pipeline {
default_tags: startedByTimer() ? 'pr daily_regression' : 'pr',
default_nvme: 'auto_md_on_ssd',
job_status: job_status_internal,
image_version: 'el9.7'
// image_version: 'el9.7'
image_version: 'sles15.7'
),
'Functional Cluster Box Medium MD on SSD': getFunctionalTestStage(
name: 'Functional Cluster Box Medium MD on SSD',
Expand All @@ -1374,7 +1425,8 @@ pipeline {
run_if_pr: true,
run_if_landing: false,
job_status: job_status_internal,
image_version: 'el9.7'
// image_version: 'el9.7'
image_version: 'sles15.7'
),
'Functional Cluster Box Medium Verbs Provider MD on SSD': getFunctionalTestStage(
name: 'Functional Cluster Box Medium Verbs Provider MD on SSD',
Expand All @@ -1390,7 +1442,8 @@ pipeline {
run_if_pr: true,
run_if_landing: false,
job_status: job_status_internal,
image_version: 'el9.7'
// image_version: 'el9.7'
image_version: 'sles15.7'
),
)
}
Expand Down
3 changes: 2 additions & 1 deletion ci/provisioning/post_provision_config_nodes_LEAP.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@ bootstrap_dnf() {
rm -rf "$REPOS_DIR"
ln -s ../zypp/repos.d "$REPOS_DIR"
dnf -y remove lua-lmod
dnf -y --nogpgcheck install lua-lmod --repo '*lua*' --repo '*network-cluster*' --repo '*oss-proxy*'
dnf -y --nogpgcheck install lua-lmod \
--repo '*network-cluster*' --repo '*oss-proxy*'
}

group_repo_post() {
Expand Down
5 changes: 4 additions & 1 deletion ftest.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,7 @@ args="${1:-quick}"
shift || true
args+=" $*"

_HTTPS_PROXY=${HTTPS_PROXY:-}
_DAOS_HTTPS_PROXY=${DAOS_HTTPS_PROXY:-}
_DAOS_NO_PROXY=${DAOS_NO_PROXY:-}
# shellcheck disable=SC2029
Expand All @@ -134,7 +135,9 @@ if ! ssh -A $SSH_KEY_ARGS ${REMOTE_ACCT:-jenkins}@"${nodes[0]}" \
LAUNCH_OPT_ARGS=\"$LAUNCH_OPT_ARGS\"
WITH_VALGRIND=\"$WITH_VALGRIND\"
STAGE_NAME=\"$STAGE_NAME\"
DAOS_HTTPS_PROXY=\"$_DAOS_HTTPS_PROXY\"
HTTPS_PROXY=\"$_HTTPS_PROXY\"
# TEMP DISABLED escape hatch: uncomment to force DAOS_HTTPS_PROXY through as a quick fix.
# DAOS_HTTPS_PROXY=\"$_DAOS_HTTPS_PROXY\"
Comment on lines +139 to +140

@daltonbohning daltonbohning Aug 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we remove this if it is not needed?

@JohnMalmberg JohnMalmberg Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am still smoke testing this, you apparently got added automatically to the PR as a codeowner.

I am expecting a few passes at this which I will be doing forced pushes before I have something ready for an official review.

DAOS_NO_PROXY=\"$_DAOS_NO_PROXY\"
DAOS_FTEST_VENV=\"$DAOS_FTEST_VENV\"
$(sed -e '1,/^$/d' "$SCRIPT_LOC"/main.sh)"; then
Expand Down
11 changes: 7 additions & 4 deletions src/tests/ftest/scripts/main.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,10 +80,13 @@ export TEST_RPMS
export DAOS_BASE
export DAOS_TEST_APP_SRC=${DAOS_TEST_APP_SRC:-"/CIShare/daos_test/apps"}
export DAOS_TEST_APP_DIR=${DAOS_TEST_APP_DIR:-"${DAOS_TEST_SHARED_DIR}/daos_test/apps"}
if [ -n "$DAOS_HTTPS_PROXY" ]; then
# shellcheck disable=SC2154
export HTTPS_PROXY="${DAOS_HTTPS_PROXY:-""}"
fi
# HTTPS_PROXY is passed in directly by ftest.sh. A high-level script like this
# should not be the one assigning HTTPS_PROXY=$DAOS_HTTPS_PROXY; revisit after
# smoke testing and document proper CI/lab proxy usage.
# if [ -n "$DAOS_HTTPS_PROXY" ]; then
# # shellcheck disable=SC2154
# export HTTPS_PROXY="${DAOS_HTTPS_PROXY:-""}"
# fi
Comment on lines +83 to +89

@daltonbohning daltonbohning Aug 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should also run the dfuse build tests since they rely on the proxy env. Unfortunately one of them is currently failing, but the others should be working

Features: build
Allow-unstable-test: true

@JohnMalmberg JohnMalmberg Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add that to the next smoke run, thanks.

if [ -n "$DAOS_NO_PROXY" ]; then
export NO_PROXY="${DAOS_NO_PROXY:-""}"
fi
Expand Down
54 changes: 24 additions & 30 deletions utils/docker/Dockerfile.leap.15
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,26 +31,35 @@ ARG JENKINS_URL
ARG REPOS
ARG DAOS_LAB_CA_FILE_URL

# Accept DAOS_HTTP_PROXY, DAOS_HTTPS_PROXY, and DAOS_NO_PROXY at build time
ARG DAOS_HTTP_PROXY
# Accept DAOS_HTTPS_PROXY, and DAOS_NO_PROXY at build time
# DAOS_HTTPS_PROXY is used safely pass a PROXY to the build environment with
# out making everything use it, which can cause problems.
# DAOS_HTTPS_PROXY is used when you do not have a local artifact server
# and you need to use a proxy server that is not a smart proxy.
#
# HTTPS_PROXY is an automatic ARG, that will be global to the build environment
# if it is passed. If HTTPS_PROXY is passed, normally the DAOS_HTTPS_PROXY
# should not be passed. HTTPS_PROXY should be a smart proxy that can
# can pass through to URLs that resolve to local IP addresses.
#
# DAOS_NO_PROXY is used to safely pass a NO_PROXY list to the build
# environment without making everything use it, which can cause problems.
# The no_proxy/NO_PROXY environment variables are risky to use as there are
# conflicts in how they should be defined for different applications.
ARG DAOS_HTTPS_PROXY
ARG DAOS_NO_PROXY
# Propagate into the build environment
ENV http_proxy=${DAOS_HTTP_PROXY} \
HTTP_PROXY=${DAOS_HTTP_PROXY} \
https_proxy=${DAOS_HTTPS_PROXY} \
HTTPS_PROXY=${DAOS_HTTPS_PROXY} \
no_proxy=${DAOS_NO_PROXY} \
ENV no_proxy=${DAOS_NO_PROXY} \
NO_PROXY=${DAOS_NO_PROXY}
# Persist into /etc/environment for use by shells and services
RUN set -e; \
if [ -n "$DAOS_HTTP_PROXY" ]; then \
echo "http_proxy=$DAOS_HTTP_PROXY" >> /etc/environment; \
echo "HTTP_PROXY=$DAOS_HTTP_PROXY" >> /etc/environment; \
fi; \
if [ -n "$DAOS_HTTPS_PROXY" ]; then \
echo "https_proxy=$DAOS_HTTPS_PROXY" >> /etc/environment; \
echo "HTTPS_PROXY=$DAOS_HTTPS_PROXY" >> /etc/environment; \
echo "daos_https_proxy=$DAOS_HTTPS_PROXY" >> /etc/environment; \
echo "DAOS_HTTPS_PROXY=$DAOS_HTTPS_PROXY" >> /etc/environment; \
fi; \
if [ -n "$HTTPS_PROXY" ]; then \
echo "https_proxy=$HTTPS_PROXY" >> /etc/environment; \
echo "HTTPS_PROXY=$HTTPS_PROXY" >> /etc/environment; \
fi; \
if [ -n "$DAOS_NO_PROXY" ]; then \
echo "no_proxy=$DAOS_NO_PROXY" >> /etc/environment; \
Expand All @@ -75,29 +84,14 @@ RUN dnf remove -y $DNF_UPGRADE_EXCLUDES || true && \
# Setup docker image for gcc build only. This is done in a separate stage
# to allow for a smaller image for CI building only.
FROM basic AS build-ci
ARG JENKINS_URL

# Install minimal set of packages for building DAOS with gcc compiler and
# no Java support.
RUN --mount=type=bind,source=utils/scripts/install-leap15.sh,target=/tmp/daos/install.sh \
INSTALL_BUILD_CI_ONLY="true" /tmp/daos/install.sh && \
JENKINS_URL="$JENKINS_URL" INSTALL_BUILD_CI_ONLY="true" /tmp/daos/install.sh && \
dnf clean all

# According to https://pkgs.org/search/?q=lua-lmod, Leap 15.6 only has lua-lmod-8.7.34.
# This version has a problem with loading modules, as described in https://github.com/TACC/Lmod/issues/687.
# This affects MPI detection in scons, so DAOS cannot be built with MPI support.
# A custom source of the package is required in CI to install a valid version
# of lua-lmod and its dependencies:
# opensuse-network-cluster for lua-lmod (>=8.7.55)
# opensuse-oss for lua53, lua53-luaterm,, sqlite3-tcl, tcl
# opensuse-devel-languages-lua for lua53-luafilesystem, lua53-luaposix
ARG JENKINS_URL
RUN if [ -n "$JENKINS_URL" ]; then \
dnf -y remove lua-lmod; \
dnf -y --nogpgcheck install lua-lmod --repo '*lua*' \
--repo '*network-cluster*' --repo '*oss-proxy*' && \
dnf clean all; \
fi

# Add DAOS users
ARG UID=1000
RUN --mount=type=bind,source=utils/scripts/helpers/daos-server-user-setup.sh,target=/tmp/daos/daos-server-user-setup.sh \
Expand Down
Loading
Loading