Native SGC support for the operator

[s-alad]

What does this PR do?

Adds native SGC support for resolving the Operator's own ENC[...] credentials. When -secretBackendType is set without -secretBackendCommand, the Operator invokes /usr/local/bin/secret-generic-connector using the SGC v1.1 payload, including the backend type, nested configuration, and a 30-second timeout.

The public Operator image includes the normal or FIPS SGC binary selected by FIPS_ENABLED. Both flavors currently use SGC 7.84.0-rc.2 through the SGC_VERSION build argument.

The existing command-based secret backend remains unchanged and takes precedence when configured.

Motivation

This allows the Operator to use the same out-of-the-box secret management implementation as the Agent, so deployments can replace datadog-vault-secrets after rollout validation.

Additional Notes

Documentation for the operator will have to be updated

The internal GBI wrapper image preserves the bundled SGC binary in https://github.com/DataDog/images/pull/11615. Enabling SGC in deployment values is handled separately.

Minimum Agent Versions

None. This change resolves the Operator's own credentials and does not depend on an Agent or Cluster Agent version.

Describe your test plan

go test -count=1 ./pkg/secrets ./cmd
make managergobuild
make lint

Local Operator + SGC + Vault end-to-end validation

1. Build the Operator with SGC

From the datadog-operator PR branch:

export ARCH="$(go env GOARCH)"

docker build \
  --build-arg FIPS_ENABLED=false \
  --build-arg GOARCH="$ARCH" \
  -t local/datadog-operator:sgc-e2e \
  .

docker run --rm \
  --entrypoint /usr/local/bin/secret-generic-connector \
  local/datadog-operator:sgc-e2e \
  --version

Expected:

secret-generic-connector 7.84.0-rc.2

2. Create a kind cluster and install Vault

export KUBECONFIG=/tmp/sgc-operator-e2e.kubeconfig

kind create cluster \
  --name sgc-operator-e2e \
  --kubeconfig "$KUBECONFIG"

kind load docker-image \
  --name sgc-operator-e2e \
  local/datadog-operator:sgc-e2e

kubectl create namespace datadog-agent

# Required by the internal Operator chart's node affinity.
kubectl label node sgc-operator-e2e-control-plane \
  node-role.kubernetes.io/nodeless=true \
  --overwrite

helm repo add hashicorp https://helm.releases.hashicorp.com --force-update
helm repo update hashicorp

helm upgrade --install vault hashicorp/vault \
  --namespace vault \
  --create-namespace \
  --set server.dev.enabled=true \
  --set server.dev.devRootToken=root \
  --set injector.enabled=false \
  --wait \
  --timeout 180s

kubectl -n vault wait \
  --for=condition=Ready pod/vault-0 \
  --timeout=120s

kubectl -n vault exec vault-0 -- vault status

3. Configure Vault Kubernetes authentication

kubectl create clusterrolebinding vault-tokenreview-binding \
  --clusterrole=system:auth-delegator \
  --serviceaccount=vault:vault

kubectl -n vault exec vault-0 -- vault auth enable kubernetes

kubectl -n vault exec vault-0 -- sh -ec '
vault write auth/kubernetes/config \
  kubernetes_host=https://kubernetes.default.svc:443 \
  token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
'

kubectl -n vault exec vault-0 -- \
  vault secrets enable -path=applications kv-v2

Create the policy and role:

kubectl -n vault exec -i vault-0 -- \
  vault policy write datadog-operator /dev/stdin <<'EOF'
path "applications/data/datadog-operator/*" {
  capabilities = ["read"]
}
EOF

kubectl -n vault exec vault-0 -- \
  vault write auth/kubernetes/role/datadog-operator \
  bound_service_account_names=datadog-operator \
  bound_service_account_namespaces=datadog-agent \
  policies=datadog-operator \
  ttl=1h

Add dummy credentials and enable audit logging:

kubectl -n vault exec vault-0 -- \
  vault kv put applications/datadog-operator/api \
  value=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

kubectl -n vault exec vault-0 -- \
  vault kv put applications/datadog-operator/app \
  value=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb

kubectl -n vault exec vault-0 -- \
  vault audit enable file file_path=/tmp/audit.log

4. Deploy the Operator

Set the path to a k8s-datadog-agent-ops checkout:

export OPS_REPO=/path/to/k8s-datadog-agent-ops
export ARM_ENABLED=false

if [ "$(go env GOARCH)" = "arm64" ]; then
  export ARM_ENABLED=true
fi

Create the Helm values:

cat >/tmp/operator-sgc-values.yaml <<EOF
global:
  docker:
    registry: local
  cni:
    cilium:
      enabled: false
  datacenter:
    arm_enabled: ${ARM_ENABLED}

replicaCount: 1

image:
  name: datadog-operator
  tag: sgc-e2e
  pullPolicy: Never
  useFips: false

apiKey: "ENC[vault://applications/data/datadog-operator/api#/data/value]"
appKey: "ENC[vault://applications/data/datadog-operator/app#/data/value]"

supportExtendedDaemonset: false
enableDatadogMonitor: false
enableDatadogSLO: false
enableIntrospection: false
enableDatadogDashboard: false
enableDatadogAgentProfile: false
enableDatadogAgentInternal: false
dpa:
  enabled: false
EOF

Install the chart:

helm upgrade --install datadog-operator \
  "$OPS_REPO/charts/datadog-operator" \
  --namespace datadog-agent \
  -f /tmp/operator-sgc-values.yaml \
  --no-hooks

The chart wiring is handled separately, so patch the Deployment with the new SGC flags:

cat >/tmp/operator-sgc-patch.yaml <<'EOF'
spec:
  template:
    spec:
      containers:
        - name: datadog-operator
          args:
            - -secretBackendCommand=
            - -secretBackendType=hashicorp.vault
            - '-secretBackendConfig={"vault_session":{"vault_auth_type":"kubernetes"}}'
            - -secretRefreshInterval=5s
            - -datadogMonitorEnabled=false
            - -datadogSLOEnabled=false
            - -introspectionEnabled=false
            - -operatorMetricsEnabled=false
            - -logEncoder=json
            - -metrics-addr=:8383
            - -loglevel=debug
          env:
            - name: VAULT_ADDR
              value: http://vault.vault.svc:8200
            - name: DD_SECRETS_VAULT_AUTH_PATH
              value: auth/kubernetes/login
            - name: DD_SECRETS_VAULT_ROLE
              value: datadog-operator
            - name: DD_SECRETS_SA_TOKEN_PATH
              value: /var/run/secrets/kubernetes.io/serviceaccount/token
EOF

kubectl -n datadog-agent patch deployment datadog-operator \
  --type=strategic \
  --patch-file=/tmp/operator-sgc-patch.yaml

kubectl -n datadog-agent rollout status \
  deployment/datadog-operator \
  --timeout=180s

5. Verify secret retrieval

export POD="$(
  kubectl -n datadog-agent get pod \
    -l app.kubernetes.io/name=datadog-operator \
    --field-selector=status.phase=Running \
    -o jsonpath='{.items[0].metadata.name}'
)"

kubectl -n datadog-agent exec "$POD" -- \
  /usr/local/bin/secret-generic-connector --version

kubectl -n datadog-agent logs "$POD" | \
  grep -E 'Unable to get credentials|Failed to refresh credentials|unsupported backend|failed to provide a vault address' || true

No decryption errors should be returned.

Confirm Vault received the authentication and secret-read requests:

kubectl -n vault exec vault-0 -- \
  cat /tmp/audit.log >/tmp/vault-audit.log

jq -r '
  select(.type == "request") |
  [.request.operation, .request.path] |
  @tsv
' /tmp/vault-audit.log | sort | uniq -c

Expected paths:

update auth/kubernetes/login
read applications/data/datadog-operator/api
read applications/data/datadog-operator/app

6. Verify secret rotation

export ROTATION_START="$(date -u +%Y-%m-%dT%H:%M:%SZ)"

kubectl -n vault exec vault-0 -- \
  vault kv put applications/datadog-operator/api \
  value=cccccccccccccccccccccccccccccccc

kubectl -n vault exec vault-0 -- \
  vault kv put applications/datadog-operator/app \
  value=dddddddddddddddddddddddddddddddddddddddd

sleep 10

kubectl -n datadog-agent logs "$POD" \
  --since-time="$ROTATION_START" | \
  grep 'Credentials have changed, cache updated'

Expected:

Credentials have changed, cache updated

This validates that the Operator retrieves and refreshes its own ENC[vault://...] credentials through SGC using Vault Kubernetes authentication.

Checklist

• PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
• PR has a milestone or the qa/skip-qa label
• All commits are signed (see: signing commits)

[datadog-official]

   

⚠️ Warnings

❌ Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 1 Pipeline job failed

DataDog/datadog-operator | verify_licenses

View more details · View in GitLab

ℹ️ Info

🔄 Datadog auto-retried 1 job - 1 passed on retry

🎯 Code Coverage (details)
• Patch Coverage: 87.50%
• Overall Coverage: 51.16% (+0.15%)

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 6e54f65 | Docs | View more details | Give us feedback!}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-11T15:21:43.328795Z	cfa1fb4	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cfa1fb45e5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve the legacy backend's five-second timeout

When an explicitly configured legacy secret-backend command hangs, this global default now allows every invocation to block for 30 seconds instead of the previous 5 seconds. Since CredentialManager.fetchCredentials retries retriable command failures up to five times, a bad legacy backend can now delay Operator startup or credential retrieval for roughly 150 seconds. Use the longer timeout only for the embedded SGC path so existing command-based backends retain their prior behavior.

Useful? React with 👍 / 👎.

[s-alad]

changed to 30s since both the regular datadog-agent and sgc default to 30s