Skip to content

Kubernetes admission webhook fix #2638

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mathieuancelin merged 1 commit into from
Aug 20, 2026
Merged

Kubernetes admission webhook fix #2638

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions kubernetes/helm/otoroshi/templates/NOTES.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,27 @@ Otoroshi is running in cluster mode:
Workers connect to the leader API service at:
https://{{ .Values.cluster.leaderApiServiceName }}.{{ .Release.Namespace }}.svc.cluster.local:{{ .Values.service.https }}
{{- end }}
{{- if or .Values.webhooks.validation .Values.webhooks.sidecarInject }}

============================================================
7. Admission webhooks (upgrades)
============================================================
{{ if .Values.webhooks.validation }}- ValidatingWebhookConfiguration "otoroshi-admission-webhook-validation" (CRD validation)
{{ end }}{{ if .Values.webhooks.sidecarInject }}- MutatingWebhookConfiguration "otoroshi-admission-webhook-injector" (sidecar injection)
{{ end }}
They are answered by request sinks registered in the Otoroshi global config through
OTOROSHI_INITIAL_CUSTOMIZATION, which only applies on the FIRST boot, while the datastore
is still empty. If you upgraded from an earlier chart version, or pointed this release at a
pre-existing Redis, check that Danger zone -> Global plugins contains
"Request sink plugin wrapper" entries for:

cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookCRDValidator
cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookSidecarInjector

Without them Otoroshi answers 404, and once the CRDs controller job has flipped failurePolicy
from Ignore to Fail (first sync loop, <= 60s), every `kubectl apply` of a proxy.otoroshi.io
resource fails with "the server could not find the requested resource".
{{- end }}

============================================================
More
Expand Down
50 changes: 50 additions & 0 deletions kubernetes/helm/otoroshi/templates/deployment-cluster.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,16 @@ spec:
value: "{{ .Values.loadbalancer.nodePort.http }}"
- name: OTOROSHI_EXPOSED_PORTS_HTTPS
value: "{{ .Values.loadbalancer.nodePort.https }}"
# Only applied on the very first boot, when the datastore is still empty. On an
# upgrade (or a pre-existing Redis) Otoroshi keeps the global config it already has,
# so any change here must also be applied by hand on existing installs.
#
# The admission-webhook request sinks are declared TWICE on purpose:
# - under `plugins.config.ng` (wrapped in RequestSinkWrapper) because the next-gen
# proxy engine — the default — only looks at that list;
# - under `scripts.sinkRefs` for the legacy engine.
# Note that `plugins.config.ng` REPLACES the built-in list (deepMerge does not merge
# arrays), hence the NgClientCredentials entry below which is an Otoroshi default.
- name: OTOROSHI_INITIAL_CUSTOMIZATION
value: >
{
Expand All @@ -144,6 +154,46 @@ spec:
"defaultDomain": "www.{{ .Values.env.domain }}",
"randomIfNotFound": false
},
"plugins": {
"enabled": true,
"config": {
"ng": [
{
"enabled": true,
"debug": false,
"include": [],
"exclude": [],
"plugin": "cp:otoroshi.next.plugins.NgClientCredentials",
"config": {
"expiration": 3600000,
"default_key_pair": "otoroshi-jwt-signing",
"domain": "*",
"secure": true
}
},
{
"enabled": true,
"debug": false,
"include": [],
"exclude": [],
"plugin": "cp:otoroshi.next.plugins.wrappers.RequestSinkWrapper",
"config": {
"plugin": "cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookCRDValidator"
}
},
{
"enabled": true,
"debug": false,
"include": [],
"exclude": [],
"plugin": "cp:otoroshi.next.plugins.wrappers.RequestSinkWrapper",
"config": {
"plugin": "cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookSidecarInjector"
}
}
]
}
},
"scripts":{
"enabled":true,
"sinkRefs":[
Expand Down
50 changes: 50 additions & 0 deletions kubernetes/helm/otoroshi/templates/deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,16 @@ spec:
value: "{{ .Values.loadbalancer.nodePort.http }}"
- name: OTOROSHI_EXPOSED_PORTS_HTTPS
value: "{{ .Values.loadbalancer.nodePort.https }}"
# Only applied on the very first boot, when the datastore is still empty. On an
# upgrade (or a pre-existing Redis) Otoroshi keeps the global config it already has,
# so any change here must also be applied by hand on existing installs.
#
# The admission-webhook request sinks are declared TWICE on purpose:
# - under `plugins.config.ng` (wrapped in RequestSinkWrapper) because the next-gen
# proxy engine — the default — only looks at that list;
# - under `scripts.sinkRefs` for the legacy engine.
# Note that `plugins.config.ng` REPLACES the built-in list (deepMerge does not merge
# arrays), hence the NgClientCredentials entry below which is an Otoroshi default.
- name: OTOROSHI_INITIAL_CUSTOMIZATION
value: >
{
Expand All @@ -116,6 +126,46 @@ spec:
"defaultDomain": "www.{{ .Values.env.domain }}",
"randomIfNotFound": false
},
"plugins": {
"enabled": true,
"config": {
"ng": [
{
"enabled": true,
"debug": false,
"include": [],
"exclude": [],
"plugin": "cp:otoroshi.next.plugins.NgClientCredentials",
"config": {
"expiration": 3600000,
"default_key_pair": "otoroshi-jwt-signing",
"domain": "*",
"secure": true
}
},
{
"enabled": true,
"debug": false,
"include": [],
"exclude": [],
"plugin": "cp:otoroshi.next.plugins.wrappers.RequestSinkWrapper",
"config": {
"plugin": "cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookCRDValidator"
}
},
{
"enabled": true,
"debug": false,
"include": [],
"exclude": [],
"plugin": "cp:otoroshi.next.plugins.wrappers.RequestSinkWrapper",
"config": {
"plugin": "cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookSidecarInjector"
}
}
]
}
},
"scripts":{
"enabled":true,
"sinkRefs":[
Expand Down
9 changes: 8 additions & 1 deletion kubernetes/helm/otoroshi/templates/webhooks.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,11 @@
{{- $webhookServiceName := ternary .Values.cluster.leaderApiServiceName .Values.service.name .Values.cluster.enabled -}}
{{/*
The webhook MUST target the very service passed as `otoroshiServiceName` in the
KubernetesConfig of the deployment templates: the sinks that answer these calls only
match a Host of `<otoroshiServiceName>.<namespace>.svc[.cluster.local]` (see
KubernetesAdmissionWebhookCRDValidator.matches), and the certificate Otoroshi generates
for that endpoint is issued for those two SANs only (see createWebhookCerts).
*/}}
{{- $webhookServiceName := ternary .Values.cluster.workerServiceName .Values.service.name .Values.cluster.enabled -}}
{{- if .Values.webhooks.validation }}
---
apiVersion: admissionregistration.k8s.io/v1
Expand Down
9 changes: 9 additions & 0 deletions kubernetes/helm/otoroshi/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -209,8 +209,17 @@ autoscaling:
# HPA scaling behavior (v2). Empty means use Kubernetes defaults.
behavior: {}

# Admission webhooks. They are answered by request sinks registered in the Otoroshi
# global config by OTOROSHI_INITIAL_CUSTOMIZATION — which only applies on the FIRST boot,
# while the datastore is still empty. On an upgrade, or against a pre-existing Redis,
# register them by hand (Danger zone -> Global plugins) before turning these on,
# otherwise Otoroshi answers 404 and, once the CRDs controller job has flipped
# failurePolicy from Ignore to Fail, every `kubectl apply` of a proxy.otoroshi.io
# resource fails with "the server could not find the requested resource".
webhooks:
# kube-apiserver validates every proxy.otoroshi.io CR against Otoroshi before persisting it.
validation: true
# Injects an otoroshi-sidecar container into pods labeled `otoroshi.io/sidecar: inject`.
sidecarInject: false

# PodDisruptionBudget.
Expand Down
163 changes: 104 additions & 59 deletions kubernetes/kustomize/components/initial-customization-cluster/initial-customization.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,63 +1,108 @@
{
"config": {},
"scripts": {
"enabled": true,
"sinkRefs": [
"cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookCRDValidator",
"cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookSidecarInjector"
],
"sinkConfig": {},
"jobRefs": [
"cp:otoroshi.plugins.jobs.kubernetes.KubernetesOtoroshiCRDsControllerJob"
],
"jobConfig": {
"KubernetesConfig": {
"trust": false,
"namespaces": ["*"],
"labels": {},
"namespacesLabels": {},
"ingressClasses": ["otoroshi"],
"defaultGroup": "default",
"ingresses": false,
"crds": true,
"coreDnsIntegration": false,
"coreDnsIntegrationDryRun": false,
"kubeLeader": false,
"restartDependantDeployments": false,
"watch": false,
"syncDaikokuApikeysOnly": false,
"kubeSystemNamespace": "kube-system",
"coreDnsConfigMapName": "coredns",
"coreDnsDeploymentName": "coredns",
"corednsPort": 53,
"otoroshiServiceName": "otoroshi-worker-service",
"otoroshiNamespace": "otoroshi",
"clusterDomain": "cluster.local",
"syncIntervalSeconds": 60,
"coreDnsEnv": null,
"watchTimeoutSeconds": 60,
"watchGracePeriodSeconds": 5,
"mutatingWebhookName": "otoroshi-admission-webhook-injector",
"validatingWebhookName": "otoroshi-admission-webhook-validation",
"templates": {
"service-group": {},
"service-descriptor": {},
"apikeys": {},
"global-config": {},
"jwt-verifier": {},
"tcp-service": {},
"certificate": {},
"auth-module": {},
"script": {},
"organizations": {},
"teams": {},
"webhooks": {
"flags": {
"requestCert": true,
"originCheck": true,
"tokensCheck": true,
"displayEnv": false,
"tlsTrace": false
"config": {
"plugins": {
"enabled": true,
"config": {
"ng": [
{
"enabled": true,
"debug": false,
"include": [],
"exclude": [],
"plugin": "cp:otoroshi.next.plugins.NgClientCredentials",
"config": {
"expiration": 3600000,
"default_key_pair": "otoroshi-jwt-signing",
"domain": "*",
"secure": true
}
},
{
"enabled": true,
"debug": false,
"include": [],
"exclude": [],
"plugin": "cp:otoroshi.next.plugins.wrappers.RequestSinkWrapper",
"config": {
"plugin": "cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookCRDValidator"
}
},
{
"enabled": true,
"debug": false,
"include": [],
"exclude": [],
"plugin": "cp:otoroshi.next.plugins.wrappers.RequestSinkWrapper",
"config": {
"plugin": "cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookSidecarInjector"
}
}
]
}
},
"scripts": {
"enabled": true,
"sinkRefs": [
"cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookCRDValidator",
"cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookSidecarInjector"
],
"sinkConfig": {},
"jobRefs": [
"cp:otoroshi.plugins.jobs.kubernetes.KubernetesOtoroshiCRDsControllerJob"
],
"jobConfig": {
"KubernetesConfig": {
"trust": false,
"namespaces": [
"*"
],
"labels": {},
"namespacesLabels": {},
"ingressClasses": [
"otoroshi"
],
"defaultGroup": "default",
"ingresses": false,
"crds": true,
"coreDnsIntegration": false,
"coreDnsIntegrationDryRun": false,
"kubeLeader": false,
"restartDependantDeployments": false,
"watch": false,
"syncDaikokuApikeysOnly": false,
"kubeSystemNamespace": "kube-system",
"coreDnsConfigMapName": "coredns",
"coreDnsDeploymentName": "coredns",
"corednsPort": 53,
"otoroshiServiceName": "otoroshi-worker-service",
"otoroshiNamespace": "otoroshi",
"clusterDomain": "cluster.local",
"syncIntervalSeconds": 60,
"coreDnsEnv": null,
"watchTimeoutSeconds": 60,
"watchGracePeriodSeconds": 5,
"mutatingWebhookName": "otoroshi-admission-webhook-injector",
"validatingWebhookName": "otoroshi-admission-webhook-validation",
"templates": {
"service-group": {},
"service-descriptor": {},
"apikeys": {},
"global-config": {},
"jwt-verifier": {},
"tcp-service": {},
"certificate": {},
"auth-module": {},
"script": {},
"organizations": {},
"teams": {},
"webhooks": {
"flags": {
"requestCert": true,
"originCheck": true,
"tokensCheck": true,
"displayEnv": false,
"tlsTrace": false
}
}
}
}
Expand Down
25 changes: 25 additions & 0 deletions kubernetes/kustomize/components/initial-customization-cluster/readme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,31 @@ setups, use [`initial-customization-single`](../initial-customization-single/) i

This component is enabled by default in the `cluster*` overlays.

## Shape of the file

The whole payload lives under a top-level `config` key — that is the only key
Otoroshi merges into the global config (a sibling top-level `scripts` key means
something else entirely: a list of Script *entities* to import).

The admission-webhook request sinks are declared twice on purpose:

- under `config.plugins.config.ng`, wrapped in `RequestSinkWrapper`, because the
next-gen proxy engine — the default — only reads its request sinks from there;
- under `config.scripts.sinkRefs`, for the legacy engine.

`config.plugins.config.ng` *replaces* Otoroshi's built-in list rather than
merging into it (a JSON deep merge does not merge arrays), which is why the
default `NgClientCredentials` entry is repeated in the file.

This JSON is only applied on the very first boot, while the datastore is still
empty. On an upgrade — or against a pre-existing Redis — Otoroshi keeps the
global config it already has, so changes here must also be applied by hand on
existing installs (Danger zone → Global plugins).

Because `otoroshiServiceName` is `otoroshi-worker-service` here, the
[`webhooks`](../webhooks/) component needs its `clientConfig.service.name`
patched to that same service — see its readme.

## Customize

Edit `initial-customization.json` next to this `kustomization.yaml` to tweak
Expand Down
Loading
Loading