Repository navigation
Replies: 5 comments
|
If a subtenant just correlates to a subset of namespaces within a tenant, managed by the Tenant Owner, this is already working. If you can outline your use-case a bit further i can assist you with a concrete design and point you to the necessary resources. If this is about a |
|
@oliverbaehler Thanks for the quick reply, and for offering to help with the design. Let me lay out the real topology, because it narrows the question a fair bit. Platform team runs several clusters. In each cluster they create N tenants. A tenant is pinned to its own nodepool and a fixed resource envelope. Between tenants we need hard isolation: no node sharing, and no cross-tenant traffic (enforced with Istio on top of the namespace boundary). That part Capsule already does well, and it stays the platform team's job. The mental model we want is: a tenant is a cluster, a subtenant is a namespace. Inside a tenant, soft isolation is enough. The thing that breaks namespace-as-a-service for us is CRDs. Tenants need their own operators and CRs, but CRDs are cluster-scoped, so handing tenants the ability to install them means version and schema conflicts between teams. The usual escape hatch is vcluster, which gives every team a control plane to operate and makes their life worse, not better. What we actually want is for the platform team to whitelist a set of CRDs per tenant, and for the tenant admin team to serve those to their subtenants without ever touching cluster scope. So the responsibility split is:
Given a subtenant is just a namespace, you're right that most of the RBAC side already works. The gaps are:
So the question is narrower than the original issue: can a tenant owner subdivide their own tenant's resources and API surface across their namespaces, without cluster-admin involvement? Nested If part of this is already reachable with |
|
Okay, not all requirementns are going to be covered as per today. Before i start i want to hint towards the new rulesystem: https://projectcapsule.dev/docs/rules/enforcement/ While this is currently deployed via Tenant CR it ends up in a RuleStatus CR, the idea in the future is that there can be aggregates of RuleStatuses. Meaning Rules can also be created per namespace by whomever has the permission to do so. But this will take some more thinking from my side until a implementation lands (months). One other thing, we are also adding a Break-The-Glass Functionality, essentially where users can provide Templates, which require approvement and then something is kept for a time-period, this may also work well with your timed requirement: # This template grants temporary CRUD access to all instances of one custom
# resource in the namespace of the BreakRequest.
#
# Example BreakRequest parameters:
# params:
# name: widgets-editor-alice
# crdName: widgets.platform.example.io
# subjectKind: User
# subjectName: alice@example.com
apiVersion: capsule.clastix.io/v1beta2
kind: BreakRequestTemplate
metadata:
name: crd-resource-crud
spec:
# Optional. Selectors are ORed; omit this field to allow BreakRequests from
# every namespace. Matching namespace names are published in status.namespaces.
# namespaceSelectors:
# - matchLabels:
# projectcapsule.dev/break-the-glass: enabled
autoApprove: true
defaultDuration: 1h
maxDuration: 8h
keepFor: 7d
paramSchema:
type: object
additionalProperties: false
required:
- name
- crdName
- subjectKind
- subjectName
properties:
name:
type: string
description: Unique DNS-compatible name for the temporary Role and RoleBinding.
pattern: '^[a-z0-9]([-a-z0-9]*[a-z0-9])?$'
maxLength: 63
crdName:
type: string
description: Name of the CustomResourceDefinition, for example widgets.platform.example.io.
minLength: 1
maxLength: 253
subjectKind:
type: string
description: Kind of RBAC subject receiving temporary access.
enum:
- User
- Group
subjectName:
type: string
description: Name of the Kubernetes user or group receiving temporary access.
minLength: 1
context:
resources:
- apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
name: '{{ .crdName }}'
index: crd
optional: false
resources:
- policy:
# Owner requires Capsule to create the resource. Merge adopts an
# existing resource when possible and creates it otherwise.
creation: Owner
# Prevent changes through admission while the request is active.
protect: true
force: false
targets:
- apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: '{{ .name }}'
rules:
- apiGroups:
- '{{ (index .crd 0).spec.group }}'
resources:
- '{{ (index .crd 0).spec.names.plural }}'
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- deletecollection
template: |
# Multi YAML
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: {{ $.params.name }}
rules:
- apiGroups:
- {{ (index $.context.resources.crd 0).spec.group }}
resources:
- {{ (index $.context.resources.crd 0).spec.names.plural }}
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- deletecollection
- policy:
creation: Owner
protect: true
force: false
targets:
- apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: '{{ .name }}'
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: '{{ .name }}'
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: '{{ .subjectKind }}'
name: '{{ .subjectName }}'I will try to solve these issues with tools available as per release 0.14.2: 1. Quota Delegation: Okay, but why can't they. In your setup you would probably have a Resourcepool per Tenant, which equals the capacity of the Node, since these are also dedicated to the Tenant. This is provisioned by the platform, we can also write a GTR to automate that process so that Resourcepools scale based on the nodes but that would be a bit more advanced.So if the TenantOwner can now distribute the Claims to the namespaces, don't we create a ceiling per namespace and the total potential quantity matches the available resources from the node? 2. CRD permissions: "tenant A may use these CRDs, and its tenant admin decides which subtenants get which"", indicates that it's just about being able to use CRDs, that's more a question of native RBAC than anything else. If we provide Tenant-Owners with the proper permissions to access CRDs, they can then also distribute further bindings within their Tenant with We want to allow Tenant-Owners to distribute access to Gateway API, for this we must first distribute these privileges to the Tenant-Owners, in a strict scenario the controller needs to be able to bind the role as well: ---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: gateway-api-role-distributor
rules:
- apiGroups:
- rbac.authorization.k8s.io
resources:
- rolebindings
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- rbac.authorization.k8s.io
resources:
- clusterroles
resourceNames:
- gateway-api-role-distributor
verbs:
- get
- bind
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: capsule:controller:gateway-api-role-distributor
labels:
projectcapsule.dev/aggregate-to-controller: "true"
rules:
- apiGroups:
- rbac.authorization.k8s.io
resources:
- clusterroles
resourceNames:
- gateway-api-role-distributor
verbs:
- get
- bindOn the Tenant we will leverage Promotions ---
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
labels:
customer: renewable
annotations:
info.projectcapsule.dev/icon: https://cdn-icons-png.flaticon.com/512/3463/3463440.png
info.projectcapsule.dev/description: "Production tenant for the solar team"
info.projectcapsule.dev/links: '[{"title":"Grafana","url":"https://grafana.example.com/d/payments","icon":"mdi:grafana"},{"title":"Runbook","url":"https://wiki.example.com/payments-runbook", "icon": "icon": "fa-solid fa-chart-line"}]'
spec:
rules:
# Promote only the dedicated TenantResource ServiceAccount. Capsule binds
# this distributor role in every solar namespace so the TenantResource can
# create the temporary user-facing RoleBindings there.
- permissions:
promotions:
- clusterRoles:
- gateway-api-role-distributor
- adminLet's assume every Tenant has ---
apiVersion: v1
kind: ServiceAccount
metadata:
name: gateway-api-role-distributor
namespace: solar-system
labels:
projectcapsule.dev/promote: "true"
---
apiVersion: capsule.clastix.io/v1beta2
kind: TenantResource
metadata:
name: rbac-distribution
namespace: solar-system
spec:
resyncPeriod: 60s
serviceAccount:
name: gateway-api-role-distributor
resources:
- rawItems:
- apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: "tenant:cr:gateway-api"
namespace: "{{namespace}}"
subjects:
- kind: User
name: dave
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: gateway-api-role-distributor
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ConfigMap
metadata:
name: tenant-management
namespace: solar-system
data:
team-a: |
owners:
- kind: User
name: alice
- kind: User
name: bob
team-b: |
owners:
- kind: User
name: alice
- kind: User
name: bob
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: gateway-api-role-distributor
namespace: solar-system
labels:
projectcapsule.dev/promote: "true"
---
apiVersion: capsule.clastix.io/v1beta2
kind: TenantResource
metadata:
name: tenant-distribution
namespace: solar-system
spec:
resyncPeriod: 60s
serviceAccount:
name: gateway-api-role-distributor
resources:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: solar-system
context:
resources:
- index: mgmt
apiVersion: v1
kind: ConfigMap
name: tenant-management
namespace: solar-system
generators:
- template: |
{{- range $team, $config := (index $.mgmt 0 "data") }}
{{- $team_config := fromYAML $config }}
---
apiVersion: capsule.clastix.io/v1beta2
kind: TenantResource
metadata:
name: "{{ $team }}-tenant-distribution"
namespace: solar-system
ownerReferences:
- apiVersion: capsule.clastix.io/v1beta2
kind: TenantResource
name: {{ $.replications.metadata.name | quote }}
uid: {{ $.replications.metadata.uid | quote }}
controller: true
blockOwnerDeletion: true
spec:
resyncPeriod: 60s
serviceAccount:
name: gateway-api-role-distributor
resources:
- namespaceSelector:
matchLabels:
team: {{ $team }}
rawItems:
- apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: "tenant:{{`{{namespace}}`}}:cr:gateway-api"
namespace: "{{`{{namespace}}`}}"
subjects:
{{- range $team_config.owners }}
- kind: {{ .kind }}
name: {{ .name }}
apiGroup: rbac.authorization.k8s.io
{{- end }}
roleRef:
kind: ClusterRole
name: gateway-api-role-distributor
apiGroup: rbac.authorization.k8s.io
{{- end }}We have an example inventory in a ConfigMap, based on that we spawn for each "team" a dedicated
Hope that helps, we are obiously happy to add features, if we think they might be a benefit for your case and the project. Maybe in the end we can add a reference documentation for this case on the website, WDYT? (,,>﹏<,,) |
|
@oliverbaehler This helps a lot, thanks. Model I'm confirming: tenant = cluster, subtenant = namespace, soft isolation inside. 1. Quota. ResourcePool per tenant + owner handing out claims gets me most of the way. The bit I'm stuck on: claims are self-service per namespace, so nothing stops one namespace draining the whole pool. 2. CRDs. Makes sense, it's RBAC + Promotions + 3. Slice object. The 4. Inheritance. Understood, nodeSelector/tolerations/netpol need an external policy engine for now. The scheduling enforcement + So 2 and 3 I can validate now, 1 is my one open question, 4 is roadmap. |
---
apiVersion: capsule.clastix.io/v1beta2
kind: ResourcePool
metadata:
name: solar
labels:
projectcapsule.dev/tenant: solar
spec:
quota:
hard:
limits.cpu: "0"
requests.cpu: "2"
requests.memory: "2Gi"
limits.memory: "4Gi"
requests.nvidia.com/gpu: 8
selectors:
- matchLabels:
capsule.clastix.io/tenant: solar
---
apiVersion: v1
kind: ConfigMap
metadata:
name: tenant-management
namespace: solar-system
data:
team-a: |
quota:
requests.cpu: "1"
requests.memory: "1Gi"
limits.memory: "2Gi"
requests.nvidia.com/gpu: 4
owners:
- kind: User
name: alice
- kind: User
name: bob
team-b: |
quota:
requests.cpu: "1"
requests.memory: "1Gi"
limits.memory: "2Gi"
requests.nvidia.com/gpu: 4
owners:
- kind: User
name: alice
- kind: User
name: bob
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: gateway-api-role-distributor
namespace: solar-system
labels:
projectcapsule.dev/promote: "true"
---
apiVersion: capsule.clastix.io/v1beta2
kind: TenantResource
metadata:
name: tenant-distribution
namespace: solar-system
spec:
resyncPeriod: 60s
serviceAccount:
name: gateway-api-role-distributor
resources:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: solar-system
context:
resources:
- index: mgmt
apiVersion: v1
kind: ConfigMap
name: tenant-management
namespace: solar-system
generators:
- template: |
{{- range $team, $config := (index $.mgmt 0 "data") }}
{{- $team_config := fromYAML $config }}
---
apiVersion: capsule.clastix.io/v1beta2
kind: TenantResource
metadata:
name: "{{ $team }}-tenant-distribution"
namespace: solar-system
ownerReferences:
- apiVersion: capsule.clastix.io/v1beta2
kind: TenantResource
name: {{ $.replications.metadata.name | quote }}
uid: {{ $.replications.metadata.uid | quote }}
controller: true
blockOwnerDeletion: true
spec:
resyncPeriod: 60s
serviceAccount:
name: gateway-api-role-distributor
resources:
- namespaceSelector:
matchLabels:
team: {{ $team }}
generators:
- template: |
---
apiVersion: capsule.clastix.io/v1beta2
kind: ResourcePoolClaim
metadata:
name: "managed-quota"
namespace: "{{`{{ $.namespace.metadata.name }}`}}"
spec:
pool: "solar"
claim:
{{- $team_config.quota | toYAML | nindent 16 }}
rawItems:
- apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: "tenant:{{`{{namespace}}`}}:cr:gateway-api"
namespace: "{{`{{namespace}}`}}"
subjects:
{{- range $team_config.owners }}
- kind: {{ .kind }}
name: {{ .name }}
apiGroup: rbac.authorization.k8s.io
{{- end }}
roleRef:
kind: ClusterRole
name: gateway-api-role-distributor
apiGroup: rbac.authorization.k8s.io
{{- end }} |
Uh oh!
There was an error while loading. Please reload this page.
Summary
Support a three-level permission hierarchy in Capsule:
Today Capsule has two effective tiers: cluster administrators and tenant owners. There is no way for a tenant owner to delegate a bounded portion of their tenant to a sub-team without involving a cluster admin.
Problem / Use case
Larger tenants (e.g. a business unit) need to hand a subset of their tenant to sub-teams (squads, projects, environments) so those sub-teams can self-serve without getting full tenant-owner power and without cluster-admin intervention.
There is currently no first-class concept for this nested, self-service delegation.
What I want
A Subtenant tier owned by a Tenant Owner, where:
Out of scope for this issue
Implementation/design details (API shape, controllers, webhooks) — this issue is only to capture the feature requirement. Happy to discuss the approach separately.
All reactions