Skip to content

Narrow wildcard RBAC verbs to least-privilege on manager ClusterRole - #2027

Open
abays wants to merge 1 commit into
openstack-k8s-operators:mainfrom
abays:narrow-wildcard-rbac-verbs
Open

Narrow wildcard RBAC verbs to least-privilege on manager ClusterRole#2027
abays wants to merge 1 commit into
openstack-k8s-operators:mainfrom
abays:narrow-wildcard-rbac-verbs

Conversation

@abays

@abays abays commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Replace verbs: ["*"] with explicit minimal verb sets on the openstack-operator controller's kubebuilder RBAC markers and regenerated manifests:

  • admissionregistration.k8s.io webhooks: get/list/watch/create/update/delete/deletecollection
  • apiextensions.k8s.io: narrow resources from "*" to customresourcedefinitions
  • core serviceaccounts: get/list/delete (only used for cleanup)
  • core namespaces: get/create/update
  • core services: get/list/create/update/delete
  • Remove configmaps from init operator controller markers (unused; other controllers declare their own)

@abays
abays requested review from dprince and stuggi August 11, 2026 10:25
@openshift-ci
openshift-ci Bot requested review from rebtoor and slagle August 11, 2026 10:26
@openshift-ci

openshift-ci Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abays

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown

OpenStackControlPlane CRD Size Report

Metric Value
CRD JSON size 337535 bytes (330KB)
Base branch size 337535 bytes
Change +0.00%
Status yellow — growing
Threshold reference
Color Range Meaning
🟢 green < 300KB Comfortable
🟡 yellow 300–400KB Growing
🟠 orange 400–750KB Concerning
🔴 red > 750KB Approaching 1.5MB etcd limit (cut in half to allow space for update)

@stuggi

stuggi commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openstack-k8s-operators_openstack-operator/2027/pull-ci-openstack-k8s-operators-openstack-operator-main-openstack-operator-build-deploy-kuttl-4-20/2087123376488321024/artifacts/openstack-operator-build-deploy-kuttl-4-20/openstack-k8s-operators-gather/artifacts/must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-e951c596b1c06e42cc41260bfe9b929fa850a06bae1d25cda4a1289fdbebaf5b/namespaces/openstack-operators/pods/openstack-operator-controller-init-fc6c4fbf4-drmsh/logs/operator.log

go@v0.33.13/tools/cache/reflector.go:285", "type": "*v1.ServiceAccount", "error": "serviceaccounts is forbidden: User \"system:serviceaccount:openstack-operators:openstack-operator-controller-init\" cannot watch resource \"serviceaccounts\" in API group \"\" at the cluster scope"}
k8s.io/apimachinery/pkg/util/runtime.logError
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/runtime/runtime.go:226
k8s.io/apimachinery/pkg/util/runtime.handleError
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/runtime/runtime.go:217
k8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/runtime/runtime.go:203
k8s.io/client-go/tools/cache.DefaultWatchErrorHandler
	/opt/app-root/src/go/pkg/mod/k8s.io/client-go@v0.33.13/tools/cache/reflector.go:200
k8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1
	/opt/app-root/src/go/pkg/mod/k8s.io/client-go@v0.33.13/tools/cache/reflector.go:360
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/wait/backoff.go:233
k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/wait/backoff.go:255
k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/wait/backoff.go:256
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/wait/backoff.go:233
k8s.io/client-go/tools/cache.(*Reflector).RunWithContext
	/opt/app-root/src/go/pkg/mod/k8s.io/client-go@v0.33.13/tools/cache/reflector.go:358
k8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/wait/wait.go:63
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/wait/wait.go:72

config/operator/rbac/role.yaml narrows serviceaccounts and services verbs to get,list,delete / get,list,create,update,delete — dropping watch. But deleteAllOwnedResources in openstack_controller.go:408,413 does typed List() calls (corev1.ServiceAccountList, corev1.ServiceList), which go through the controller-runtime cache and need watch for the informer's reflector. Without it, the manager will hit continuous Forbidden errors on Watch() and relist/backoff loops once that code path runs (triggered by OPENSTACK_RELEASE_VERSION changes). Not a hard break (List still works via periodic relist), but will spam errors for the life of the process, like above.

I think we either add the watch back, or switching those two List() calls to unstructured.

@abays

abays commented Aug 13, 2026

Copy link
Copy Markdown
Contributor Author

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openstack-k8s-operators_openstack-operator/2027/pull-ci-openstack-k8s-operators-openstack-operator-main-openstack-operator-build-deploy-kuttl-4-20/2087123376488321024/artifacts/openstack-operator-build-deploy-kuttl-4-20/openstack-k8s-operators-gather/artifacts/must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-e951c596b1c06e42cc41260bfe9b929fa850a06bae1d25cda4a1289fdbebaf5b/namespaces/openstack-operators/pods/openstack-operator-controller-init-fc6c4fbf4-drmsh/logs/operator.log

go@v0.33.13/tools/cache/reflector.go:285", "type": "*v1.ServiceAccount", "error": "serviceaccounts is forbidden: User \"system:serviceaccount:openstack-operators:openstack-operator-controller-init\" cannot watch resource \"serviceaccounts\" in API group \"\" at the cluster scope"}
k8s.io/apimachinery/pkg/util/runtime.logError
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/runtime/runtime.go:226
k8s.io/apimachinery/pkg/util/runtime.handleError
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/runtime/runtime.go:217
k8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/runtime/runtime.go:203
k8s.io/client-go/tools/cache.DefaultWatchErrorHandler
	/opt/app-root/src/go/pkg/mod/k8s.io/client-go@v0.33.13/tools/cache/reflector.go:200
k8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1
	/opt/app-root/src/go/pkg/mod/k8s.io/client-go@v0.33.13/tools/cache/reflector.go:360
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/wait/backoff.go:233
k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/wait/backoff.go:255
k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/wait/backoff.go:256
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/wait/backoff.go:233
k8s.io/client-go/tools/cache.(*Reflector).RunWithContext
	/opt/app-root/src/go/pkg/mod/k8s.io/client-go@v0.33.13/tools/cache/reflector.go:358
k8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/wait/wait.go:63
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/wait/wait.go:72

config/operator/rbac/role.yaml narrows serviceaccounts and services verbs to get,list,delete / get,list,create,update,delete — dropping watch. But deleteAllOwnedResources in openstack_controller.go:408,413 does typed List() calls (corev1.ServiceAccountList, corev1.ServiceList), which go through the controller-runtime cache and need watch for the informer's reflector. Without it, the manager will hit continuous Forbidden errors on Watch() and relist/backoff loops once that code path runs (triggered by OPENSTACK_RELEASE_VERSION changes). Not a hard break (List still works via periodic relist), but will spam errors for the life of the process, like above.

I think we either add the watch back, or switching those two List() calls to unstructured.

Looks like we might need watch for Services as well, which I think you implied:

controller-runtime's cache-backed client creates an informer on-demand for any typed List() call, even for unregistered types. When deleteOwnedResources calls r.List(ctx, &corev1.ServiceList{}, ...) on line 414, the cache spins up a Service informer at that moment — same as  what happened with ServiceAccount. The services watch error just hasn't appeared yet because the code hits ServiceAccount first (line 409) and the reflector backoff keeps retrying that; once ServiceAccount resolves, the Service informer would hit the same wall on line 414.

@abays
abays force-pushed the narrow-wildcard-rbac-verbs branch 2 times, most recently from 247ac04 to 88c5cf8 Compare August 13, 2026 11:33
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/313aba5016294d4d9ef40afefe45231a

✔️ openstack-k8s-operators-content-provider SUCCESS in 4h 53m 17s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 37m 04s
cifmw-crc-podified-edpm-baremetal MERGE_CONFLICT in 2m 18s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 56m 46s
✔️ openstack-operator-edpm-baremetal-minor-update SUCCESS in 2h 37m 05s

Replace verbs: ["*"] with explicit minimal verb sets on the
openstack-operator controller's kubebuilder RBAC markers and
regenerated manifests:

- admissionregistration.k8s.io webhooks: get/list/watch/create/update/delete/deletecollection
- apiextensions.k8s.io: narrow resources from "*" to customresourcedefinitions
- core serviceaccounts: get/list/delete (only used for cleanup)
- core namespaces: get/create/update
- core services: get/list/create/update/delete
- Remove configmaps from this controller's markers (unused; other controllers declare their own)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@abays
abays force-pushed the narrow-wildcard-rbac-verbs branch from 88c5cf8 to 3532763 Compare August 13, 2026 19:28
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/a52c1d65a35e4b2a93c05592d2033fb8

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 55m 37s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 34m 28s
cifmw-crc-podified-edpm-baremetal FAILURE in 1h 17m 23s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 51m 28s
✔️ openstack-operator-edpm-baremetal-minor-update SUCCESS in 2h 27m 10s

@stuggi

stuggi commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

recheck

@stuggi

stuggi commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

/retest

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/b35774e09a994cfbaa8ed3c7bfb03c69

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 25m 10s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 30m 53s
cifmw-crc-podified-edpm-baremetal FAILURE in 35m 33s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 45m 19s
✔️ openstack-operator-edpm-baremetal-minor-update SUCCESS in 2h 10m 39s

@openshift-ci

openshift-ci Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

@abays: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/openstack-operator-build-deploy-kuttl-4-20 3532763 link true /test openstack-operator-build-deploy-kuttl-4-20

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@abays

abays commented Aug 14, 2026

Copy link
Copy Markdown
Contributor Author

Build failed (check pipeline). Post recheck (without leading slash) to rerun all jobs. Make sure the failure cause has been resolved before you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/b35774e09a994cfbaa8ed3c7bfb03c69

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 25m 10s ✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 30m 53s ❌ cifmw-crc-podified-edpm-baremetal FAILURE in 35m 33s ✔️ openstack-operator-tempest-multinode SUCCESS in 1h 45m 19s ✔️ openstack-operator-edpm-baremetal-minor-update SUCCESS in 2h 10m 39s

Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_011_fetch_openshift_token.log
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: NoneType: None
task path: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/openshift_login/tasks/try_login.yml:29
fatal: [localhost]: FAILED! => 
    changed: true
    msg: non-zero return code
    rc: 1
    stderr: ''
    stderr_lines: []
    stdout: |
      Unable to connect to the server: EOF
    stdout_lines:
    - 'Unable to connect to the server: EOF'

@abays

abays commented Aug 14, 2026

Copy link
Copy Markdown
Contributor Author

recheck

@abays

abays commented Aug 14, 2026

Copy link
Copy Markdown
Contributor Author

@abays: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:
Test name Commit Details Required Rerun command
ci/prow/openstack-operator-build-deploy-kuttl-4-20 3532763 link true /test openstack-operator-build-deploy-kuttl-4-20

Full PR test history. Your PR dashboard.
Details

No resources found in openstack-operators namespace.
No resources found in openstack-operators namespace.
No resources found in openstack-operators namespace.
No resources found in openstack-operators namespace.
make: *** [Makefile:899: openstack_wait] Error 124

/test openstack-operator-build-deploy-kuttl-4-20

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants