Page MenuHomePhabricator

Review the upstream opensearch-operator chart to determine whether is is suitable for us to use
Closed, ResolvedPublic

Event Timeline

BTullis triaged this task as High priority.Jul 17 2025, 2:55 PM
BTullis updated the task description. (Show Details)

One of the first things that I noticed about this operator was the fact that they have added the capability to use Roles and Rolebindings in preference to ClusterRoles and ClusterRoleBindings.
This is potentially beneficial in terms of the overall cluster security, but it does impose some limitations due to the way that this operator and chart are written.

From here: https://github.com/opensearch-project/opensearch-k8s-operator/blob/main/charts/opensearch-operator/values.yaml#L119-L123

## If this is set to true, RoleBindings will be used instead of ClusterRoleBindings, inorder to restrict ClusterRoles
## to the namespace where the operator and OpenSearch cluster are in. In that case, specify the namespace where they
## are in in manager.watchNamespace field.
## If false, ClusterRoleBindings will be used
useRoleBindings: false

If we compare this to the way that flink-operator or the cloudnativepg operator work, both of those allow us to have a list of multiple watched namespaces, where the Roles and RoleBindings are created.
Those operators are installed into their own namespace, as part of the admin_ng deployment, and they manage the custom resources in a separate target namespace.

Someone has asked for this feature to be added to the opensearch-operator:
https://github.com/opensearch-project/opensearch-k8s-operator/issues/374
...but it is not yet available.

Generally, it seems that if people cannot use the ClusterRoles and ClusterRoleBindings, then they install one operator per cluster, in the same namespace as the cluster.
That might work for us, but it is less than ideal.

While not ideal that they do not support the watchNamespaces feature natively, we've also added it after the fact to the cloudnative-pg, ceph-csi-rbd and ceph-csi-cephfs operators. We can add it there as well.

While not ideal that they do not support the watchNamespaces feature natively, we've also added it after the fact to the cloudnative-pg, ceph-csi-rbd and ceph-csi-cephfs operators. We can add it there as well.

+1 this is the same comment that I added in the code reviews that Brian sent :)

BTullis updated Other Assignee, removed: bking.
bking changed the task status from Open to In Progress.Aug 19 2025, 8:24 PM

Long-overdue update:

After a lot of finagling, I've settled on chart version 2.7.0 as the latest chart (2.8.0) has a bug related to RBAC (the bug was closed, but I'm still hitting it).

I've made the changes requested by @elukey on the aforementioned code review and I can confirm that the operator starts cleanly and is able to provision OpenSearch clusters using our custom OpenSearch image.

I'm currently building the OpenSearch operator image and my next step will be that the our custom operator image works with the helm chart installation.

Some info about the new charts:

  • The overall code quality seems good.
  • The community/upstream is active and it seems responsive to github issues, even if I have to say that there are a lot of them for an helm chart. It may be due to the popularity and need for custom changes, so not a real concern.
  • The only big security concern was related to a very permissive ClusteRole needed to watch all namespaces with high privileges, that got replaced by a setup where every namespace in need of an opensearch cluster would need also the operator deployed alongside with it (so its permissions are restricted to the namespace where it gets deployed into). Not ideal but so far the operator doesn't support watching a list of namespaces, a feature that may be added by us in the future.

+1 from my side.