Page MenuHomePhabricator

Investigate Bitnami alternatives for the MariaDB chart
Closed, ResolvedPublic2 Estimated Story Points

Description

Investigation spike!

We have three different databases we maintain:

  • Catalyst
  • Patch Demo
  • Environments

Evaluation criteria:

  • Can we migrate safely from the bitnami chart to the new chart
  • Stability of the upstream (or a proxy thereof)
    • How long have they been around
    • How many users do they have
    • How often do they seem to make breaking changes without clear migration instructions
  • Ease of use
    • Options are easy to configure
    • Docs exist and are easy to use
  • Ease of ongoing maintenance

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Draft: POC of the shared envDB created using the mariadb-operatorrepos/test-platform/catalyst/catalyst-api!137jnucheT403763-operator-POCmain
Customize query in GitLab

Event Timeline

thcipriani set the point value for this task to 2.
thcipriani moved this task from Backlog to Ready on the Catalyst (tenpo o wawa) board.
  • Mariab offers a kubernetes CRD (https://github.com/mariadb-operator/mariadb-operator), which can be used to define a mariadb instance. It was created in 2022 and is an active project with 66 contributors and the latest release on Aug. 8. Since it’s supported by mariadb itself I think we can put our confidence in its continued maintenance and stability. Releases are well documented with regards to upgrading for breaking changes, which has happened a few times in the past year.

I was able to create a mariadb deployment for catalyst-api in Minikube by doing the following commands:

$ helm repo add mariadb-operator https://helm.mariadb.com/mariadb-operator
$ helm install mariadb-operator-crds mariadb-operator/mariadb-operator-crds 
$ helm install mariadb-operator mariadb-operator/mariadb-operator

And then applying the following helm templates:

---
apiVersion: v1
kind: Secret
metadata:
  name: {{.Release.Name}}-mariadb-secrets
type: Opaque
data:
  rootPassword: {{ .Values.app.db.rootPassword | b64enc }}
  password: {{ .Values.app.db.password | b64enc}}
---
apiVersion: k8s.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: {{ .Release.Name }}-mariadb
spec:
  rootPasswordSecretKeyRef:
    name: {{.Release.Name}}-mariadb-secrets
    key: rootPassword

  port: 3306

  storage:
    size: 1Gi
    # storageClassName: csi-hostpath-sc

  myCnf: |
    [mariadb]
    bind-address=*
    default_storage_engine=InnoDB
    binlog_format=row
    innodb_autoinc_lock_mode=2
    innodb_buffer_pool_size=1024M
    max_allowed_packet=256M

  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      memory: 1Gi
---
apiVersion: k8s.mariadb.com/v1alpha1
kind: Database
metadata:
  name: {{ .Values.app.db.database }}
spec:
  mariaDbRef:
    name: {{ .Release.Name }}-mariadb
---
apiVersion: k8s.mariadb.com/v1alpha1
kind: User
metadata:
  name: {{ .Values.app.db.username }}
spec:
  mariaDbRef:
    name: {{ .Release.Name }}-mariadb
  passwordSecretKeyRef:
    name: {{.Release.Name}}-mariadb-secrets
    key:  password
  maxUserConnections: 0
---
apiVersion: k8s.mariadb.com/v1alpha1
kind: Grant
metadata:
  name: catalyst-api
spec:
  mariaDbRef:
    name: {{ .Release.Name }}-mariadb
  privileges:
    - ALL PRIVILEGES
  database: {{ .Values.app.db.database }}
  username: {{ .Values.app.db.username }}

For catalyst-api and patch demo, it looks like we’ll need to restore from a backup of the existing db in order to migrate.

TODO: investigate https://github.com/CloudPirates-io/helm-charts/tree/main/charts/mariadb

Open question: is this, effectively, a drop-in for the bitnami chart?

CloudPirates mariadb helm chart:
According to https://www.cloudpirates.io/en/about they have been around since 2021. The helm chart was created in August and updated 3 days ago. The CloudPirates helm charts project has 18 contrubutors.

At first I thought this was a port of the bitnami helm chart, but it is missing some values that are available in the bitnami chart, notably, the ability to use an existing pvc.


Documentation consists of the README in the repository.

I was unable to directly upgrade to the cloupirates chart due to: Error: UPGRADE FAILED: cannot patch "patchdemo-mariadb" with kind StatefulSet: StatefulSet.apps "patchdemo-mariadb" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden.

The initdb scripts must be moved to a configmap. I did this but it doesn’t seem to be working as expected, since the tables didn’t get created, however, I was able to shell into the container and run the script (created by the chart from the configmap) to create the tables:

mysql@patchdemo-mariadb-0:/docker-entrypoint-initdb.d$ mariadb -uroot -predacted < 01-init.sql 
Table	Op	Msg_type	Msg_text
patchdemo.tasks	optimize	note	Table does not support optimize, doing recreate + analyze instead
patchdemo.tasks	optimize	status	OK
Table	Op	Msg_type	Msg_text
patchdemo.wikis	optimize	note	Table does not support optimize, doing recreate + analyze instead
patchdemo.wikis	optimize	status	OK
Table	Op	Msg_type	Msg_text
patchdemo.patches	optimize	note	Table does not support optimize, doing recreate + analyze instead
patchdemo.patches	optimize	status	OK
mysql@patchdemo-mariadb-0:/docker-entrypoint-initdb.d$ mariadb -u patchdemo -predacted
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 205
Server version: 12.0.2-MariaDB-ubu2404 mariadb.org binary distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> use patchdemo;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [patchdemo]> show tables;
+---------------------+
| Tables_in_patchdemo |
+---------------------+
| patches             |
| tasks               |
| wikis               |
+---------------------+
3 rows in set (0.001 sec)

MariaDB [patchdemo]> ^DBye

It seems we’ll need to experiment more with this chart if we want to use it.

thcipriani moved this task from In progress to Ready on the Catalyst (noka) board.
thcipriani added a subscriber: jeena.
thcipriani edited projects, added Catalyst; removed Catalyst (Ken).
jnuche moved this task from Backlog to Ken on the Catalyst board.
jnuche edited projects, added Catalyst (Ken); removed Catalyst.
jnuche moved this task from Ready to In progress on the Catalyst (Ken) board.
jnuche closed this task as Resolved.EditedOct 15 2025, 4:02 PM
jnuche moved this task from In progress to Done on the Catalyst (Ken) board.

I looked at:

And also did some extra searching for extra alternatives.

TL;DR: I propose we use the MariaDB operator

I couldn't really find any options more attractive than the two listed above.

When comparing CloudPirates vs Operator: Originally my hope was that the CloudPirates could be a drop-in replacement for the bitnami chart, but unfortunately that's not the case. When it comes to ease-of-use I didn't see big differences between both options, the operator comes with its own set of charts that make configuring the deployment very similar to what we already have. Full comparison:

  • None is a drop-in for bitnami
  • Similar complexity regarding configuration for our use case, the operator also uses charts
  • The operator has more and better (matter of opinion I guess) documentation
  • The operator seems to have a much more detailed and flexible configuration, allowing for greater control in the future
  • The operator has much more attention from the community from what I could see: posts in the official MariaDB foundation blog, stars in GitHub, references in Reddit's Kubernetes sub, and more

In order to use the operator to create DBs we would first need to install in the cluster its CRDs and the operator itself:

helm repo add mariadb-operator https://helm.mariadb.com/mariadb-operator
helm -n default install mariadb-operator-crds mariadb-operator/mariadb-operator-crds
helm -n default install mariadb-operator mariadb-operator/mariadb-operator

The installation of those charts is a good candidate to be placed in our Tofu stack here.

A MariaDB cluster can then be created using the cluster chart.


PoC

To illustrate what would be needed for the migration, I put together a PoC for our most complex use case: The shared DB for environments deployed by the API.

The chart in this draft deploys a single instance MariaDB with almost identical configuration to the bitnami one we're currently using: https://gitlab.wikimedia.org/repos/test-platform/catalyst/catalyst-api/-/merge_requests/137. The only exception I could identify is it's not possible to define DB users with underscores, so the user used by API will need to be renamed from catalyst_api to catalyst-api.

Data migration

The operator allows to prime a DB from an existing source, such as an existing PVC, as in this example. But from what I've seen it's probably simpler to just create a dump from the bitnami DB and restore it into the operator one.

Configuration comparison

To compare the relative complexity, this is the values.yaml for the new DB with the operator:

envdb:
  rootPassword: secretpassword
  # Available keys defined at: https://github.com/mariadb-operator/mariadb-operator/blob/main/docs/api_reference.md#mariadbspec
  mariadb:
    rootPasswordSecretKeyRef:
      name: envdb
      key: root-password
    galera:
      # At time of writing the flag has no default, so we explicitly set it out of caution
      # https://github.com/mariadb-operator/mariadb-operator/blob/main/docs/api_reference.md#galera
      enabled: false
    replicas: 1
    resources:
      requests:
        cpu: 4
        memory: 4Gi
      limits:
        cpu: 8
        memory: 12Gi
    storage:
      size: 12Gi
    tolerations:
      - key: "dedicated"
        operator: "Equal"
        value: "db"
        effect: "NoSchedule"
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: dedicated
                  operator: In
                  values:
                    - db

  usernamePassword: catalyst
  users:
    - name: catalyst-api
      passwordSecretKeyRef:
        name: envdb
        key: username-password
      host: "%"
      maxUserConnections: 0
      cleanupPolicy: Delete
      requeueInterval: 1h
      retryInterval: 10s

  grants:
    - name: catalyst-api
      username: catalyst-api
      privileges:
      - ALL PRIVILEGES
      database: "*"
      table: "*"
      host: "%"
      grantOption: true
      cleanupPolicy: Delete
      requeueInterval: 1h
      retryInterval: 10s

and this the same file for the old bitnami DB:

envdb:
  # T403766 using bitnami legacy images requires this directive
  global:
    security:
      allowInsecureImages: true
  image:
    repository: bitnamilegacy/mariadb
  volumePermissions:
    image:
      repository: bitnamilegacy/os-shell
  metrics:
    image:
      repository: bitnamilegacy/mysqld-exporter
  auth:
    username: catalyst_api
    password: catalyst
    rootPassword: secretpassword
  primary:
    # The CPU and memory are oversized. We should reduce them once we have a better idea of what the
    # reqs in production are
    resources:
      requests:
        cpu: 4
        memory: 4Gi
      limits:
        cpu: 8
        memory: 12Gi
    persistence:
      size: 12Gi
    tolerations:
      - key: dedicated
        operator: Equal
        value: db
        effect: NoSchedule
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: dedicated
                  operator: In
                  values:
                    - db
  initdbScriptsConfigMap: env-db-setup

In the case of bitnami the grants are configured in a separate file.