see: [ALERT] (wbaas-3): critical SQL primary PV utilization on wikibase-cloud Kubernetes Pod labels {project_id=wikibase-cloud, cluster_name=wbaas-3, pod_name=sql-mariadb-primary-0}
Description
Description
Related Objects
Related Objects
Event Timeline
Comment Actions
Going to remove the statefulset
Merge https://github.com/wmde/wbaas-deploy/pull/638
Deploy that
Then adjust the PVC
Finally restart the pods.
Copying-ish the plan run by Adam back in 2020 (see: https://github.com/wbstack/deploy/blob/main/docs/services/sql.md#increasing-allocated-disk-space-replica-example and also the serverfault docs at: https://serverfault.com/questions/955293/how-to-increase-disk-size-in-a-stateful-set)
Comment Actions
Yesterday we happily went through this procedure on staging; interestingly we noted that restarting the pod wasn't necessary; seems that this automatically resized while the workloads were still running!
Comment Actions
We followed these same steps; again without restarting the pods and we now happily see out volume utilisation well below 0.85.
What was slightly unpleasant was that we discovered that the sites seemed unavailable for around 5 minutes after the SQL secondary restarted. The logs looked like this:
2023-01-04 12:35:52 0 [Note] Added new Master_info '' to hash table [25/1963]
2023-01-04 12:35:52 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: ready for connections.
Version: '10.5.15-MariaDB-log' socket: '/opt/bitnami/mariadb/tmp/mysql.sock' port: 3306 Source distribution
2023-01-04 12:36:09 0 [Note] InnoDB: Buffer pool(s) load completed at 230104 12:36:09
2023-01-04 12:41:50 74 [Warning] 'proxies_priv' entry '@% root@sql-mariadb-secondary-0' ignored in --skip-name-resolve mode.
find: '/docker-entrypoint-startdb.d/': No such file or directory
mariadb 12:41:50.11 INFO ==> Stopping mariadb
2023-01-04 12:41:50 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (initiated by: unknown): Normal shutdown
2023-01-04 12:41:50 0 [Note] Event Scheduler: Purging the queue. 0 events
2023-01-04 12:41:50 0 [Note] InnoDB: FTS optimize thread exiting.
2023-01-04 12:41:50 0 [Note] InnoDB: Starting shutdown...
2023-01-04 12:41:50 0 [Note] InnoDB: Dumping buffer pool(s) to /bitnami/mariadb/data/ib_buffer_pool
2023-01-04 12:41:50 0 [Note] InnoDB: Restricted to 44357 pages due to innodb_buf_pool_dump_pct=25
2023-01-04 12:41:50 0 [Note] InnoDB: Buffer pool(s) dump completed at 230104 12:41:50
2023-01-04 12:41:50 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2023-01-04 12:41:50 0 [Note] InnoDB: Shutdown completed; log sequence number 86580569765; transaction id 63619717
2023-01-04 12:41:50 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: Shutdown complete
mariadb 12:41:51.33 INFO ==> ** MariaDB setup finished! **
mariadb 12:41:51.42 INFO ==> ** Starting MariaDB **
2023-01-04 12:41:51 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (mysqld 10.5.15-MariaDB-log) starting as process 1 ...
2023-01-04 12:41:51 0 [Note] InnoDB: Uses event mutexes
2023-01-04 12:41:51 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2023-01-04 12:41:51 0 [Note] InnoDB: Number of pools: 1
2023-01-04 12:41:51 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions