We haven't tested these applications with MariaDB 10.6 yet.
Some of them (Hive, Druid, DataHub) will be possible to test lightly by using the test cluster and an-test-coord001, so it is probably best to upgrade this host first.

Once all DBs are upgraded, edit https://docs.google.com/spreadsheets/d/1Obj5ozGQYl7Zei0MBLELVD8eDGqqsF_t9T3ZbrOsmZg/edit#gid=1305106294

Details

Related Changes in Gerrit:

Subject	Author	Repo	Branch	Lines +/-
Revert the change to disable the gobbin timers on an-launcher	Btullis	operations/puppet	production	+1 -1
Temporarily disable gobblin timers to permit hive maintenance	Btullis	operations/puppet	production	+1 -1
Facilitate a role swap from an-mariadb1001 to an-mariadb1002	Btullis	operations/deployment-charts	master	+6 -6
Configure analytics_meta MariaDB clients to connect to an-mariadb1002	Btullis	operations/puppet	production	+7 -4
Temporarily pause gobblin to facilitate Hive/MariaDB maintenance	Btullis	operations/puppet	production	+1 -1
Configure reuse-parts for an-mariadb servers	Btullis	operations/puppet	production	+2 -2
analytics_test_cluster_coordinator: upgrade mariadb to version 10.6	Brouberol	operations/puppet	production	+2 -0

Customize query in gerrit

Related Objects

Mentioned In: T394498: SSD firmware update for an-mariadb100[1-2]
T369737: Site Issue: Delayed data in the `webrequest_sampled_live` kafka topic
T351731: Turnilo: invalid transforms on wmf_netflow dashboard
Mentioned Here: T365996: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f1-eqiad
T348977: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2

Event Timeline

BTullis created this task.May 21 2024, 4:22 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 21 2024, 4:22 PM

Gehel triaged this task as High priority.May 24 2024, 8:43 AM

Gehel moved this task from Incoming to MariaDB (Analytics / WMCS / Replicas / ...) on the Data-Platform-SRE board.

BTullis edited projects, added: Data-Platform-SRE (2024.05.27 - 2024.06.16); removed: Data-Platform-SRE.May 24 2024, 3:17 PM

brouberol updated the task description. (Show Details)Jun 4 2024, 12:47 PM

It seems that we need to install a new deb package

brouberol@an-test-coord1001:~$ dpkg -l | grep wmf-mariadb
ii  wmf-mariadb104                        10.4.28+deb11u1                amd64        MariaDB plus patches.
brouberol@an-test-coord1001:~$ sudo apt-cache search wmf-mariadb106
wmf-mariadb106 - MariaDB plus patches.
wmf-mariadb106-client - MariaDB 10.4 client only with Wikimedia-specific patches.

@brouberol first you need to stop mariadb, then remove wmf-mariadb104 and then install wmf-mariadb106 via puppet (or however it is handling from your end).
Once done, start mariadb again and don't forget to run: mysql_upgrade

Thanks, that's perfectly clear!

Change #1038771 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] analytics_test_cluster_coordinator: upgrade mariadb to version 10.6

https://gerrit.wikimedia.org/r/1038771

gerritbot added a project: Patch-For-Review.Jun 4 2024, 1:00 PM

brouberol claimed this task.Jun 4 2024, 1:10 PM

brouberol moved this task from Backlog to In Progress on the Data-Platform-SRE (2024.05.27 - 2024.06.16) board.

brouberol@an-test-coord1001:~$ sudo puppet agent --disable 'brouberol - T365503 - MariaDB upgrade'

brouberol@an-test-coord1001:~$ sudo systemctl stop mariadb.service
subrouberol@an-test-coord1001:~$ sudo apt-get remove wmf-mariadb104
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libcrack2 libjemalloc2 linux-image-5.10.0-28-amd64 puppet python3-debconf
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  wmf-mariadb104
0 upgraded, 0 newly installed, 1 to remove and 9 not upgraded.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
su: warning: cannot change directory to /nonexistent: No such file or directory
INFO:debmonitor:Got 1 updates from dpkg hook version 3
INFO:debmonitor:Successfully sent the dpkg_hook update to the DebMonitor server
(Reading database ... 200305 files and directories currently installed.)
Removing wmf-mariadb104 (10.4.28+deb11u1) ...
update-alternatives: warning: alternative /opt/wmf-mariadb104/bin/mysql (part of link group mariadb) doesn't exist; removing from list of alternatives
update-alternatives: warning: /etc/alternatives/mariadb is dangling; it will be updated with best choice
update-alternatives: warning: alternative /opt/wmf-mariadb104/bin/mysqldump (part of link group mariadbdump) doesn't exist; removing from list of alternatives
update-alternatives: warning: /etc/alternatives/mariadbdump is dangling; it will be updated with best choice
update-alternatives: warning: alternative /opt/wmf-mariadb104/bin/mysql (part of link group mysql) doesn't exist; removing from list of alternatives
update-alternatives: warning: /etc/alternatives/mysql is dangling; it will be updated with best choice
update-alternatives: warning: alternative /opt/wmf-mariadb104/bin/mysqldump (part of link group mysqldump) doesn't exist; removing from list of alternatives
update-alternatives: warning: /etc/alternatives/mysqldump is dangling; it will be updated with best choice
update-alternatives: warning: alternative /opt/wmf-mariadb104/bin/mysqlbinlog (part of link group mysqlbinlog) doesn't exist; removing from list of alternatives
update-alternatives: warning: /etc/alternatives/mysqlbinlog is dangling; it will be updated with best choice
update-alternatives: warning: alternative /opt/wmf-mariadb104/bin/mysql_upgrade (part of link group mysql_upgrade) doesn't exist; removing from list of alternatives
update-alternatives: warning: /etc/alternatives/mysql_upgrade is dangling; it will be updated with best choice
update-alternatives: warning: alternative /opt/wmf-mariadb104/bin/mysqlcheck (part of link group mysqlcheck) doesn't exist; removing from list of alternatives
update-alternatives: warning: /etc/alternatives/mysqlcheck is dangling; it will be updated with best choice
update-alternatives: warning: alternative /opt/wmf-mariadb104/bin/mariabackup (part of link group xtrabackup) doesn't exist; removing from list of alternatives
update-alternatives: warning: /etc/alternatives/xtrabackup is dangling; it will be updated with best choice
update-alternatives: warning: alternative /opt/wmf-mariadb104/bin/mbstream (part of link group mbstream) doesn't exist; removing from list of alternatives
update-alternatives: warning: /etc/alternatives/mbstream is dangling; it will be updated with best choice
brouberol@an-test-coord1001:~$

Change #1038771 merged by Brouberol:

[operations/puppet@production] analytics_test_cluster_coordinator: upgrade mariadb to version 10.6

https://gerrit.wikimedia.org/r/1038771

brouberol@an-test-coord1001:~$ sudo puppet agent --enable
brouberol@an-test-coord1001:~$ sudo run-puppet-agent
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for an-test-coord1001.eqiad.wmnet
Info: Unable to serialize catalog to json, retrying with pson
Info: Applying configuration version '(2f10486ba4) Brouberol - analytics_test_cluster_coordinator: upgrade mariadb to version 10.6'
Notice: /Stage[main]/Geoip::Data::Puppet/File[/usr/share/GeoIP/GeoIP2-City.mmdb]/content: content changed '{sha256}c11297fa5c4ecbdd956e4585dead37486d7dc465e20eadf1417dcc5beb4ae1f8' to '{sha256}4abc3bd15f72be47e9190fa3135e6c960856146119b49a937b97ea90db085134'
Notice: /Stage[main]/Geoip::Data::Puppet/File[/usr/share/GeoIP/GeoIP2-Connection-Type.mmdb]/content: content changed '{sha256}1fcddb42fe0026f62d54996c935c204c5f828d9d4c5bf6b4c46e6912334a7fbc' to '{sha256}12d8b9ffd5041c386776c6d944b0107a32e6cf17d1acccc00fe5a9340b4eb247'
Notice: /Stage[main]/Geoip::Data::Puppet/File[/usr/share/GeoIP/GeoIP2-Country.mmdb]/content: content changed '{sha256}4ef62c259e0806343cc2872cfc4185870faebcb3be4315018c3afa084647efb8' to '{sha256}870a5317a49de350beee7a9017be45bd27108ae964a2a8e5c92b71d172f26db9'
Notice: /Stage[main]/Geoip::Data::Puppet/File[/usr/share/GeoIP/GeoIP2-ISP.mmdb]/content: content changed '{sha256}1c0b5c1f5b5599f1f2211c626e1f44c77506ba424d0713621c371689687374b3' to '{sha256}2cc4c4dac86d0aa369a30c9e9c60f77596ef7f4a600a39d1b16842d0b0cef663'
Notice: /Stage[main]/Mariadb::Config/File[/etc/my.cnf]/content:
--- /etc/my.cnf	2023-11-23 15:20:31.634577599 +0000
+++ /tmp/puppet-file20240605-3737358-1p8wqrx	2024-06-05 08:27:48.317025381 +0000
@@ -22,7 +22,7 @@
 socket    = /run/mysqld/mysqld.sock
 port      = 3306
 datadir   = /srv/mysql
-basedir   = /opt/wmf-mariadb104
+basedir   = /opt/wmf-mariadb106
 tmpdir    = /srv/tmp
 server_id = 171980073
 read_only = 0

Notice: /Stage[main]/Mariadb::Config/File[/etc/my.cnf]/content: content changed '{sha256}c89dd3ac7ebb2a489072c404f5567b6be4237bcafd9baafbbfb33ced9587ae4b' to '{sha256}fa61e3999fdfb39d45e6cae74d3ce6a811e9eeb07b3ec15feb3cfec39662da8e'
Notice: /Stage[main]/Mariadb::Packages_wmf/Package[wmf-mariadb106]/ensure: created
Notice: /Stage[main]/Mariadb::Packages_wmf/File[/usr/local/bin/mbstream]/target: target changed '/etc/alternatives/mbstream' to '/opt/wmf-mariadb106/bin/mbstream' (corrective)
Notice: /Stage[main]/Mariadb::Service/Service[mariadb]/ensure: ensure changed 'stopped' to 'running' (corrective)
Info: /Stage[main]/Mariadb::Service/Service[mariadb]: Unscheduling refresh on Service[mariadb]
Notice: Applied catalog in 70.47 seconds

brouberol@an-test-coord1001:~$ sudo systemctl status mariadb
● mariadb.service - mariadb database server
     Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2024-06-05 08:28:13 UTC; 48s ago
   Main PID: 3737977 (mysqld)
     Status: "Taking your SQL requests now..."
      Tasks: 25 (limit: 154177)
     Memory: 1.1G
        CPU: 2.106s
     CGroup: /system.slice/mariadb.service
             └─3737977 /opt/wmf-mariadb106/bin/mysqld

Jun 05 08:28:12 an-test-coord1001 systemd[1]: Starting mariadb database server...
Jun 05 08:28:13 an-test-coord1001 systemd[1]: Started mariadb database server.
brouberol@an-test-coord1001:~$ sudo mysql_upgrade
Phase 1/8: Checking and upgrading mysql database
Processing databases
mysql
mysql.column_stats                                 OK
mysql.columns_priv                                 OK
mysql.db                                           OK
mysql.event                                        OK
mysql.func                                         OK
mysql.global_priv                                  OK
mysql.gtid_slave_pos                               OK
mysql.help_category                                OK
mysql.help_keyword                                 OK
mysql.help_relation                                OK
mysql.help_topic                                   OK
mysql.index_stats                                  OK
mysql.innodb_index_stats                           OK
mysql.innodb_table_stats                           OK
mysql.plugin                                       OK
mysql.proc                                         OK
mysql.procs_priv                                   OK
mysql.proxies_priv                                 OK
mysql.roles_mapping                                OK
mysql.servers                                      OK
mysql.table_stats                                  OK
mysql.tables_priv                                  OK
mysql.time_zone                                    OK
mysql.time_zone_leap_second                        OK
mysql.time_zone_name                               OK
mysql.time_zone_transition                         OK
mysql.time_zone_transition_type                    OK
mysql.transaction_registry                         OK
Phase 2/8: Installing used storage engines... Skipped
Phase 3/8: Running 'mysql_fix_privilege_tables'
Phase 4/8: Fixing views
mysql.user                                         OK
sys.host_summary                                   OK
sys.host_summary_by_file_io                        OK
sys.host_summary_by_file_io_type                   OK
sys.host_summary_by_stages                         OK
sys.host_summary_by_statement_latency              OK
sys.host_summary_by_statement_type                 OK
sys.innodb_buffer_stats_by_schema                  OK
sys.innodb_buffer_stats_by_table                   OK
sys.innodb_lock_waits                              OK
sys.io_by_thread_by_latency                        OK
sys.io_global_by_file_by_bytes                     OK
sys.io_global_by_file_by_latency                   OK
sys.io_global_by_wait_by_bytes                     OK
sys.io_global_by_wait_by_latency                   OK
sys.latest_file_io                                 OK
sys.memory_by_host_by_current_bytes                OK
sys.memory_by_thread_by_current_bytes              OK
sys.memory_by_user_by_current_bytes                OK
sys.memory_global_by_current_bytes                 OK
sys.memory_global_total                            OK
sys.metrics                                        OK
sys.processlist                                    OK
sys.ps_check_lost_instrumentation                  OK
sys.schema_auto_increment_columns                  OK
sys.schema_index_statistics                        OK
sys.schema_object_overview                         OK
sys.schema_redundant_indexes                       OK
sys.schema_table_lock_waits                        OK
sys.schema_table_statistics                        OK
sys.schema_table_statistics_with_buffer            OK
sys.schema_tables_with_full_table_scans            OK
sys.schema_unused_indexes                          OK
sys.session                                        OK
sys.session_ssl_status                             OK
sys.statement_analysis                             OK
sys.statements_with_errors_or_warnings             OK
sys.statements_with_full_table_scans               OK
sys.statements_with_runtimes_in_95th_percentile    OK
sys.statements_with_sorting                        OK
sys.statements_with_temp_tables                    OK
sys.user_summary                                   OK
sys.user_summary_by_file_io                        OK
sys.user_summary_by_file_io_type                   OK
sys.user_summary_by_stages                         OK
sys.user_summary_by_statement_latency              OK
sys.user_summary_by_statement_type                 OK
sys.version                                        OK
sys.wait_classes_global_by_avg_latency             OK
sys.wait_classes_global_by_latency                 OK
sys.waits_by_host_by_latency                       OK
sys.waits_by_user_by_latency                       OK
sys.waits_global_by_latency                        OK
sys.x$host_summary                                 OK
sys.x$host_summary_by_file_io                      OK
sys.x$host_summary_by_file_io_type                 OK
sys.x$host_summary_by_stages                       OK
sys.x$host_summary_by_statement_latency            OK
sys.x$host_summary_by_statement_type               OK
sys.x$innodb_buffer_stats_by_schema                OK
sys.x$innodb_buffer_stats_by_table                 OK
sys.x$innodb_lock_waits                            OK
sys.x$io_by_thread_by_latency                      OK
sys.x$io_global_by_file_by_bytes                   OK
sys.x$io_global_by_file_by_latency                 OK
sys.x$io_global_by_wait_by_bytes                   OK
sys.x$io_global_by_wait_by_latency                 OK
sys.x$latest_file_io                               OK
sys.x$memory_by_host_by_current_bytes              OK
sys.x$memory_by_thread_by_current_bytes            OK
sys.x$memory_by_user_by_current_bytes              OK
sys.x$memory_global_by_current_bytes               OK
sys.x$memory_global_total                          OK
sys.x$processlist                                  OK
sys.x$ps_digest_95th_percentile_by_avg_us          OK
sys.x$ps_digest_avg_latency_distribution           OK
sys.x$ps_schema_table_statistics_io                OK
sys.x$schema_flattened_keys                        OK
sys.x$schema_index_statistics                      OK
sys.x$schema_table_lock_waits                      OK
sys.x$schema_table_statistics                      OK
sys.x$schema_table_statistics_with_buffer          OK
sys.x$schema_tables_with_full_table_scans          OK
sys.x$session                                      OK
sys.x$statement_analysis                           OK
sys.x$statements_with_errors_or_warnings           OK
sys.x$statements_with_full_table_scans             OK
sys.x$statements_with_runtimes_in_95th_percentile  OK
sys.x$statements_with_sorting                      OK
sys.x$statements_with_temp_tables                  OK
sys.x$user_summary                                 OK
sys.x$user_summary_by_file_io                      OK
sys.x$user_summary_by_file_io_type                 OK
sys.x$user_summary_by_stages                       OK
sys.x$user_summary_by_statement_latency            OK
sys.x$user_summary_by_statement_type               OK
sys.x$wait_classes_global_by_avg_latency           OK
sys.x$wait_classes_global_by_latency               OK
sys.x$waits_by_host_by_latency                     OK
sys.x$waits_by_user_by_latency                     OK
sys.x$waits_global_by_latency                      OK
Phase 5/8: Fixing table and database names
Phase 6/8: Checking and upgrading tables
Processing databases
airflow_analytics_test
airflow_analytics_test.ab_permission               OK
airflow_analytics_test.ab_permission_view          OK
airflow_analytics_test.ab_permission_view_role     OK
airflow_analytics_test.ab_register_user            OK
airflow_analytics_test.ab_role                     OK
airflow_analytics_test.ab_user                     OK
airflow_analytics_test.ab_user_role                OK
airflow_analytics_test.ab_view_menu                OK
airflow_analytics_test.alembic_version             OK
airflow_analytics_test.connection                  OK
airflow_analytics_test.dag                         OK
airflow_analytics_test.dag_code                    OK
airflow_analytics_test.dag_pickle                  OK
airflow_analytics_test.dag_run                     OK
airflow_analytics_test.dag_tag                     OK
airflow_analytics_test.import_error                OK
airflow_analytics_test.job                         OK
airflow_analytics_test.log                         OK
airflow_analytics_test.rendered_task_instance_fields OK
airflow_analytics_test.sensor_instance             OK
airflow_analytics_test.serialized_dag              OK
airflow_analytics_test.sla_miss                    OK
airflow_analytics_test.slot_pool                   OK
airflow_analytics_test.task_fail                   OK
airflow_analytics_test.task_instance               OK
airflow_analytics_test.task_reschedule             OK
airflow_analytics_test.variable                    OK
airflow_analytics_test.xcom                        OK
datahub
datahub.metadata_aspect_v2                         OK
datahub_prototype
datahub_prototype.metadata_aspect_v2               OK
datahub_prototype.metadata_index                   OK
druid
druid.druid_audit                                  OK
druid.druid_config                                 OK
druid.druid_dataSource                             OK
druid.druid_pendingSegments                        OK
druid.druid_rules                                  OK
druid.druid_segments                               OK
druid.druid_supervisors                            OK
druid.druid_tasklocks                              OK
druid.druid_tasklogs                               OK
druid.druid_tasks                                  OK
hive_metastore
hive_metastore.AUX_TABLE                           OK
hive_metastore.BUCKETING_COLS                      OK
hive_metastore.CDS                                 OK
hive_metastore.COLUMNS_V2                          OK
hive_metastore.COMPACTION_QUEUE                    OK
hive_metastore.COMPLETED_COMPACTIONS               OK
hive_metastore.COMPLETED_TXN_COMPONENTS            OK
hive_metastore.DATABASE_PARAMS                     OK
hive_metastore.DBS                                 OK
hive_metastore.DB_PRIVS                            OK
hive_metastore.DELEGATION_TOKENS                   OK
hive_metastore.FUNCS                               OK
hive_metastore.FUNC_RU                             OK
hive_metastore.GLOBAL_PRIVS                        OK
hive_metastore.HIVE_LOCKS                          OK
hive_metastore.IDXS                                OK
hive_metastore.INDEX_PARAMS                        OK
hive_metastore.KEY_CONSTRAINTS                     OK
hive_metastore.MASTER_KEYS                         OK
hive_metastore.METASTORE_DB_PROPERTIES             OK
hive_metastore.NEXT_COMPACTION_QUEUE_ID            OK
hive_metastore.NEXT_LOCK_ID                        OK
hive_metastore.NEXT_TXN_ID                         OK
hive_metastore.NOTIFICATION_LOG                    OK
hive_metastore.NOTIFICATION_SEQUENCE               OK
hive_metastore.NUCLEUS_TABLES                      OK
hive_metastore.PARTITIONS                          OK
hive_metastore.PARTITION_EVENTS                    OK
hive_metastore.PARTITION_KEYS                      OK
hive_metastore.PARTITION_KEY_VALS                  OK
hive_metastore.PARTITION_PARAMS                    OK
hive_metastore.PART_COL_PRIVS                      OK
hive_metastore.PART_COL_STATS                      OK
hive_metastore.PART_PRIVS                          OK
hive_metastore.ROLES                               OK
hive_metastore.ROLE_MAP                            OK
hive_metastore.SDS                                 OK
hive_metastore.SD_PARAMS                           OK
hive_metastore.SEQUENCE_TABLE                      OK
hive_metastore.SERDES                              OK
hive_metastore.SERDE_PARAMS                        OK
hive_metastore.SKEWED_COL_NAMES                    OK
hive_metastore.SKEWED_COL_VALUE_LOC_MAP            OK
hive_metastore.SKEWED_STRING_LIST                  OK
hive_metastore.SKEWED_STRING_LIST_VALUES           OK
hive_metastore.SKEWED_VALUES                       OK
hive_metastore.SORT_COLS                           OK
hive_metastore.TABLE_PARAMS                        OK
hive_metastore.TAB_COL_STATS                       OK
hive_metastore.TBLS                                OK
hive_metastore.TBL_COL_PRIVS                       OK
hive_metastore.TBL_PRIVS                           OK
hive_metastore.TXNS                                OK
hive_metastore.TXN_COMPONENTS                      OK
hive_metastore.TYPES                               OK
hive_metastore.TYPE_FIELDS                         OK
hive_metastore.VERSION                             OK
hive_metastore.WRITE_SET                           OK
hue_test
hue_test.auth_group                                OK
hue_test.auth_group_permissions                    OK
hue_test.auth_permission                           OK
hue_test.auth_user                                 OK
hue_test.auth_user_groups                          OK
hue_test.auth_user_user_permissions                OK
hue_test.axes_accessattempt                        OK
hue_test.axes_accesslog                            OK
hue_test.beeswax_metainstall                       OK
hue_test.beeswax_queryhistory                      OK
hue_test.beeswax_savedquery                        OK
hue_test.beeswax_session                           OK
hue_test.defaultconfiguration_groups               OK
hue_test.desktop_connector                         OK
hue_test.desktop_defaultconfiguration              OK
hue_test.desktop_document                          OK
hue_test.desktop_document2                         OK
hue_test.desktop_document2_dependencies            OK
hue_test.desktop_document2permission               OK
hue_test.desktop_document_tags                     OK
hue_test.desktop_documentpermission                OK
hue_test.desktop_documenttag                       OK
hue_test.desktop_settings                          OK
hue_test.desktop_userpreferences                   OK
hue_test.django_admin_log                          OK
hue_test.django_content_type                       OK
hue_test.django_migrations                         OK
hue_test.django_session                            OK
hue_test.django_site                               OK
hue_test.documentpermission2_groups                OK
hue_test.documentpermission2_users                 OK
hue_test.documentpermission_groups                 OK
hue_test.documentpermission_users                  OK
hue_test.jobsub_checkforsetup                      OK
hue_test.jobsub_jobdesign                          OK
hue_test.jobsub_jobhistory                         OK
hue_test.jobsub_oozieaction                        OK
hue_test.jobsub_ooziedesign                        OK
hue_test.jobsub_ooziejavaaction                    OK
hue_test.jobsub_ooziemapreduceaction               OK
hue_test.jobsub_ooziestreamingaction               OK
hue_test.oozie_bundle                              OK
hue_test.oozie_bundledcoordinator                  OK
hue_test.oozie_coordinator                         OK
hue_test.oozie_datainput                           OK
hue_test.oozie_dataoutput                          OK
hue_test.oozie_dataset                             OK
hue_test.oozie_decision                            OK
hue_test.oozie_decisionend                         OK
hue_test.oozie_distcp                              OK
hue_test.oozie_email                               OK
hue_test.oozie_end                                 OK
hue_test.oozie_fork                                OK
hue_test.oozie_fs                                  OK
hue_test.oozie_generic                             OK
hue_test.oozie_history                             OK
hue_test.oozie_hive                                OK
hue_test.oozie_java                                OK
hue_test.oozie_job                                 OK
hue_test.oozie_join                                OK
hue_test.oozie_kill                                OK
hue_test.oozie_link                                OK
hue_test.oozie_mapreduce                           OK
hue_test.oozie_node                                OK
hue_test.oozie_pig                                 OK
hue_test.oozie_shell                               OK
hue_test.oozie_sqoop                               OK
hue_test.oozie_ssh                                 OK
hue_test.oozie_start                               OK
hue_test.oozie_streaming                           OK
hue_test.oozie_subworkflow                         OK
hue_test.oozie_workflow                            OK
hue_test.pig_document                              OK
hue_test.pig_pigscript                             OK
hue_test.useradmin_grouppermission                 OK
hue_test.useradmin_huepermission                   OK
hue_test.useradmin_ldapgroup                       OK
hue_test.useradmin_userprofile                     OK
information_schema
oozie
oozie.BUNDLE_ACTIONS                               OK
oozie.BUNDLE_JOBS                                  OK
oozie.COORD_ACTIONS                                OK
oozie.COORD_JOBS                                   OK
oozie.OOZIE_SYS                                    OK
oozie.OPENJPA_SEQUENCE_TABLE                       OK
oozie.SLA_EVENTS                                   OK
oozie.SLA_REGISTRATION                             OK
oozie.SLA_SUMMARY                                  OK
oozie.VALIDATE_CONN                                OK
oozie.WF_ACTIONS                                   OK
oozie.WF_JOBS                                      OK
performance_schema
sys
sys.sys_config                                     OK
test
Phase 7/8: uninstalling plugins
Phase 8/8: Running 'FLUSH PRIVILEGES'
OK

MariaDB [(none)]> SELECT VERSION();
+---------------------+
| VERSION()           |
+---------------------+
| 10.6.17-MariaDB-log |
+---------------------+
1 row in set (0.000 sec)

MariaDB [(none)]> SHOW DATABASES;
+------------------------+
| Database               |
+------------------------+
| airflow_analytics_test |
| datahub                |
| datahub_prototype      |
| druid                  |
| hive_metastore         |
| hue_test               |
| information_schema     |
| mysql                  |
| oozie                  |
| performance_schema     |
| sys                    |
| test                   |
+------------------------+
12 rows in set (0.001 sec)

brouberol updated the task description. (Show Details)Jun 5 2024, 8:30 AM

Maintenance_bot removed a project: Patch-For-Review.Jun 5 2024, 8:30 AM

@brouberol do you use replication?

show slave status\G

If so, double check if it is running (check Seconds_Behind_Master) if it says NULL you'll need to issue start slave;

@Marostegui This is our test instance, that runs in a standalone fashion. However, an-mariadb100{1,2} run in a primary/replica setup, meaning that I will have to run this check when I upgratde them. Thanks!

datahub-next.wikimedia.org, which is configured to use the following JDBC URI jdbc:mysql://an-test-coord1001.eqiad.wmnet:3306/datahub?verifyServerCertificate=true&useSSL=true&useUnicode=yes&characterEncoding=UTF-8&enabledTLSProtocols=TLSv1.2, is working correclty.

brouberol updated the task description. (Show Details)Jun 5 2024, 9:08 AM

Gehel edited projects, added: Data-Platform-SRE (2024.06.17 - 2024.07.07); removed: Data-Platform-SRE (2024.05.27 - 2024.06.16).Jun 17 2024, 3:05 PM

Gehel moved this task from Backlog to In Progress on the Data-Platform-SRE (2024.06.17 - 2024.07.07) board.

Thanks @brouberol. I'll pick up the work on this for now. I'm happy to see that we have had no reports of incompatibilities, so I'm hopeful that we can get the production systems upgraded soon. We also need to be very careful about the upgrade schedule in T348977: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 which will cause connectivity losses for an-mariadb100[1-2] at different times.

An upgrade to bookworm if that's doable, is probably also worth it.

In T365503#9943776, @Marostegui wrote:

An upgrade to bookworm if that's doable, is probably also worth it.

Yes, good point, thanks. Should be do-able.

Change #1052712 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Configure reuse-parts for an-mariadb servers

https://gerrit.wikimedia.org/r/1052712

gerritbot added a project: Patch-For-Review.Jul 8 2024, 11:54 AM

Change #1052712 merged by Btullis:

[operations/puppet@production] Configure reuse-parts for an-mariadb servers

https://gerrit.wikimedia.org/r/1052712

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host an-mariadb1002.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1002 for host an-mariadb1002.eqiad.wmnet with OS bookworm completed:

an-mariadb1002 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202407081235_btullis_3488808_an-mariadb1002.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
- Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)

BTullis updated the task description. (Show Details)Jul 8 2024, 1:05 PM

I have upgraded an-mariadb1002 to bookworm and that has successfully installed MariaDB 10.6.
Replication seems fine. I now have to prepare a few patches to switch the primary role from an-mariadb1001 to an-mariadb1002 before Wednesday's switch reboot in T348977: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2

I also have to re-deploy external-services so that the networkpolicy allows access to an-mariadb1002.

Gehel edited projects, added: Data-Platform-SRE (2024.07.08 - 2024.07.28); removed: Data-Platform-SRE (2024.06.17 - 2024.07.07).Jul 8 2024, 6:33 PM

Gehel moved this task from Backlog to In Progress on the Data-Platform-SRE (2024.07.08 - 2024.07.28) board.

Change #1052932 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Configure analytics_meta MariaDB clients to connect to an-mariadb1002

https://gerrit.wikimedia.org/r/1052932

Change #1052937 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Facilitate a role swap from an-mariadb1001 to an-mariadb1002

https://gerrit.wikimedia.org/r/1052937

Change #1052944 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Temporarily pause gobblin to facilitate Hive/MariaDB maintenance

https://gerrit.wikimedia.org/r/1052944

Change #1052945 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Revert the change to disable the gobbin timers on an-launcher

https://gerrit.wikimedia.org/r/1052945

I have scheduled a role swap for tomorrow at 10:00 UTC and I will have paused ingestions to the data lake via gobblin an hour beforehand. This is to facilitate the swith upgrade in: T348977 at 14:00 tomorrow.
We will then need to do the reverse operation on the following day, as T365996 will make an-mariadb1002 unavailable.

We don't have a readily available cookbook for doing this role swap, so I'll just note down the procedure that I will carry out here.
I'm also referring to the steps outlined here: https://mariadb.com/kb/en/changing-a-replica-to-become-the-primary/
It's also pretty similar to https://wikitech.wikimedia.org/wiki/Fundraising/techops/procedures/services-mariadb_primary_swap

an-mariadb1002: SHOW SLAVE STATUS\G; and verify that Seconds_Behind_Master is 0

an-mariadb1001: SET @@global.read_only=1;
an-mariadb1001: FLUSH TABLES WITH READ LOCK;
an-mariadb1001: SHOW MASTER STATUS; and SELECT @@global.gtid_binlog_pos;

an-mariadb1002: SHOW SLAVE STATUS\G; and verify that Master_Log_File and Exec_Master_Log_Pos match the output shown in the previous SHOW MASTER STATUS; command.
an-mariadb1002: STOP SLAVE; and RESET SLAVE ALL;
an-mariadb1002: SHOW MASTER STATUS; and SELECT @@global.gtid_binlog_pos; - Very that global.gtid_binlog_pos is the same as was previously shown.
an-mariadb1002: SET @@global.read_only=0;

an-mariadb1001: RESET MASTER;
an-mariadb1001: CHANGE MASTER TO MASTER_HOST="10.64.142.7",MASTER_PORT=3306, MASTER_USER="redacted",MASTER_PASSWORD="redacted",MASTER_USE_GTID=current_pos,MASTER_SSL=1;
an-mariadb1001: START SLAVE;
an-mariadb1001: SHOW SLAVE STATUS\G; and check for Slave_IO_Running: Yes, Slave_SQL_Running: Yes, Seconds_Behind_Master: 0

Merge Configure analytics_meta MariaDB clients to connect to an-mariadb1002 on gerrit:
Execute: sudo cumin 'A:druid or A:an-mariadb or A:hadoop-coordinator' run-puppet-agent This will update:
- /etc/my.cnf on an-mariadb100[1-2]
- /etc/hive/conf.analytics-hadoop/hive-site.xml on an-coord100[3-4]
- /etc/druid/common.runtime.properties on all druid servers
Execute: sudo cumin A:hadoop-coordinator systemctl restart hive-server2 hive-metastore to restart hive with the new configuration
Execute: sudo cumin A:druid systemctl restart druid-*

Merge: Facilitate a role swap from an-mariadb1001 to an-mariadb1002 to the deployment-charts repo on gerrit
Deploy superset
Deploy datahub
Deploy mpic

Change #1052944 merged by Btullis:

[operations/puppet@production] Temporarily pause gobblin to facilitate Hive/MariaDB maintenance

https://gerrit.wikimedia.org/r/1052944

Mentioned in SAL (#wikimedia-analytics) [2024-07-10T10:10:18Z] <btullis> proceeding with an-mariadb role swap as per https://phabricator.wikimedia.org/T365503#9965173

Change #1052932 merged by Btullis:

[operations/puppet@production] Configure analytics_meta MariaDB clients to connect to an-mariadb1002

https://gerrit.wikimedia.org/r/1052932

Change #1052937 merged by Btullis:

[operations/deployment-charts@master] Facilitate a role swap from an-mariadb1001 to an-mariadb1002

https://gerrit.wikimedia.org/r/1052937

The role swap is done. There were a few issues, but nothing major.

The superset_staging database is not replicated, which means that the superset-next deployment to dse-k8s isn't working. It is excluded from the replication here. This was done in order to support the process of syncing the staging database from production here, but I'm not keen on it. I am tempted to remove this configuration and allow the two databases to be an exact replica of each other.

Once the puppet change had been merged, I also needed to:
- run puppet on the deployment server to update the external-services values
- deploy admin_ng on dse-k8s in order to update the analytics_meta primary server

I had forgotten about the fact that db1208 was replicating from an-mariadb1001, so when I issued the RESET MASTER the replication to this replica failed. I updated MASTER_HOST on db1208 to point to an-mariadb1002 and everything was then fine.

A few systemd based refine jobs failed, so I re-ran them.

I'll try to make this a bit more streamlined tomorrow, when I fail back.

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host an-mariadb1001.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1002 for host an-mariadb1001.eqiad.wmnet with OS bookworm completed:

an-mariadb1001 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202407101318_btullis_3886281_an-mariadb1001.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
- Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)

I have now reimaged an-mariadb1001 and it has installed mariadb 10.6.

btullis@an-mariadb1001:~$ sudo mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 491
Server version: 10.6.18-MariaDB-log MariaDB Server

I'll leave this ticket open until after tomorrow's failback from an-mariadb1002.

BTullis mentioned this in T351731: Turnilo: invalid transforms on wmf_netflow dashboard.Jul 10 2024, 3:55 PM

BTullis mentioned this in T369737: Site Issue: Delayed data in the `webrequest_sampled_live` kafka topic.Jul 10 2024, 4:21 PM

Change #1053624 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Temporarily disable gobblin timers to permit hive maintenance

https://gerrit.wikimedia.org/r/1053624

Change #1053624 merged by Btullis:

[operations/puppet@production] Temporarily disable gobblin timers to permit hive maintenance

https://gerrit.wikimedia.org/r/1053624

I believe that this is all done now. I have swapped the roles back, so that an-mariadb1001 is the active master and an-mariadb1002 is the replica.
The backup host, db1208, is set to replicate from an-mariadb1001 again.

Change #1052945 abandoned by Btullis:

[operations/puppet@production] Revert the change to disable the gobbin timers on an-launcher

Reason:

Reverted in another patch

https://gerrit.wikimedia.org/r/1052945

BTullis mentioned this in T394498: SSD firmware update for an-mariadb100[1-2].Sep 1 2025, 11:42 AM

Maintenance_bot removed a project: Patch-For-Review.Sep 1 2025, 12:31 PM

Upgrade mariadb on analytics_meta from 10.4 to 10.6Closed, ResolvedPublicActions

Description

Details

Related Objects

Event Timeline

Upgrade mariadb on analytics_meta from 10.4 to 10.6
Closed, ResolvedPublic
Actions