CWilliams-WMF
User

Projects (6)
View All

Calendar

User Details

User Since: May 5 2026, 7:27 AM (4 w, 6 d)
Availability: Available
LDAP User: CWilliams
MediaWiki User: CWilliams-WMF [ Global Accounts ]

Recent Activity
View All

Today

CWilliams-WMF closed T427780: Provide downtime duration information in sre.mysql cookbooks as Resolved.

Mon, Jun 8, 8:53 AM · Infrastructure-Foundations, Spicerack, SRE-tools, DBA

CWilliams-WMF closed T427780: Provide downtime duration information in sre.mysql cookbooks, a subtask of T426318: Add support for automatic downtime when depooling instances using sre.mysql.depool, as Resolved.

Mon, Jun 8, 8:53 AM · DBA

Thu, Jun 4

CWilliams-WMF closed T426725: Migrate x3 section to Debian Trixie as Resolved.

The remaining hosts are out of scope for this ticket:

% sudo cumin 'A:db-section-x3 and A:bookworm'
6 hosts will be targeted:
clouddb[1016,1020,1022-1023].eqiad.wmnet,db2200.codfw.wmnet,db1216.eqiad.wmnet
DRY-RUN mode enabled, aborting

Thu, Jun 4, 8:26 AM · DBA

CWilliams-WMF closed T426725: Migrate x3 section to Debian Trixie, a subtask of T422365: Migration to Debian Trixie of production database-related hosts, as Resolved.

Thu, Jun 4, 8:26 AM · DBA

CWilliams-WMF added a comment to P93843 (An Untitled Masterwork).

Thu, Jun 4, 7:39 AM

CWilliams-WMF updated the task description for T426725: Migrate x3 section to Debian Trixie.

Thu, Jun 4, 6:54 AM · DBA

CWilliams-WMF closed T427895: Switchover x3 master (db1255 -> db1258) as Resolved.

Proceeding with the reimage of db1255 for T426725

Thu, Jun 4, 6:10 AM · Patch-For-Review, DBA

CWilliams-WMF closed T427895: Switchover x3 master (db1255 -> db1258), a subtask of T426725: Migrate x3 section to Debian Trixie, as Resolved.

Thu, Jun 4, 6:10 AM · DBA

CWilliams-WMF updated the task description for T427895: Switchover x3 master (db1255 -> db1258).

Thu, Jun 4, 6:09 AM · Patch-For-Review, DBA

Wed, Jun 3

CWilliams-WMF closed T427031: Migrate s8 section to Debian Trixie, a subtask of T422365: Migration to Debian Trixie of production database-related hosts, as Resolved.

Wed, Jun 3, 12:03 PM · DBA

CWilliams-WMF closed T427031: Migrate s8 section to Debian Trixie as Resolved.

The remaining hosts are out of scope for this ticket, marking as resolved:

% sudo cumin 'A:db-section-s8 and A:bookworm'
6 hosts will be targeted:
an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db2198.codfw.wmnet,db1171.eqiad.wmnet,dbstore1009.eqiad.wmnet
DRY-RUN mode enabled, aborting

Wed, Jun 3, 12:03 PM · DBA

CWilliams-WMF added a comment to T427780: Provide downtime duration information in sre.mysql cookbooks.

@elukey yes, I did have an idea... but @Volans suggesting that making it part of the log messages from the calls adding the downtime would be preferential, given that it makes it available to every cookbook, presuming that there are no complications in doing that.
@Marostegui would the log message be enough for you?

Wed, Jun 3, 9:51 AM · Infrastructure-Foundations, Spicerack, SRE-tools, DBA

CWilliams-WMF updated the task description for T426725: Migrate x3 section to Debian Trixie.

Wed, Jun 3, 8:47 AM · DBA

CWilliams-WMF closed T427892: Switchover s8 master (db2161 -> db2165) as Resolved.

Wed, Jun 3, 7:46 AM · DBA

CWilliams-WMF added a comment to T427884: Create database grants for cumin2003.

Thanks!

Wed, Jun 3, 7:17 AM · DBA

CWilliams-WMF claimed T427884: Create database grants for cumin2003.

Wed, Jun 3, 7:17 AM · DBA

Tue, Jun 2

CWilliams-WMF added a comment to T427897: Upgrade Cumin hosts to Trixie.

AttributeError: module 'urllib3.exceptions' has no attribute 'SubjectAltNameWarning'

Tue, Jun 2, 4:12 PM · Infrastructure-Foundations, SRE

CWilliams-WMF added a comment to T427031: Migrate s8 section to Debian Trixie.

Another time that Icinga held red on "MariaDB sustained replica lag on <section>", despite replication having caught up about 10-15 minutes beforehand.
Manually cleared downtime once green and then repooled.

Tue, Jun 2, 2:23 PM · DBA

CWilliams-WMF updated the task description for T427031: Migrate s8 section to Debian Trixie.

Tue, Jun 2, 2:19 PM · DBA

CWilliams-WMF updated the task description for T427031: Migrate s8 section to Debian Trixie.

Tue, Jun 2, 12:09 PM · DBA

CWilliams-WMF changed the status of T427892: Switchover s8 master (db2161 -> db2165) from Open to In Progress.

Tue, Jun 2, 11:20 AM · DBA

CWilliams-WMF updated the task description for T427892: Switchover s8 master (db2161 -> db2165).

Tue, Jun 2, 11:18 AM · DBA

CWilliams-WMF updated the task description for T427892: Switchover s8 master (db2161 -> db2165).

Tue, Jun 2, 11:16 AM · DBA

CWilliams-WMF updated the task description for T427892: Switchover s8 master (db2161 -> db2165).

Tue, Jun 2, 11:05 AM · DBA

CWilliams-WMF updated the task description for T427892: Switchover s8 master (db2161 -> db2165).

Tue, Jun 2, 11:01 AM · DBA

CWilliams-WMF claimed T427895: Switchover x3 master (db1255 -> db1258).

Tue, Jun 2, 9:10 AM · Patch-For-Review, DBA

CWilliams-WMF closed T427893: Switchover s8 master (db2161 -> db2165) as Invalid.

Tue, Jun 2, 9:07 AM · DBA

CWilliams-WMF claimed T427892: Switchover s8 master (db2161 -> db2165).

Tue, Jun 2, 9:05 AM · DBA

CWilliams-WMF claimed T427893: Switchover s8 master (db2161 -> db2165).

Tue, Jun 2, 9:01 AM · DBA

Mon, Jun 1

CWilliams-WMF updated the task description for T427031: Migrate s8 section to Debian Trixie.

Mon, Jun 1, 3:41 PM · DBA

CWilliams-WMF created T427803: New VictorOps user request for cwilliams.

Mon, Jun 1, 1:35 PM · observability

CWilliams-WMF updated the task description for T427031: Migrate s8 section to Debian Trixie.

Mon, Jun 1, 10:06 AM · DBA

CWilliams-WMF added a comment to T426318: Add support for automatic downtime when depooling instances using sre.mysql.depool.

There is no default downtime, you need to pass --downtime=n and then it wil perform the downtime. Given that depooling can include decommissioning, it didn't seem to make sense to have a default value.

Mon, Jun 1, 10:04 AM · DBA

CWilliams-WMF created T427780: Provide downtime duration information in sre.mysql cookbooks.

Mon, Jun 1, 8:55 AM · Infrastructure-Foundations, Spicerack, SRE-tools, DBA

CWilliams-WMF closed T426318: Add support for automatic downtime when depooling instances using sre.mysql.depool as Resolved.

Mon, Jun 1, 8:41 AM · DBA

CWilliams-WMF added a comment to T426318: Add support for automatic downtime when depooling instances using sre.mysql.depool.

@FCeratto-WMF as mentioned, this is not something unique to pooling. For example, in sre.mysql.clone

step("icinga", "Disabling monitoring for source and target host")                                                                                                                                                                     
source_alerter = self.alerting_hosts(self.source_host.hosts)                                                                                                                                                                          
source_downtime_id = source_alerter.downtime(self.admin_reason, duration=timedelta(hours=8))

Mon, Jun 1, 8:40 AM · DBA

CWilliams-WMF added a comment to T426318: Add support for automatic downtime when depooling instances using sre.mysql.depool.

@Marostegui no, the spicerack code does not do that nor does it look like other sre.mysql cookbooks show that information. However, the sre.hosts.downtime cookbook updates Phabricator with a comment. So, if you want it to be logged either to the console, Phabricator, or both then it looks like that is a more general task for sre.mysql, or do use the downtime cookbook directly. I will wait to hear your reply before I closed this ticket, as it was merged.

Mon, Jun 1, 7:46 AM · DBA

Wed, May 27

CWilliams-WMF claimed T427377: sre.mysql.pool: remove downtime before pooling.

Wed, May 27, 3:59 PM · DBA

CWilliams-WMF added a comment to T426318: Add support for automatic downtime when depooling instances using sre.mysql.depool.

START - Cookbook sre.mysql.depool depool db2163: Testing cookbook
[cookbooks.sre.mysql.pool.depool] Setting downtime
Scheduling downtime on Icinga server alert1002.wikimedia.org for hosts: db2163
Created silence ID da7a2b6d-a0f4-46d9-a37a-b29ef98b503b
Previous configuration saved. To restore it run: dbctl config restore /var/cache/conftool/dbconfig/20260527-155100-cwilliams.json
dbctl commit (dc=codfw): 'Testing cookbook', diff saved to https://phabricator.wikimedia.org/P93277 and previous config saved to /var/cache/conftool/dbconfig/20260527-155100-cwilliams.json
Monitoring number of wikiuser* connections
Connection drain completed
Unable to access task : not adding comment 'Completed depooling of db2163 by cwilliams@cumin1003: Testing cookbook'
Released lock for key /spicerack/locks/cookbooks/sre.mysql.depool:db2163: {'concurrency': 1, 'created': '2026-05-27 15:50:51.057298', 'owner': 'cwilliams@cumin1003 [1809630]', 'ttl': 60}
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2163: Testing cookbook

Wed, May 27, 3:56 PM · DBA

CWilliams-WMF updated the title for P93266 spicerack.icinga.IcingaError from spicerack.icinga.IcingaErro to spicerack.icinga.IcingaError.

Wed, May 27, 3:45 PM · DBA

CWilliams-WMF added a comment to T427031: Migrate s8 section to Debian Trixie.

Whilst the original cookbook was waiting for replication to catch-up, an exception caused the process to bail out: https://phabricator.wikimedia.org/P93266
Judging by Icinga, it appears that the service recovered at:

[2026-05-27 15:10:44] SERVICE ALERT: db1178;MariaDB sustained replica lag on s8;OK;HARD;5;(C)10 ge (W)5 ge 0

Wed, May 27, 3:30 PM · DBA

CWilliams-WMF updated the task description for T427031: Migrate s8 section to Debian Trixie.

Wed, May 27, 3:26 PM · DBA

CWilliams-WMF created P93266 spicerack.icinga.IcingaError.

Wed, May 27, 3:03 PM · DBA

CWilliams-WMF updated the task description for T427031: Migrate s8 section to Debian Trixie.

Wed, May 27, 1:25 PM · DBA

CWilliams-WMF updated the task description for T427031: Migrate s8 section to Debian Trixie.

Wed, May 27, 11:37 AM · DBA

CWilliams-WMF updated the task description for T427031: Migrate s8 section to Debian Trixie.

Wed, May 27, 10:00 AM · DBA

Tue, May 26

CWilliams-WMF updated the task description for T427031: Migrate s8 section to Debian Trixie.

Tue, May 26, 2:28 PM · DBA

CWilliams-WMF updated the task description for T427031: Migrate s8 section to Debian Trixie.

Tue, May 26, 11:50 AM · DBA

CWilliams-WMF updated the task description for T427031: Migrate s8 section to Debian Trixie.

Tue, May 26, 11:23 AM · DBA

CWilliams-WMF updated the task description for T427031: Migrate s8 section to Debian Trixie.

Tue, May 26, 9:53 AM · DBA

CWilliams-WMF updated the task description for T427031: Migrate s8 section to Debian Trixie.

Tue, May 26, 9:18 AM · DBA

CWilliams-WMF added a comment to T427059: Discussion for adding support of test-s4 in dbctl.

@Ladsgroup thanks for the quick response!

but we can easily test this

What is involved in doing this?

Tue, May 26, 8:30 AM · Patch-For-Review, DBA

CWilliams-WMF updated the task description for T427031: Migrate s8 section to Debian Trixie.

Tue, May 26, 8:21 AM · DBA

CWilliams-WMF updated the task description for T427059: Discussion for adding support of test-s4 in dbctl.

Tue, May 26, 7:48 AM · Patch-For-Review, DBA

Fri, May 22

CWilliams-WMF created T427059: Discussion for adding support of test-s4 in dbctl.

Fri, May 22, 12:32 PM · Patch-For-Review, DBA

CWilliams-WMF updated the task description for T427031: Migrate s8 section to Debian Trixie.

Fri, May 22, 10:47 AM · DBA

CWilliams-WMF updated the task description for T427031: Migrate s8 section to Debian Trixie.

Fri, May 22, 10:31 AM · DBA

CWilliams-WMF changed the status of T427031: Migrate s8 section to Debian Trixie from Open to In Progress.

Fri, May 22, 9:07 AM · DBA

CWilliams-WMF changed the status of T427031: Migrate s8 section to Debian Trixie, a subtask of T422365: Migration to Debian Trixie of production database-related hosts, from Open to In Progress.

Fri, May 22, 9:07 AM · DBA

Thu, May 21

CWilliams-WMF updated the task description for T426725: Migrate x3 section to Debian Trixie.

Thu, May 21, 2:15 PM · DBA

CWilliams-WMF closed T426936: Switchover x3 master (db2241 -> db2162) as Resolved.

Thu, May 21, 2:13 PM · DBA

CWilliams-WMF closed T426936: Switchover x3 master (db2241 -> db2162), a subtask of T426725: Migrate x3 section to Debian Trixie, as Resolved.

Thu, May 21, 2:13 PM · DBA

CWilliams-WMF updated the task description for T426936: Switchover x3 master (db2241 -> db2162).

Thu, May 21, 2:13 PM · DBA

CWilliams-WMF updated the task description for T426725: Migrate x3 section to Debian Trixie.

Thu, May 21, 1:21 PM · DBA

CWilliams-WMF added a comment to T426936: Switchover x3 master (db2241 -> db2162).

Pending repool whilst reimaging for T426725

Thu, May 21, 1:16 PM · DBA

CWilliams-WMF updated the task description for T426936: Switchover x3 master (db2241 -> db2162).

Thu, May 21, 1:15 PM · DBA

CWilliams-WMF updated the task description for T426936: Switchover x3 master (db2241 -> db2162).

Thu, May 21, 1:12 PM · DBA

CWilliams-WMF updated the task description for T426936: Switchover x3 master (db2241 -> db2162).

Thu, May 21, 1:07 PM · DBA

CWilliams-WMF updated the task description for T426936: Switchover x3 master (db2241 -> db2162).

Thu, May 21, 1:02 PM · DBA

CWilliams-WMF updated the task description for T426936: Switchover x3 master (db2241 -> db2162).

Thu, May 21, 11:42 AM · DBA

CWilliams-WMF added a comment to T426936: Switchover x3 master (db2241 -> db2162).

Noting that there is a difference in MariaDB on the minor version, the other difference seem to be as expected:

24 config differences
Variable                  db2241                    db2162
========================= ========================= =========================
general_log_file          db2241.log                db2162.log
gtid_binlog_pos           171966580-171966580-79... 171966560-171966560-15...
gtid_binlog_state         171966580-171966580-79... 171966560-171966560-15...
gtid_current_pos          0-180359179-5751637176... 0-180359179-5751637176...
gtid_domain_id            180356619                 180359385
gtid_slave_pos            0-180359179-5751637176... 0-180359179-5751637176...
hostname                  db2241                    db2162
innodb_buffer_pool_size   405874409472              404800667648
innodb_buffer_pool_siz... 405874409472              404800667648
innodb_buffer_pool_siz... 405874409472              404800667648
log_bin_basename          /srv/sqldata/db2241-bin   /srv/sqldata/db2162-bin
log_bin_index             /srv/sqldata/db2241-bi... /srv/sqldata/db2162-bi...
log_slow_query_file       db2241-slow.log           db2162-slow.log
pid_file                  /srv/sqldata/db2241.pid   /srv/sqldata/db2162.pid
report_host               db2241.codfw.wmnet        db2162.codfw.wmnet
rpl_semi_sync_master_e... ON                        OFF
rpl_semi_sync_slave_en... OFF                       ON
server_id                 180356619                 180359385
server_uid                NJgxPfGLXWwZjiC6baS4tu... IhJ+cAGEva1poSuFcK3mtg...
slow_query_log_file       db2241-slow.log           db2162-slow.log
version                   10.11.13-MariaDB-log      10.11.16-MariaDB-log
version_source_revision   8fb09426b98583916ccfd4... 3218602d3100db9ce7a875...
version_ssl_library       OpenSSL 3.0.16 11 Feb ... OpenSSL 3.5.6 7 Apr 2026
wsrep_node_name           db2241                    db2162

Thu, May 21, 11:42 AM · DBA

CWilliams-WMF claimed T426936: Switchover x3 master (db2241 -> db2162).

Thu, May 21, 11:36 AM · DBA

CWilliams-WMF updated the task description for T426936: Switchover x3 master (db2241 -> db2162).

Thu, May 21, 10:16 AM · DBA

CWilliams-WMF updated the task description for T426725: Migrate x3 section to Debian Trixie.

Thu, May 21, 8:13 AM · DBA

CWilliams-WMF added a parent task for T426936: Switchover x3 master (db2241 -> db2162): T426725: Migrate x3 section to Debian Trixie.

Thu, May 21, 8:12 AM · DBA

CWilliams-WMF added a subtask for T426725: Migrate x3 section to Debian Trixie: T426936: Switchover x3 master (db2241 -> db2162).

Thu, May 21, 8:12 AM · DBA

CWilliams-WMF updated the task description for T426725: Migrate x3 section to Debian Trixie.

Thu, May 21, 8:02 AM · DBA

Wed, May 20

CWilliams-WMF updated the task description for T426725: Migrate x3 section to Debian Trixie.

Wed, May 20, 4:16 PM · DBA

CWilliams-WMF updated the task description for T426725: Migrate x3 section to Debian Trixie.

Wed, May 20, 1:23 PM · DBA

CWilliams-WMF added a comment to T426725: Migrate x3 section to Debian Trixie.

Upgrading db1258.eqiad.wmnet

This ended up in a broken state as the management password that was entered was incorrect.
See https://phabricator.wikimedia.org/P92681 for related output

Wed, May 20, 12:26 PM · DBA

CWilliams-WMF created P92681 Invalid passwords are not re-requested with retry.

Wed, May 20, 12:15 PM · DBA

CWilliams-WMF updated the task description for T426725: Migrate x3 section to Debian Trixie.

Wed, May 20, 11:41 AM · DBA

CWilliams-WMF changed the status of T426725: Migrate x3 section to Debian Trixie, a subtask of T422365: Migration to Debian Trixie of production database-related hosts, from Open to In Progress.

Wed, May 20, 10:37 AM · DBA

CWilliams-WMF changed the status of T426725: Migrate x3 section to Debian Trixie from Open to In Progress.

Wed, May 20, 10:37 AM · DBA

CWilliams-WMF updated the task description for T426725: Migrate x3 section to Debian Trixie.

Wed, May 20, 8:29 AM · DBA