Rebuild db2203
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Marostegui
	Jan 21 2025, 6:54 AM

Description

This host got stuck when doing a switchover. The process is in a very weird state and given it is the candidate master, it is safer to reclone and upgrade.

root@db2203:~# ps aux | grep mariadb
mysql       1510  225 77.4 436050024 408705824 ? Ssl   2024 568765:55 /opt/wmf-mariadb106/bin/mysqld

root@db2203:~# w
 06:53:30 up 174 days, 21:25,  1 user,  load average: 0.21, 0.42, 1.48
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	db2203: Disable notifications	operations/puppet	production	+2 -1

Customize query in gerrit

Related Objects

Mentioned Here: P72169 dbctl commit (dc=all): 'Depool db2216 T384273'

Event Timeline

Marostegui created this task.Jan 21 2025, 6:54 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 21 2025, 6:54 AM

Marostegui triaged this task as High priority.Jan 21 2025, 6:55 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-21T06:56:40Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2216 T384273', diff saved to https://phabricator.wikimedia.org/P72169 and previous config saved to /var/cache/conftool/dbconfig/20250121-065640-marostegui.json

Icinga downtime and Alertmanager silence (ID=9654e11c-3dfa-405c-94e3-10fc5d04d96f) set by marostegui@cumin1002 for 1 day, 0:00:00 on 1 host(s) and their services with reason: rebuilding index