User Details
- User Since
- Jan 7 2025, 6:49 PM (56 w, 6 d)
- Availability
- Available
- IRC Nick
- federico3
- LDAP User
- Federico Ceratto
- MediaWiki User
- FCeratto-WMF [ Global Accounts ]
Yesterday
Thu, Feb 5
Tue, Feb 3
Thanks, closing!
Currently we wait for Icinga to be/go green before pooling (and this implies the replication lag has to be low enough).
The check can be bypassed using --skip-safety-checks and in that case we do the gradual pooling regardless of lag or Icinga status in general.
sudo db-switchover --timeout=25 --replicating-master --read-only-master --only-slave-move db1176.eqiad.wmnet db-test1002.eqiad.wmnet
$ hostname cumin1003 $ sudo db-switchover --timeout=25 --replicating-master --read-only-master --only-slave-move db1176 db-test2001 Starting preflight checks... [ERROR]: Read only status could be not read from one or more servers $
Thu, Jan 29
Tue, Jan 27
Mon, Jan 26
Later on with T374026 we could also repool the two hosts at the same time.
Thu, Jan 22
Change deployed, closing task. Please reopen it if there's any issue.
Wed, Jan 21
Thanks, closing task.
I'm closing the task as resolved for now. @dr0ptp4kt if there's any issue please reopen the task.
Change deployed on Puppet, closing task.
Tue, Jan 20
SSH key verified, opening CR.
@FRomeo_WMF and @greg hello - could you please review the request for approval?
Change deployed, closing task.
Change deployed, closing task.
Mon, Jan 19
@KReid-WMF access configured - can you please confirm it works so we can close the task? Thanks
If I'm understanding correctly that this is a request for the deployment group, @thcipriani can you please approve it?
As discussed I'll add a lock on the section to update one host at a time and show it on https://zarcillo.wikimedia.org/ui/locks
Pending OOB confirmation of the SSH key
Related to T411679 where the access was initially granted.
Hello @Milimetric @Ahoelzl @Ottomata - can you please review this access request for analytics-privatedata-users? Thanks
Implemented in T400056
Thanks, closing task.
Fri, Jan 16
@Johannes_Richter_WMDE i updated the permissions, can you please confirm that you have access now? Thanks
@dr0ptp4kt the change was deployed, can you please confirm if the access works for you now?
Ryan confirmed on IRC, opening https://gerrit.wikimedia.org/r/c/operations/puppet/+/1227848
@Marostegui ok - I'll put it back into Open/Pending in the meantime.
Hello @DannyS712 sorry for the ping, when you have a second could you please reply to https://phabricator.wikimedia.org/T413634#11505670
I updated the task summary: the cookbook is doing depooling, replication lag checks, icinga checks and repooling and was tested on different hosts.
[Pinged RKemper on IRC]
Pending out of band SSH verification.
Thu, Jan 15
we've done quite a bunch of work to get the wmfmariadbpy repo to build and publish packages that can be installed on cumin hosts - we can add auto schema to it in a dedicated package and dedicated directory
Wed, Jan 14
Tue, Jan 13
Mon, Jan 12
Completed on codfw.
Completed on codfw
Completed on eqiad DC master
Completed on eqiad DC master
I saw the alert on alertmanager:
Jan 9 2026
Jan 8 2026
Completed on codfw on replicas.
Completed on codfw on replicas.
Jan 7 2026
Completed in eqiad on replicas.
Completed in eqiad on replicas.