TZ: UTC +1/+2
User Details
User Details
- User Since
- Sep 1 2016, 6:48 AM (518 w, 5 d)
- Availability
- Available
- IRC Nick
- marostegui
- LDAP User
- Marostegui
- MediaWiki User
- MArostegui (WMF) [ Global Accounts ]
Yesterday
Yesterday
Marostegui moved T434541: Drop the local wikifunctionsclient_usage tables from all wikis, replaced by wikifunctions_usage / wikifunctions_usage_wiki on x1 from Triage to Blocked on the DBA board.
@Jdforrester-WMF could you let us know when we can proceed?
Thanks
Marostegui added a comment to T434532: db2201's both mysql instances crashed in the last few days.
[Tue Aug 4 17:48:49 2026] {6}[Hardware Error]: DIMM location: not present. DMI handle: 0x0000
[Tue Aug 4 17:48:49 2026] EDAC MC3: 1 UE memory read error on CPU_SrcID#0_MC#3_Chan#1_DIMM#0 (channel:1 slot:0 page:0x15e95e1 offset:0x240 grain:32 - err_code:0x0000:0x009f SystemAddress:0x15e95e1240 ProcessorSocketId:0x0 MemoryControllerId:0x3 ChannelAddress:0x2ad2bc240 ChannelId:0x1 RankAddress:0x15695e240 PhysicalRankId:0x0 DimmSlotId:0x0 Row:0x9b4f Column:0x48 Bank:0x3 BankGroup:0x0 ChipSelect:0x0 ChipId:0x0)Marostegui added a comment to T434532: db2201's both mysql instances crashed in the last few days.
This is a few days old:
Description: A critical diagnostic event occurred in the memory device at A8. Contact your service provider for assistance in replacing the device. (Extended ID: 0x4E41).
Marostegui moved T434538: decommission db1151.eqiad.wmnet from Triage to In progress on the DBA board.
Marostegui added a parent task for T434538: decommission db1151.eqiad.wmnet: T433475: Decommission db11[50-82,84].
Marostegui added a subtask for T433475: Decommission db11[50-82,84]: T434538: decommission db1151.eqiad.wmnet.
Marostegui added a comment to T434538: decommission db1151.eqiad.wmnet.
Not yet ready
Marostegui added a comment to T433474: decommission db1177.eqiad.wmnet.
This is ready for DCOps.
Marostegui added a parent task for T434498: decommission db1164.eqiad.wmnet: T433475: Decommission db11[50-82,84].
Marostegui added a subtask for T433475: Decommission db11[50-82,84]: T434498: decommission db1164.eqiad.wmnet.
Marostegui closed T422365: Migration to Debian Trixie of production database-related hosts as Resolved.
All done!
All done
Marostegui closed T433463: Migrate m1 to Debian Trixie, a subtask of T422365: Migration to Debian Trixie of production database-related hosts, as Resolved.
Marostegui closed T434493: Switchover m1 master (db1164 -> db1213), a subtask of T433463: Migrate m1 to Debian Trixie, as Resolved.
Marostegui added a subtask for T433475: Decommission db11[50-82,84]: T434480: decommission db1152.eqiad.wmnet.
Marostegui added a parent task for T434480: decommission db1152.eqiad.wmnet: T433475: Decommission db11[50-82,84].
Marostegui added a comment to T434480: decommission db1152.eqiad.wmnet.
Not ready yet, this host was switched today as part of https://phabricator.wikimedia.org/T434288 so giving it a few days before decom
Marostegui added a comment to T433463: Migrate m1 to Debian Trixie.
db1164 was promoted to master, db1213 is no longer a master and needs to be reimaged and then switched back.
Marostegui closed T434043: Switchover m1 master (db1213 -> db1164), a subtask of T433463: Migrate m1 to Debian Trixie, as Resolved.
This was done
Fri, Aug 7
Fri, Aug 7
Marostegui added a comment to P95944 (An Untitled Masterwork).
[11:29:48] marostegui@cumin1003:~$ time sudo dbctl config commit -m "Adding db1271 to dbctl"
eqiad/hostsByName live eqiad/hostsByName generated
"db1260": "10.64.0.187", "db1260": "10.64.0.187",
"db1261": "10.64.16.77", "db1261": "10.64.16.77",
"db1262": "10.64.32.64", "db1262": "10.64.32.64",
"db1263": "10.64.48.47", "db1263": "10.64.48.47",
"db1264": "10.64.16.6", "db1264": "10.64.16.6",
"db1271": "10.64.0.155",
"es1035": "10.64.152.4", "es1035": "10.64.152.4",
"es1036": "10.64.154.3", "es1036": "10.64.154.3",
"es1037": "10.64.156.3", "es1037": "10.64.156.3",
"es1038": "10.64.160.4", "es1038": "10.64.160.4",
"es1039": "10.64.162.3", "es1039": "10.64.162.3",Marostegui updated subscribers of T434287: Move s2 sanitarium (db1155:3312) from db1156 to db1271.
Marostegui triaged T434287: Move s2 sanitarium (db1155:3312) from db1156 to db1271 as Medium priority.
Marostegui moved T434043: Switchover m1 master (db1213 -> db1164) from Triage to In progress on the DBA board.
Marostegui added a comment to T407942: Productionize db12[65-90].
Yes, that is correct. I've +1ed it.
Marostegui added a comment to T433471: decommission db1178.eqiad.wmnet.
Ready for DC-Ops!
Marostegui placed T433471: decommission db1178.eqiad.wmnet up for grabs.
Thu, Aug 6
Thu, Aug 6
Marostegui added a comment to T407942: Productionize db12[65-90].
I've started replication on db1266 in ms2, this means ms2 topology is a bit strange at the moment, but I am going to leave it replicating for a few days before going ahead and setting up as a master
Marostegui added a comment to T407942: Productionize db12[65-90].
I've started replication on db1268 in ms3, this means ms3 topology is a bit strange at the moment, but I am going to leave it replicating for a few days before going ahead and setting up as a master
Marostegui added a comment to T434043: Switchover m1 master (db1213 -> db1164).
@jcrespo I will schedule this for Tuesday in the end, I'll do it after checking the backups table to makes sure nothing is in progress. The RO time would be around 5-10 seconds and I will merge your patch inmediately after.
Wed, Aug 5
Wed, Aug 5
Marostegui added a comment to T373436: cookbook to change a host's master.
I think if you document this in wikitech we can close this?
Marostegui updated subscribers of T433853: Create a table to store calculated properties for SuggestedInvestigations.
Marostegui added a comment to T407942: Productionize db12[65-90].
I've started replication on db1267 in ms1, this means ms1 topology is a bit strange at the moment, but I am going to leave it replicating for a few days before going ahead and setting up as a master
Marostegui updated subscribers of T434048: Decommission clouddb1013-clouddb1020.
Marostegui closed T409557: Productionize new clouddb* hosts (clouddb1022-1033), a subtask of T408692: Set up replication on new hosts clouddb102[2-5], as Resolved.
All new hosts are now serving traffic. Closing this task and will open a new one to decommission the old ones if all goes fine, in a month.
Marostegui closed T409557: Productionize new clouddb* hosts (clouddb1022-1033), a subtask of T409162: Q2:rack/setup/install clouddb1026-1033, as Resolved.
Marostegui added a comment to T434043: Switchover m1 master (db1213 -> db1164).
Ok thanks! I will adjust
Marostegui added a comment to T434043: Switchover m1 master (db1213 -> db1164).
Connectivity between proxies and db1164 has been tested.
Marostegui updated subscribers of T434043: Switchover m1 master (db1213 -> db1164).
@jcrespo could you let me know where it would be a BAD moment to do this switchover from a backups point of view so I can avoid that window?
Thanks