Page MenuHomePhabricator
Feed Search

Thu, Dec 4

Marostegui updated the task description for T411570: Migrate database hosts from BIOS to UEFI.
Thu, Dec 4, 10:38 AM · Epic, DBA
Marostegui added a comment to T315642: Monitor GTID status.

Excellent thanks!

Thu, Dec 4, 9:55 AM · DBA
Marostegui added a comment to T407472: Install a testing db with Debian Trixie.

db1169 is in s1

Thu, Dec 4, 6:30 AM · DBA
Marostegui triaged T411652: db1229 crashed - Broken memory module at B7 as Medium priority.
Thu, Dec 4, 6:08 AM · SRE, DC-Ops, ops-eqiad, DBA
Marostegui updated the task description for T409162: Q2:rack/setup/install clouddb1026-1033.
Thu, Dec 4, 5:59 AM · ops-eqiad, cloud-services-team (Hardware), DC-Ops, SRE
Marostegui updated the task description for T411164: Drop rev_sha1 from revision table in wmf production.
Thu, Dec 4, 5:53 AM · Data-Engineering, Schema-change-in-production, DBA
Marostegui updated the task description for T411163: Drop ar_sha1 from archive table in wmf production.
Thu, Dec 4, 5:53 AM · Data-Engineering, Schema-change-in-production, DBA

Wed, Dec 3

Marostegui updated the task description for T411570: Migrate database hosts from BIOS to UEFI.
Wed, Dec 3, 3:07 PM · Epic, DBA
Marostegui updated the task description for T411164: Drop rev_sha1 from revision table in wmf production.
Wed, Dec 3, 3:06 PM · Data-Engineering, Schema-change-in-production, DBA
Marostegui updated the task description for T411163: Drop ar_sha1 from archive table in wmf production.
Wed, Dec 3, 3:06 PM · Data-Engineering, Schema-change-in-production, DBA
Marostegui closed T411527: Remove sockpuppet database as Resolved.

Thanks! Closing this!

Wed, Dec 3, 10:30 AM · database-backups, Data-Persistence-Backup, DBA, Data-Persistence
Marostegui closed T411498: Reclone db1169 (s1) as Resolved.

Done

Wed, Dec 3, 7:26 AM · DBA
Marostegui closed T411498: Reclone db1169 (s1), a subtask of T410400: Support UEFI on databases instead of Legacy BIOS, as Resolved.
Wed, Dec 3, 7:26 AM · Data-Persistence, Infrastructure-Foundations
Marostegui updated the task description for T411570: Migrate database hosts from BIOS to UEFI.
Wed, Dec 3, 7:03 AM · Epic, DBA
Marostegui added a comment to T360029: Integrate dbctl IP changes as part of VLAN changes. .

Thanks for the heads-up, @Marostegui.

Wed, Dec 3, 6:49 AM · conftool, Data-Persistence, SRE, Infrastructure-Foundations
Marostegui added a subtask for T383674: Abstract away different database depooling mechanisms into a cookbook: T411573: sre.mysql.parsercache: make it work with msX sections.
Wed, Dec 3, 6:23 AM · Patch-For-Review, DBA
Marostegui added a parent task for T411573: sre.mysql.parsercache: make it work with msX sections: T383674: Abstract away different database depooling mechanisms into a cookbook.
Wed, Dec 3, 6:23 AM · Patch-For-Review, DBA
Marostegui added a comment to T411573: sre.mysql.parsercache: make it work with msX sections.

Yeah, I think this should go under the wrapper discussed at: https://phabricator.wikimedia.org/T383674

Wed, Dec 3, 6:23 AM · Patch-For-Review, DBA
Marostegui triaged T411573: sre.mysql.parsercache: make it work with msX sections as Medium priority.
Wed, Dec 3, 6:20 AM · Patch-For-Review, DBA
Marostegui created T411573: sre.mysql.parsercache: make it work with msX sections.
Wed, Dec 3, 6:20 AM · Patch-For-Review, DBA
Marostegui added a comment to T411497: Drop modtoken and flags from cache tables.

Of course this needs depooling for each alter as the table is around 50G

Wed, Dec 3, 6:13 AM · Schema-change-in-production, DBA, Data-Engineering
Marostegui added a comment to T411498: Reclone db1169 (s1).

I am reimagining this host with Trixie, which was the original point of the task, which ended up with all the nokia/uefi things

Wed, Dec 3, 6:02 AM · DBA
Marostegui moved T411497: Drop modtoken and flags from cache tables from Triage to In progress on the DBA board.
Wed, Dec 3, 5:59 AM · Schema-change-in-production, DBA, Data-Engineering
Marostegui added a comment to T411497: Drop modtoken and flags from cache tables.

For the ALTER I will just proceed on msX for now.

Wed, Dec 3, 5:59 AM · Schema-change-in-production, DBA, Data-Engineering
Marostegui added a comment to T411497: Drop modtoken and flags from cache tables.

Thank you both. After @Krinkle's message above, I am not sure whether I can proceed deleting them (on both parsercache and ms*) or I should wait until the code is cleaned up?

Wed, Dec 3, 5:58 AM · Schema-change-in-production, DBA, Data-Engineering
Marostegui added a comment to T411527: Remove sockpuppet database.

wikitech mariadb documentation updated.

Wed, Dec 3, 5:34 AM · database-backups, Data-Persistence-Backup, DBA, Data-Persistence
Marostegui added a comment to T411527: Remove sockpuppet database.

The database has been removed.
Grants have been removed

Wed, Dec 3, 5:32 AM · database-backups, Data-Persistence-Backup, DBA, Data-Persistence
Marostegui moved T411527: Remove sockpuppet database from Triage to In progress on the DBA board.
Wed, Dec 3, 5:30 AM · database-backups, Data-Persistence-Backup, DBA, Data-Persistence
Marostegui claimed T411527: Remove sockpuppet database.
Wed, Dec 3, 5:28 AM · database-backups, Data-Persistence-Backup, DBA, Data-Persistence
Marostegui updated the task description for T407991: Q2:rack/setup/install db2249.
Wed, Dec 3, 5:17 AM · ops-codfw, SRE, Data-Persistence, DC-Ops
Marostegui triaged T411570: Migrate database hosts from BIOS to UEFI as Medium priority.
Wed, Dec 3, 5:14 AM · Epic, DBA
Marostegui updated the task description for T411570: Migrate database hosts from BIOS to UEFI.
Wed, Dec 3, 5:14 AM · Epic, DBA
Marostegui created T411570: Migrate database hosts from BIOS to UEFI.
Wed, Dec 3, 5:12 AM · Epic, DBA
Marostegui updated the task description for T411164: Drop rev_sha1 from revision table in wmf production.
Wed, Dec 3, 4:56 AM · Data-Engineering, Schema-change-in-production, DBA
Marostegui updated the task description for T411163: Drop ar_sha1 from archive table in wmf production.
Wed, Dec 3, 4:56 AM · Data-Engineering, Schema-change-in-production, DBA

Tue, Dec 2

Marostegui added a comment to T391581: Accept both FQDN and bare hostname in DB cookbooks.

I thought that was done when it was mentioned at: https://phabricator.wikimedia.org/T391581#11319934

Tue, Dec 2, 5:43 PM · Patch-For-Review, DBA
Marostegui added a comment to T411497: Drop modtoken and flags from cache tables.

@Ladsgroup I am confused, the patch you linked talks about objectcache table but ms sections do not have that, they have objectstash instead (which does have modtoken), I guess it is a name error?

Tue, Dec 2, 2:30 PM · Schema-change-in-production, DBA, Data-Engineering
Marostegui added a comment to T411497: Drop modtoken and flags from cache tables.

Yeah they aren't written since 2022 from what I can see

Tue, Dec 2, 2:27 PM · Schema-change-in-production, DBA, Data-Engineering
Marostegui moved T411498: Reclone db1169 (s1) from Triage to In progress on the DBA board.

Recloning

Tue, Dec 2, 2:25 PM · DBA
Marostegui reopened T391581: Accept both FQDN and bare hostname in DB cookbooks as "Open".

Is the clone cookbook missing something?

Tue, Dec 2, 1:57 PM · Patch-For-Review, DBA
Marostegui renamed T411498: Reclone db1169 (s1) from Reclone db1169 to Reclone db1169 (s1).
Tue, Dec 2, 1:55 PM · DBA
Marostegui created T411498: Reclone db1169 (s1).
Tue, Dec 2, 1:54 PM · DBA
Marostegui added a comment to T410400: Support UEFI on databases instead of Legacy BIOS.

Thank you!!!!!

Tue, Dec 2, 1:47 PM · Data-Persistence, Infrastructure-Foundations
Marostegui added a comment to T411497: Drop modtoken and flags from cache tables.
$ sudo db-mysql pc1011 parsercache -e "show tables like 'objec%'";
+--------------------------------+
| Tables_in_parsercache (objec%) |
+--------------------------------+
| objectcache                    |
+--------------------------------+
Tue, Dec 2, 1:46 PM · Schema-change-in-production, DBA, Data-Engineering
Marostegui claimed T411497: Drop modtoken and flags from cache tables.

I remember seeing them in parsercache, I will check anyway.

Tue, Dec 2, 1:46 PM · Schema-change-in-production, DBA, Data-Engineering
Marostegui created P86306 (An Untitled Masterwork).
Tue, Dec 2, 12:43 PM
Marostegui updated the task description for T411163: Drop ar_sha1 from archive table in wmf production.
Tue, Dec 2, 11:36 AM · Data-Engineering, Schema-change-in-production, DBA
Marostegui updated the task description for T411164: Drop rev_sha1 from revision table in wmf production.
Tue, Dec 2, 11:36 AM · Data-Engineering, Schema-change-in-production, DBA
Marostegui updated the task description for T411164: Drop rev_sha1 from revision table in wmf production.
Tue, Dec 2, 11:35 AM · Data-Engineering, Schema-change-in-production, DBA
Marostegui updated the task description for T411163: Drop ar_sha1 from archive table in wmf production.
Tue, Dec 2, 11:35 AM · Data-Engineering, Schema-change-in-production, DBA
Marostegui added a comment to T411111: Database Creation request for requestctl.wikimedia.org.

Remember the user needs to have privileges from the proxies IPs only and the connection should be made to m2-master.eqiad.wmnet as that is rerouted via haproxy to the appropiated host.

Tue, Dec 2, 11:23 AM · DBA, Hiddenparma
Marostegui added a parent task for T411374: Broadcom Nic not supporting uefi with older firmware: T410400: Support UEFI on databases instead of Legacy BIOS.
Tue, Dec 2, 7:35 AM · Infrastructure-Foundations, SRE
Marostegui added a subtask for T410400: Support UEFI on databases instead of Legacy BIOS: T411374: Broadcom Nic not supporting uefi with older firmware.
Tue, Dec 2, 7:35 AM · Data-Persistence, Infrastructure-Foundations
Marostegui updated the task description for T411164: Drop rev_sha1 from revision table in wmf production.
Tue, Dec 2, 6:56 AM · Data-Engineering, Schema-change-in-production, DBA
Marostegui updated the task description for T411163: Drop ar_sha1 from archive table in wmf production.
Tue, Dec 2, 6:56 AM · Data-Engineering, Schema-change-in-production, DBA
Marostegui updated the task description for T411164: Drop rev_sha1 from revision table in wmf production.
Tue, Dec 2, 6:47 AM · Data-Engineering, Schema-change-in-production, DBA
Marostegui closed T410508: Auto_schema broken in HEAD as Resolved.

This seems to be fixed, everything seems to be running fine at T411163 and T411164
Thanks!

Tue, Dec 2, 6:40 AM · DBA
Marostegui updated the task description for T411164: Drop rev_sha1 from revision table in wmf production.
Tue, Dec 2, 6:31 AM · Data-Engineering, Schema-change-in-production, DBA
Marostegui updated the task description for T411164: Drop rev_sha1 from revision table in wmf production.
Tue, Dec 2, 6:31 AM · Data-Engineering, Schema-change-in-production, DBA

Mon, Dec 1

Marostegui updated subscribers of T360029: Integrate dbctl IP changes as part of VLAN changes. .

I've been talking to @ayounsi about this ticket, who could help us doing the automation/integration of dbctl and this cookbook? @Scott_French maybe as he's been working with dbctl "recently"?

Mon, Dec 1, 4:08 PM · conftool, Data-Persistence, SRE, Infrastructure-Foundations
Marostegui closed T400056: Core DB testbed on VMs as Resolved.

The issue was the same as the one at https://phabricator.wikimedia.org/T400056#11378959
The fix

Mon, Dec 1, 2:44 PM · DBA
Marostegui closed T400056: Core DB testbed on VMs, a subtask of T384810: MariaDB lifetime management system, as Resolved.
Mon, Dec 1, 2:44 PM · Patch-For-Review, DBA
Marostegui added a comment to T410084: sre.mysql.clone cookbook not using --ignore-existing.

As discussed during the meeting, we should simply inform the user for situations where the host wasn't there.
There is no situation where we want a host to be in production but not on zarcilo.
If the entry is there, it should be good, if the entry isn't there, we should create it (and let the operator know)

Mon, Dec 1, 1:46 PM · Patch-For-Review, DBA
Marostegui added a comment to T410508: Auto_schema broken in HEAD.

I have just started it, I will report back if there are errors

Mon, Dec 1, 11:55 AM · DBA
Marostegui updated the task description for T411163: Drop ar_sha1 from archive table in wmf production.
Mon, Dec 1, 11:48 AM · Data-Engineering, Schema-change-in-production, DBA
Marostegui updated the task description for T411163: Drop ar_sha1 from archive table in wmf production.
Mon, Dec 1, 11:48 AM · Data-Engineering, Schema-change-in-production, DBA
Marostegui updated the task description for T411163: Drop ar_sha1 from archive table in wmf production.
Mon, Dec 1, 11:48 AM · Data-Engineering, Schema-change-in-production, DBA
Marostegui added a comment to T410508: Auto_schema broken in HEAD.

Ok thanks! I can git pull from the latest HEAD and test with a fresh copy with T411163 and T411164

Mon, Dec 1, 8:02 AM · DBA
Marostegui closed T333118: Slow recentchanges query on 10.6 and 10.11, a subtask of T301879: Test MariaDB 10.6 on Bullseye, as Invalid.
Mon, Dec 1, 6:55 AM · DBA
Marostegui closed T333118: Slow recentchanges query on 10.6 and 10.11 as Invalid.

rc_type has been removed T410531 - so these queries are no longer valid. We should capture new slow ones and create a task for them.

Mon, Dec 1, 6:55 AM · mariadb-optimizer-bug, DBA
Marostegui closed Restricted Task, a subtask of T206103: recentchanges table indexes: tmp1, tmp2 and tmp3, as Invalid.
Mon, Dec 1, 6:54 AM · Moderator-Tools-Team, DBA, Schema-change, MediaWiki-Recent-changes, Growth-Team
Marostegui updated the task description for T411164: Drop rev_sha1 from revision table in wmf production.
Mon, Dec 1, 6:44 AM · Data-Engineering, Schema-change-in-production, DBA
Marostegui updated the task description for T411163: Drop ar_sha1 from archive table in wmf production.
Mon, Dec 1, 6:44 AM · Data-Engineering, Schema-change-in-production, DBA
Marostegui updated the task description for T411163: Drop ar_sha1 from archive table in wmf production.
Mon, Dec 1, 5:49 AM · Data-Engineering, Schema-change-in-production, DBA
Marostegui closed T410531: Drop rc_type from recentchanges in wmf production, a subtask of T74157: [Story] Use rc_source and drop rc_type, as Resolved.
Mon, Dec 1, 5:48 AM · DBA, MW-1.46-notes (1.46.0-wmf.4; 2025-11-25), MW-1.45-notes (1.45.0-wmf.25; 2025-10-28), wmde-wikidata-tech, Patch-For-Review, Moderator-Tools-Team, Story, Wikidata, MediaWiki-General, MediaWiki-Recent-changes, Technical-Debt
Marostegui closed T410531: Drop rc_type from recentchanges in wmf production as Resolved.

This is all done

Mon, Dec 1, 5:48 AM · Data-Engineering, DBA, Schema-change-in-production
Marostegui updated the task description for T410531: Drop rc_type from recentchanges in wmf production.
Mon, Dec 1, 5:48 AM · Data-Engineering, DBA, Schema-change-in-production
Marostegui updated the task description for T410531: Drop rc_type from recentchanges in wmf production.
Mon, Dec 1, 5:42 AM · Data-Engineering, DBA, Schema-change-in-production
Marostegui added a comment to T410400: Support UEFI on databases instead of Legacy BIOS.

Just fyi while I was working on R440 servers I came across issue with Broadcom nic when you switch to uefi the 10g card disappears from bios

firmware series 21.x shipped with
Legacy PXE ROM only
No UEFI PXE driver

Mon, Dec 1, 5:39 AM · Data-Persistence, Infrastructure-Foundations

Sun, Nov 30

Marostegui updated the task description for T411164: Drop rev_sha1 from revision table in wmf production.
Sun, Nov 30, 6:55 AM · Data-Engineering, Schema-change-in-production, DBA
Marostegui updated the task description for T411164: Drop rev_sha1 from revision table in wmf production.
Sun, Nov 30, 6:54 AM · Data-Engineering, Schema-change-in-production, DBA
Marostegui updated the task description for T411163: Drop ar_sha1 from archive table in wmf production.
Sun, Nov 30, 6:53 AM · Data-Engineering, Schema-change-in-production, DBA
Marostegui updated the task description for T410531: Drop rc_type from recentchanges in wmf production.
Sun, Nov 30, 6:53 AM · Data-Engineering, DBA, Schema-change-in-production
Marostegui updated the task description for T410531: Drop rc_type from recentchanges in wmf production.
Sun, Nov 30, 6:53 AM · Data-Engineering, DBA, Schema-change-in-production

Fri, Nov 28

Marostegui updated the task description for T410531: Drop rc_type from recentchanges in wmf production.
Fri, Nov 28, 1:52 PM · Data-Engineering, DBA, Schema-change-in-production
Marostegui updated the task description for T411163: Drop ar_sha1 from archive table in wmf production.
Fri, Nov 28, 7:32 AM · Data-Engineering, Schema-change-in-production, DBA
Marostegui updated the task description for T411164: Drop rev_sha1 from revision table in wmf production.
Fri, Nov 28, 6:33 AM · Data-Engineering, Schema-change-in-production, DBA
Marostegui updated the task description for T410531: Drop rc_type from recentchanges in wmf production.
Fri, Nov 28, 6:28 AM · Data-Engineering, DBA, Schema-change-in-production
Marostegui updated the task description for T410531: Drop rc_type from recentchanges in wmf production.
Fri, Nov 28, 6:27 AM · Data-Engineering, DBA, Schema-change-in-production
Marostegui updated the task description for T410531: Drop rc_type from recentchanges in wmf production.
Fri, Nov 28, 6:24 AM · Data-Engineering, DBA, Schema-change-in-production
Marostegui updated the task description for T410531: Drop rc_type from recentchanges in wmf production.
Fri, Nov 28, 6:06 AM · Data-Engineering, DBA, Schema-change-in-production
Marostegui updated the task description for T410531: Drop rc_type from recentchanges in wmf production.
Fri, Nov 28, 6:03 AM · Data-Engineering, DBA, Schema-change-in-production
Marostegui updated the task description for T410531: Drop rc_type from recentchanges in wmf production.
Fri, Nov 28, 5:43 AM · Data-Engineering, DBA, Schema-change-in-production
Marostegui updated the task description for T410531: Drop rc_type from recentchanges in wmf production.
Fri, Nov 28, 4:54 AM · Data-Engineering, DBA, Schema-change-in-production
Marostegui updated the task description for T410531: Drop rc_type from recentchanges in wmf production.
Fri, Nov 28, 4:51 AM · Data-Engineering, DBA, Schema-change-in-production
Marostegui updated the task description for T409557: Productionize new clouddb* hosts (clouddb1022-1033).
Fri, Nov 28, 4:30 AM · cloud-services-team, Data-Services, DBA
Marostegui added a comment to T409557: Productionize new clouddb* hosts (clouddb1022-1033).

@taavi @fnegri clouddb1022 and clouddb1023 (both x3 and s3) are ready to be added to the LB.

Fri, Nov 28, 4:29 AM · cloud-services-team, Data-Services, DBA
Marostegui updated the task description for T411164: Drop rev_sha1 from revision table in wmf production.
Fri, Nov 28, 4:23 AM · Data-Engineering, Schema-change-in-production, DBA
Marostegui updated the task description for T411163: Drop ar_sha1 from archive table in wmf production.
Fri, Nov 28, 4:22 AM · Data-Engineering, Schema-change-in-production, DBA

Thu, Nov 27

Marostegui added a comment to T410145: Build a way to duplicate a whole wiki from one section to another.

Another point to consider: https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Community_Affairs_Committee/Sister_Projects_Task_Force/Results_of_the_consultation_about_Wikispore_and_Wikinews#Wikinews it's very likely most if not all wikinews projects will be closed and archived which means number of closed wikis will reach ~170. Having a small archive cluster that we could move all of these wikis to would help s3 a lot.

Thu, Nov 27, 1:28 PM · DBA, MediaWiki-libs-Rdbms