In T360029#9726677, @CDanis wrote:

Anyway I think that all that is needed to unblock VLAN migrations has been done or documented on this ticket? Optimistically closing but please re-open if you disagree.

Thu, Apr 18, 2:28 PM · conftool, Data-Persistence, SRE, Infrastructure-Foundations

• Marostegui created P60924 (An Untitled Masterwork).

Thu, Apr 18, 1:06 PM

• Marostegui updated the task description for T362745: Upgrade s7 to MariaDB 10.6.

Thu, Apr 18, 12:57 PM · DBA

• Marostegui updated the task description for T355609: Make cuc_id a bigint.

Thu, Apr 18, 10:27 AM · Schema-change-in-production, DBA

• Marostegui updated the task description for T356166: Drop cl_collation_ext index from categorylinks in production.

Thu, Apr 18, 9:12 AM · Schema-change-in-production, DBA

Thu, Apr 18, 6:33 AM · CheckUser, Schema-change-in-production, DBA

Thu, Apr 18, 6:32 AM · CheckUser, Schema-change-in-production, DBA

• Marostegui added a comment to T362786: Enable dbctl for parsercache.

In T362786#9723982, @Scott_French wrote:

Is the idea to start treating pc sections like everything else in dbctl or something more limited? (e.g., just managing the section to (singular) host mapping, as we do today in mediawiki-config)

Thu, Apr 18, 5:38 AM · Infrastructure-Foundations, Data-Persistence, conftool

Wed, Apr 17

• Marostegui added a comment to T360029: Integrate dbctl IP changes as part of VLAN changes. .

That's awesome!! Then I guess the cookbook to orchestrate all this can be done? Do we need something else?

Wed, Apr 17, 4:21 PM · conftool, Data-Persistence, SRE, Infrastructure-Foundations

• Marostegui added a project to T362786: Enable dbctl for parsercache: Infrastructure-Foundations.

Wed, Apr 17, 3:37 PM · Infrastructure-Foundations, Data-Persistence, conftool

• Marostegui added a comment to T362786: Enable dbctl for parsercache.

We do need to list the spares somehow somewhere because otherwise we will forget where they are (or even their hoatnames)

Wed, Apr 17, 3:37 PM · Infrastructure-Foundations, Data-Persistence, conftool

• Marostegui created T362786: Enable dbctl for parsercache.

Wed, Apr 17, 3:22 PM · Infrastructure-Foundations, Data-Persistence, conftool

• Marostegui updated the task description for T362745: Upgrade s7 to MariaDB 10.6.

Wed, Apr 17, 12:55 PM · DBA

• Marostegui updated the task description for T362745: Upgrade s7 to MariaDB 10.6.

Wed, Apr 17, 7:54 AM · DBA

• Marostegui updated the task description for T362745: Upgrade s7 to MariaDB 10.6.

Wed, Apr 17, 5:29 AM · DBA

• Marostegui updated the task description for T362745: Upgrade s7 to MariaDB 10.6.

Wed, Apr 17, 5:17 AM · DBA

• Marostegui updated the task description for T362745: Upgrade s7 to MariaDB 10.6.

Wed, Apr 17, 5:17 AM · DBA

• Marostegui updated the task description for T362745: Upgrade s7 to MariaDB 10.6.

Wed, Apr 17, 5:16 AM · DBA

• Marostegui updated the task description for T356960: Upgrade hosts to MariaDB 10.6.

Wed, Apr 17, 5:11 AM · DBA

• Marostegui assigned T362746: Upgrade s4 to MariaDB 10.6 to ABran-WMF.

Wed, Apr 17, 5:10 AM · DBA

• Marostegui triaged T362746: Upgrade s4 to MariaDB 10.6 as Medium priority.

Wed, Apr 17, 5:10 AM · DBA

• Marostegui created T362746: Upgrade s4 to MariaDB 10.6.

Wed, Apr 17, 5:10 AM · DBA

• Marostegui added a comment to T362229: Drop MachineVision tables from beta and production.

Renamed this on db1249:

./commonswiki/T362229_machine_vision_safe_search.frm
./commonswiki/T362229_machine_vision_provider.ibd
./commonswiki/T362229_machine_vision_provider.frm
./commonswiki/T362229_machine_vision_image.ibd
./commonswiki/T362229_machine_vision_image.frm
./commonswiki/T362229_machine_vision_label.ibd
./commonswiki/T362229_machine_vision_label.frm
./commonswiki/T362229_machine_vision_suggestion.ibd
./commonswiki/T362229_machine_vision_suggestion.frm
./commonswiki/T362229_machine_vision_freebase_mapping.ibd
./commonswiki/T362229_machine_vision_freebase_mapping.frm
./commonswiki/T362229_machine_vision_safe_search.ibd
./testcommonswiki/T362229_machine_vision_suggestion.ibd
./testcommonswiki/T362229_machine_vision_suggestion.frm
./testcommonswiki/T362229_machine_vision_freebase_mapping.ibd
./testcommonswiki/T362229_machine_vision_provider.ibd
./testcommonswiki/T362229_machine_vision_provider.frm
./testcommonswiki/T362229_machine_vision_image.ibd
./testcommonswiki/T362229_machine_vision_image.frm
./testcommonswiki/T362229_machine_vision_label.ibd
./testcommonswiki/T362229_machine_vision_label.frm
./testcommonswiki/T362229_machine_vision_freebase_mapping.frm
./testcommonswiki/T362229_machine_vision_safe_search.ibd
./testcommonswiki/T362229_machine_vision_safe_search.frm

Wed, Apr 17, 5:04 AM · DBA, Structured-Data-Backlog, MachineVision

• Marostegui updated the task description for T362745: Upgrade s7 to MariaDB 10.6.

Wed, Apr 17, 4:57 AM · DBA

• Marostegui added a comment to T360751: Upgrade backup sources to MariaDB 10.6.

@jcrespo I am starting to work with s7 codfw

Wed, Apr 17, 4:56 AM · Patch-For-Review, Data-Persistence, Data-Persistence-Backup

• Marostegui added a subtask for T356960: Upgrade hosts to MariaDB 10.6: T362745: Upgrade s7 to MariaDB 10.6.

Wed, Apr 17, 4:55 AM · DBA

• Marostegui added a parent task for T362745: Upgrade s7 to MariaDB 10.6: T356960: Upgrade hosts to MariaDB 10.6.

Wed, Apr 17, 4:55 AM · DBA

• Marostegui updated the task description for T356960: Upgrade hosts to MariaDB 10.6.

Wed, Apr 17, 4:55 AM · DBA

• Marostegui triaged T362745: Upgrade s7 to MariaDB 10.6 as Medium priority.

Wed, Apr 17, 4:53 AM · DBA

• Marostegui created T362745: Upgrade s7 to MariaDB 10.6.

Wed, Apr 17, 4:52 AM · DBA

• Marostegui added a comment to T361584: decommission db2100.codfw.wmnet.

This host wasn't removed from zarcillo - I have done so.

Wed, Apr 17, 4:50 AM · SRE, ops-codfw, Patch-For-Review, DBA, decommission-hardware

Wed, Apr 17, 4:45 AM · CheckUser, Schema-change-in-production, DBA

Wed, Apr 17, 4:39 AM · CheckUser, Schema-change-in-production, DBA

Tue, Apr 16

• Marostegui updated the task description for T362134: Upgrade s3 to MariaDB 10.6.

Tue, Apr 16, 3:49 PM · DBA

• Marostegui added a comment to T360332: Make the cupe_actor column nullable on WMF wikis.

In T360332#9711388, @Marostegui wrote:

@ABran-WMF I switched s2 master a few days ago (T362036) - check if you need/can run this schema change on the OLD master.

Tue, Apr 16, 3:15 PM · DBA, Schema-change-in-production

• Marostegui closed T362616: Switchover s3 master (db2127 -> db2205) as Resolved.

All done

Tue, Apr 16, 3:13 PM · Patch-For-Review, DBA

• Marostegui added a comment to T360751: Upgrade backup sources to MariaDB 10.6.

s3 codfw master switched.

Tue, Apr 16, 3:13 PM · Patch-For-Review, Data-Persistence, Data-Persistence-Backup

• Marostegui closed T362616: Switchover s3 master (db2127 -> db2205), a subtask of T362134: Upgrade s3 to MariaDB 10.6, as Resolved.

Tue, Apr 16, 3:11 PM · DBA

• Marostegui updated the task description for T362616: Switchover s3 master (db2127 -> db2205).

Tue, Apr 16, 3:11 PM · Patch-For-Review, DBA

• Marostegui updated the task description for T362616: Switchover s3 master (db2127 -> db2205).

Tue, Apr 16, 3:08 PM · Patch-For-Review, DBA

• Marostegui updated the task description for T362616: Switchover s3 master (db2127 -> db2205).

Tue, Apr 16, 2:51 PM · Patch-For-Review, DBA

• Marostegui triaged T362616: Switchover s3 master (db2127 -> db2205) as Medium priority.

Tue, Apr 16, 2:50 PM · Patch-For-Review, DBA

• Marostegui claimed T362616: Switchover s3 master (db2127 -> db2205).

Tue, Apr 16, 2:48 PM · Patch-For-Review, DBA

Tue, Apr 16, 1:06 PM · CheckUser, Schema-change-in-production, DBA

• Marostegui added a comment to T360149: Create a database for Striker test instance.

In T360149#9716836, @ABran-WMF wrote:

@taavi would you be OK to drop & recreate that db so we discard the underscore?

Tue, Apr 16, 9:07 AM · Data-Persistence-Backup, Patch-For-Review, DBA, cloud-services-team, Striker

• Marostegui added a parent task for T362616: Switchover s3 master (db2127 -> db2205): T362134: Upgrade s3 to MariaDB 10.6.

Tue, Apr 16, 8:49 AM · Patch-For-Review, DBA

• Marostegui added a subtask for T362134: Upgrade s3 to MariaDB 10.6: T362616: Switchover s3 master (db2127 -> db2205).

Tue, Apr 16, 8:49 AM · DBA

• Marostegui updated the task description for T362134: Upgrade s3 to MariaDB 10.6.

Tue, Apr 16, 8:48 AM · DBA

• Marostegui added a comment to T360751: Upgrade backup sources to MariaDB 10.6.

Thanks, s3 codfw is almost done. Only pending the master which I'll try to do this week

Tue, Apr 16, 8:14 AM · Patch-For-Review, Data-Persistence, Data-Persistence-Backup

• Marostegui updated the task description for T362134: Upgrade s3 to MariaDB 10.6.

Tue, Apr 16, 5:41 AM · DBA

• Marostegui updated the task description for T362134: Upgrade s3 to MariaDB 10.6.

Tue, Apr 16, 5:25 AM · DBA

Tue, Apr 16, 5:20 AM · CheckUser, Schema-change-in-production, DBA

• Marostegui moved T361627: Create cuc_agent_id, cule_agent_id and cupe_agent_id columns in cu_changes, cu_log_event and cu_private_event tables respectively on WMF wikis from Ready to In progress on the DBA board.

Tue, Apr 16, 5:12 AM · CheckUser, Schema-change-in-production, DBA

• Marostegui claimed T361627: Create cuc_agent_id, cule_agent_id and cupe_agent_id columns in cu_changes, cu_log_event and cu_private_event tables respectively on WMF wikis.

Tue, Apr 16, 5:07 AM · CheckUser, Schema-change-in-production, DBA

• Marostegui updated the task description for T356166: Drop cl_collation_ext index from categorylinks in production.

Tue, Apr 16, 5:05 AM · Schema-change-in-production, DBA

Mon, Apr 15

• Marostegui updated the task description for T356166: Drop cl_collation_ext index from categorylinks in production.

Mon, Apr 15, 5:29 AM · Schema-change-in-production, DBA

• Marostegui updated the task description for T356166: Drop cl_collation_ext index from categorylinks in production.

Mon, Apr 15, 5:25 AM · Schema-change-in-production, DBA

Sun, Apr 14

• Marostegui updated the task description for T362482: x2 codfw master (db2144) TCP errors.

Sun, Apr 14, 12:23 PM · Patch-For-Review, netops, Infrastructure-Foundations, DBA

• Marostegui added a member for WMF-NDA: VRiley-WMF.

Sun, Apr 14, 11:59 AM

• Marostegui closed T362482: x2 codfw master (db2144) TCP errors as Resolved.

Per the irc chat, closing this as fixed

Sun, Apr 14, 11:48 AM · Patch-For-Review, netops, Infrastructure-Foundations, DBA

• Marostegui added a comment to T362482: x2 codfw master (db2144) TCP errors.

Thank you @cmooney! Up to you if you want to keep this task open for that or you want to open an specific one for that operation

Sun, Apr 14, 11:46 AM · Patch-For-Review, netops, Infrastructure-Foundations, DBA

• Marostegui added a comment to T362482: x2 codfw master (db2144) TCP errors.

The new master is showing the same issue - from IRC:

13:38:30 <topranks> I think it could be on a network link in eqiad 
13:38:39 <topranks> showing errors on that side - sorry missed it at first 
13:39:05 <topranks> errors have increased - same shape as the retransmits 
13:39:46 <claime> db2142 is showing the same retransmits now
13:39:52 <topranks> yeah ok 
13:40:00 <topranks> I'm depooling the Aerlion transport link

Sun, Apr 14, 11:42 AM · Patch-For-Review, netops, Infrastructure-Foundations, DBA

• Marostegui added a comment to T362482: x2 codfw master (db2144) TCP errors.

The reboot didn't work, going to start to switch the master

Sun, Apr 14, 11:33 AM · Patch-For-Review, netops, Infrastructure-Foundations, DBA

• Marostegui added a comment to T362482: x2 codfw master (db2144) TCP errors.

For now I have rebooted x2 master to see if it comes back in a better state (as doing a switch is really painful on x2)

Sun, Apr 14, 11:29 AM · Patch-For-Review, netops, Infrastructure-Foundations, DBA

• Marostegui created T362482: x2 codfw master (db2144) TCP errors.

Sun, Apr 14, 11:25 AM · Patch-For-Review, netops, Infrastructure-Foundations, DBA

Sat, Apr 13

• Marostegui added a comment to T360332: Make the cupe_actor column nullable on WMF wikis.

@ABran-WMF I switched s2 master a few days ago (T362036) - check if you need/can run this schema change on the OLD master.

Sat, Apr 13, 6:09 AM · DBA, Schema-change-in-production

Fri, Apr 12

• Marostegui updated the task description for T362134: Upgrade s3 to MariaDB 10.6.

Fri, Apr 12, 5:53 AM · DBA

• Marostegui added a comment to T360751: Upgrade backup sources to MariaDB 10.6.

@jcrespo eqiad s3 is almost done T362134

Fri, Apr 12, 5:42 AM · Patch-For-Review, Data-Persistence, Data-Persistence-Backup

• Marostegui updated the task description for T362134: Upgrade s3 to MariaDB 10.6.

Fri, Apr 12, 5:42 AM · DBA

• Marostegui updated the task description for T362134: Upgrade s3 to MariaDB 10.6.

Fri, Apr 12, 5:42 AM · DBA

• Marostegui moved T362229: Drop MachineVision tables from beta and production from Ready to In progress on the DBA board.

Fri, Apr 12, 5:28 AM · DBA, Structured-Data-Backlog, MachineVision

• Marostegui added a comment to T362229: Drop MachineVision tables from beta and production.

They've not been written since March 27th:

root@db1238:/srv# find . -name machine*.ibd | xargs ls -lh
-rw-rw---- 1 mysql mysql  52M May 30  2020 ./sqldata/commonswiki/machine_vision_freebase_mapping.ibd
-rw-rw---- 1 mysql mysql 3.1G Mar 27 14:16 ./sqldata/commonswiki/machine_vision_image.ibd
-rw-rw---- 1 mysql mysql  13G Mar 27 14:15 ./sqldata/commonswiki/machine_vision_label.ibd
-rw-rw---- 1 mysql mysql  64K May 30  2020 ./sqldata/commonswiki/machine_vision_provider.ibd
-rw-rw---- 1 mysql mysql 728M Mar 27 14:11 ./sqldata/commonswiki/machine_vision_safe_search.ibd
-rw-rw---- 1 mysql mysql 6.1G Mar 27 14:12 ./sqldata/commonswiki/machine_vision_suggestion.ibd
-rw-rw---- 1 mysql mysql  52M Jun  1  2020 ./sqldata/testcommonswiki/machine_vision_freebase_mapping.ibd
-rw-rw---- 1 mysql mysql  88K Sep  4  2022 ./sqldata/testcommonswiki/machine_vision_image.ibd
-rw-rw---- 1 mysql mysql 144K Sep  4  2022 ./sqldata/testcommonswiki/machine_vision_label.ibd
-rw-rw---- 1 mysql mysql  64K Jun  1  2020 ./sqldata/testcommonswiki/machine_vision_provider.ibd
-rw-rw---- 1 mysql mysql  64K Sep  4  2022 ./sqldata/testcommonswiki/machine_vision_safe_search.ibd
-rw-rw---- 1 mysql mysql  88K Sep  4  2022 ./sqldata/testcommonswiki/machine_vision_suggestion.ibd

Fri, Apr 12, 5:23 AM · DBA, Structured-Data-Backlog, MachineVision

• Marostegui claimed T362229: Drop MachineVision tables from beta and production.

Fri, Apr 12, 5:22 AM · DBA, Structured-Data-Backlog, MachineVision