Page MenuHomePhabricator
Feed Advanced Search

Mon, Jun 10

Marostegui updated the task description for T367017: Switchover s4 master (db2140 -> db2179).
Mon, Jun 10, 6:42 AM · DBA
Marostegui updated the task description for T367017: Switchover s4 master (db2140 -> db2179).
Mon, Jun 10, 6:38 AM · DBA
Marostegui updated the task description for T367017: Switchover s4 master (db2140 -> db2179).
Mon, Jun 10, 6:36 AM · DBA
Marostegui added a comment to T367017: Switchover s4 master (db2140 -> db2179).

Old: 100
API: 300

Mon, Jun 10, 6:19 AM · DBA
Marostegui updated the task description for T364299: Make rc_id a bigint.
Mon, Jun 10, 6:18 AM · Schema-change-in-production, DBA
Marostegui updated subscribers of T366414: Wikidata restbase requests for HTML content returning 500 for many (newer?) pages/revisions.
Mon, Jun 10, 6:17 AM · Data-Persistence, RESTBase-Cassandra, Wikidata, RESTBase
Marostegui updated the task description for T367017: Switchover s4 master (db2140 -> db2179).
Mon, Jun 10, 5:45 AM · DBA
Marostegui claimed T367017: Switchover s4 master (db2140 -> db2179).
Mon, Jun 10, 5:43 AM · DBA
Marostegui updated the task description for T367017: Switchover s4 master (db2140 -> db2179).
Mon, Jun 10, 5:42 AM · DBA
Marostegui updated the task description for T364069: Rebuild pagelinks tables.
Mon, Jun 10, 5:13 AM · DBA
Marostegui closed T366875: Switchover s7 master (db2218 -> db2121) as Resolved.

This is completed

Mon, Jun 10, 5:10 AM · DBA
Marostegui updated the task description for T366875: Switchover s7 master (db2218 -> db2121).
Mon, Jun 10, 5:10 AM · DBA
Marostegui updated the task description for T366875: Switchover s7 master (db2218 -> db2121).
Mon, Jun 10, 5:06 AM · DBA
Marostegui added a comment to T365568: Drop flaggedpage_pending from production.

I have renamed the table entirely on s5 dewiki

Mon, Jun 10, 4:47 AM · DBA
Marostegui updated the task description for T366875: Switchover s7 master (db2218 -> db2121).
Mon, Jun 10, 4:38 AM · DBA
Marostegui updated the task description for T366875: Switchover s7 master (db2218 -> db2121).
Mon, Jun 10, 4:38 AM · DBA
Marostegui added a comment to T366875: Switchover s7 master (db2218 -> db2121).

Old weight 300

Mon, Jun 10, 4:37 AM · DBA
Marostegui updated the task description for T364069: Rebuild pagelinks tables.
Mon, Jun 10, 4:35 AM · DBA
Marostegui updated the task description for T364299: Make rc_id a bigint.
Mon, Jun 10, 4:34 AM · Schema-change-in-production, DBA

Sat, Jun 8

Marostegui updated the task description for T364069: Rebuild pagelinks tables.
Sat, Jun 8, 1:58 PM · DBA
Marostegui updated the task description for T364069: Rebuild pagelinks tables.
Sat, Jun 8, 1:54 PM · DBA

Fri, Jun 7

Marostegui added a comment to T366354: Create the a new DB schema to store the worklist data.

@Ladsgroup could you review this request? Thanks!
@cmelo a quick glance shows that you are not having any single PK on any of the tables, is that correct? Or should we assume cew_id cewa_id cewuba_id would be PK for each table?
If you could provide the full "show create table" for each table, that would be clearer to review.

Fri, Jun 7, 1:05 PM · MW-1.43-notes (1.43.0-wmf.13; 2024-07-09), DBA, Campaign-Tools (Campaign-Tools-Current-Sprint), Event-Discovery
Marostegui closed T365338: MariaDB 10.6.18 released as Resolved.

This was pushed to the repo

Fri, Jun 7, 12:24 PM · DBA
Marostegui updated the task description for T366875: Switchover s7 master (db2218 -> db2121).
Fri, Jun 7, 6:23 AM · DBA
Marostegui claimed T366875: Switchover s7 master (db2218 -> db2121).
Fri, Jun 7, 6:23 AM · DBA
Marostegui updated the task description for T366875: Switchover s7 master (db2218 -> db2121).
Fri, Jun 7, 6:23 AM · DBA
Marostegui added a comment to T365568: Drop flaggedpage_pending from production.

Done this for now:

cumin2024@db1185.eqiad.wmnet[dewiki]> rename table flaggedpage_pending to T365568_flaggedpage_pending;
Query OK, 0 rows affected (0.003 sec)

cumin2024@db1185.eqiad.wmnet[dewiki]>
Fri, Jun 7, 6:00 AM · DBA
Marostegui updated the task description for T365568: Drop flaggedpage_pending from production.
Fri, Jun 7, 5:59 AM · DBA
Marostegui moved T366354: Create the a new DB schema to store the worklist data from Pending comment to Blocked on the DBA board.
Fri, Jun 7, 5:57 AM · MW-1.43-notes (1.43.0-wmf.13; 2024-07-09), DBA, Campaign-Tools (Campaign-Tools-Current-Sprint), Event-Discovery
Marostegui added a comment to T365356: Document new parsercache failover process.

Thank you so much Scott.
Arnaud, good idea! Let's review things on that other task.

Fri, Jun 7, 5:10 AM · DBA
Marostegui added a comment to T366354: Create the a new DB schema to store the worklist data.

@cmelo is this task meant to be for DBAS? There is no mention to the storage you want to use, is this request for a MariaDB database in our misc sections? If so please follow: https://wikitech.wikimedia.org/wiki/MariaDB#Database_creation_template

Fri, Jun 7, 4:50 AM · MW-1.43-notes (1.43.0-wmf.13; 2024-07-09), DBA, Campaign-Tools (Campaign-Tools-Current-Sprint), Event-Discovery
Marostegui updated the task description for T364299: Make rc_id a bigint.
Fri, Jun 7, 4:44 AM · Schema-change-in-production, DBA
Marostegui updated the task description for T364299: Make rc_id a bigint.
Fri, Jun 7, 4:43 AM · Schema-change-in-production, DBA

Jun 5 2024

Ladsgroup awarded T366699: Test a different InnoDB compression algorithm a Love token.
Jun 5 2024, 12:56 PM · DBA
Marostegui added a parent task for T366699: Test a different InnoDB compression algorithm: T365805: Test MariaDB 10.11.
Jun 5 2024, 12:51 PM · DBA
Marostegui added a subtask for T365805: Test MariaDB 10.11: T366699: Test a different InnoDB compression algorithm.
Jun 5 2024, 12:51 PM · DBA
Marostegui triaged T366699: Test a different InnoDB compression algorithm as Low priority.
Jun 5 2024, 12:51 PM · DBA
Marostegui created T366699: Test a different InnoDB compression algorithm.
Jun 5 2024, 12:51 PM · DBA
Marostegui updated subscribers of T365356: Document new parsercache failover process.

This looks good to me - thank you.
@ABran-WMF as a person who's not dealt with parsercache much (yet) - can you check https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting and see if with that documentation you'd be able to swap a dead parsercache host or you are missing some bits.

Jun 5 2024, 11:14 AM · DBA
Marostegui updated the task description for T366687: Switchover s2 master (db1222 -> db1162).
Jun 5 2024, 11:01 AM · DBA
Marostegui awarded T239378: Disable parent task metadata by default for new sub tasks a Like token.
Jun 5 2024, 10:59 AM · Patch-For-Review, User-brennen, Release-Engineering-Team, Phabricator, Developer Productivity
Marostegui triaged T366687: Switchover s2 master (db1222 -> db1162) as Medium priority.
Jun 5 2024, 10:38 AM · DBA
Marostegui added a comment to T366687: Switchover s2 master (db1222 -> db1162).

I will get this done on Tuesday next week

Jun 5 2024, 10:37 AM · DBA
Marostegui closed T366677: Reimage x2 eqiad master as Resolved.

This is done

Jun 5 2024, 10:14 AM · DBA
Marostegui updated the task description for T366677: Reimage x2 eqiad master.
Jun 5 2024, 10:11 AM · DBA
Marostegui closed T365098: Add external store templates to SwitchMaster tool as Resolved.

Thank you Amir - this is great to have! Thanks for working on it!

Jun 5 2024, 10:04 AM · DBA
Marostegui closed T366682: Switchover es7 master (es1035 -> es1039) as Declined.

This was a test

Jun 5 2024, 10:04 AM · DBA
Marostegui added a comment to T365098: Add external store templates to SwitchMaster tool.

DNS merged

Jun 5 2024, 9:52 AM · DBA
Marostegui updated the task description for T366677: Reimage x2 eqiad master.
Jun 5 2024, 9:38 AM · DBA
Marostegui triaged T366677: Reimage x2 eqiad master as Medium priority.
Jun 5 2024, 9:34 AM · DBA
Marostegui created T366677: Reimage x2 eqiad master.
Jun 5 2024, 9:33 AM · DBA
Marostegui added a comment to T365503: Upgrade mariadb on analytics_meta from 10.4 to 10.6.

@brouberol do you use replication?

show slave status\G
Jun 5 2024, 8:32 AM · Data-Platform-SRE (2024.06.17 - 2024.07.07)
Marostegui updated the task description for T364299: Make rc_id a bigint.
Jun 5 2024, 8:30 AM · Schema-change-in-production, DBA
Marostegui closed T366038: Switchover s2 master (db2207 -> db2204) as Resolved.

Done

Jun 5 2024, 7:29 AM · DBA
Marostegui updated the task description for T366038: Switchover s2 master (db2207 -> db2204).
Jun 5 2024, 7:28 AM · DBA
Marostegui updated the task description for T366038: Switchover s2 master (db2207 -> db2204).
Jun 5 2024, 7:25 AM · DBA
Marostegui updated the task description for T366038: Switchover s2 master (db2207 -> db2204).
Jun 5 2024, 7:24 AM · DBA
Marostegui updated the task description for T366038: Switchover s2 master (db2207 -> db2204).
Jun 5 2024, 7:13 AM · DBA
Marostegui updated the task description for T366038: Switchover s2 master (db2207 -> db2204).
Jun 5 2024, 7:10 AM · DBA
Marostegui added a comment to T201662: Change the MySQL passwords.

Yes, I think we can. Let's do it in a controlled way just in case. Let's drop all codfw and we can check in a few hours the event_log and make sure events are still being killed, if so, we can drop eqiad.

Jun 5 2024, 5:11 AM · SecTeam-Processed, Security, DBA
Marostegui added a comment to T365098: Add external store templates to SwitchMaster tool.

So far looks okay but we need to set candidate masters in puppet as comments. None has it.

Jun 5 2024, 5:09 AM · DBA

Jun 4 2024

Marostegui closed T366552: Switchover s1 master (db2203 -> db2212) as Resolved.

The old master is being repooled now automatically. Closing this.

Jun 4 2024, 2:52 PM · DBA
Marostegui closed T366552: Switchover s1 master (db2203 -> db2212), a subtask of T364299: Make rc_id a bigint, as Resolved.
Jun 4 2024, 2:52 PM · Schema-change-in-production, DBA
Marostegui updated the task description for T366552: Switchover s1 master (db2203 -> db2212).
Jun 4 2024, 2:51 PM · DBA
Marostegui added a comment to T366552: Switchover s1 master (db2203 -> db2212).

Leaving this ticket opened as a reminder to move the backup source to the new master

Jun 4 2024, 2:51 PM · DBA
Marostegui updated the task description for T364299: Make rc_id a bigint.
Jun 4 2024, 2:33 PM · Schema-change-in-production, DBA
Marostegui updated the task description for T364299: Make rc_id a bigint.
Jun 4 2024, 2:30 PM · Schema-change-in-production, DBA
Marostegui added a comment to T201662: Change the MySQL passwords.

Yes, the ops database is where the events are installed and that is not being touched.

Jun 4 2024, 2:29 PM · SecTeam-Processed, Security, DBA
Marostegui added a comment to T201662: Change the MySQL passwords.

Yes, I can see events being killed on hosts which don't have ops user.

cumin2024@db1163.eqiad.wmnet[ops]> select stamp from event_log order by stamp desc limit 1;
+---------------------+
| stamp               |
+---------------------+
| 2024-06-04 04:54:33 |
+---------------------+
1 row in set (0.001 sec)
Jun 4 2024, 12:58 PM · SecTeam-Processed, Security, DBA
Marostegui added a comment to T365503: Upgrade mariadb on analytics_meta from 10.4 to 10.6.

@brouberol first you need to stop mariadb, then remove wmf-mariadb104 and then install wmf-mariadb106 via puppet (or however it is handling from your end).
Once done, start mariadb again and don't forget to run: mysql_upgrade

Jun 4 2024, 12:56 PM · Data-Platform-SRE (2024.06.17 - 2024.07.07)
Marostegui added a comment to T201662: Change the MySQL passwords.

@Ladsgroup can you give me a lost of hosts without that user?

Jun 4 2024, 10:51 AM · SecTeam-Processed, Security, DBA
Marostegui added a comment to T201662: Change the MySQL passwords.

If I remember correctly the ops user was migrated to root (I haven't checked but I recall something like that). The database is still ops but I believe the definer was migrated to root. Otherwise, if the user is missing in so many hosts, that's definitely something we need to look at.

Jun 4 2024, 10:51 AM · SecTeam-Processed, Security, DBA
Marostegui added a comment to T366552: Switchover s1 master (db2203 -> db2212).

Leaving this ticket opened as a reminder to move the backup source to the new master

Jun 4 2024, 10:44 AM · DBA
Marostegui updated the task description for T366552: Switchover s1 master (db2203 -> db2212).
Jun 4 2024, 10:44 AM · DBA
Marostegui updated the task description for T366552: Switchover s1 master (db2203 -> db2212).
Jun 4 2024, 10:43 AM · DBA
Marostegui added a comment to T366552: Switchover s1 master (db2203 -> db2212).

Topology is moved but db2141:3311 remains unchanged as the backups are running. Will move it once it is finished and in sync

Jun 4 2024, 10:41 AM · DBA
Marostegui added a comment to T356166: Drop cl_collation_ext index from categorylinks in production.

All done

Jun 4 2024, 10:07 AM · Schema-change-in-production, DBA
Marostegui closed T356166: Drop cl_collation_ext index from categorylinks in production, a subtask of T342697: Audit of unused indexes, 2023, as Resolved.
Jun 4 2024, 10:07 AM · MW-1.42-notes (1.42.0-wmf.17; 2024-02-06), MediaWiki-Platform-Team (Radar), DBA
Marostegui closed T356166: Drop cl_collation_ext index from categorylinks in production as Resolved.
Jun 4 2024, 10:07 AM · Schema-change-in-production, DBA
Marostegui added a comment to T366552: Switchover s1 master (db2203 -> db2212).

Old weight 400

Jun 4 2024, 10:01 AM · DBA
Marostegui updated the task description for T366552: Switchover s1 master (db2203 -> db2212).
Jun 4 2024, 10:01 AM · DBA
Marostegui updated the task description for T364299: Make rc_id a bigint.
Jun 4 2024, 8:32 AM · Schema-change-in-production, DBA
Marostegui closed T355609: Make cuc_id a bigint, a subtask of T63111: Convert primary key integers and references thereto from int to bigint (unsigned), as Resolved.
Jun 4 2024, 7:27 AM · MW-1.43-notes (1.43.0-wmf.4; 2024-05-07), MW-1.42-notes (1.42.0-wmf.15; 2024-01-23), MediaWiki-General, Schema-change, DBA
Marostegui closed T355609: Make cuc_id a bigint as Resolved.

Done

Jun 4 2024, 7:27 AM · Schema-change-in-production, DBA
Marostegui updated the task description for T355609: Make cuc_id a bigint.
Jun 4 2024, 7:26 AM · Schema-change-in-production, DBA
Marostegui updated the task description for T364299: Make rc_id a bigint.
Jun 4 2024, 7:22 AM · Schema-change-in-production, DBA
Marostegui closed T356960: Upgrade hosts to MariaDB 10.6 as Resolved.

Every production database has been migrated to 10.6 - pending only some backup sources tracked at https://phabricator.wikimedia.org/T360751

Jun 4 2024, 7:08 AM · DBA
Marostegui updated the task description for T366552: Switchover s1 master (db2203 -> db2212).
Jun 4 2024, 6:22 AM · DBA
Marostegui updated the task description for T366552: Switchover s1 master (db2203 -> db2212).
Jun 4 2024, 6:21 AM · DBA
Marostegui claimed T366552: Switchover s1 master (db2203 -> db2212).
Jun 4 2024, 6:20 AM · DBA
Marostegui added a parent task for T366552: Switchover s1 master (db2203 -> db2212): T364299: Make rc_id a bigint.
Jun 4 2024, 6:20 AM · DBA
Marostegui added a subtask for T364299: Make rc_id a bigint: T366552: Switchover s1 master (db2203 -> db2212).
Jun 4 2024, 6:20 AM · Schema-change-in-production, DBA
Marostegui changed the status of T365568: Drop flaggedpage_pending from production from Stalled to Open.

Done this for now:

cumin2024@db1185.eqiad.wmnet[dewiki]> rename table flaggedpage_pending to T365568_flaggedpage_pending;
Query OK, 0 rows affected (0.003 sec)
Jun 4 2024, 6:15 AM · DBA
Marostegui claimed T365568: Drop flaggedpage_pending from production.
Jun 4 2024, 6:13 AM · DBA
Marostegui added a comment to T365356: Document new parsercache failover process.

Copying over from T362786#9806963, spots to update include:

  1. (definitely) https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting - update instructions for depooling a pc instance
  2. (definitely) https://wikitech.wikimedia.org/wiki/Dbctl - clarify "add a section" instructions to include the pc section case
  3. (maybe) https://wikitech.wikimedia.org/wiki/Parser_cache - note that the configuration uses etcd as a source of truth in production
Jun 4 2024, 5:47 AM · DBA

Jun 3 2024

Marostegui added a comment to T365568: Drop flaggedpage_pending from production.

Let's give it two more days just in case.

Jun 3 2024, 1:22 PM · DBA
Marostegui added a comment to T365568: Drop flaggedpage_pending from production.

@Ladsgroup is this good to go?

Jun 3 2024, 5:15 AM · DBA
Marostegui closed T366429: db1213 replication broken (Index for table dewiki.page_props is corrupt) as Resolved.
Jun 3 2024, 4:59 AM · DBA
Marostegui added a comment to T366429: db1213 replication broken (Index for table dewiki.page_props is corrupt).

Host back in sync with the master, I am going remove downtime and start repooling.

Jun 3 2024, 4:48 AM · DBA
Marostegui updated the task description for T364299: Make rc_id a bigint.
Jun 3 2024, 4:44 AM · Schema-change-in-production, DBA