Page MenuHomePhabricator
Feed Advanced Search

Thu, Jun 13

Marostegui updated the task description for T367261: Rebuild recentchanges table everywhere.
Thu, Jun 13, 4:56 AM · DBA
Marostegui closed T367146: Switchover s5 master (db1230 -> db1183), a subtask of T364069: Rebuild pagelinks tables, as Resolved.
Thu, Jun 13, 4:54 AM · DBA
Marostegui closed T367146: Switchover s5 master (db1230 -> db1183), a subtask of T364299: Make rc_id a bigint, as Resolved.
Thu, Jun 13, 4:54 AM · Schema-change-in-production, DBA
Marostegui closed T367146: Switchover s5 master (db1230 -> db1183) as Resolved.

Done

Thu, Jun 13, 4:54 AM · DBA
Marostegui updated the task description for T367146: Switchover s5 master (db1230 -> db1183).
Thu, Jun 13, 4:53 AM · DBA
Marostegui added a comment to T367146: Switchover s5 master (db1230 -> db1183).

Old 500

Thu, Jun 13, 4:52 AM · DBA
Marostegui updated the task description for T367146: Switchover s5 master (db1230 -> db1183).
Thu, Jun 13, 4:51 AM · DBA
Marostegui updated the task description for T367261: Rebuild recentchanges table everywhere.
Thu, Jun 13, 4:38 AM · DBA
Marostegui moved T367261: Rebuild recentchanges table everywhere from Ready to In progress on the DBA board.
Thu, Jun 13, 4:37 AM · DBA
Marostegui updated the task description for T367146: Switchover s5 master (db1230 -> db1183).
Thu, Jun 13, 4:34 AM · DBA
Marostegui updated the task description for T367146: Switchover s5 master (db1230 -> db1183).
Thu, Jun 13, 4:34 AM · DBA

Wed, Jun 12

Marostegui added a comment to T201662: Change the MySQL passwords.

Works for me!
Thank you all

Wed, Jun 12, 4:12 PM · SecTeam-Processed, Security, DBA
Marostegui added a comment to T366982: Simple filter restrictions on meta causes Recent Changes to time out.

Any chances the query can be written in a different way like:

cumin2024@dbstore1008.eqiad.wmnet[metawiki]> EXPLAIN SELECT      rc_id,     rc_timestamp,     rc_namespace,     rc_title,     rc_minor,     rc_bot,     rc_new,     rc_cur_id,     rc_this_oldid,     rc_last_oldid,     rc_type,     rc_source,     rc_patrolled,     rc_ip,     rc_old_len,     rc_new_len,     rc_deleted,     rc_logid,     rc_log_type,     rc_log_action,     rc_params,     rc_actor,     recentchanges_actor.actor_user AS `rc_user`,     recentchanges_actor.actor_name AS `rc_user_text`,     comment_rc_comment.comment_text AS `rc_comment_text`,     comment_rc_comment.comment_data AS `rc_comment_data`,     comment_rc_comment.comment_id AS `rc_comment_cid`,     rc_title,     rc_namespace,     wl_user,     wl_notificationtimestamp,     we_expiry,     page_latest,     (SELECT GROUP_CONCAT(ctd_name SEPARATOR ',')       FROM `change_tag`       JOIN `change_tag_def` ON (ct_tag_id = ctd_id)       WHERE ct_rc_id = rc_id) AS `ts_tags` FROM `recentchanges` STRAIGHT_JOIN `actor` `recentchanges_actor` ON (actor_id = rc_actor) LEFT JOIN `comment` `comment_rc_comment` ON (comment_id = rc_comment_id) LEFT JOIN `watchlist` ON (wl_user = 2134281 AND wl_title = rc_title AND wl_namespace = rc_namespace) LEFT JOIN `watchlist_expiry` ON (wl_id = we_item) LEFT JOIN `page` ON (rc_cur_id = page_id) WHERE actor_user IS NOT NULL AND rc_bot = 0  AND rc_type != 6  AND rc_source != 'wb'  AND rc_namespace NOT IN (1198, 1199, 866, 867) AND rc_timestamp >= '20240601170235' AND rc_new IN (0, 1) ORDER BY rc_timestamp DESC  LIMIT 50;
+------+--------------------+---------------------+--------+-----------------------------------------------------------------------------------------------------------------------+--------------+---------+---------------------------------------------------------------------------+--------+-------------+
| id   | select_type        | table               | type   | possible_keys                                                                                                         | key          | key_len | ref                                                                       | rows   | Extra       |
+------+--------------------+---------------------+--------+-----------------------------------------------------------------------------------------------------------------------+--------------+---------+---------------------------------------------------------------------------+--------+-------------+
|    1 | PRIMARY            | recentchanges       | range  | rc_timestamp,rc_name_type_patrolled_timestamp,rc_ns_actor,rc_actor,rc_namespace_title_timestamp,rc_new_name_timestamp | rc_timestamp | 14      | NULL                                                                      | 160905 | Using where |
|    1 | PRIMARY            | comment_rc_comment  | eq_ref | PRIMARY                                                                                                               | PRIMARY      | 8       | metawiki.recentchanges.rc_comment_id                                      | 1      |             |
|    1 | PRIMARY            | watchlist           | eq_ref | wl_user,wl_user_notificationtimestamp,wl_namespace_title                                                              | wl_user      | 265     | const,metawiki.recentchanges.rc_namespace,metawiki.recentchanges.rc_title | 1      |             |
|    1 | PRIMARY            | watchlist_expiry    | eq_ref | PRIMARY                                                                                                               | PRIMARY      | 4       | metawiki.watchlist.wl_id                                                  | 1      | Using where |
|    1 | PRIMARY            | page                | eq_ref | PRIMARY                                                                                                               | PRIMARY      | 4       | metawiki.recentchanges.rc_cur_id                                          | 1      |             |
|    1 | PRIMARY            | recentchanges_actor | eq_ref | PRIMARY,actor_user                                                                                                    | PRIMARY      | 8       | metawiki.recentchanges.rc_actor                                           | 1      | Using where |
|    2 | DEPENDENT SUBQUERY | change_tag          | ref    | ct_rc_tag_id,ct_tag_id_id                                                                                             | ct_rc_tag_id | 9       | metawiki.recentchanges.rc_id                                              | 1      | Using index |
|    2 | DEPENDENT SUBQUERY | change_tag_def      | eq_ref | PRIMARY                                                                                                               | PRIMARY      | 4       | metawiki.change_tag.ct_tag_id                                             | 1      |             |
+------+--------------------+---------------------+--------+-----------------------------------------------------------------------------------------------------------------------+--------------+---------+---------------------------------------------------------------------------+--------+-------------+
8 rows in set (0.003 sec)
Wed, Jun 12, 12:07 PM · DBA, mariadb-optimizer-bug, Wikimedia-Slow-DB-Query, Growth-Team, MediaWiki-Recent-changes
Marostegui added projects to T366982: Simple filter restrictions on meta causes Recent Changes to time out: Data-Persistence, mariadb-optimizer-bug.

db1227 is the only host where the optimizer is fine and the query runs perfectly fast:

Wed, Jun 12, 11:46 AM · DBA, mariadb-optimizer-bug, Wikimedia-Slow-DB-Query, Growth-Team, MediaWiki-Recent-changes
Marostegui added a comment to T363812: Setup backups for es6, es7 and archive old read only backups.

@Marostegui, in order to resolve this ticket, now that read activity I assume is lower, do you think I could get a host from es4 and es5 on both dcs depooled for a day and with exclusive usage in order to take a final, archivable, full backup of those sections? Doesn't have to happen at the same time on the 4 hosts:

Wed, Jun 12, 9:17 AM · Patch-For-Review, database-backups, Data-Persistence-Backup
Marostegui added a comment to T367162: db1240.s3 index issues.

From the logs (UTC):

Jun 11 11:51:23 db1240 mysqld[2665119]: 2024-06-11 11:51:23 7512754 [ERROR] InnoDB: We detected index corruption in an InnoDB type table. You have to dump + drop + reimport the table or, in a case of widespread corruption, dump all InnoDB tables and recreate the whole ta>
Jun 11 11:51:23 db1240 mysqld[2665119]: 2024-06-11 11:51:23 7512754 [ERROR] mysqld: Index for table 'recentchanges' is corrupt; try to repair it
Jun 11 11:51:23 db1240 mysqld[2665119]: 2024-06-11 11:51:23 7512754 [ERROR] Slave SQL: Error 'Index for table 'recentchanges' is corrupt; try to repair it' on query. Default database: 'cywiki'. Query: 'INSERT /* RecentChange::save  */ INTO `recentchanges` (rc_type,rc_min>
Jun 11 11:51:23 db1240 mysqld[2665119]: 2024-06-11 11:51:23 7512754 [Warning] Slave: Index for table 'recentchanges' is corrupt; try to repair it Error_code: 1034
Wed, Jun 12, 8:47 AM · Data-Persistence-Backup
Marostegui committed rSCHCHf8b0d6a6cbe1: optimize_recentchanges.py: Optimize this table.
optimize_recentchanges.py: Optimize this table
Wed, Jun 12, 8:15 AM
Marostegui updated the task description for T367261: Rebuild recentchanges table everywhere.
Wed, Jun 12, 8:05 AM · DBA
Marostegui updated the task description for T364069: Rebuild pagelinks tables.
Wed, Jun 12, 7:59 AM · DBA
Marostegui closed T367262: Switchover s6 master (db2214 -> db2129) as Resolved.

This is done

Wed, Jun 12, 7:17 AM · DBA
Marostegui updated the task description for T367262: Switchover s6 master (db2214 -> db2129).
Wed, Jun 12, 7:16 AM · DBA
Marostegui updated the task description for T367262: Switchover s6 master (db2214 -> db2129).
Wed, Jun 12, 7:15 AM · DBA
Marostegui added a comment to T367262: Switchover s6 master (db2214 -> db2129).

Old 400

Wed, Jun 12, 7:15 AM · DBA
Marostegui updated the task description for T367262: Switchover s6 master (db2214 -> db2129).
Wed, Jun 12, 7:15 AM · DBA
Marostegui updated the task description for T367262: Switchover s6 master (db2214 -> db2129).
Wed, Jun 12, 6:43 AM · DBA
Marostegui claimed T367262: Switchover s6 master (db2214 -> db2129).
Wed, Jun 12, 5:05 AM · DBA
Marostegui added a parent task for T367142: db1233 recentchanges index corrupted: T367261: Rebuild recentchanges table everywhere.
Wed, Jun 12, 5:03 AM · DBA
Marostegui added a parent task for T367162: db1240.s3 index issues: T367261: Rebuild recentchanges table everywhere.
Wed, Jun 12, 5:03 AM · Data-Persistence-Backup
Marostegui added subtasks for T367261: Rebuild recentchanges table everywhere: T367162: db1240.s3 index issues, T367142: db1233 recentchanges index corrupted.
Wed, Jun 12, 5:03 AM · DBA
Marostegui triaged T367261: Rebuild recentchanges table everywhere as High priority.
Wed, Jun 12, 5:02 AM · DBA
Marostegui created T367261: Rebuild recentchanges table everywhere.
Wed, Jun 12, 5:02 AM · DBA

Tue, Jun 11

Marostegui updated the task description for T364069: Rebuild pagelinks tables.
Tue, Jun 11, 4:01 PM · DBA
Marostegui edited projects for T330173: page_assessments table uses an inadequate VARBINARY length for pa_class/pa_importance, added: Schema-change; removed Schema-change-in-production.

Removing Schema-change-in-production in production as this is not ready for DBAs to execute.

Tue, Jun 11, 11:21 AM · Schema-change, Data-Persistence (work done), Community-Tech, MediaWiki-extensions-PageAssessments
Marostegui updated the task description for T364069: Rebuild pagelinks tables.
Tue, Jun 11, 11:14 AM · DBA
Marostegui closed T367145: Switchover s5 master (db2123 -> db2213) as Resolved.

This is done

Tue, Jun 11, 10:47 AM · DBA
Marostegui closed T367145: Switchover s5 master (db2123 -> db2213), a subtask of T364069: Rebuild pagelinks tables, as Resolved.
Tue, Jun 11, 10:47 AM · DBA
Marostegui closed T367145: Switchover s5 master (db2123 -> db2213), a subtask of T364299: Make rc_id a bigint, as Resolved.
Tue, Jun 11, 10:47 AM · Schema-change-in-production, DBA
Marostegui updated the task description for T367145: Switchover s5 master (db2123 -> db2213).
Tue, Jun 11, 10:44 AM · DBA
Marostegui updated the task description for T367145: Switchover s5 master (db2123 -> db2213).
Tue, Jun 11, 10:41 AM · DBA
Marostegui updated the task description for T367155: Gerrit error: Error while fetching results for wm-patch-demo.
Tue, Jun 11, 10:37 AM · Patch-For-Review, Release-Engineering-Team, Gerrit
Marostegui updated the task description for T367155: Gerrit error: Error while fetching results for wm-patch-demo.
Tue, Jun 11, 10:35 AM · Patch-For-Review, Release-Engineering-Team, Gerrit
Marostegui created T367155: Gerrit error: Error while fetching results for wm-patch-demo.
Tue, Jun 11, 10:35 AM · Patch-For-Review, Release-Engineering-Team, Gerrit
Marostegui updated the task description for T367145: Switchover s5 master (db2123 -> db2213).
Tue, Jun 11, 10:29 AM · DBA
Marostegui added a comment to T367145: Switchover s5 master (db2123 -> db2213).

Old 300
vslow, dump 100

Tue, Jun 11, 10:28 AM · DBA
Marostegui updated the task description for T367145: Switchover s5 master (db2123 -> db2213).
Tue, Jun 11, 8:56 AM · DBA
Marostegui updated the task description for T367146: Switchover s5 master (db1230 -> db1183).
Tue, Jun 11, 8:47 AM · DBA
Marostegui claimed T367146: Switchover s5 master (db1230 -> db1183).
Tue, Jun 11, 8:46 AM · DBA
Marostegui claimed T367145: Switchover s5 master (db2123 -> db2213).
Tue, Jun 11, 8:46 AM · DBA
Marostegui added a parent task for T367145: Switchover s5 master (db2123 -> db2213): T364069: Rebuild pagelinks tables.
Tue, Jun 11, 8:45 AM · DBA
Marostegui added a parent task for T367146: Switchover s5 master (db1230 -> db1183): T364069: Rebuild pagelinks tables.
Tue, Jun 11, 8:45 AM · DBA
Marostegui added subtasks for T364069: Rebuild pagelinks tables: T367146: Switchover s5 master (db1230 -> db1183), T367145: Switchover s5 master (db2123 -> db2213).
Tue, Jun 11, 8:45 AM · DBA
Marostegui added a parent task for T367145: Switchover s5 master (db2123 -> db2213): T364299: Make rc_id a bigint.
Tue, Jun 11, 8:45 AM · DBA
Marostegui added a parent task for T367146: Switchover s5 master (db1230 -> db1183): T364299: Make rc_id a bigint.
Tue, Jun 11, 8:45 AM · DBA
Marostegui added subtasks for T364299: Make rc_id a bigint: T367145: Switchover s5 master (db2123 -> db2213), T367146: Switchover s5 master (db1230 -> db1183).
Tue, Jun 11, 8:45 AM · Schema-change-in-production, DBA
Marostegui added a comment to T365805: Test MariaDB 10.11.

Installed 10.11 on db1153 (non used x2 replica) - will test here for a few weeks.

Tue, Jun 11, 8:32 AM · DBA
Marostegui updated subscribers of T366440: Add type column on GlobalRenameQueue table.
Tue, Jun 11, 8:02 AM · MW-1.43-notes (1.43.0-wmf.10; 2024-06-18), Schema-change, MediaWiki-Platform-Team (Radar), Account-Vanishing, MediaWiki-extensions-CentralAuth
Marostegui updated the task description for T364299: Make rc_id a bigint.
Tue, Jun 11, 6:57 AM · Schema-change-in-production, DBA
Marostegui updated the task description for T364299: Make rc_id a bigint.
Tue, Jun 11, 6:14 AM · Schema-change-in-production, DBA
Marostegui closed T367142: db1233 recentchanges index corrupted as Resolved.

All done - also upgraded the host.
It is repooling now

Tue, Jun 11, 6:10 AM · DBA
Marostegui moved T367142: db1233 recentchanges index corrupted from Triage to In progress on the DBA board.
Tue, Jun 11, 6:04 AM · DBA
Marostegui added a comment to T367142: db1233 recentchanges index corrupted.

This is fixed. I am going to rebuild it in all the databases just in case.

Tue, Jun 11, 6:03 AM · DBA
Marostegui created T367142: db1233 recentchanges index corrupted.
Tue, Jun 11, 6:00 AM · DBA
Marostegui added a comment to T366982: Simple filter restrictions on meta causes Recent Changes to time out.

Which host is this?

Tue, Jun 11, 5:34 AM · DBA, mariadb-optimizer-bug, Wikimedia-Slow-DB-Query, Growth-Team, MediaWiki-Recent-changes
Marostegui added a parent task for T366687: Switchover s2 master (db1222 -> db1162): T364069: Rebuild pagelinks tables.
Tue, Jun 11, 5:25 AM · DBA
Marostegui added a parent task for T367140: Switchover s3 master (db1223 -> db1157): T364069: Rebuild pagelinks tables.
Tue, Jun 11, 5:25 AM · DBA
Marostegui added subtasks for T364069: Rebuild pagelinks tables: T367140: Switchover s3 master (db1223 -> db1157), T366687: Switchover s2 master (db1222 -> db1162).
Tue, Jun 11, 5:25 AM · DBA
Marostegui closed T367140: Switchover s3 master (db1223 -> db1157) as Resolved.

This is done

Tue, Jun 11, 5:22 AM · DBA
Marostegui closed T367140: Switchover s3 master (db1223 -> db1157), a subtask of T364299: Make rc_id a bigint, as Resolved.
Tue, Jun 11, 5:22 AM · Schema-change-in-production, DBA
Marostegui updated the task description for T367140: Switchover s3 master (db1223 -> db1157).
Tue, Jun 11, 5:21 AM · DBA
Marostegui updated the task description for T367140: Switchover s3 master (db1223 -> db1157).
Tue, Jun 11, 5:19 AM · DBA
Marostegui claimed T367140: Switchover s3 master (db1223 -> db1157).
Tue, Jun 11, 5:06 AM · DBA
Marostegui moved T367140: Switchover s3 master (db1223 -> db1157) from Triage to In progress on the DBA board.
Tue, Jun 11, 5:05 AM · DBA
Marostegui updated the task description for T367140: Switchover s3 master (db1223 -> db1157).
Tue, Jun 11, 5:05 AM · DBA
Marostegui added a comment to T367140: Switchover s3 master (db1223 -> db1157).

Old 500

Tue, Jun 11, 5:05 AM · DBA
Marostegui added a parent task for T367140: Switchover s3 master (db1223 -> db1157): T364299: Make rc_id a bigint.
Tue, Jun 11, 5:03 AM · DBA
Marostegui added a subtask for T364299: Make rc_id a bigint: T367140: Switchover s3 master (db1223 -> db1157).
Tue, Jun 11, 5:03 AM · Schema-change-in-production, DBA
Marostegui added a parent task for T366687: Switchover s2 master (db1222 -> db1162): T364299: Make rc_id a bigint.
Tue, Jun 11, 5:02 AM · DBA
Marostegui added a subtask for T364299: Make rc_id a bigint: T366687: Switchover s2 master (db1222 -> db1162).
Tue, Jun 11, 5:02 AM · Schema-change-in-production, DBA
Marostegui closed T366687: Switchover s2 master (db1222 -> db1162) as Resolved.

This is done

Tue, Jun 11, 4:57 AM · DBA
Marostegui updated the task description for T366687: Switchover s2 master (db1222 -> db1162).
Tue, Jun 11, 4:55 AM · DBA
Marostegui updated the task description for T366687: Switchover s2 master (db1222 -> db1162).
Tue, Jun 11, 4:53 AM · DBA
Marostegui updated the task description for T364069: Rebuild pagelinks tables.
Tue, Jun 11, 4:46 AM · DBA
Marostegui updated the task description for T364069: Rebuild pagelinks tables.
Tue, Jun 11, 4:45 AM · DBA
Marostegui merged T367020: Switchover s2 master (db1222 -> db1162) into T366687: Switchover s2 master (db1222 -> db1162).
Tue, Jun 11, 4:45 AM · DBA
Marostegui merged task T367020: Switchover s2 master (db1222 -> db1162) into T366687: Switchover s2 master (db1222 -> db1162).
Tue, Jun 11, 4:44 AM · Patch-For-Review, DBA
Marostegui updated the task description for T364069: Rebuild pagelinks tables.
Tue, Jun 11, 4:42 AM · DBA
Marostegui updated the task description for T366687: Switchover s2 master (db1222 -> db1162).
Tue, Jun 11, 4:35 AM · DBA
Marostegui updated the task description for T366687: Switchover s2 master (db1222 -> db1162).
Tue, Jun 11, 4:34 AM · DBA
Marostegui added a comment to T366687: Switchover s2 master (db1222 -> db1162).

Old 500

Tue, Jun 11, 4:33 AM · DBA

Mon, Jun 10

Marostegui added a comment to T366852: Discover and fix under-utlizied replicas.

Be careful when increasing weights. Some of those hosts got those new weights because they got overloaded with "normal" weights for some reason.

Mon, Jun 10, 6:27 PM · DBA
Marostegui closed T365568: Drop flaggedpage_pending from production as Resolved.

This is done

Mon, Jun 10, 4:21 PM · DBA
Marostegui updated the task description for T365568: Drop flaggedpage_pending from production.
Mon, Jun 10, 4:20 PM · DBA
Marostegui updated the task description for T365568: Drop flaggedpage_pending from production.
Mon, Jun 10, 4:19 PM · DBA
Marostegui updated the task description for T365568: Drop flaggedpage_pending from production.
Mon, Jun 10, 3:47 PM · DBA
Marostegui added a comment to T365568: Drop flaggedpage_pending from production.

Dropping in s3 with sleep in between the drops. It will take around 25 minutes.

Mon, Jun 10, 3:47 PM · DBA
Marostegui updated the task description for T365568: Drop flaggedpage_pending from production.
Mon, Jun 10, 3:45 PM · DBA
Marostegui updated the task description for T365568: Drop flaggedpage_pending from production.
Mon, Jun 10, 3:42 PM · DBA
Marostegui updated the task description for T365568: Drop flaggedpage_pending from production.
Mon, Jun 10, 3:41 PM · DBA
Marostegui updated the task description for T364299: Make rc_id a bigint.
Mon, Jun 10, 1:27 PM · Schema-change-in-production, DBA
Marostegui closed T367017: Switchover s4 master (db2140 -> db2179) as Resolved.

Done

Mon, Jun 10, 6:43 AM · DBA