Feed Advanced Search

Today

Marostegui added a comment to T203709: Schema change for adding indexes of ct_tag_id.

s1 progress:

Thu, Sep 20, 6:38 AM · Blocked-on-schema-change, User-Ladsgroup, Wikidata-Campsite, MediaWiki-Database, MediaWiki-Change-tagging
Marostegui added a comment to T203709: Schema change for adding indexes of ct_tag_id.

I have altered db2070 (enwiki) and I will leave it like that for a few days before going ahead on eqiad directly on the master.

Thu, Sep 20, 6:36 AM · Blocked-on-schema-change, User-Ladsgroup, Wikidata-Campsite, MediaWiki-Database, MediaWiki-Change-tagging
Marostegui added a comment to T202764: Wikidata produces a lot of failed requests for recentchanges API.

These are the last 24h: https://logstash.wikimedia.org/goto/cd0af28f39b7ad679b9d1e1130636fdf
Errors are almost gone now.

Thu, Sep 20, 6:29 AM · Datacenter-Switchover-2018, Performance-Team (Radar), DBA, Patch-For-Review, User-Addshore, Operations, Wikidata-Query-Service, Wikidata
Marostegui moved T203709: Schema change for adding indexes of ct_tag_id from Backlog to In progress on the Blocked-on-schema-change board.
Thu, Sep 20, 6:00 AM · Blocked-on-schema-change, User-Ladsgroup, Wikidata-Campsite, MediaWiki-Database, MediaWiki-Change-tagging
Marostegui closed T51593: Rename two indexes in the Echo extension, a subtask of T104459: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves, as Resolved.
Thu, Sep 20, 5:56 AM · Release-Engineering-Team (Watching / External), Wikimedia-Incident, Datasets-General-or-Unknown, Patch-For-Review, WorkType-NewFunctionality, DBA
Marostegui closed T51593: Rename two indexes in the Echo extension as Resolved.

This is all done - one less drift from code and production!

Thu, Sep 20, 5:56 AM · Blocked-on-schema-change, DBA, Schema-change
Marostegui updated the task description for T51593: Rename two indexes in the Echo extension.
Thu, Sep 20, 5:55 AM · Blocked-on-schema-change, DBA, Schema-change
Marostegui added a comment to T51593: Rename two indexes in the Echo extension.

codfw s3 mediawikiwiki progress:

Thu, Sep 20, 5:49 AM · Blocked-on-schema-change, DBA, Schema-change
Marostegui updated the task description for T51593: Rename two indexes in the Echo extension.
Thu, Sep 20, 5:46 AM · Blocked-on-schema-change, DBA, Schema-change

Yesterday

Marostegui added a comment to T186188: Failover DB masters in row D.

I just sync'ed with @ayounsi about the network maintenance. It is still blocked on the cables.

Wed, Sep 19, 5:49 PM · DBA
Marostegui edited projects for T204838: Make Wikibase wb_terms term_search_key index the same as wb_term_search_key in WMF production, added: User-Marostegui; removed DBA.
Wed, Sep 19, 5:30 PM · User-Marostegui, Wikidata-Campsite, MediaWiki-extensions-WikibaseRepository, Wikidata
Marostegui edited projects for T204836: Create wb_terms_entity_id wb_terms index for Wikibase on install and upgrade, added: User-Marostegui; removed DBA.
Wed, Sep 19, 5:30 PM · User-Marostegui, Wikidata-Campsite, MediaWiki-extensions-WikibaseRepository, Wikidata
Marostegui edited projects for T204837: Make Wikibase wb_terms term_text index the same as wb_terms_text in WMF production, added: User-Marostegui; removed DBA.
Wed, Sep 19, 5:30 PM · User-Marostegui, Wikidata-Campsite, MediaWiki-extensions-WikibaseRepository, Wikidata
Marostegui moved T202764: Wikidata produces a lot of failed requests for recentchanges API from In progress to Done on the DBA board.
Wed, Sep 19, 3:17 PM · Datacenter-Switchover-2018, Performance-Team (Radar), DBA, Patch-For-Review, User-Addshore, Operations, Wikidata-Query-Service, Wikidata
Marostegui awarded T204593: dbstore2002 s2 crashed a Mountain of Wealth token.
Wed, Sep 19, 2:39 PM · Patch-For-Review, DBA
Marostegui added a comment to T204593: dbstore2002 s2 crashed.

This oneliner will comress the tables from the direction of the smallest towards the largest:

mysql -BN -S /run/mysqld/mysqld.s2.sock -e "SELECT table_schema, table_name FROM information_Schema.tables WHERE engine='INNODB' and row_format <> 'COMPRESSED' ORDER BY DATA_LENGTH ASC" | while read db table; do mysql --skip-ssl --socket /run/mysqld/mysqld.s2.sock -e "ALTER TABLE $db.$table ROW_FORMAT=COMPRESSED;"; done

It shall be run inside a tmux/screen

Wed, Sep 19, 2:32 PM · Patch-For-Review, DBA
Marostegui added a comment to T51593: Rename two indexes in the Echo extension.

codfw x1 enwiki progress:

Wed, Sep 19, 1:48 PM · Blocked-on-schema-change, DBA, Schema-change
Marostegui updated the task description for T51593: Rename two indexes in the Echo extension.
Wed, Sep 19, 1:36 PM · Blocked-on-schema-change, DBA, Schema-change
Marostegui updated the task description for T51593: Rename two indexes in the Echo extension.
Wed, Sep 19, 1:21 PM · Blocked-on-schema-change, DBA, Schema-change
Marostegui added a comment to T172489: Monitor read_only on all databases, make it page on masters.

@jcrespo is this done after today's merge or still missing things?

Wed, Sep 19, 12:58 PM · Patch-For-Review, Wikimedia-Incident, monitoring, DBA
Marostegui updated the task description for T51593: Rename two indexes in the Echo extension.
Wed, Sep 19, 12:56 PM · Blocked-on-schema-change, DBA, Schema-change
Marostegui updated the task description for T51593: Rename two indexes in the Echo extension.
Wed, Sep 19, 12:56 PM · Blocked-on-schema-change, DBA, Schema-change
Marostegui added a comment to T85414: wikibase: synchronize schema on production with what is created on install.

A quick check about unused indexes on that table reports that only wb_terms_entity_id and wb_terms_search_key aren't in use:

root@neodymium:/home/marostegui# ./section s8 | grep codfw | egrep -v "dbstore|db2094" | while read host port; do echo "$host:$port"; mysql.py -h$host:$port sys -e "select * from schema_unused_indexes where object_name='wb_terms';";done
db2086.codfw.wmnet:3318
object_schema	object_name	index_name
wikidatawiki	wb_terms	wb_terms_entity_id
wikidatawiki	wb_terms	wb_terms_search_key
db2085.codfw.wmnet:3318
object_schema	object_name	index_name
wikidatawiki	wb_terms	wb_terms_entity_id
wikidatawiki	wb_terms	wb_terms_search_key
db2083.codfw.wmnet:3306
object_schema	object_name	index_name
wikidatawiki	wb_terms	wb_terms_entity_id
wikidatawiki	wb_terms	wb_terms_search_key
db2082.codfw.wmnet:3306
object_schema	object_name	index_name
wikidatawiki	wb_terms	wb_terms_entity_id
wikidatawiki	wb_terms	wb_terms_search_key
db2081.codfw.wmnet:3306
object_schema	object_name	index_name
wikidatawiki	wb_terms	wb_terms_entity_id
wikidatawiki	wb_terms	wb_terms_search_key
db2080.codfw.wmnet:3306
object_schema	object_name	index_name
wikidatawiki	wb_terms	wb_terms_entity_id
wikidatawiki	wb_terms	wb_terms_search_key
db2079.codfw.wmnet:3306
object_schema	object_name	index_name
wikidatawiki	wb_terms	wb_terms_entity_id
wikidatawiki	wb_terms	wb_terms_search_key
db2045.codfw.wmnet:3306
object_schema	object_name	index_name
wikidatawiki	wb_terms	wb_terms_entity_id
wikidatawiki	wb_terms	wb_terms_search_key
Wed, Sep 19, 12:43 PM · wikidata-tech-focus, DBA, MediaWiki-extensions-WikibaseRepository, Wikidata
Marostegui added a comment to T203565: Reclone db1114 (s1 api) from another API host.

It is not entirely clear to me what the differences are?
Which hosts did you compare?

Wed, Sep 19, 10:31 AM · Patch-For-Review, DBA
Marostegui added a comment to T203565: Reclone db1114 (s1 api) from another API host.

Good catch.
Can you open a task to get that checked and fixed across all the hosts where this difference exists?

Wed, Sep 19, 10:28 AM · Patch-For-Review, DBA
Marostegui updated the task description for T51593: Rename two indexes in the Echo extension.
Wed, Sep 19, 10:27 AM · Blocked-on-schema-change, DBA, Schema-change
Marostegui moved T204006: Execute the schema change for Partial Blocks from Backlog to Next on the DBA board.
Wed, Sep 19, 10:17 AM · DBA, Blocked-on-schema-change, Anti-Harassment
Marostegui added a comment to T204006: Execute the schema change for Partial Blocks.

I will try to get this deployed on eqiad before we switch back to it.

Wed, Sep 19, 10:17 AM · DBA, Blocked-on-schema-change, Anti-Harassment
Marostegui closed T200509: Make sure multi-instance slaves page as Resolved.

This is done

Wed, Sep 19, 9:23 AM · Patch-For-Review, DBA
Marostegui closed T200509: Make sure multi-instance slaves page, a subtask of T159423: Meta ticket: Migrate multi-source database hosts to multi-instance, as Resolved.
Wed, Sep 19, 9:23 AM · Epic, DBA
Marostegui updated the task description for T200509: Make sure multi-instance slaves page.
Wed, Sep 19, 9:22 AM · Patch-For-Review, DBA
Marostegui added a comment to T204594: db1118 mysql process crashed (mysql 8.0 test host).

There is also the fact that I executed an ALTER table

It would fit in my theory because it would be (maybe) the first event with GTID that the host received in a long time.

Wed, Sep 19, 7:28 AM · DBA
Marostegui added a comment to T204594: db1118 mysql process crashed (mysql 8.0 test host).

There is also the fact that I executed an ALTER table on the DC master for this host (db1067) with replication. It was done early that day:

17th Sept
05:28 marostegui: Deploy schema change on s1 eqiad master (db1067) - T67448 T114117 T51191
Wed, Sep 19, 7:25 AM · DBA
Marostegui closed T153638: Drop echo tables from local wiki databases as Resolved.
Wed, Sep 19, 6:13 AM · Patch-For-Review, DBA
Marostegui closed T153638: Drop echo tables from local wiki databases, a subtask of T51593: Rename two indexes in the Echo extension, as Resolved.
Wed, Sep 19, 6:13 AM · Blocked-on-schema-change, DBA, Schema-change
Marostegui added a comment to T202764: Wikidata produces a lot of failed requests for recentchanges API.

Looking at logstash: https://logstash.wikimedia.org/goto/39a6fe9edd787798129b66ae9d61ed90 there's definitely a drop in timeouts, but there are still present, so I will monitor this further.

Wed, Sep 19, 5:07 AM · Datacenter-Switchover-2018, Performance-Team (Radar), DBA, Patch-For-Review, User-Addshore, Operations, Wikidata-Query-Service, Wikidata

Tue, Sep 18

Marostegui added a comment to T202764: Wikidata produces a lot of failed requests for recentchanges API.

@Smalyshev have you noticed any improvements since the above comment was done, and the index is gone from everywhere but the recetchanges slaves (like in eqiad)?

Tue, Sep 18, 3:09 PM · Datacenter-Switchover-2018, Performance-Team (Radar), DBA, Patch-For-Review, User-Addshore, Operations, Wikidata-Query-Service, Wikidata
Marostegui added a comment to P7562 Masterwork From Distant Lands.

SELECT rev_id,rev_page,rev_timestamp,rev_minor_edit,rev_deleted,rev_len,rev_parent_id,rev_sha1,COALESCE( comment_rev_comment.comment_text, rev_comment ) AS rev_comment_text,comment_rev_comment.comment_data AS rev_comment_data,comment_rev_comment.comment_id AS rev_comment_cid,rev_user,rev_user_text,NULL AS rev_actor,rev_text_id,rev_content_format,rev_content_model,page_namespace,page_title,page_id,page_latest,page_is_redirect,page_len FROM revision LEFT JOIN revision_comment_temp temp_rev_comment ON ((temp_rev_comment.revcomment_rev = rev_id)) LEFT JOIN comment comment_rev_comment ON ((comment_rev_comment.comment_id = temp_rev_comment.revcomment_comment_id)) INNER JOIN page ON ((page_id = rev_page)) WHERE (rev_timestamp>='19991130000000') AND rev_page = '33952837' ORDER BY rev_timestamp ,rev_id LIMIT 2 ;

Tue, Sep 18, 2:56 PM
Marostegui removed a project from T204669: Slow access to Special:Contributions on mediawiki.org (due to enabling actor table WRITE_BOTH mode): DBA.
Tue, Sep 18, 1:44 PM · Core-Platform-Team (CPT-Q1-Jul-Sep-2018), Patch-For-Review, Regression, MediaWiki-Special-pages, Performance
Marostegui moved T204669: Slow access to Special:Contributions on mediawiki.org (due to enabling actor table WRITE_BOTH mode) from Triage to Blocked external/Not db team on the DBA board.
Tue, Sep 18, 1:01 PM · Core-Platform-Team (CPT-Q1-Jul-Sep-2018), Patch-For-Review, Regression, MediaWiki-Special-pages, Performance
Marostegui added a subtask for T51593: Rename two indexes in the Echo extension: T153638: Drop echo tables from local wiki databases.
Tue, Sep 18, 9:44 AM · Blocked-on-schema-change, DBA, Schema-change
Marostegui added a parent task for T153638: Drop echo tables from local wiki databases: T51593: Rename two indexes in the Echo extension.
Tue, Sep 18, 9:44 AM · Patch-For-Review, DBA
Marostegui added a comment to T153638: Drop echo tables from local wiki databases.

From s3 the following wikis will need to get the table deleted:

Tue, Sep 18, 9:19 AM · Patch-For-Review, DBA
Marostegui created P7560 (An Untitled Masterwork).
Tue, Sep 18, 9:16 AM
Marostegui moved T153638: Drop echo tables from local wiki databases from Backlog to In progress on the DBA board.
Tue, Sep 18, 9:04 AM · Patch-For-Review, DBA
Marostegui added a comment to T153638: Drop echo tables from local wiki databases.

This bite us today when changing some indexes, so I am going to start deleting these tables in core. They were renamed at T153638#3097450 and nothing broken, and nothing has broken since codfw is active.
So I am going to slowly start dropping those tables in all the wikis apart from the following wikis:

Tue, Sep 18, 9:04 AM · Patch-For-Review, DBA
Marostegui updated the task description for T51593: Rename two indexes in the Echo extension.
Tue, Sep 18, 6:56 AM · Blocked-on-schema-change, DBA, Schema-change
Marostegui raised the priority of T51593: Rename two indexes in the Echo extension from Low to Normal.

I have altered testwiki on db2033 for now.

Tue, Sep 18, 6:48 AM · Blocked-on-schema-change, DBA, Schema-change
Marostegui closed T195836: Cannot reliably get the EXPLAIN for a query on analytics wiki replica cluster as Declined.

I am going to close this as there is nothing else we can do at the moment as per T195836#4291244
If someone feels this needs to remain open, please reopen.

Tue, Sep 18, 6:34 AM · Data-Services, Quarry, DBA
Marostegui added a comment to T100501: mysql user and group should be a system user/group.

Blocked on full stretch migration.

Tue, Sep 18, 6:31 AM · Patch-For-Review, Operations, DBA
Marostegui added a comment to T202764: Wikidata produces a lot of failed requests for recentchanges API.

@Smalyshev eqiad and codfw are not the same.
The index only exists on recentchanges replicas and the masters (you can ignore dbstore1002, it is not used in production).

Tue, Sep 18, 6:25 AM · Datacenter-Switchover-2018, Performance-Team (Radar), DBA, Patch-For-Review, User-Addshore, Operations, Wikidata-Query-Service, Wikidata
Marostegui added a comment to T202764: Wikidata produces a lot of failed requests for recentchanges API.

The API requests for recentchanges now seem to be faster, but I still get exceptions in the log :( I also get a bunch of errors for Wikidata URLs like: https://www.wikidata.org/wiki/Special:EntityData/Q33799921.ttl?nocache=1537250691109&flavor=dump
These are supposed to be pretty fast but still produce "no response" sometimes. I'll try to see what else can be causing those. Individual requests that I am testing seem to be fine, but I wonder if it's possible that the request still occasionally uses the DB host with wrong index?

Tue, Sep 18, 6:17 AM · Datacenter-Switchover-2018, Performance-Team (Radar), DBA, Patch-For-Review, User-Addshore, Operations, Wikidata-Query-Service, Wikidata
Marostegui moved T204593: dbstore2002 s2 crashed from Triage to In progress on the DBA board.
Tue, Sep 18, 5:42 AM · Patch-For-Review, DBA
Marostegui triaged T204593: dbstore2002 s2 crashed as High priority.

Triaging this as high as it is the backup source.
I think we should just reimport s2 there.

Tue, Sep 18, 5:41 AM · Patch-For-Review, DBA
Marostegui added a comment to T202764: Wikidata produces a lot of failed requests for recentchanges API.

@Smalyshev - the indexes have been removed from the API hosts.
The queries on those two servers (db2080 and db2081) now take around 0.05 sec to run. Can you check if this makes a difference from your end?.
Keep in mind that the indexes still exists on other hosts (recentchanges and main traffic ones).

Tue, Sep 18, 5:26 AM · Datacenter-Switchover-2018, Performance-Team (Radar), DBA, Patch-For-Review, User-Addshore, Operations, Wikidata-Query-Service, Wikidata
Marostegui added a comment to T204594: db1118 mysql process crashed (mysql 8.0 test host).

I started an alter table early in the morning on eqiad master for externallinks (130G) and recentchanges, but that went thru at around lunch time, so I don't think it could be the cause of the crash, but maybe put some more pressure on the host or something....or it could just be coincidence.

Tue, Sep 18, 5:19 AM · DBA
Marostegui added a comment to T202764: Wikidata produces a lot of failed requests for recentchanges API.

I am a bit confused by now - is the original problem because recentchanges is using a wrong host, or it's using right host and the indexes there are wrong, or something else? And how can it be fixed? WDQS poller depends on RC API, and having it take 30+ seconds instead of usual sub-second response time is a serious issue.

Tue, Sep 18, 4:58 AM · Datacenter-Switchover-2018, Performance-Team (Radar), DBA, Patch-For-Review, User-Addshore, Operations, Wikidata-Query-Service, Wikidata
Marostegui added a comment to T204593: dbstore2002 s2 crashed.

Nothing on HW logs that could indicate something is wrong with storage or similar.

Tue, Sep 18, 4:56 AM · Patch-For-Review, DBA

Mon, Sep 17

Marostegui awarded T204311: Upgrade all core (mediawiki) database servers to mariadb 10.1 a Cookie token.
Mon, Sep 17, 6:12 PM · Patch-For-Review, Operations, DBA
Marostegui closed T204462: Degraded disk on db1069 (x1 master) as Resolved.

RAID rebuilt correctly:

Number of Virtual Disks: 1
Virtual Drive: 0 (Target Id: 0)
Name                :
RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0
Size                : 3.271 TB
Sector Size         : 512
Mirror Data         : 3.271 TB
State               : Optimal
Strip Size          : 256 KB
Number Of Drives per span:2
Span Depth          : 6
Default Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Disk's Default
Encryption Type     : None
Default Power Savings Policy: Controller Defined
Current Power Savings Policy: None
Can spin up in 1 minute: Yes
Mon, Sep 17, 4:56 PM · Operations, ops-eqiad, DBA
Marostegui added a comment to T114117: Drop externallinks.el_from_namespace on wmf databases.

Yeah, it was just a column drop :-)

Mon, Sep 17, 4:54 PM · DBA, Schema-change
Marostegui added a comment to T204554: externallinks table no longer accessible on the replicas.

@Bstorm finished the run:

root@labsdb1009.eqiad.wmnet[enwiki_p]> describe externallinks;
+-------------+------------------+------+-----+---------+-------+
| Field       | Type             | Null | Key | Default | Extra |
+-------------+------------------+------+-----+---------+-------+
| el_id       | int(10) unsigned | NO   |     | 0       |       |
| el_from     | int(8) unsigned  | NO   |     | 0       |       |
| el_to       | blob             | NO   |     | NULL    |       |
| el_index    | blob             | NO   |     | NULL    |       |
| el_index_60 | varbinary(60)    | NO   |     | NULL    |       |
+-------------+------------------+------+-----+---------+-------+
5 rows in set (0.01 sec)
Mon, Sep 17, 4:48 PM · DBA, Data-Services
Marostegui merged task T204554: externallinks table no longer accessible on the replicas into T114117: Drop externallinks.el_from_namespace on wmf databases.
Mon, Sep 17, 4:44 PM · DBA, Data-Services
Marostegui merged T204554: externallinks table no longer accessible on the replicas into T114117: Drop externallinks.el_from_namespace on wmf databases.
Mon, Sep 17, 4:44 PM · DBA, Schema-change
Marostegui updated subscribers of T204554: externallinks table no longer accessible on the replicas.

Yes, this is part of: T114117#4589703
@Bstorm is going to re-run the views

Mon, Sep 17, 4:43 PM · DBA, Data-Services
Marostegui added a comment to T114117: Drop externallinks.el_from_namespace on wmf databases.

@Bstorm can you run the views again? This is the last time, as s1 was the only pending wiki to alter in eqiad.
Thank you!

Mon, Sep 17, 4:25 PM · DBA, Schema-change
Marostegui added a comment to T202764: Wikidata produces a lot of failed requests for recentchanges API.

The plan is for us dbas to test setting up a single API with the same structure than eqiad and do all assuming that fixies it, and later we will have to evaluate what is the right long-term status, given some unknowns and related tasks such as T202167:

Mon, Sep 17, 4:20 PM · Datacenter-Switchover-2018, Performance-Team (Radar), DBA, Patch-For-Review, User-Addshore, Operations, Wikidata-Query-Service, Wikidata
Marostegui merged T204539: Degraded RAID on db1069 into T204462: Degraded disk on db1069 (x1 master).
Mon, Sep 17, 3:44 PM · Operations, ops-eqiad, DBA
Marostegui merged task T204539: Degraded RAID on db1069 into T204462: Degraded disk on db1069 (x1 master).
Mon, Sep 17, 3:44 PM · ops-eqiad, Operations
Marostegui added a comment to T204462: Degraded disk on db1069 (x1 master).

Disk replaced by @Cmjohnson

root@db1069:~#  megacli -PDRbld -ShowProg -PhysDrv [32:7] -aALL
Mon, Sep 17, 3:39 PM · Operations, ops-eqiad, DBA
Marostegui added a comment to T203709: Schema change for adding indexes of ct_tag_id.

I am planning to deploy this first on an active enwiki (codfw) replica, and leave it for a couple of days to make sure nothing gets weird with its reads or writes, before deploying it to eqiad.

Mon, Sep 17, 3:13 PM · Blocked-on-schema-change, User-Ladsgroup, Wikidata-Campsite, MediaWiki-Database, MediaWiki-Change-tagging
Marostegui claimed T203709: Schema change for adding indexes of ct_tag_id.
Mon, Sep 17, 3:11 PM · Blocked-on-schema-change, User-Ladsgroup, Wikidata-Campsite, MediaWiki-Database, MediaWiki-Change-tagging
Marostegui claimed T51593: Rename two indexes in the Echo extension.
Mon, Sep 17, 3:11 PM · Blocked-on-schema-change, DBA, Schema-change
Marostegui added a comment to T51593: Rename two indexes in the Echo extension.

I will try to get this deployed in eqiad at least, before we switch back

Mon, Sep 17, 3:10 PM · Blocked-on-schema-change, DBA, Schema-change
Marostegui added a subtask for T104459: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves: T51593: Rename two indexes in the Echo extension.
Mon, Sep 17, 3:05 PM · Release-Engineering-Team (Watching / External), Wikimedia-Incident, Datasets-General-or-Unknown, Patch-For-Review, WorkType-NewFunctionality, DBA
Marostegui added a parent task for T51593: Rename two indexes in the Echo extension: T104459: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves.
Mon, Sep 17, 3:05 PM · Blocked-on-schema-change, DBA, Schema-change
Marostegui added a parent task for T202764: Wikidata produces a lot of failed requests for recentchanges API: T104459: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves.
Mon, Sep 17, 3:05 PM · Datacenter-Switchover-2018, Performance-Team (Radar), DBA, Patch-For-Review, User-Addshore, Operations, Wikidata-Query-Service, Wikidata
Marostegui added a subtask for T104459: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves: T202764: Wikidata produces a lot of failed requests for recentchanges API.
Mon, Sep 17, 3:05 PM · Release-Engineering-Team (Watching / External), Wikimedia-Incident, Datasets-General-or-Unknown, Patch-For-Review, WorkType-NewFunctionality, DBA
Marostegui added a comment to T203709: Schema change for adding indexes of ct_tag_id.

I will try to get this deployed in eqiad before we switch back.

Mon, Sep 17, 2:42 PM · Blocked-on-schema-change, User-Ladsgroup, Wikidata-Campsite, MediaWiki-Database, MediaWiki-Change-tagging
Marostegui triaged T204006: Execute the schema change for Partial Blocks as Normal priority.
Mon, Sep 17, 2:38 PM · DBA, Blocked-on-schema-change, Anti-Harassment
Marostegui updated the task description for T203548: Remove partitions from s7 masters (db1062 and db2040) for metawiki.pagelinks.
Mon, Sep 17, 2:08 PM · DBA
Marostegui closed T203548: Remove partitions from s7 masters (db1062 and db2040) for metawiki.pagelinks as Resolved.

db1062 is now done:

root@db1062.eqiad.wmnet[metawiki]> set session sql_log_bin=0; alter table pagelinks remove partitioning;
Query OK, 0 rows affected (0.00 sec)
Mon, Sep 17, 2:07 PM · DBA
Marostegui closed T203548: Remove partitions from s7 masters (db1062 and db2040) for metawiki.pagelinks, a subtask of T189107: DB meta task for next DC failover issues, as Resolved.
Mon, Sep 17, 2:07 PM · Patch-For-Review, Epic, Operations, DBA
Marostegui closed T187089: Fix WMF schemas to not break when comment store goes WRITE_NEW as Resolved.

This is all done

Mon, Sep 17, 12:56 PM · Core-Platform-Team, Patch-For-Review, Schema-change, DBA
Marostegui closed T187089: Fix WMF schemas to not break when comment store goes WRITE_NEW, a subtask of T166733: Deploy refactored comment storage, as Resolved.
Mon, Sep 17, 12:56 PM · Core-Platform-Team (CPT-Q1-Jul-Sep-2018), User-notice, Epic, Release-Engineering-Team (Watching / External)
Marostegui closed T187089: Fix WMF schemas to not break when comment store goes WRITE_NEW, a subtask of T188132: Merge image_comment_temp table into the image table, as Resolved.
Mon, Sep 17, 12:56 PM · Core-Platform-Team (CPT-Q1-Jul-Sep-2018), MW-1.31-release-notes (WMF-deploy-2018-04-03 (1.31.0-wmf.28)), Patch-For-Review, MediaWiki-Database
Marostegui updated the task description for T187089: Fix WMF schemas to not break when comment store goes WRITE_NEW.
Mon, Sep 17, 12:56 PM · Core-Platform-Team, Patch-For-Review, Schema-change, DBA
Marostegui updated the task description for T204311: Upgrade all core (mediawiki) database servers to mariadb 10.1.
Mon, Sep 17, 10:19 AM · Patch-For-Review, Operations, DBA
Marostegui updated the task description for T51191: Dropping rc_moved_to_title/rc_moved_to_ns on wmf databases.
Mon, Sep 17, 9:05 AM · Patch-For-Review, Blocked-on-schema-change, DBA, Schema-change
Marostegui updated the task description for T114117: Drop externallinks.el_from_namespace on wmf databases.
Mon, Sep 17, 9:05 AM · DBA, Schema-change
Marostegui updated the task description for T67448: Dropping rc_cur_time on wmf databases.
Mon, Sep 17, 9:05 AM · Patch-For-Review, Blocked-on-schema-change, DBA, Schema-change
Marostegui added a comment to T204476: Lock wait timeout exceeded from WikiPage::insertOn (Unable to import "Template:IPA_Symbol").

We have an innodb_lock_wait_timeout of 50 seconds (https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_lock_wait_timeout) so I assume your process is taking more than that and hence MySQL is giving up.
Is there anyway for you to do that import in smaller chunks?

Mon, Sep 17, 8:42 AM · Wikimedia-production-error, MediaWiki-Export-or-Import
Marostegui added a comment to T195293: 503 error attempting to open multiple projects (Wikipedia and meta wiki are loading very slowly) .

This awaits T201011

Mon, Sep 17, 7:29 AM · MW-1.32-release-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)), Schema-change, Language-2018-July-September, User-Nikerabbit, MediaWiki-extensions-Translate, Wikimedia-Incident, Wikimedia-production-error, Operations
Marostegui added a watcher for Blocked-on-schema-change: Marostegui.
Mon, Sep 17, 7:27 AM
Marostegui closed T201011: Apply schema change to translate_reviews in WMF as Resolved.

This is all done.

Mon, Sep 17, 7:27 AM · MediaWiki-extensions-Translate, Blocked-on-schema-change, Language-2018-July-September
Marostegui closed T201011: Apply schema change to translate_reviews in WMF, a subtask of T17441: Some tables lack unique or primary keys, may allow confusing duplicate data, as Resolved.
Mon, Sep 17, 7:27 AM · PostgreSQL, DBA, Schema-change, MediaWiki-Database
Marostegui closed T201011: Apply schema change to translate_reviews in WMF, a subtask of T195293: 503 error attempting to open multiple projects (Wikipedia and meta wiki are loading very slowly) , as Resolved.
Mon, Sep 17, 7:27 AM · MW-1.32-release-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)), Schema-change, Language-2018-July-September, User-Nikerabbit, MediaWiki-extensions-Translate, Wikimedia-Incident, Wikimedia-production-error, Operations
Marostegui updated the task description for T201011: Apply schema change to translate_reviews in WMF.
Mon, Sep 17, 7:26 AM · MediaWiki-extensions-Translate, Blocked-on-schema-change, Language-2018-July-September
Marostegui added a comment to T201011: Apply schema change to translate_reviews in WMF.

In s8, db1092 has the same issue. So I am going to re-import the table before executing the change so replication doesn't break once I get it deployed directly on the master.

Mon, Sep 17, 7:13 AM · MediaWiki-extensions-Translate, Blocked-on-schema-change, Language-2018-July-September
Marostegui added a comment to T201011: Apply schema change to translate_reviews in WMF.

For some reason db1094 (s7) already had that PK added there so it broke replication while adding it. Just to be completely sure and safe about consistency, I am going to re-import the table there from the master.

Mon, Sep 17, 6:45 AM · MediaWiki-extensions-Translate, Blocked-on-schema-change, Language-2018-July-September