I have altered db2070 (enwiki) and I will leave it like that for a few days before going ahead on eqiad directly on the master.
These are the last 24h: https://logstash.wikimedia.org/goto/cd0af28f39b7ad679b9d1e1130636fdf
Errors are almost gone now.
This is all done - one less drift from code and production!
codfw s3 mediawikiwiki progress:
I just sync'ed with @ayounsi about the network maintenance. It is still blocked on the cables.
This oneliner will comress the tables from the direction of the smallest towards the largest:mysql -BN -S /run/mysqld/mysqld.s2.sock -e "SELECT table_schema, table_name FROM information_Schema.tables WHERE engine='INNODB' and row_format <> 'COMPRESSED' ORDER BY DATA_LENGTH ASC" | while read db table; do mysql --skip-ssl --socket /run/mysqld/mysqld.s2.sock -e "ALTER TABLE $db.$table ROW_FORMAT=COMPRESSED;"; done
It shall be run inside a tmux/screen
codfw x1 enwiki progress:
@jcrespo is this done after today's merge or still missing things?
A quick check about unused indexes on that table reports that only wb_terms_entity_id and wb_terms_search_key aren't in use:
root@neodymium:/home/marostegui# ./section s8 | grep codfw | egrep -v "dbstore|db2094" | while read host port; do echo "$host:$port"; mysql.py -h$host:$port sys -e "select * from schema_unused_indexes where object_name='wb_terms';";done db2086.codfw.wmnet:3318 object_schema object_name index_name wikidatawiki wb_terms wb_terms_entity_id wikidatawiki wb_terms wb_terms_search_key db2085.codfw.wmnet:3318 object_schema object_name index_name wikidatawiki wb_terms wb_terms_entity_id wikidatawiki wb_terms wb_terms_search_key db2083.codfw.wmnet:3306 object_schema object_name index_name wikidatawiki wb_terms wb_terms_entity_id wikidatawiki wb_terms wb_terms_search_key db2082.codfw.wmnet:3306 object_schema object_name index_name wikidatawiki wb_terms wb_terms_entity_id wikidatawiki wb_terms wb_terms_search_key db2081.codfw.wmnet:3306 object_schema object_name index_name wikidatawiki wb_terms wb_terms_entity_id wikidatawiki wb_terms wb_terms_search_key db2080.codfw.wmnet:3306 object_schema object_name index_name wikidatawiki wb_terms wb_terms_entity_id wikidatawiki wb_terms wb_terms_search_key db2079.codfw.wmnet:3306 object_schema object_name index_name wikidatawiki wb_terms wb_terms_entity_id wikidatawiki wb_terms wb_terms_search_key db2045.codfw.wmnet:3306 object_schema object_name index_name wikidatawiki wb_terms wb_terms_entity_id wikidatawiki wb_terms wb_terms_search_key
It is not entirely clear to me what the differences are?
Which hosts did you compare?
Can you open a task to get that checked and fixed across all the hosts where this difference exists?
I will try to get this deployed on eqiad before we switch back to it.
This is done
There is also the fact that I executed an ALTER table
It would fit in my theory because it would be (maybe) the first event with GTID that the host received in a long time.
There is also the fact that I executed an ALTER table on the DC master for this host (db1067) with replication. It was done early that day:
17th Sept 05:28 marostegui: Deploy schema change on s1 eqiad master (db1067) - T67448 T114117 T51191
Looking at logstash: https://logstash.wikimedia.org/goto/39a6fe9edd787798129b66ae9d61ed90 there's definitely a drop in timeouts, but there are still present, so I will monitor this further.
Tue, Sep 18
@Smalyshev have you noticed any improvements since the above comment was done, and the index is gone from everywhere but the recetchanges slaves (like in eqiad)?
SELECT rev_id,rev_page,rev_timestamp,rev_minor_edit,rev_deleted,rev_len,rev_parent_id,rev_sha1,COALESCE( comment_rev_comment.comment_text, rev_comment ) AS rev_comment_text,comment_rev_comment.comment_data AS rev_comment_data,comment_rev_comment.comment_id AS rev_comment_cid,rev_user,rev_user_text,NULL AS rev_actor,rev_text_id,rev_content_format,rev_content_model,page_namespace,page_title,page_id,page_latest,page_is_redirect,page_len FROM revision LEFT JOIN revision_comment_temp temp_rev_comment ON ((temp_rev_comment.revcomment_rev = rev_id)) LEFT JOIN comment comment_rev_comment ON ((comment_rev_comment.comment_id = temp_rev_comment.revcomment_comment_id)) INNER JOIN page ON ((page_id = rev_page)) WHERE (rev_timestamp>='19991130000000') AND rev_page = '33952837' ORDER BY rev_timestamp ,rev_id LIMIT 2 ;
From s3 the following wikis will need to get the table deleted:
This bite us today when changing some indexes, so I am going to start deleting these tables in core. They were renamed at T153638#3097450 and nothing broken, and nothing has broken since codfw is active.
So I am going to slowly start dropping those tables in all the wikis apart from the following wikis:
I have altered testwiki on db2033 for now.
I am going to close this as there is nothing else we can do at the moment as per T195836#4291244
If someone feels this needs to remain open, please reopen.
@Smalyshev eqiad and codfw are not the same.
The index only exists on recentchanges replicas and the masters (you can ignore dbstore1002, it is not used in production).
The API requests for recentchanges now seem to be faster, but I still get exceptions in the log :( I also get a bunch of errors for Wikidata URLs like: https://www.wikidata.org/wiki/Special:EntityData/Q33799921.ttl?nocache=1537250691109&flavor=dump
These are supposed to be pretty fast but still produce "no response" sometimes. I'll try to see what else can be causing those. Individual requests that I am testing seem to be fine, but I wonder if it's possible that the request still occasionally uses the DB host with wrong index?
Triaging this as high as it is the backup source.
I think we should just reimport s2 there.
@Smalyshev - the indexes have been removed from the API hosts.
The queries on those two servers (db2080 and db2081) now take around 0.05 sec to run. Can you check if this makes a difference from your end?.
Keep in mind that the indexes still exists on other hosts (recentchanges and main traffic ones).
I started an alter table early in the morning on eqiad master for externallinks (130G) and recentchanges, but that went thru at around lunch time, so I don't think it could be the cause of the crash, but maybe put some more pressure on the host or something....or it could just be coincidence.
I am a bit confused by now - is the original problem because recentchanges is using a wrong host, or it's using right host and the indexes there are wrong, or something else? And how can it be fixed? WDQS poller depends on RC API, and having it take 30+ seconds instead of usual sub-second response time is a serious issue.
Nothing on HW logs that could indicate something is wrong with storage or similar.
Mon, Sep 17
RAID rebuilt correctly:
Number of Virtual Disks: 1 Virtual Drive: 0 (Target Id: 0) Name : RAID Level : Primary-1, Secondary-0, RAID Level Qualifier-0 Size : 3.271 TB Sector Size : 512 Mirror Data : 3.271 TB State : Optimal Strip Size : 256 KB Number Of Drives per span:2 Span Depth : 6 Default Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU Current Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU Default Access Policy: Read/Write Current Access Policy: Read/Write Disk Cache Policy : Disk's Default Encryption Type : None Default Power Savings Policy: Controller Defined Current Power Savings Policy: None Can spin up in 1 minute: Yes
Yeah, it was just a column drop :-)
@Bstorm finished the run:
email@example.com[enwiki_p]> describe externallinks; +-------------+------------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------------+------------------+------+-----+---------+-------+ | el_id | int(10) unsigned | NO | | 0 | | | el_from | int(8) unsigned | NO | | 0 | | | el_to | blob | NO | | NULL | | | el_index | blob | NO | | NULL | | | el_index_60 | varbinary(60) | NO | | NULL | | +-------------+------------------+------+-----+---------+-------+ 5 rows in set (0.01 sec)
@Bstorm can you run the views again? This is the last time, as s1 was the only pending wiki to alter in eqiad.
The plan is for us dbas to test setting up a single API with the same structure than eqiad and do all assuming that fixies it, and later we will have to evaluate what is the right long-term status, given some unknowns and related tasks such as T202167:
Disk replaced by @Cmjohnson
root@db1069:~# megacli -PDRbld -ShowProg -PhysDrv [32:7] -aALL
I am planning to deploy this first on an active enwiki (codfw) replica, and leave it for a couple of days to make sure nothing gets weird with its reads or writes, before deploying it to eqiad.
I will try to get this deployed in eqiad at least, before we switch back
I will try to get this deployed in eqiad before we switch back.
db1062 is now done:
firstname.lastname@example.org[metawiki]> set session sql_log_bin=0; alter table pagelinks remove partitioning; Query OK, 0 rows affected (0.00 sec)
This is all done
We have an innodb_lock_wait_timeout of 50 seconds (https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_lock_wait_timeout) so I assume your process is taking more than that and hence MySQL is giving up.
Is there anyway for you to do that import in smaller chunks?
This is all done.
In s8, db1092 has the same issue. So I am going to re-import the table before executing the change so replication doesn't break once I get it deployed directly on the master.
For some reason db1094 (s7) already had that PK added there so it broke replication while adding it. Just to be completely sure and safe about consistency, I am going to re-import the table there from the master.