Reported by @Dreamy_Jazz . Query on that table (on centralauth) that are done around 3K to 4K times a second in some cases become slow and take more than 5s to respond. See https://logstash.wikimedia.org/goto/63d827ac6c7aebbb683e2590c0cb0e12
It might be related or not, we are seeing replag from time to time on s7. Maybe a large write that's choking s7? Investigating.
It seems a bot or a refreshlinks update is going ballistic in arwiki:
This is for just an hour and half:
root@dbstore1008:/srv/sqldata.s7# grep '### UPDATE ' binlog_to_check_amir | sort | uniq -c | sort -nr | head 4840548 ### UPDATE `arwiki`.`page` 91607 ### UPDATE `centralauth`.`global_edit_count` 13589 ### UPDATE `metawiki`.`discussiontools_item_revisions` 13437 ### UPDATE `metawiki`.`page` 11329 ### UPDATE `metawiki`.`discussiontools_item_pages` 10567 ### UPDATE `eswiki`.`page` 5614 ### UPDATE `heartbeat`.`heartbeat` 3771 ### UPDATE `eswiki`.`discussiontools_item_revisions` 3591 ### UPDATE `hewiki`.`discussiontools_item_revisions` 3562 ### UPDATE `ukwiki`.`page` root@dbstore1008:/srv/sqldata.s7# grep '### INSERT ' binlog_to_check_amir | sort | uniq -c | sort -nr | head 17332 ### INSERT INTO `eswiki`.`cu_useragent_clienthints_map` 12121 ### INSERT INTO `metawiki`.`cu_useragent_clienthints_map` 5401 ### INSERT INTO `arwiki`.`wbc_entity_usage` 5399 ### INSERT INTO `centralauth`.`localuser` 5381 ### INSERT INTO `centralauth`.`localnames` 4568 ### INSERT INTO `metawiki`.`wbc_entity_usage` 4559 ### INSERT INTO `eswiki`.`linter` 4517 ### INSERT INTO `hewiki`.`pagelinks` 4193 ### INSERT INTO `hewiki`.`cu_useragent_clienthints_map` 4070 ### INSERT INTO `ukwiki`.`wbc_entity_usage` root@dbstore1008:/srv/sqldata.s7# grep '### DELETE ' binlog_to_check_amir | sort | uniq -c | sort -nr | head 12556 ### DELETE FROM `eswiki`.`cu_useragent_clienthints_map` 6850 ### DELETE FROM `metawiki`.`cu_useragent_clienthints_map` 4815 ### DELETE FROM `arwiki`.`wbc_entity_usage` 4511 ### DELETE FROM `viwiki`.`cu_log_event` 4491 ### DELETE FROM `eswiki`.`linter` 3776 ### DELETE FROM `ukwiki`.`cu_useragent_clienthints_map` 3161 ### DELETE FROM `hewiki`.`pagelinks` 2976 ### DELETE FROM `arwiki`.`recentchanges` 2908 ### DELETE FROM `eswiki`.`recentchanges` 2695 ### DELETE FROM `hewiki`.`cu_useragent_clienthints_map`
Mentioned in SAL (#wikimedia-operations) [2026-04-14T23:11:28Z] <Amir1> optimizing globalblocks table on s7 (T423349)
Manuel and I discussed this and we decided that one or two slow queries every couple of hours is not worth fixing and having a bit of slowdown from time to time is normal: network slowdown/retry, data getting out of memory, lock contention with another request, etc. could cause this. If it happens in large enough volume, we could investigate but the very nature of large infra systems means we have small issues like this all the time.
Thanks, in which case we will exclude these warnings from our chore wheel to avoid future confusion about whether to address these logs
Thanks again for looking into this