Page MenuHomePhabricator

logging views are incredibly slow for specific targets
Closed, ResolvedPublic

Description

Following {T363633}, any query against logging, logging_userindex and logging_logindex and filtered by log_action and/or log_title and log_namespace are incredibly slow.

SELECT log_action, log_timestamp, log_params
FROM `enwiki_p`.`logging_logindex`
WHERE log_type = 'block'
AND log_action IN ('block', 'reblock', 'unblock')
AND log_timestamp > 0
AND log_title = 'GoldenRing'
AND log_namespace = 2
ORDER BY log_timestamp ASC

This used to be very fast, now the query plan looks like:

select_typetabletypepossible_keyskeykey_lenrefrowsExtra
SIMPLEloggingreflog_type_action, log_type_time, log_timeslog_type_action34const52360612Using index condition; Using where; Using filesort

Related Objects

Event Timeline

MusikAnimal created this task.

This effects the XTools Edit Counter and hence is very widespread. Having spoken with @taavi it sounds like there isn't a quick fix, so I will disable the effected queries in XTools as a temporary measure.

QuicoleJR raised the priority of this task from High to Unbreak Now!.May 4 2024, 12:20 AM
QuicoleJR subscribed.

Much of XTools is completely disabled because of this. This needs to be unbroken now!

JJMC89 lowered the priority of this task from Unbreak Now! to Needs Triage.May 4 2024, 12:50 AM
JJMC89 subscribed.

Priority should normally be set by product managers, maintainers, community liaisons, or developers who plan to work on the task, or by the bugwrangler or experienced community members, not by the reporter filing the bug report or by outside observers. When in doubt, do not change the Priority field value, but add a comment suggesting the change and convincing reasons for it.

@JJMC89 XTools is really important, and this glitch has shut off a lot of its most important features, Revision History Statistics being one example that affected me personally.

This is also affecting AntiCompositeBot's NoLicense task. I've pointed it at the analytics cluster for now in the hopes that it might be able to complete before the query killer, but given the size of the Commons database I'm not hopeful.

Change #1027062 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] maintain-views: Redact entire logging_logindex rows without a target

https://gerrit.wikimedia.org/r/1027062

taavi triaged this task as High priority.
taavi added a project: cloud-services-team.

Change #1027062 merged by Majavah:

[operations/puppet@production] maintain-views: Redact entire logging_logindex rows without a target

https://gerrit.wikimedia.org/r/1027062

Cookbook cookbooks.sre.wikireplicas.update-views run by taavi: Started updating wiki replica views

Cookbook cookbooks.sre.wikireplicas.update-views started by taavi completed:

  • clouddb1021.eqiad.wmnet (PASS)
    • Ran 'maintain-views --all-databases --replace-all --auto-depool --table logging'
  • clouddb1017.eqiad.wmnet (PASS)
    • Ran 'maintain-views --all-databases --replace-all --auto-depool --table logging'
  • clouddb1018.eqiad.wmnet (PASS)
    • Ran 'maintain-views --all-databases --replace-all --auto-depool --table logging'
  • clouddb1019.eqiad.wmnet (PASS)
    • Ran 'maintain-views --all-databases --replace-all --auto-depool --table logging'
  • clouddb1020.eqiad.wmnet (PASS)
    • Ran 'maintain-views --all-databases --replace-all --auto-depool --table logging'
  • clouddb1013.eqiad.wmnet (PASS)
    • Ran 'maintain-views --all-databases --replace-all --auto-depool --table logging'
  • clouddb1014.eqiad.wmnet (PASS)
    • Ran 'maintain-views --all-databases --replace-all --auto-depool --table logging'
  • clouddb1015.eqiad.wmnet (PASS)
    • Ran 'maintain-views --all-databases --replace-all --auto-depool --table logging'
  • clouddb1016.eqiad.wmnet (PASS)
    • Ran 'maintain-views --all-databases --replace-all --auto-depool --table logging'