Page MenuHomePhabricator
Create Task
Maniphest T297708

Set max execution time for several expensive mediawiki actions
Closed, ResolvedPublic
Actions

Description

Since T195792: Add support for setting individual query timeout in wikimedia/rdbms is done now we can set a max execution time. For the initial deployment:

  • Five special pages and their API counterparts will get max query time of 30s (why 30s?): RecentChanges, RecentChangesLinked, Watchlist, Log, Contributions
  • All of direct queries done by DPL will get a 10s timeout.

Rolling out plan for the special pages:

  • Before 2022, we roll it out on 1/3rd of requests randomly (with a rand() function in CommonSettings.php)
  • First week of January: We roll it out everywhere

DPL will get it without gradual deployment.

Detailed announcement: https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/IPJNO75HYAQWIGTHI5LJHTDVLVOC4LJP/

Details

SubjectRepoBranchLines +/-
Add a config to pass the testmediawiki/extensions/WikibaseREL1_35+1 -0
Add a config to pass the testmediawiki/extensions/WikibaseREL1_36+1 -0
Add a config to pass the testmediawiki/extensions/WikibaseREL1_37+1 -0
Allow setting max execution time to several special pagesmediawiki/coreREL1_36+51 -4
Allow setting max execution time to several special pagesmediawiki/coreREL1_37+52 -4
Allow setting max execution time to several special pagesmediawiki/coreREL1_35+51 -4
Full roll out of wgMaxExecutionTimeForExpensiveQueriesoperations/mediawiki-configmaster+4 -4
logspam: Consolidate max_statement_time errorsoperations/puppetproduction+1 -0
Set a maximum allowed time for db queriesmediawiki/extensions/intersectionwmf/1.38.0-wmf.13+8 -1
Set a maximum allowed time for db queriesmediawiki/extensions/intersectionwmf/1.38.0-wmf.12+8 -1
Gradual roll out of $wgMaxExecutionTimeForExpensiveQueriesoperations/mediawiki-configmaster+5 -0
Allow setting max execution time to several special pagesmediawiki/corewmf/1.38.0-wmf.13+52 -4
Allow setting max execution time to several special pagesmediawiki/corewmf/1.38.0-wmf.12+52 -4
Set a maximum allowed time for db queriesmediawiki/extensions/intersectionmaster+8 -1
Add a config to pass the testmediawiki/extensions/Wikibasewmf/1.38.0-wmf.12+1 -0
Add a config to pass the testmediawiki/extensions/Wikibasewmf/1.38.0-wmf.13+1 -0
beta: Set wgMaxExecutionTimeForExpensiveQueriesoperations/mediawiki-configmaster+4 -0
Allow setting max execution time to several special pagesmediawiki/coremaster+52 -4
Add a config to pass the testmediawiki/extensions/Wikibasemaster+1 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptDec 14 2021, 3:47 PM
gerritbot added a comment.Dec 14 2021, 3:47 PM

Change 747142 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] [WIP] Allow setting max execution time to several special pages

https://gerrit.wikimedia.org/r/747142

Ladsgroup triaged this task as High priority.Dec 14 2021, 3:47 PM
gerritbot added a project: Patch-For-Review.Dec 14 2021, 3:47 PM
Ladsgroup mentioned this in T297754: CVE-2022-28204: Whatlinkshere of heavily used properties in wikidata can be easily utilized as a DDoS vector.Dec 14 2021, 9:12 PM
Ladsgroup mentioned this in T195792: Add support for setting individual query timeout in wikimedia/rdbms.Dec 15 2021, 5:19 PM
gerritbot added a comment.Dec 15 2021, 6:27 PM

Change 747564 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/extensions/Wikibase@master] Add a config to pass the test

https://gerrit.wikimedia.org/r/747564

gerritbot added a comment.Dec 16 2021, 1:52 AM

Change 747564 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Add a config to pass the test

https://gerrit.wikimedia.org/r/747564

gerritbot added a comment.Dec 16 2021, 1:54 AM

Change 747142 merged by jenkins-bot:

[mediawiki/core@master] Allow setting max execution time to several special pages

https://gerrit.wikimedia.org/r/747142

ReleaseTaggerBot added a project: MW-1.38-notes (1.38.0-wmf.16; 2022-01-03).Dec 16 2021, 2:00 AM
Maintenance_bot removed a project: Patch-For-Review.Dec 16 2021, 2:10 AM
gerritbot added a comment.Dec 16 2021, 1:33 PM

Change 747840 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] beta: Set wgMaxExecutionTimeForExpensiveQueries

https://gerrit.wikimedia.org/r/747840

gerritbot added a project: Patch-For-Review.Dec 16 2021, 1:33 PM

Change 747840 merged by jenkins-bot:

[operations/mediawiki-config@master] beta: Set wgMaxExecutionTimeForExpensiveQueries

https://gerrit.wikimedia.org/r/747840

gerritbot added a comment.Dec 16 2021, 1:48 PM

Change 747845 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/extensions/intersection@master] Set a maximum allowed time for db queries

https://gerrit.wikimedia.org/r/747845

gerritbot added a comment.Dec 16 2021, 2:17 PM

Change 747693 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/extensions/Wikibase@wmf/1.38.0-wmf.12] Add a config to pass the test

https://gerrit.wikimedia.org/r/747693

gerritbot added a comment.Dec 16 2021, 2:17 PM

Change 747694 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/extensions/Wikibase@wmf/1.38.0-wmf.13] Add a config to pass the test

https://gerrit.wikimedia.org/r/747694

gerritbot added a comment.Dec 16 2021, 2:18 PM

Change 747695 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@wmf/1.38.0-wmf.13] Allow setting max execution time to several special pages

https://gerrit.wikimedia.org/r/747695

gerritbot added a comment.Dec 16 2021, 2:18 PM

Change 747696 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@wmf/1.38.0-wmf.12] Allow setting max execution time to several special pages

https://gerrit.wikimedia.org/r/747696

gerritbot added a comment.Dec 16 2021, 2:36 PM

Change 747697 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/extensions/intersection@wmf/1.38.0-wmf.13] Set a maximum allowed time for db queries

https://gerrit.wikimedia.org/r/747697

gerritbot added a comment.Dec 16 2021, 2:37 PM

Change 747698 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/extensions/intersection@wmf/1.38.0-wmf.12] Set a maximum allowed time for db queries

https://gerrit.wikimedia.org/r/747698

gerritbot added a comment.Dec 16 2021, 2:44 PM

Change 747693 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@wmf/1.38.0-wmf.12] Add a config to pass the test

https://gerrit.wikimedia.org/r/747693

gerritbot added a comment.Dec 16 2021, 2:44 PM

Change 747694 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@wmf/1.38.0-wmf.13] Add a config to pass the test

https://gerrit.wikimedia.org/r/747694

gerritbot added a comment.Dec 16 2021, 2:47 PM

Change 747845 merged by jenkins-bot:

[mediawiki/extensions/intersection@master] Set a maximum allowed time for db queries

https://gerrit.wikimedia.org/r/747845

gerritbot added a comment.Dec 16 2021, 2:48 PM

Change 747857 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] Gradual roll out of $wgMaxExecutionTimeForExpensiveQueries

https://gerrit.wikimedia.org/r/747857

ReleaseTaggerBot edited projects, added MW-1.38-notes (1.38.0-wmf.12; 2021-12-06); removed MW-1.38-notes (1.38.0-wmf.16; 2022-01-03).Dec 16 2021, 3:00 PM
gerritbot added a comment.Dec 16 2021, 3:18 PM

Change 747696 merged by jenkins-bot:

[mediawiki/core@wmf/1.38.0-wmf.12] Allow setting max execution time to several special pages

https://gerrit.wikimedia.org/r/747696

gerritbot added a comment.Dec 16 2021, 3:19 PM

Change 747695 merged by jenkins-bot:

[mediawiki/core@wmf/1.38.0-wmf.13] Allow setting max execution time to several special pages

https://gerrit.wikimedia.org/r/747695

Stashbot added a comment.Dec 16 2021, 3:30 PM

Mentioned in SAL (#wikimedia-operations) [2021-12-16T15:30:49Z] <ladsgroup@deploy1002> Synchronized php-1.38.0-wmf.13/includes/DefaultSettings.php: Backport: [[gerrit:747695|Allow setting max execution time to several special pages (T297708)], Part I (duration: 01m 06s)

Stashbot added a comment.Dec 16 2021, 3:32 PM

Mentioned in SAL (#wikimedia-operations) [2021-12-16T15:32:09Z] <ladsgroup@deploy1002> Synchronized php-1.38.0-wmf.13/includes/: Backport: [[gerrit:747695|Allow setting max execution time to several special pages (T297708)], Part II (duration: 01m 12s)

gerritbot added a comment.Dec 16 2021, 3:34 PM

Change 747857 merged by jenkins-bot:

[operations/mediawiki-config@master] Gradual roll out of $wgMaxExecutionTimeForExpensiveQueries

https://gerrit.wikimedia.org/r/747857

Stashbot added a comment.Dec 16 2021, 3:34 PM

Mentioned in SAL (#wikimedia-operations) [2021-12-16T15:34:19Z] <ladsgroup@deploy1002> Synchronized php-1.38.0-wmf.12/includes/DefaultSettings.php: Backport: [[gerrit:747696|Allow setting max execution time to several special pages (T297708)], Part I (duration: 01m 05s)

Stashbot added a comment.Dec 16 2021, 3:35 PM

Mentioned in SAL (#wikimedia-operations) [2021-12-16T15:35:40Z] <ladsgroup@deploy1002> Synchronized php-1.38.0-wmf.12/includes/: Backport: [[gerrit:747696|Allow setting max execution time to several special pages (T297708)], Part II (duration: 01m 11s)

Stashbot added a comment.Dec 16 2021, 3:36 PM

Mentioned in SAL (#wikimedia-operations) [2021-12-16T15:36:57Z] <ladsgroup@deploy1002> Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:747857|Gradual roll out of $wgMaxExecutionTimeForExpensiveQueries (T297708)]] (duration: 01m 06s)

Ladsgroup added a comment.Dec 16 2021, 3:42 PM

This is rolled out for one third of requests and we will see its result soon. After new year, we will roll this out to everyone.

ReleaseTaggerBot edited projects, added MW-1.38-notes (1.38.0-wmf.13; 2021-12-13); removed MW-1.38-notes (1.38.0-wmf.12; 2021-12-06).Dec 16 2021, 4:00 PM
gerritbot added a comment.Dec 16 2021, 4:18 PM

Change 747698 merged by jenkins-bot:

[mediawiki/extensions/intersection@wmf/1.38.0-wmf.12] Set a maximum allowed time for db queries

https://gerrit.wikimedia.org/r/747698

gerritbot added a comment.Dec 16 2021, 4:18 PM

Change 747697 merged by jenkins-bot:

[mediawiki/extensions/intersection@wmf/1.38.0-wmf.13] Set a maximum allowed time for db queries

https://gerrit.wikimedia.org/r/747697

Stashbot added a comment.Dec 16 2021, 4:30 PM

Mentioned in SAL (#wikimedia-operations) [2021-12-16T16:30:08Z] <ladsgroup@deploy1002> Synchronized php-1.38.0-wmf.13/extensions/intersection: Backport: [[gerrit:747697|Set a maximum allowed time for db queries (T297708)]] (duration: 01m 05s)

Stashbot added a comment.Dec 16 2021, 4:32 PM

Mentioned in SAL (#wikimedia-operations) [2021-12-16T16:32:41Z] <ladsgroup@deploy1002> Synchronized php-1.38.0-wmf.12/extensions/intersection: Backport: [[gerrit:747698|Set a maximum allowed time for db queries (T297708)]] (duration: 01m 06s)

Ladsgroup moved this task from In progress to Blocked on the DBA board.Dec 16 2021, 4:36 PM
In T297708#7575477, @Ladsgroup wrote:

This is rolled out for one third of requests and we will see its result soon. After new year, we will roll this out to everyone.

This goes to freezer until Jan 3rd.

Ladsgroup mentioned this in T287380: Decide on the future of DPL.Dec 16 2021, 4:41 PM
gerritbot added a comment.Dec 16 2021, 4:44 PM

Change 747876 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[operations/puppet@production] logspam: Consolidate max_statement_time errors

https://gerrit.wikimedia.org/r/747876

ReleaseTaggerBot edited projects, added MW-1.38-notes (1.38.0-wmf.12; 2021-12-06); removed MW-1.38-notes (1.38.0-wmf.13; 2021-12-13).Dec 16 2021, 5:00 PM
Wellverywell subscribed.Dec 16 2021, 5:03 PM
gerritbot added a comment.Dec 16 2021, 9:58 PM

Change 747876 merged by Ladsgroup:

[operations/puppet@production] logspam: Consolidate max_statement_time errors

https://gerrit.wikimedia.org/r/747876

RhinosF1 mentioned this in T298225: Database query error occurs when visiting Special:RecentChanges&tagfilter=wikieditor.Dec 23 2021, 4:21 PM
gerritbot added a comment.Jan 3 2022, 4:55 AM

Change 750831 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] Full roll out of wgMaxExecutionTimeForExpensiveQueries

https://gerrit.wikimedia.org/r/750831

Ladsgroup moved this task from Blocked to In progress on the DBA board.Jan 3 2022, 4:56 AM
Ladsgroup added a project: User-notice.Jan 3 2022, 5:13 AM

I will send an announcement today but I think it should be in the tech news as well.

gerritbot added a comment.Jan 3 2022, 6:56 AM

Change 750831 merged by jenkins-bot:

[operations/mediawiki-config@master] Full roll out of wgMaxExecutionTimeForExpensiveQueries

https://gerrit.wikimedia.org/r/750831

Stashbot added a comment.Jan 3 2022, 7:00 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-03T07:00:28Z] <ladsgroup@deploy1002> Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:750831|Full roll out of wgMaxExecutionTimeForExpensiveQueries (T297708)]], Part I (duration: 00m 58s)

Stashbot added a comment.Jan 3 2022, 7:02 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-03T07:02:08Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:750831|Full roll out of wgMaxExecutionTimeForExpensiveQueries (T297708)]], Part I (duration: 01m 20s)

Ladsgroup closed this task as Resolved.Jan 3 2022, 7:07 AM
Ladsgroup moved this task from In progress to Done on the DBA board.
Maintenance_bot moved this task from Incoming to Done on the User-Ladsgroup board.Jan 3 2022, 7:15 AM
thcipriani subscribed.Jan 6 2022, 5:04 PM

Seeing the Query execution was interrupted increase (as expected, probably) in the logs since this rolled out.

For the purposes of train is it safe to ignore all of these messages?

query-execution-interrupted-2022-01-06.png (241×663 px, 15 KB)

SD0001 subscribed.Jan 6 2022, 6:15 PM

Per T298010, the API is currently returning errors that look like this: internal_api_error_DBQueryError: [12099b97-17b0-42c5-883d-464328a6e662] Caught exception of type Wikimedia\Rdbms\DBQueryError I suggest a more informative error message be used in case of timeouts.

Quiddity updated the task description. (Show Details)Jan 6 2022, 8:42 PM
Quiddity subscribed.Jan 6 2022, 8:57 PM
In T297708#7593380, @Ladsgroup wrote:

I will send an announcement today but I think it should be in the tech news as well.

For Tech News, I'd suggest something like this as the summary (please edit or approve):

Five special pages (and their API counterparts) now have a maximum database query execution time of 30 seconds. These special pages are: RecentChanges, RecentChangesLinked, Watchlist, Contributions, and Log. This change will help with site performance and stability. You can read more details about this change including some possible solutions if this affects your workflows.

Ladsgroup added a comment.Jan 7 2022, 3:22 AM
In T297708#7602141, @thcipriani wrote:

Seeing the Query execution was interrupted increase (as expected, probably) in the logs since this rolled out.

For the purposes of train is it safe to ignore all of these messages?

query-execution-interrupted-2022-01-06.png (241×663 px, 15 KB)

Yes. Thank you. That's intentional (like php timeout errors basically, if they go really high suddenly, that's an issue but a baseline is fine)

In T297708#7602421, @SD0001 wrote:

Per T298010, the API is currently returning errors that look like this: internal_api_error_DBQueryError: [12099b97-17b0-42c5-883d-464328a6e662] Caught exception of type Wikimedia\Rdbms\DBQueryError I suggest a more informative error message be used in case of timeouts.

Agreed. Please file a task and subscribe me.

In T297708#7602849, @Quiddity wrote:
In T297708#7593380, @Ladsgroup wrote:

I will send an announcement today but I think it should be in the tech news as well.

For Tech News, I'd suggest something like this as the summary (please edit or approve):

Five special pages (and their API counterparts) now have a maximum database query execution time of 30 seconds. These special pages are: RecentChanges, RecentChangesLinked, Watchlist, Contributions, and Log. This change will help with site performance and stability. You can read more details about this change including some possible solutions if this affects your workflows.

Looks good to me but That's four now. I had to drop RCLinked (I will work on it).

Ladsgroup added a comment.Jan 7 2022, 4:16 AM

Added to tech news.

Quiddity moved this task from To Triage to In current Tech/News draft on the User-notice board.Jan 7 2022, 9:54 PM
SD0001 added a comment.Jan 8 2022, 7:25 AM

This is also sometimes causing the API to emit status code 500, apart from internal_api_errors (which have status 200).

Zabe mentioned this in T298847: Access to Watchlist in Wikidata returns a Fatal exception of type "Wikimedia\Rdbms\DBQueryError" (due to too many objects?).Jan 9 2022, 1:32 PM
Aklapper mentioned this in T298845: internal_api_error_DBQueryError when query for logevents.Jan 9 2022, 1:41 PM
RP88 subscribed.Jan 11 2022, 2:08 AM
Quiddity moved this task from In current Tech/News draft to Already announced/Archive on the User-notice board.Jan 13 2022, 10:03 PM
Krinkle mentioned this in T276699: Loading RC on en.wikipedia with a specific set of filters throws out an exception.Feb 4 2022, 1:54 AM
gerritbot added a comment.Feb 28 2022, 4:04 PM

Change 766791 had a related patch set uploaded (by Reedy; author: Amir Sarabadani):

[mediawiki/core@REL1_37] Allow setting max execution time to several special pages

https://gerrit.wikimedia.org/r/766791

gerritbot added a comment.Feb 28 2022, 4:04 PM

Change 766792 had a related patch set uploaded (by Reedy; author: Amir Sarabadani):

[mediawiki/core@REL1_36] Allow setting max execution time to several special pages

https://gerrit.wikimedia.org/r/766792

gerritbot added a comment.Feb 28 2022, 4:13 PM

Change 766793 had a related patch set uploaded (by Reedy; author: Amir Sarabadani):

[mediawiki/core@REL1_35] Allow setting max execution time to several special pages

https://gerrit.wikimedia.org/r/766793

gerritbot added a comment.Feb 28 2022, 4:36 PM

Change 766791 merged by jenkins-bot:

[mediawiki/core@REL1_37] Allow setting max execution time to several special pages

https://gerrit.wikimedia.org/r/766791

gerritbot added a comment.Feb 28 2022, 4:36 PM

Change 766793 merged by jenkins-bot:

[mediawiki/core@REL1_35] Allow setting max execution time to several special pages

https://gerrit.wikimedia.org/r/766793

gerritbot added a comment.Feb 28 2022, 4:37 PM

Change 766792 merged by jenkins-bot:

[mediawiki/core@REL1_36] Allow setting max execution time to several special pages

https://gerrit.wikimedia.org/r/766792

ReleaseTaggerBot added projects: MW-1.35-notes, MW-1.37-notes, MW-1.36-notes.Feb 28 2022, 5:00 PM
Quiddity unsubscribed.Feb 28 2022, 9:59 PM
brennen mentioned this in T303010: Query execution timeout for certain Special:Contributions queries.Mar 5 2022, 12:02 AM
Krinkle mentioned this in T304085: PHP Deprecated: Caller ignored an error raised from SpecialRecentChanges::doMainQuery (max_statement_time exceeded).Mar 22 2022, 2:06 PM
gerritbot added a comment.Mar 31 2022, 7:15 PM

Change 775930 had a related patch set uploaded (by Umherirrender; author: Amir Sarabadani):

[mediawiki/extensions/Wikibase@REL1_37] Add a config to pass the test

https://gerrit.wikimedia.org/r/775930

gerritbot added a comment.Mar 31 2022, 7:15 PM

Change 775931 had a related patch set uploaded (by Umherirrender; author: Amir Sarabadani):

[mediawiki/extensions/Wikibase@REL1_36] Add a config to pass the test

https://gerrit.wikimedia.org/r/775931

gerritbot added a comment.Mar 31 2022, 7:15 PM

Change 775933 had a related patch set uploaded (by Umherirrender; author: Amir Sarabadani):

[mediawiki/extensions/Wikibase@REL1_35] Add a config to pass the test

https://gerrit.wikimedia.org/r/775933

gerritbot added a comment.Mar 31 2022, 7:45 PM

Change 775930 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@REL1_37] Add a config to pass the test

https://gerrit.wikimedia.org/r/775930

gerritbot added a comment.Mar 31 2022, 7:45 PM

Change 775933 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@REL1_35] Add a config to pass the test

https://gerrit.wikimedia.org/r/775933

gerritbot added a comment.Mar 31 2022, 7:45 PM

Change 775931 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@REL1_36] Add a config to pass the test

https://gerrit.wikimedia.org/r/775931

Zabe removed a project: Patch-For-Review.Apr 7 2022, 5:11 PM
Zabe subscribed.
Umherirrender mentioned this in T306584: Commons Action API DB queries for logevents timeout (internal_api_error_DBQueryTimeoutError).Apr 21 2022, 9:56 AM
Ladsgroup edited projects, added User-notice-archive; removed User-notice.Aug 13 2022, 1:53 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL