Page MenuHomePhabricator
Create Task
Maniphest T194950

Include Wikibase dispatch lag in API "maxlag" enforcing
Closed, ResolvedPublic
Actions

Description

Per Wikimedia-Hackathon-2018 discussion:

There should be a setting which maps the Wikidata dispatch lag to the API's max lag. This should be a set factor so that the max lag is determined by:

maxLag = max( maxLag, WikibaseDispatchLag / DispatchLagToMaxLagFactor ) where DispatchLagToMaxLagFactor is probably 5 (which would mean bots stop at around 250s of dispatch lag) or 6 (which would make bots stop at 300s… if we were to do this, we should also make sure the respective alert is only fired at maybe 330s of dispatch lag).

This is similar to $wgJobQueueIncludeInMaxLagFactor.

Note: This will need a hook in ApiMain::getMaxLag first. Alternatively Wikibase could overwrite ApiMain::checkMaxLag in its API subclasses… but that's probably the worse solution.

Details

SubjectRepoBranchLines +/-
Set dispatchLagToMaxLagFactor to 60 for wikidataoperations/mediawiki-configmaster+1 -0
labs: set dispatchLagToMaxLagFactor to 60operations/mediawiki-configmaster+2 -0
Include Wikibase dispatch lag in API "maxlag" enforcingmediawiki/extensions/Wikibasemaster+39 -0
Introduce ApiMaxLagInfo hookmediawiki/coremaster+16 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

hoo created this task.May 18 2018, 1:34 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 18 2018, 1:34 PM
hoo mentioned this in T192026: Add an API parameter analogue to "maxLag" for dispatch lag.May 18 2018, 1:35 PM
hoo added a subscriber: Magnus.
• Huonenwiki closed this task as a duplicate of T27720: substitute LOCALTIMESTAMP can differ from REVISIONTIMESTAMP.May 18 2018, 5:39 PM
Sjoerddebruin reopened this task as Open.May 18 2018, 5:46 PM
Anomie added a comment.May 19 2018, 9:19 AM
In T194950#4213723, @hoo wrote

Alternatively Wikibase could overwrite ApiMain::checkMaxLag in its API subclasses… but that's probably the worse solution.

To actually make that work you'd need to override the ApiMain object itself. That's probably a very bad idea.

MisterSynergy subscribed.May 23 2018, 8:13 PM
Ladsgroup claimed this task.May 31 2018, 2:04 PM
Ladsgroup added a project: Wikidata-Ministry-Of-Magic.
Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptMay 31 2018, 2:04 PM
gerritbot subscribed.May 31 2018, 2:05 PM

Change 436543 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/core@master] Introduce MaxLagInfo hook

https://gerrit.wikimedia.org/r/436543

gerritbot added a project: Patch-For-Review.May 31 2018, 2:05 PM
gerritbot added a comment.May 31 2018, 3:03 PM

Change 436561 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/extensions/Wikibase@master] Include Wikibase dispatch lag in API "maxlag" enforcing

https://gerrit.wikimedia.org/r/436561

Ladsgroup moved this task from Tasks to Needs Review on the Wikidata-Ministry-Of-Magic board.Jun 2 2018, 3:10 PM
Ladsgroup moved this task from incoming to in progress on the Wikidata board.Jun 2 2018, 6:10 PM
Lucas_Werkmeister_WMDE added a project: Wikidata-Campsite.Jun 5 2018, 1:09 PM
Addshore moved this task from Incoming to Peer Review on the Wikidata-Campsite board.Jun 5 2018, 1:09 PM
Addshore subscribed.

@Ladsgroup will you continue this as part of you 20% campsite time? or would you rather someone else take this over?

Ladsgroup added a comment.Jun 5 2018, 1:43 PM

I get this done in my 20% campsite time 💃

gerritbot added a comment.Jun 5 2018, 6:01 PM

Change 436543 merged by jenkins-bot:
[mediawiki/core@master] Introduce ApiMaxLagInfo hook

https://gerrit.wikimedia.org/r/436543

ReleaseTaggerBot added a project: MW-1.32-notes (WMF-deploy-2018-06-12 (1.32.0-wmf.8)).Jun 5 2018, 7:01 PM
WMDE-leszek moved this task from Peer Review to In Progress on the Wikidata-Campsite board.Jun 8 2018, 10:04 AM
Addshore added a comment.Jun 11 2018, 7:27 AM

I stand by what I said before in that exposing the dispatch lage is not as clear cut as exposing max DB lag.
It requires clients to make the decision, at what point is the dispatch lag too much? when should I stop?

Legoktm subscribed.Jun 11 2018, 7:31 AM
In T194950#4270978, @Addshore wrote:

I stand by what I said before in that exposing the dispatch lage is not as clear cut as exposing max DB lag.
It requires clients to make the decision, at what point is the dispatch lag too much? when should I stop?

The general advice is that clients should back off at maxlag=5. So Wikibase should determine some threshold of dispatch lag of when it wants clients to backoff, and then come up with a factor to scale it down to 5 "seconds". I don't think clients should make decisions here, aside from picking how many "seconds" (not really) to backoff at.

Addshore added a comment.Jun 11 2018, 7:34 AM
In T194950#4270985, @Legoktm wrote:
In T194950#4270978, @Addshore wrote:

I stand by what I said before in that exposing the dispatch lage is not as clear cut as exposing max DB lag.
It requires clients to make the decision, at what point is the dispatch lag too much? when should I stop?

The general advice is that clients should back off at maxlag=5. So Wikibase should determine some threshold of dispatch lag of when it wants clients to backoff, and then come up with a factor to scale it down to 5 "seconds". I don't think clients should make decisions here, aside from picking how many "seconds" (not really) to backoff at.

Oh good, that's already been done in the patch :) I had this comment sitting here for a while but hadn't hit submit yet!

Addshore added a parent task: T196868: Track maxlag for wikidata on grafana dashboard.Jun 11 2018, 7:36 AM
Ladsgroup moved this task from In Progress to Peer Review on the Wikidata-Campsite board.Jun 11 2018, 11:34 AM
gerritbot added a comment.Jun 20 2018, 9:45 AM

Change 436561 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Include Wikibase dispatch lag in API "maxlag" enforcing

https://gerrit.wikimedia.org/r/436561

• Jonas moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite board.Jun 20 2018, 10:03 AM
Lydia_Pintscher moved this task from Test (Verification) to Needs Announcement on the Wikidata-Campsite board.Jun 25 2018, 9:27 AM
Addshore added a comment.Jun 25 2018, 12:10 PM

How can this maxlag be retrieved by the API?

https://wikidata.beta.wmflabs.org/w/api.php?action=query&titles=MediaWiki&format=json&maxlag=-1 seems to just show the old max lag still

Ladsgroup added a comment.Jun 25 2018, 12:18 PM
In T194950#4311445, @Addshore wrote:

How can this maxlag be retrieved by the API?

https://wikidata.beta.wmflabs.org/w/api.php?action=query&titles=MediaWiki&format=json&maxlag=-1 seems to just show the old max lag still

We need to enable it. I do it right now (on beta)

gerritbot added a comment.Jun 25 2018, 12:55 PM

Change 441857 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[operations/mediawiki-config@master] labs: set dispatchLagToMaxLagFactor to 60

https://gerrit.wikimedia.org/r/441857

gerritbot added a comment.Jun 25 2018, 12:58 PM

Change 441857 merged by jenkins-bot:
[operations/mediawiki-config@master] labs: set dispatchLagToMaxLagFactor to 60

https://gerrit.wikimedia.org/r/441857

Ladsgroup added a comment.Jun 25 2018, 1:03 PM

^ This enables it on beta cluster. We need to do similar thing for prod once the patch gets deployed (Wednesday afternoon)

Addshore mentioned this in T196868: Track maxlag for wikidata on grafana dashboard.Jun 26 2018, 9:59 AM
Addshore added a comment.Jun 26 2018, 10:23 AM

@Ladsgroup with the current implementation will a query with maxlag=-1 show both maxlag values, or just one?

Ladsgroup added a comment.Jun 26 2018, 10:24 AM

The highest one only

Addshore added a comment.Jun 26 2018, 10:42 AM
In T194950#4314730, @Ladsgroup wrote:

The highest one only

In which case without any sort of dedicated api endpoint it will be impossible for T196868 to track only the new maxlag that we have added.
Instead the task will just have to track the maxlag of wikidata in general.

Ladsgroup added a comment.Jun 27 2018, 7:50 PM
In T194950#4314787, @Addshore wrote:
In T194950#4314730, @Ladsgroup wrote:

The highest one only

In which case without any sort of dedicated api endpoint it will be impossible for T196868 to track only the new maxlag that we have added.
Instead the task will just have to track the maxlag of wikidata in general.

Getting dispatch-related maxlag of wikidata is already done, You just get the value of median of dispatch lag and divide it by 60.

Lea_Lacroix_WMDE subscribed.Jun 28 2018, 12:32 PM

Will be activated on July 3rd. Has been announced today.

Lea_Lacroix_WMDE moved this task from Needs Announcement to Done on the Wikidata-Campsite board.Jun 28 2018, 12:34 PM
Ladsgroup mentioned this in T198396: Remove ratelimit from Wikidata for some groups again.Jun 28 2018, 1:21 PM
Magnus added a comment.Jun 28 2018, 2:47 PM

Excellent, that means my existing code should Just Work (tm).

Is the "lag" value in the API reply in seconds? maxlag does not appear to be documented for the query action...

ValterVB subscribed.Jun 28 2018, 5:57 PM
Multichill added a comment.Jun 28 2018, 6:41 PM
In T194950#4322078, @Magnus wrote:

Excellent, that means my existing code should Just Work (tm).

Is the "lag" value in the API reply in seconds? maxlag does not appear to be documented for the query action...

Yes it is in seconds. See https://www.mediawiki.org/wiki/Manual:Maxlag_parameter for more info

Multichill added a project: Pywikibot-Wikidata.Jun 29 2018, 2:50 PM
Restricted Application added a subscriber: pywikibot-bugs-list. · View Herald TranscriptJun 29 2018, 2:50 PM
Multichill moved this task from Backlog to Upstream on the Pywikibot-Wikidata board.Jun 29 2018, 2:50 PM
ReleaseTaggerBot edited projects, added MW-1.32-notes (WMF-deploy-2018-07-10 (1.32.0-wmf.12)); removed MW-1.32-notes (WMF-deploy-2018-06-12 (1.32.0-wmf.8)).Jun 30 2018, 2:01 PM
• Vvjjkkii renamed this task from Include Wikibase dispatch lag in API "maxlag" enforcing to 1rcaaaaaaa.Jul 1 2018, 1:09 AM
• Vvjjkkii removed Ladsgroup as the assignee of this task.
• Vvjjkkii triaged this task as High priority.
• Vvjjkkii edited projects, added CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish); removed MW-1.32-notes (WMF-deploy-2018-07-10 (1.32.0-wmf.12)).
• Vvjjkkii updated the task description. (Show Details)
• Vvjjkkii edited subscribers, added: Ladsgroup; removed: gerritbot, Aklapper.
CommunityTechBot renamed this task from 1rcaaaaaaa to Include Wikibase dispatch lag in API "maxlag" enforcing.Jul 2 2018, 4:01 PM
CommunityTechBot assigned this task to Ladsgroup.
CommunityTechBot raised the priority of this task from High to Needs Triage.
CommunityTechBot updated the task description. (Show Details)
CommunityTechBot added a project: MW-1.32-notes (WMF-deploy-2018-07-10 (1.32.0-wmf.12)).
CommunityTechBot removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.
CommunityTechBot edited subscribers, added: gerritbot, Aklapper; removed: Ladsgroup.
Anomie added a comment.Jul 2 2018, 5:49 PM
In T194950#4322078, @Magnus wrote:

maxlag does not appear to be documented for the query action...

That's because it's a parameter to the "main" module, not the query action.

Addshore added a comment.Jul 2 2018, 6:10 PM
In T194950#4322078, @Magnus wrote:

Excellent, that means my existing code should Just Work (tm).

I just checked some of the API logs and it looks like QuickStatementsBot isn't currently passing a maxlag?
It's currently the biggest api caller for wikidata that isn't passing it.

Lea_Lacroix_WMDE added a comment.Jul 3 2018, 12:30 PM

Small change: the feature will be activated on Thursday July 5th instead of today.

Magnus added a comment.Jul 3 2018, 1:37 PM
In T194950#4391332, @Addshore wrote:
In T194950#4322078, @Magnus wrote:

Excellent, that means my existing code should Just Work (tm).

I just checked some of the API logs and it looks like QuickStatementsBot isn't currently passing a maxlag?
It's currently the biggest api caller for wikidata that isn't passing it.

I should:
https://bitbucket.org/magnusmanske/magnustools/src/c76cc1a06cbd015c12046755a58608cb2aa496d1/public_html/php/oauth.php#lines-416

Unless I'm using it wrong... It does look for the error anyway:
https://bitbucket.org/magnusmanske/magnustools/src/c76cc1a06cbd015c12046755a58608cb2aa496d1/public_html/php/oauth.php#lines-516

daniel unsubscribed.Jul 3 2018, 1:39 PM
gerritbot added a comment.Jul 5 2018, 10:30 AM

Change 443939 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[operations/mediawiki-config@master] Set dispatchLagToMaxLagFactor to 60 for wikidata

https://gerrit.wikimedia.org/r/443939

gerritbot added a comment.Jul 5 2018, 6:16 PM

Change 443939 merged by jenkins-bot:
[operations/mediawiki-config@master] Set dispatchLagToMaxLagFactor to 60 for wikidata

https://gerrit.wikimedia.org/r/443939

Stashbot subscribed.Jul 5 2018, 6:21 PM

Mentioned in SAL (#wikimedia-operations) [2018-07-05T18:21:19Z] <thcipriani@deploy1001> Synchronized wmf-config/Wikibase-production.php: SWAT: [[gerrit:443939|Set dispatchLagToMaxLagFactor to 60 for wikidata]] T194950 (duration: 00m 51s)

Addshore added a comment.Jul 12 2018, 5:45 PM

Looks like this is all done, and it looks like yesterday for a couple of periods the dispatch lag was high and the maxlag as a result was higher.

image.png (297×1 px, 54 KB)

It looks like roughly ~6000 requests were stopped due to max lag out of ~200000 made yesterday

There are still some bots and tools that are not setting appropriate maxlag values for their requests but we can follow up elsewhere with that.

Addshore closed this task as Resolved.Jul 12 2018, 5:45 PM
Multichill added a comment.Jul 12 2018, 9:33 PM
In T194950#4420642, @Addshore wrote:

There are still some bots and tools that are not setting appropriate maxlag values for their requests but we can follow up elsewhere with that.

Please list them on https://www.wikidata.org/wiki/Wikidata:Administrators%27_noticeboard and the accounts will be blocked.

Lucas_Werkmeister_WMDE mentioned this in T209201: WDQS server/updater performance issues.Apr 24 2019, 11:26 AM
Lucas_Werkmeister_WMDE mentioned this in T221774: Add Wikidata query service lag to Wikidata maxlag.Apr 24 2019, 2:14 PM
Bugreporter mentioned this in T238798: Allow access to MediaInfo from wikis other than Commons.Nov 20 2019, 9:38 PM
Maintenance_bot removed a project: Patch-For-Review.Nov 20 2019, 10:10 PM
Restricted Application added a project: Pywikibot. · View Herald TranscriptNov 20 2019, 10:10 PM
Maintenance_bot moved this task from Incoming to Done on the User-Ladsgroup board.Nov 20 2019, 10:15 PM
matej_suchanek mentioned this in T105764: Pywikibot does not honor API etiquette regarding dispatch.Jan 18 2020, 9:47 AM
Aklapper removed a subscriber: Anomie.Oct 16 2020, 5:39 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL