Page MenuHomePhabricator

Pywikibot does not honor API etiquette regarding dispatch
Open, HighPublic

Description

"[Bots] should check before editing entities (but only every 60s) if the Median on Special:DispatchStats (also available via API as median) is 60 or higher and not edit."

https://www.wikidata.org/w/index.php?title=Wikidata:Bots&diff=0&oldid=161061964

This appears to only be relevant to Wikibase, as the statistics for en.wp does not include the dispatch entry.

Event Timeline

jayvdb created this task.Jul 14 2015, 1:15 AM
jayvdb raised the priority of this task from to Needs Triage.
jayvdb updated the task description. (Show Details)
jayvdb added a subscriber: jayvdb.
Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptJul 14 2015, 1:15 AM

This seems to become more relevant, see T171263

Restricted Application added a subscriber: PokestarFan. · View Herald TranscriptJul 30 2017, 11:51 AM
Xqt added a subscriber: Xqt.Jul 30 2017, 1:26 PM

What have bots to do in that case? Is I see the special page is available on wikibase only. We may

  • throttle wikibase edits
  • throttle edits when the client is affected which is listed with the median lag
  • throttle edits on the Client itself

We may throttle edits by

  • freshest lag
  • median lag
  • one minute
  • other delay e.g. increase the the retry_wait

but what should we do when the median lag is still higher than 60? Process the current edit and wait for the next delay number?

Xqt triaged this task as High priority.Jul 30 2017, 1:27 PM

json call sample:

{'batchcomplete': '',
 'query': {'statistics': {'activeusers': 18695,
                          'admins': 55,
                          'articles': 29901177,
                          'dispatch': {'average': {'lag': 297,
                                                   'pending': 2421},
                                       'freshest': {'lag': 7,
                                                    'pending': 76,
                                                    'position': '526031830',
                                                    'site': 'ruwikibooks',
                                                    'touched': '2017-07-30T13:03:48Z'},
                                       'median': {'lag': 67,
                                                  'pending': 659,
                                                  'position': '526031247',
                                                  'site': 'svwikiversity',
                                                  'touched': '2017-07-30T13:02:48Z'},
                                       'newest': {'id': 526031906,
                                                  'timestamp': '20170730130355'},
                                       'oldest': {'id': 523931085,
                                                  'timestamp': '20170727130005'},
                                       'stalest': {'lag': 117327,
                                                   'pending': 889773,
                                                   'position': '525142133',
                                                   'site': 'enwiki',
                                                   'touched': '2017-07-30T13:02:17Z'}},
                          'edits': 529143615,
                          'images': 0,
                          'jobs': 704269,
                          'pages': 31604526,
                          'queued-massmessages': 0,
                          'users': 2843589}}}
>>>
MisterSynergy added a subscriber: MisterSynergy.EditedJul 30 2017, 4:04 PM

Due to T171263 this is maybe not the best moment to exactly implement what's written at Wikidata:Bots#Bot accounts, but on the long run this would be useful to have. According to the corresponding Grafana chart of the median lag (you need to select "median" from the legend manually), bots would not be allowed to edit at all since 2017-06-28, 21:30. At that time, the median lag suddenly rose from robust ~15 seconds to robust ~100 seconds, with only very little variation. The median lag did not even decrease when the stalest lag had caught up at around 21:30 on 2017-07-25.

We don't really know why the dispatcher cannot handle the edit rates at Wikidata since a couple of weeks. High edit rates likely play a role here, but with regards to the stable median lag chart I would be surprised if that was the only or main reason. My impression is that there is no way to have a median dispatch lag lower than 60 seconds with the current configuration. In order not to shut down all (Pywikibot) Wikidata bots completely, we should wait for a solution of T171263 before this one is implemented.