Page MenuHomePhabricator

Remove Wikidata query service lag from Wikidata maxlag
Closed, DeclinedPublic

Description

As a maintainer of the wikidata software I want to stop propagating the WDQS lag to Wikidata maxlag if it is no longer needed.

T221774 was added to ensure that WDQS would not fall too far behind backlog of Wikidata edits. Since then from time to time bots respecting the maxLag strategy are being throttled causing sometimes frustration from bot owner (ref T243701).

Now that the WDQS Streaming Updater is in production I think we can revisit the approaches taken to circumvent the poor performances of the WDQS Updater.

AC:

  • discuss & agree or disagree that T221774 is no longer needed
  • undo or keep & adjust T221774

Event Timeline

I don’t think I agree that this is no longer needed. It’s great if the new updater has higher throughput, but presumably it’s not infinite, and if the edit rate is high enough that the new updater can’t keep up, then we probably want to start throttling edits, rather than have an increasingly outdated query service?

I don't have much opinion on this so I'll try to ponder the pros & cons:

Arguments in favor of removing it:

  • can be cumbersome to operate (maint script running via a cron on mwmaint1002)
  • could cause frustration because only well-behaved bots are actually throttled
  • if we never had such lag issues on WDQS this feature would not have seen light

Arguments in favor of keeping it:

  • Exactly what Lucas said, WDQS might become again the bottleneck and we don't want it to be increasingly outdated.

Overall if we decide to keep this system in place I would suggest to make sure it's in line with the objectives (lag < 10m) and adjust its scaling factor accordingly (e.g. if most bots use maxLag=5 make sure that it only breaks if wdqs lag is > 10m).

I agree with Lucas. I think we want to have a safeguard in place in case things go wild again and people are starting to push the limits even of the new updater.

I agree with Lucas. I think we want to have a safeguard in place in case things go wild again and people are starting to push the limits even of the new updater.

Declining then, thanks!