Change 571705 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Triple the factor of WDQS lag to maxlag for Wikidata

https://gerrit.wikimedia.org/r/571705

Change 571705 merged by jenkins-bot:
[operations/mediawiki-config@master] Triple the factor of WDQS lag to maxlag for Wikidata

https://gerrit.wikimedia.org/r/571705

Mentioned in SAL (#wikimedia-operations) [2020-02-12T12:19:22Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:571705|Triple the factor of WDQS lag to maxlag for Wikidata (T244722)]] (duration: 01m 04s)

Mentioned in SAL (#wikimedia-operations) [2020-02-12T12:21:36Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:571705|Triple the factor of WDQS lag to maxlag for Wikidata (T244722)]], take II, the cache issue (duration: 01m 03s)

https://grafana.wikimedia.org/d/000000156/wikidata-dispatch?orgId=1&fullscreen&panelId=22&from=1581481794943&to=1581524994945

Is this now doing what we want?

I think it's doing what we discussed in the storytime and making the frequency of the oscillations higher. However in addition to that it is increasing the amplitude and thus now actually triggering low edit volume alerts. See: https://grafana.wikimedia.org/d/TUJ0V-0Zk/wikidata-alerts?orgId=1&from=1581498145748&to=1581584545749

Is this a problem? I'm still a little unsure what metric we might want to try and optimise for but once we know what it is me might be able to make smarter tweaks about this maxlag parameter.

Possible metrics for us to aim for could be something like:

• Keeping WDQS lag consistently below some value
• Keeping the maximum "maxlag > 5" interval shorter than n mins (e.g. where people are now annoyed they can't edit)
• Keeping a minimum edit rate averaged over "some time period greater" than "some number"

I guess to meet the conflicting needs of our users we should probably try and "pick 2" or something?

Since the change we perform didn't have the expected results, we're going to revert it today, and keep looking for sustainable solutions.

Mentioned in SAL (#wikimedia-operations) [2020-02-13T12:20:51Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:571956|Revert: Triple the factor of WDQS lag to maxlag for Wikidata (T244722)]] (duration: 01m 04s)

The issue seems to be somewhere in the update process and query service itself rather than the wikidata edit rate (although of course the edit rate has some effect).
Over the past months even when the wikidata edit rate drops the query service lag can continue to increase.

Also before changing the factor to be more strict we probably want to complete T238751: Only generate maxlag from pooled query service servers.
The discussions on T240442: Design a continuous throttling policy for Wikidata bots are also very relevant to this as having a hard cut off at 5 is no good for anyone, a gradual increase and decrease in edit rate would be much better for both the people using wikidata and our systems.

In T244722#5888828, @Addshore wrote:

The issue seems to be somewhere in the update process and query service itself rather than the wikidata edit rate (although of course the edit rate has some effect).
Over the past months even when the wikidata edit rate drops the query service lag can continue to increase.

I agree with this. From my non-skilled observation I would say that the lag is mostly a function of the read queries from clients not really the related to edits

Also before changing the factor to be more strict we probably want to complete T238751: Only generate maxlag from pooled query service servers.
The discussions on T240442: Design a continuous throttling policy for Wikidata bots are also very relevant to this as having a hard cut off at 5 is no good for anyone, a gradual increase and decrease in edit rate would be much better for both the people using wikidata and our systems.

Sadly, as we already have the "magic number 5" used by load of people (like pywikibot) we're a little trapped with this. The alternative could be returning maxlag with some randomly calculated weighting to avoid "turning the taps totally off". e.g. 50% of the time subtract 1s from max lag due to the query service and 50% of the time add 1s. The time and the numbers would need to be chosen to to and seek some level of write rate v.s. lag.