Page MenuHomePhabricator

Ongoing dispatch problems (April 2018)
Closed, ResolvedPublic

Description

Since early April we have had repeated problems with dispatch lag, that required quite some manual intervention. This started on April 9 (which also is the day where ff4db0c87156035d79c0378ab8ba0aa2045ecf27 was merged).

As can be seen in the graph below, the mwscript switch to hhvm seems to correlate with a significant increase in the median pass time:

(from https://grafana.wikimedia.org/dashboard/db/wikidata-dispatch-script)
(the last increase here is probably due to a change in the Wikidata edit pattern over this weekend)

Due to this I suggest to raise our resources for dispatching by up to 60%, since the median dispatch time also increased by about that (from maybe 1.4s to 2.25s).

Details

Related Gerrit Patches:
operations/puppet : productionIncrease dispatching resources by about 10%
operations/puppet : productionEnable HHVM's JIT for Wikidata dispatchers

Event Timeline

hoo triaged this task as High priority.Apr 29 2018, 5:02 PM
hoo created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 29 2018, 5:02 PM
hoo claimed this task.Apr 29 2018, 5:03 PM
hoo moved this task from Tasks to In Progress on the Wikidata-Ministry-Of-Magic board.

Change 429662 had a related patch set uploaded (by Hoo man; owner: Hoo man):
[operations/puppet@production] Increase dispatching resources by about 50%

https://gerrit.wikimedia.org/r/429662

hoo updated the task description. (Show Details)Apr 29 2018, 5:15 PM
hoo updated the task description. (Show Details)

Change 429829 had a related patch set uploaded (by Hoo man; owner: Hoo man):
[operations/puppet@production] Run enable HHVM's JIT for Wikidata dispatchers

https://gerrit.wikimedia.org/r/429829

Change 429829 merged by Giuseppe Lavagetto:
[operations/puppet@production] Enable HHVM's JIT for Wikidata dispatchers

https://gerrit.wikimedia.org/r/429829

Change 429662 abandoned by Hoo man:
Increase dispatching resources by about 50%

Reason:
I don't think this is needed anymore

https://gerrit.wikimedia.org/r/429662

hoo closed this task as Resolved.Apr 30 2018, 4:07 PM

After enabling HHVM's JIT, the median pass times immediately went down! Due to this, I guess we can consider this addressed :)

hoo removed a project: Patch-For-Review.
hoo updated the task description. (Show Details)Apr 30 2018, 4:14 PM

Change 429662 restored by Hoo man:
Increase dispatching resources by about 50%

https://gerrit.wikimedia.org/r/429662

Change 429662 merged by ArielGlenn:
[operations/puppet@production] Increase dispatching resources by about 10%

https://gerrit.wikimedia.org/r/429662

Vvjjkkii renamed this task from Ongoing dispatch problems (April 2018) to i0daaaaaaa.Jul 1 2018, 1:13 AM
Vvjjkkii reopened this task as Open.
Vvjjkkii removed hoo as the assignee of this task.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed subscribers: gerritbot, Aklapper.
CommunityTechBot renamed this task from i0daaaaaaa to Ongoing dispatch problems (April 2018).Jul 2 2018, 1:58 PM
CommunityTechBot closed this task as Resolved.
CommunityTechBot assigned this task to hoo.
CommunityTechBot updated the task description. (Show Details)
CommunityTechBot added subscribers: gerritbot, Aklapper.