Page MenuHomePhabricator

Ongoing dispatch problems (April 2018)
Closed, ResolvedPublic

Description

Since early April we have had repeated problems with dispatch lag, that required quite some manual intervention. This started on April 9 (which also is the day where ff4db0c87156035d79c0378ab8ba0aa2045ecf27 was merged).

As can be seen in the graph below, the mwscript switch to hhvm seems to correlate with a significant increase in the median pass time:

(from https://grafana.wikimedia.org/dashboard/db/wikidata-dispatch-script)
(the last increase here is probably due to a change in the Wikidata edit pattern over this weekend)

Due to this I suggest to raise our resources for dispatching by up to 60%, since the median dispatch time also increased by about that (from maybe 1.4s to 2.25s).

Event Timeline

hoo created this task.Apr 29 2018, 5:02 PM
hoo triaged this task as High priority.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 29 2018, 5:02 PM
hoo claimed this task.

Change 429662 had a related patch set uploaded (by Hoo man; owner: Hoo man):
[operations/puppet@production] Increase dispatching resources by about 50%

https://gerrit.wikimedia.org/r/429662

hoo updated the task description. (Show Details)Apr 29 2018, 5:15 PM
hoo updated the task description. (Show Details)

Change 429829 had a related patch set uploaded (by Hoo man; owner: Hoo man):
[operations/puppet@production] Run enable HHVM's JIT for Wikidata dispatchers

https://gerrit.wikimedia.org/r/429829

Change 429829 merged by Giuseppe Lavagetto:
[operations/puppet@production] Enable HHVM's JIT for Wikidata dispatchers

https://gerrit.wikimedia.org/r/429829

Change 429662 abandoned by Hoo man:
Increase dispatching resources by about 50%

Reason:
I don't think this is needed anymore

https://gerrit.wikimedia.org/r/429662

hoo closed this task as Resolved.Apr 30 2018, 4:07 PM

After enabling HHVM's JIT, the median pass times immediately went down! Due to this, I guess we can consider this addressed :)

hoo removed a project: Patch-For-Review.
hoo updated the task description. (Show Details)Apr 30 2018, 4:14 PM

Change 429662 restored by Hoo man:
Increase dispatching resources by about 50%

https://gerrit.wikimedia.org/r/429662

Change 429662 merged by ArielGlenn:
[operations/puppet@production] Increase dispatching resources by about 10%

https://gerrit.wikimedia.org/r/429662

Vvjjkkii renamed this task from Ongoing dispatch problems (April 2018) to i0daaaaaaa.Jul 1 2018, 1:13 AM
Vvjjkkii reopened this task as Open.
Vvjjkkii removed hoo as the assignee of this task.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed subscribers: gerritbot, Aklapper.
CommunityTechBot assigned this task to hoo.
CommunityTechBot closed this task as Resolved.
CommunityTechBot renamed this task from i0daaaaaaa to Ongoing dispatch problems (April 2018).
CommunityTechBot added subscribers: gerritbot, Aklapper.