Page MenuHomePhabricator

Toolforge alert for "worker class instances not spread out enough"
Closed, ResolvedPublic

Description

The new and improved spreadcheck.py NPRE check is alerting:

$ sudo /usr/local/bin/spreadcheck.py -v --config /usr/local/etc/spreadcheck-tools.yaml
2019-01-23T00:05:01Z __main__     INFO    : worker: 10 instances on labvirt1012.eqiad.wmnet; expected <=9
CRITICAL: worker class instances not spread out enough

We need to move an instance off of labvirt1012 to make it happy about the balance of that sub-cluster in Toolforge.

Event Timeline

bd808 created this task.Jan 23 2019, 12:06 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 23 2019, 12:06 AM
bd808 assigned this task to Andrew.Fri, Jan 25, 6:02 AM
bd808 added a subscriber: Andrew.

@Andrew can you take care of this when you get a bit of time?

Mentioned in SAL (#wikimedia-cloud) [2019-01-25T14:22:33Z] <andrewbogott> draining and moving tools-worker-1021 to a new labvirt for T214447

Mentioned in SAL (#wikimedia-cloud) [2019-01-25T14:22:56Z] <andrewbogott> draining and moving tools-worker-1016 to a new labvirt for T214447

Andrew closed this task as Resolved.Fri, Jan 25, 4:17 PM