Page MenuHomePhabricator

mw1251 down (no ssh) but still in dsh group?
Closed, ResolvedPublic

Event Timeline

Jdforrester-WMF triaged this task as High priority.Mar 25 2020, 6:32 PM
Jdforrester-WMF created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 25 2020, 6:32 PM
Dzahn claimed this task.Mar 25 2020, 7:29 PM
Dzahn added a comment.Mar 25 2020, 7:33 PM

should have been removed by this change:

https://gerrit.wikimedia.org/r/c/operations/puppet/+/583114/2/conftool-data/node/eqiad.yaml

conftool generates dsh groups.. unless it happens to be a scap proxy

This is still broken, and was causing confusion during the 4pm SWAT deployment. Thankfully scap appears to route around broken proxies, so it didn't fail to sync 1/9th of the Apaches as I feared it would.

Joe claimed this task.Mar 26 2020, 7:44 AM
Joe raised the priority of this task from High to Unbreak Now!.
Joe added a subscriber: Dzahn.
Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptMar 26 2020, 7:44 AM
Joe added a comment.Mar 26 2020, 8:06 AM

Not only that, but also mw1252, which is a mcrouter proxy, got decommissioned yesterday.

Fixing both.

Change 583558 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] mw: remove decommissioned servers from the scap,mcrouter proxies

https://gerrit.wikimedia.org/r/583558

Change 583558 merged by Giuseppe Lavagetto:
[operations/puppet@production] mw: remove decommissioned servers from the scap,mcrouter proxies

https://gerrit.wikimedia.org/r/583558

Joe closed this task as Resolved.Mar 26 2020, 8:35 AM