Page MenuHomePhabricator

ESAMS Refresh/Rebuild (October 2019)
Open, Needs TriagePublic

Description

This will be the master task to track the refresh/rebuild of the esams caching data center, along with all its current and previous subtasks for the complete procurement, installation, and configuration of the new equipment to the decommission and recycling of the old hardware. Implementation of the onsite work is set for Oct 20-27 by Arzhel and Papaul.

Details

Related Gerrit Patches:
operations/homer/public : masterUpdate config to match new esams infra
operations/puppet : productionRemove old esams networking devices from monitoring
operations/dns : masterRepool esams
operations/dns : masterRevert "Depool esams for onsite work"
operations/dns : mastergeodns: eqiad non-primary for all public users
operations/dns : masterRename cr1-esams to cr3-esams
operations/puppet : productionRename cr1-esams to cr3-esams (same IP, new box)
operations/dns : masterAdd mgmt IPs for esams scs and asw2
operations/dns : masterDepool esams for onsite work
operations/dns : masterDepool esams for onsite work
operations/dns : masterMove most North American traffic westwards
operations/dns : masterMove GeoDNS default from eqiad to codfw

Related Objects

StatusAssignedTask
Openwiki_willy
StalledNone
ResolvedPapaul
ResolvedPapaul
ResolvedDzahn
ResolvedPapaul
ResolvedPapaul
Resolvedfaidon
ResolvedPapaul
ResolvedPapaul
Resolvedmark
ResolvedPapaul
ResolvedDzahn
Resolvedmark
ResolvedPapaul
ResolvedPapaul
ResolvedPapaul
ResolvedPapaul
Resolvedayounsi
OpenNone
Resolvedayounsi
Resolvedayounsi
OpenNone
ResolvedRobH
ResolvedPapaul
DuplicatePapaul
ResolvedPapaul
ResolvedPapaul
ResolvedPapaul
ResolvedPapaul
ResolvedPapaul
OpenRobH
Openwiki_willy
OpenNone

Event Timeline

Restricted Application added a project: Operations. · View Herald TranscriptOct 17 2019, 9:22 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
wiki_willy added a subtask: Unknown Object (Task).Oct 17 2019, 9:24 PM
wiki_willy added a subtask: Unknown Object (Task).
wiki_willy added a subtask: Unknown Object (Task).
wiki_willy added a subtask: Unknown Object (Task).
wiki_willy added a subtask: Unknown Object (Task).Oct 17 2019, 9:26 PM
wiki_willy added a subtask: Unknown Object (Task).
wiki_willy added a subtask: Unknown Object (Task).
wiki_willy added a subtask: Unknown Object (Task).
wiki_willy added a subtask: Unknown Object (Task).
Papaul closed subtask Unknown Object (Task) as Resolved.Oct 21 2019, 4:10 PM
Papaul closed subtask Unknown Object (Task) as Resolved.Oct 21 2019, 4:14 PM
Papaul closed subtask Unknown Object (Task) as Resolved.Oct 22 2019, 5:28 AM
ayounsi reopened subtask Unknown Object (Task) as Open.Oct 22 2019, 5:31 AM

Change 545270 had a related patch set uploaded (by CDanis; owner: Ayounsi):
[operations/dns@master] Depool esams for onsite work

https://gerrit.wikimedia.org/r/545270

Change 545270 merged by Ayounsi:
[operations/dns@master] Depool esams for onsite work

https://gerrit.wikimedia.org/r/545270

Mentioned in SAL (#wikimedia-operations) [2019-10-22T13:06:13Z] <XioNoX> depool esams for onsite work - T235805

Icinga downtime for 2:00:00 set by ayounsi@cumin1001 on 28 host(s) and their services with reason: Onsite work (asw)

bast3002.wikimedia.org,cp[3007-3008,3010,3030,3032-3036,3038-3047,3049].esams.wmnet,lvs[3001-3004].esams.wmnet,maerlant.wikimedia.org,multatuli.wikimedia.org,nescio.wikimedia.org

Change 545288 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/dns@master] Move GeoDNS default from eqiad to codfw

https://gerrit.wikimedia.org/r/545288

Change 545288 merged by BBlack:
[operations/dns@master] Move GeoDNS default from eqiad to codfw

https://gerrit.wikimedia.org/r/545288

Change 545294 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/dns@master] Move most North American traffic westwards

https://gerrit.wikimedia.org/r/545294

Change 545294 merged by BBlack:
[operations/dns@master] Move most North American traffic westwards

https://gerrit.wikimedia.org/r/545294

Change 545385 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/dns@master] geodns: eqiad non-primary for all public users

https://gerrit.wikimedia.org/r/545385

Change 545440 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/dns@master] Depool esams for onsite work

https://gerrit.wikimedia.org/r/545440

Change 545440 merged by Ayounsi:
[operations/dns@master] Depool esams for onsite work

https://gerrit.wikimedia.org/r/545440

Mentioned in SAL (#wikimedia-operations) [2019-10-23T06:59:12Z] <XioNoX> depool esams - T235805

Icinga downtime for 5:00:00 set by ayounsi@cumin1001 on 28 host(s) and their services with reason: Onsite work

bast3002.wikimedia.org,cp[3007-3008,3010,3030,3032-3036,3038-3047,3049].esams.wmnet,lvs[3001-3004].esams.wmnet,maerlant.wikimedia.org,multatuli.wikimedia.org,nescio.wikimedia.org

Mentioned in SAL (#wikimedia-operations) [2019-10-23T07:05:37Z] <XioNoX> redirect ns2 to eqiad - T235805

Change 545444 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/dns@master] Add mgmt IPs for esams scs and asw2

https://gerrit.wikimedia.org/r/545444

Papaul closed subtask Unknown Object (Task) as Resolved.Oct 23 2019, 11:57 AM
Papaul closed subtask Unknown Object (Task) as Resolved.Oct 23 2019, 12:08 PM
Papaul closed subtask Unknown Object (Task) as Resolved.
Papaul closed subtask Unknown Object (Task) as Resolved.

Change 545544 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/dns@master] Rename cr1-esams to cr3-esams

https://gerrit.wikimedia.org/r/545544

Change 545546 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Rename cr1-esams to cr3-esams (same IP, new box)

https://gerrit.wikimedia.org/r/545546

Change 545444 merged by Ayounsi:
[operations/dns@master] Add mgmt IPs for esams scs and asw2

https://gerrit.wikimedia.org/r/545444

Change 545546 merged by Ayounsi:
[operations/puppet@production] Rename cr1-esams to cr3-esams (same IP, new box)

https://gerrit.wikimedia.org/r/545546

Change 545544 merged by Ayounsi:
[operations/dns@master] Rename cr1-esams to cr3-esams

https://gerrit.wikimedia.org/r/545544

Change 545570 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/dns@master] Revert "Depool esams for onsite work"

https://gerrit.wikimedia.org/r/545570

Change 545570 merged by BBlack:
[operations/dns@master] Revert "Depool esams for onsite work"

https://gerrit.wikimedia.org/r/545570

Change 545571 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/dns@master] Repool esams

https://gerrit.wikimedia.org/r/545571

Change 545571 abandoned by CDanis:
Repool esams

https://gerrit.wikimedia.org/r/545571

Change 545660 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/homer/public@master] New esams stuff

https://gerrit.wikimedia.org/r/545660

Dzahn added a subscriber: Dzahn.Wed, Oct 23, 11:14 PM

for an overview of the confusing bastion naming situation please see T216199#5601054

added 2 tickets above that were not linked yet to clarify

Papaul closed subtask Unknown Object (Task) as Resolved.Thu, Oct 24, 8:14 PM
Dzahn changed the status of subtask T236329: decommission bast3002 from Stalled to Open.Fri, Oct 25, 3:14 PM

Change 546270 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Remove old esams networking devices from monitoring

https://gerrit.wikimedia.org/r/546270

Change 546270 merged by Dzahn:
[operations/puppet@production] Remove old esams networking devices from monitoring

https://gerrit.wikimedia.org/r/546270

Change 545660 merged by Ayounsi:
[operations/homer/public@master] Update config to match new esams infra

https://gerrit.wikimedia.org/r/545660

Papaul closed subtask Unknown Object (Task) as Resolved.Thu, Oct 31, 8:54 PM