Page MenuHomePhabricator
Create Task
Maniphest T235805

ESAMS Refresh/Rebuild (October 2019)
Closed, ResolvedPublic
Actions

Description

This will be the master task to track the refresh/rebuild of the esams caching data center, along with all its current and previous subtasks for the complete procurement, installation, and configuration of the new equipment to the decommission and recycling of the old hardware. Implementation of the onsite work is set for Oct 20-27 by Arzhel and Papaul.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolvedwiki_willyT235805 ESAMS Refresh/Rebuild (October 2019)
Unknown Object (Task)
Unknown Object (Task)
Unknown Object (Task)
Unknown Object (Task)
Unknown Object (Task)
 Resolved• MoritzMuehlenhoffT236216 rack/setup/install ganeti300[123]
 ResolvedPapaulT236217 rack/setup/install dns300[12]
 ResolvedPapaulT236294 rack/setup/install lvs300[567]
 ResolvedDzahnT236394 rack/setup/install bast3004
 ResolvedRequestPapaulT236329 decommission bast3002
Unknown Object (Task)
 ResolvedPapaulT233242 rack/setup/install cp30[50-65].esams.wmnet
Unknown Object (Task)
Unknown Object (Task)
Unknown Object (Task)
 ResolvedfaidonT184061 SRE 2017-18 Q3 goal Cleanup esams and refresh servers and infrastructure (tracking)
 ResolvedPapaulT184063 Remove all decommissioned hardware
 ResolvedPapaulT169518 Decommission esams ms-fe / ms-be
ResolvedmarkT95742 Decomission amssq31-62 (32 hosts)
 ResolvedPapaulT87790 decom amslvs1-4 (dc work)
 ResolvedDzahnT87729 decom amslvs1-4
 ResolvedmarkT94215 decommission cp3001 & cp3002
 ResolvedPapaulT130883 decom cp3011-22 (12 machines)
 ResolvedPapaulT159480 Decommission bast3001
 ResolvedPapaulT167376 Decommission cp300[3456]
Restricted Task
 ResolvedPapaulT184064 Prepare racks OE14, OE15 and OE16 with new infrastructure
 ResolvedayounsiT184065 Setup new access switches
 ResolvedRobHT184066 rack/setup/install ps[12]-oe1[456]-esams
 ResolvedayounsiT184067 Complete router migration from cr1-esams to cr3-esams
 ResolvedayounsiT174616 set up cr3-esams
 ResolvedfaidonT174637 Setup esams atlas anchor
 ResolvedRobHT184068 Procure and install LVS and miscellaneous servers
Unknown Object (Task)
 ResolvedPapaulT216199 decom bast3003 (65R8Q4J, formerly amslvs4)
 DuplicatePapaulT198790 Relabel hooft to bast3002
 ResolvedRequestPapaulT236451 decommission lvs300[1234]
 ResolvedRequestPapaulT236452 decommission nescio and maerlant
 ResolvedRequestPapaulT236454 decommission cp3030-3049
 ResolvedPapaulT208585 Decommission cp3007-cp3010
 ResolvedRequestPapaulT236489 decommission multatuli
 ResolvedBBlackT236497 cp3056 hardware issue
Unknown Object (Task)
 ResolvedayounsiT237030 Setup new MX204 in knams

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a project: SRE. · View Herald TranscriptOct 17 2019, 9:22 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
wiki_willy added a subtask: Unknown Object (Task).Oct 17 2019, 9:24 PM
wiki_willy added a subtask: Unknown Object (Task).
wiki_willy added a subtask: Unknown Object (Task).
wiki_willy added a subtask: Unknown Object (Task).
wiki_willy added a subtask: Unknown Object (Task).Oct 17 2019, 9:26 PM
wiki_willy added a subtask: Unknown Object (Task).
wiki_willy added a subtask: Unknown Object (Task).
wiki_willy added a subtask: Unknown Object (Task).
wiki_willy added a subtask: Unknown Object (Task).
wiki_willy added a subtask: T184061: SRE 2017-18 Q3 goal Cleanup esams and refresh servers and infrastructure (tracking).Oct 17 2019, 9:28 PM
Papaul closed subtask Unknown Object (Task) as Resolved.Oct 21 2019, 4:10 PM
Papaul closed subtask Unknown Object (Task) as Resolved.Oct 21 2019, 4:14 PM
• MoritzMuehlenhoff subscribed.Oct 21 2019, 4:14 PM
Papaul closed subtask Unknown Object (Task) as Resolved.Oct 22 2019, 5:28 AM
ayounsi reopened subtask Unknown Object (Task) as Open.Oct 22 2019, 5:31 AM
gerritbot added a comment.Oct 22 2019, 1:04 PM

Change 545270 had a related patch set uploaded (by CDanis; owner: Ayounsi):
[operations/dns@master] Depool esams for onsite work

https://gerrit.wikimedia.org/r/545270

gerritbot added a project: Patch-For-Review.Oct 22 2019, 1:04 PM

Change 545270 merged by Ayounsi:
[operations/dns@master] Depool esams for onsite work

https://gerrit.wikimedia.org/r/545270

Stashbot added a comment.Oct 22 2019, 1:06 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-22T13:06:13Z] <XioNoX> depool esams for onsite work - T235805

Maintenance_bot removed a project: Patch-For-Review.Oct 22 2019, 1:10 PM
ops-monitoring-bot added a comment.Oct 22 2019, 1:13 PM

Icinga downtime for 2:00:00 set by ayounsi@cumin1001 on 28 host(s) and their services with reason: Onsite work (asw)

bast3002.wikimedia.org,cp[3007-3008,3010,3030,3032-3036,3038-3047,3049].esams.wmnet,lvs[3001-3004].esams.wmnet,maerlant.wikimedia.org,multatuli.wikimedia.org,nescio.wikimedia.org
gerritbot added a comment.Oct 22 2019, 2:05 PM

Change 545288 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/dns@master] Move GeoDNS default from eqiad to codfw

https://gerrit.wikimedia.org/r/545288

gerritbot added a project: Patch-For-Review.Oct 22 2019, 2:05 PM

Change 545288 merged by BBlack:
[operations/dns@master] Move GeoDNS default from eqiad to codfw

https://gerrit.wikimedia.org/r/545288

Maintenance_bot removed a project: Patch-For-Review.Oct 22 2019, 2:13 PM
gerritbot added a comment.Oct 22 2019, 2:33 PM

Change 545294 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/dns@master] Move most North American traffic westwards

https://gerrit.wikimedia.org/r/545294

gerritbot added a project: Patch-For-Review.Oct 22 2019, 2:33 PM
gerritbot added a comment.Oct 22 2019, 2:35 PM

Change 545294 merged by BBlack:
[operations/dns@master] Move most North American traffic westwards

https://gerrit.wikimedia.org/r/545294

Maintenance_bot removed a project: Patch-For-Review.Oct 22 2019, 3:10 PM
gerritbot added a comment.Oct 22 2019, 7:56 PM

Change 545385 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/dns@master] geodns: eqiad non-primary for all public users

https://gerrit.wikimedia.org/r/545385

gerritbot added a project: Patch-For-Review.Oct 22 2019, 7:56 PM
gerritbot added a comment.Oct 23 2019, 6:53 AM

Change 545440 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/dns@master] Depool esams for onsite work

https://gerrit.wikimedia.org/r/545440

gerritbot added a comment.Oct 23 2019, 6:58 AM

Change 545440 merged by Ayounsi:
[operations/dns@master] Depool esams for onsite work

https://gerrit.wikimedia.org/r/545440

Stashbot added a comment.Oct 23 2019, 6:59 AM

Mentioned in SAL (#wikimedia-operations) [2019-10-23T06:59:12Z] <XioNoX> depool esams - T235805

ops-monitoring-bot added a comment.Oct 23 2019, 7:02 AM

Icinga downtime for 5:00:00 set by ayounsi@cumin1001 on 28 host(s) and their services with reason: Onsite work

bast3002.wikimedia.org,cp[3007-3008,3010,3030,3032-3036,3038-3047,3049].esams.wmnet,lvs[3001-3004].esams.wmnet,maerlant.wikimedia.org,multatuli.wikimedia.org,nescio.wikimedia.org
Stashbot added a comment.Oct 23 2019, 7:05 AM

Mentioned in SAL (#wikimedia-operations) [2019-10-23T07:05:37Z] <XioNoX> redirect ns2 to eqiad - T235805

gerritbot added a comment.Oct 23 2019, 7:12 AM

Change 545444 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/dns@master] Add mgmt IPs for esams scs and asw2

https://gerrit.wikimedia.org/r/545444

Papaul closed subtask Unknown Object (Task) as Resolved.Oct 23 2019, 11:57 AM
Papaul closed subtask Unknown Object (Task) as Resolved.Oct 23 2019, 12:08 PM
Papaul closed subtask Unknown Object (Task) as Resolved.
Papaul closed subtask Unknown Object (Task) as Resolved.
gerritbot added a comment.Oct 23 2019, 12:19 PM

Change 545544 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/dns@master] Rename cr1-esams to cr3-esams

https://gerrit.wikimedia.org/r/545544

gerritbot added a comment.Oct 23 2019, 12:33 PM

Change 545546 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Rename cr1-esams to cr3-esams (same IP, new box)

https://gerrit.wikimedia.org/r/545546

gerritbot added a comment.Oct 23 2019, 12:36 PM

Change 545444 merged by Ayounsi:
[operations/dns@master] Add mgmt IPs for esams scs and asw2

https://gerrit.wikimedia.org/r/545444

gerritbot added a comment.Oct 23 2019, 12:38 PM

Change 545546 merged by Ayounsi:
[operations/puppet@production] Rename cr1-esams to cr3-esams (same IP, new box)

https://gerrit.wikimedia.org/r/545546

gerritbot added a comment.Oct 23 2019, 12:38 PM

Change 545544 merged by Ayounsi:
[operations/dns@master] Rename cr1-esams to cr3-esams

https://gerrit.wikimedia.org/r/545544

gerritbot added a comment.Oct 23 2019, 2:18 PM

Change 545570 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/dns@master] Revert "Depool esams for onsite work"

https://gerrit.wikimedia.org/r/545570

gerritbot added a comment.Oct 23 2019, 2:18 PM

Change 545570 merged by BBlack:
[operations/dns@master] Revert "Depool esams for onsite work"

https://gerrit.wikimedia.org/r/545570

gerritbot added a comment.Oct 23 2019, 2:19 PM

Change 545571 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/dns@master] Repool esams

https://gerrit.wikimedia.org/r/545571

gerritbot added a comment.Oct 23 2019, 2:19 PM

Change 545571 abandoned by CDanis:
Repool esams

https://gerrit.wikimedia.org/r/545571

gerritbot added a comment.Oct 23 2019, 8:53 PM

Change 545660 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/homer/public@master] New esams stuff

https://gerrit.wikimedia.org/r/545660

Dzahn added subtasks: T216199: decom bast3003 (65R8Q4J, formerly amslvs4), T198790: Relabel hooft to bast3002.Oct 23 2019, 11:02 PM
Dzahn mentioned this in T198790: Relabel hooft to bast3002.Oct 23 2019, 11:04 PM
Dzahn mentioned this in T216199: decom bast3003 (65R8Q4J, formerly amslvs4).
Dzahn subscribed.Oct 23 2019, 11:14 PM

for an overview of the confusing bastion naming situation please see T216199#5601054

added 2 tickets above that were not linked yet to clarify

Dzahn added a subtask: T236329: decommission bast3002.Oct 23 2019, 11:18 PM
Dzahn added subtasks: T236394: rack/setup/install bast3004, T236217: rack/setup/install dns300[12].Oct 24 2019, 7:59 PM
Papaul closed subtask Unknown Object (Task) as Resolved.Oct 24 2019, 8:14 PM
BBlack added a subtask: T236451: decommission lvs300[1234].Oct 25 2019, 2:08 AM
BBlack added a subtask: T236452: decommission nescio and maerlant.Oct 25 2019, 3:28 AM
BBlack added a subtask: T236454: decommission cp3030-3049.Oct 25 2019, 4:04 AM
BBlack added a subtask: T208585: Decommission cp3007-cp3010.Oct 25 2019, 5:03 AM
BBlack added a subtask: T236489: decommission multatuli.Oct 25 2019, 3:03 PM
Dzahn changed the status of subtask T236329: decommission bast3002 from Stalled to Open.Oct 25 2019, 3:14 PM
Dzahn closed subtask T236394: rack/setup/install bast3004 as Resolved.Oct 25 2019, 3:21 PM
gerritbot added a comment.Oct 25 2019, 6:28 PM

Change 546270 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Remove old esams networking devices from monitoring

https://gerrit.wikimedia.org/r/546270

gerritbot added a comment.Oct 25 2019, 6:41 PM

Change 546270 merged by Dzahn:
[operations/puppet@production] Remove old esams networking devices from monitoring

https://gerrit.wikimedia.org/r/546270

Papaul closed subtask T236489: decommission multatuli as Resolved.Oct 28 2019, 9:27 AM
Papaul closed subtask T236329: decommission bast3002 as Resolved.Oct 28 2019, 9:32 AM
Papaul closed subtask T236452: decommission nescio and maerlant as Resolved.Oct 28 2019, 9:37 AM
Papaul closed subtask T236454: decommission cp3030-3049 as Resolved.Oct 28 2019, 10:12 AM
Papaul closed subtask T236451: decommission lvs300[1234] as Resolved.Oct 28 2019, 1:50 PM
BBlack added a subtask: T236497: cp3056 hardware issue.Oct 29 2019, 1:13 PM
BBlack closed subtask T236217: rack/setup/install dns300[12] as Resolved.
Papaul closed subtask T208585: Decommission cp3007-cp3010 as Resolved.Oct 29 2019, 8:26 PM
Papaul closed subtask T216199: decom bast3003 (65R8Q4J, formerly amslvs4) as Resolved.Oct 31 2019, 12:17 AM
faidon closed subtask T184061: SRE 2017-18 Q3 goal Cleanup esams and refresh servers and infrastructure (tracking) as Resolved.Oct 31 2019, 1:07 PM
gerritbot added a comment.Oct 31 2019, 5:03 PM

Change 545660 merged by Ayounsi:
[operations/homer/public@master] Update config to match new esams infra

https://gerrit.wikimedia.org/r/545660

Papaul closed subtask Unknown Object (Task) as Resolved.Oct 31 2019, 8:54 PM
wiki_willy added a subtask: Unknown Object (Task).Oct 31 2019, 9:42 PM
faidon added a subtask: T237030: Setup new MX204 in knams.Nov 1 2019, 2:53 PM
RobH closed subtask Unknown Object (Task) as Resolved.Nov 25 2019, 8:18 PM
ayounsi closed subtask T237030: Setup new MX204 in knams as Resolved.Nov 26 2019, 6:44 AM
BBlack closed subtask T236497: cp3056 hardware issue as Resolved.Nov 26 2019, 1:38 PM
RobH moved this task from Backlog to Blocked on the ops-esams board.Feb 21 2020, 4:29 PM
RobH unsubscribed.
RobH moved this task from Blocked to Backlog on the ops-esams board.Apr 13 2020, 3:29 PM
RobH closed this task as Resolved.Jul 30 2020, 5:25 PM

resolving as this now only has the single sub task to terminate the racks, also in the same project, so no need to keep a master tracking task open.

wiki_willy closed subtask Unknown Object (Task) as Resolved.Dec 22 2020, 7:11 PM
gerritbot added a comment.Jun 13 2024, 2:19 PM

Change #545385 abandoned by BBlack:

[operations/dns@master] geodns: eqiad non-primary for all public users

Reason:

Ancient unused commit, noticed it in a conflict list

https://gerrit.wikimedia.org/r/545385

Maintenance_bot removed a project: Patch-For-Review.Jun 13 2024, 2:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits