Page MenuHomePhabricator
Create Task
Maniphest T255766

cr3-eqsin to production
Closed, ResolvedPublic
Actions

Description

  • Schedule maintenance,eg. 16:00 UTC (low traffic time, but can be done during normal time too) On Monday 22nd or Tuesday 23rd.
  • Depool eqsin
  • Downtime cr1-eqsin
  • Copy cr1-eqsin IX to cr3-eqsin
  • Move VRRP master to cr2
  • Power cr1-eqsin down
  • If needed unrack cr1-eqsin
  • Connect core links:
    • Move cable 1115 from cr1-eqsin:xe-1/0/0 to cr3-eqsin:xe-0/1/5 - link should come up
    • Move cable 1116 from cr1-eqsin:xe-2/0/0 to cr3-eqsin:et-0/0/0
    • Move cable 1116 from cr2-eqsin:xe-0/1/6 to cr2-eqsin:et-0/0/0 - link should stay down until cr2 is re-configured
    • Move cable 1121 from cr1-eqsin:xe-0/1/0 to cr3-eqsin:xe-0/1/0 - link should come up
    • Move cable 1076 from cr1-eqsin:xe-2/0/3 to cr3-eqsin:et-0/0/1
    • Move cable 1076 from asw-0603-eqsin:xe-0/0/21 to asw-0603-eqsin:et-??? - link should stay down until asw1-eqsin is re-configured
    • Remove cable 1084 between cr1-eqsin:xe-1/1/0 and asw-0604-eqsin:xe-1/0/21
  • Reconfigure cr2-cr3 links on the cr2 side
  • Reconfigure asw1-eqsin for new et- interface
  • Check for connectivity with switch vlans
  • Re-configure cr2 with cr3-eqsin neighbor IP (BGP confed)
  • Check for cr3 to be reachable on loopback IP
  • Check that we're using the Transport link
  • Add cr3-eqsin to monitoring (and remove cr1) (Puppet)
  • Re-configure servers BGP sessions with new cr3 IP (anycast/LVS) (Puppet)
  • Re-configure all routers with cr3-eqsin public IP (BGP confed)
  • Connect transit/peering links
    • Move cable 1120 from cr1-eqsin:xe-0/0/0 cr3-eqsin:xe-0/1/1 - link should come up
    • Move cable 1016 from cr1-eqsin:xe-2/0/1 cr3-eqsin:xe-0/1/3 - link should come up
    • Move cable ??? from cr1-eqsin:xe-2/0/2 cr3-eqsin:xe-0/1/4 - link should come up
  • Check for properly established BGP
  • Check if no discrepancies in Homer (CR)
  • Check that all ports are populated with optics

DONE for CR3

  • Downtime cr2-eqsin
  • Fail VRRP master to cr3-eqsin
  • Recable cr2-eqsin<->asw
  • Move cable 1118 from cr1-eqsin:xe-0/1/4 to cr3-eqsin:et-0/0/1
  • Move cable 1118 from asw-0604-eqsin:xe-1/0/20 to asw-0604-eqsin:et-??? - link should stay down until re-configured
  • Remove cable 1117 between cr2-eqsin:xe-0/1/3 and asw-0603-eqsin:xe-0/0/20
  • Reconfigure cr2-eqsin<->asw (both sides)
  • Check for connectivity

Done for onsite work

  • Upgrade cr2-eqsin
  • Check for all green
  • Repool site
  • Update DNS with new neighbor hostnames (and remove cr1) (DNS)
  • Update LibreNMS (remove cr1, bills, etc)
  • Update Netbox (cables)

Details

SubjectRepoBranchLines +/-
Replace cr1-eqsin with cr3-eqsinoperations/homer/publicmaster+6 -7
Depool eqsin for cr3-eqsin setupoperations/dnsmaster+2 -0
Remove cr1-eqsin loopback & rename relevant links (cr1->cr3)operations/dnsmaster+23 -27
cr1-eqsin -> cr3-eqsinoperations/puppetproduction+10 -10
Customize query in gerrit

Related Objects
Search...

Event Timeline

• ayounsi triaged this task as Medium priority.Jun 18 2020, 12:09 PM
• ayounsi created this task.
Restricted Application added a project: SRE. · View Herald TranscriptJun 18 2020, 12:09 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
• ayounsi added a subtask: T253246: (Need By: TBD) rack/setup/install cr3-eqsin.wikimedia.org.Jun 18 2020, 12:11 PM
gerritbot added a comment.Jun 18 2020, 12:20 PM

Change 606419 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/homer/public@master] Replace cr1-eqsin with cr3-eqsin

https://gerrit.wikimedia.org/r/606419

gerritbot added a project: Patch-For-Review.Jun 18 2020, 12:20 PM
gerritbot added a comment.Jun 18 2020, 12:32 PM

Change 606423 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/dns@master] Remove cr1-eqsin loopback & rename relevant links (cr1->cr3)

https://gerrit.wikimedia.org/r/606423

gerritbot added a comment.Jun 18 2020, 12:43 PM

Change 606425 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] cr1-eqsin -> cr3-eqsin

https://gerrit.wikimedia.org/r/606425

• ayounsi updated the task description. (Show Details)Jun 18 2020, 12:46 PM
• ayounsi updated the task description. (Show Details)
RobH added a comment.Jun 18 2020, 3:21 PM

It isn't clear to me if this is for Jin (DreamICC) or for Equinix remote hands. If it is Jin, and I'll be supervising them, I prefer we never do work in eqsin on Monday, as Monday AM there is Sunday PM for me, and I rather not lose my weekend to this (having to be at the keyboard @ 6PM Sunday ruins that day for me to do anything fun.)

If we are having Equinix remote hands do the work, same timeframe applies if you would like me to easily stay late and monitor the work as it progresses. If @ayounsi will be coordinating instead, then I have no preference.

  • Will I be coordinating this? (if so Tuesday-Thursday is best)?
  • Do we want Jin to do this (need more notice than this likely) or Equinix remote hands? I prefer Jin as they follow directions better.
• ayounsi added a comment.EditedJun 18 2020, 4:16 PM

It's a tradeoff between DC traffic, work quality and working hours for both remote hands and me.

  • eqsin peak traffic is between 5am UTC and 3pm UTC
  • My only requirement is to start it no later than 4pm UTC, as a 2h maintenance means finishing at 8pm my time
  • 4pm UTC means midnight Singapore time, which is terribly late for Jin, but might be ok for Equinix if they have 24/7 staff onsite
  • I think we all agree that Jin would do better than Equinix remote hands (especially as it's not a single task we need to do but several, with the small risk of having to rollback if something goes wrong

@RobH: I'm fine coordinating it if outside of your working hours.

RobH added a comment.EditedJun 18 2020, 5:08 PM

I've created a google doc, since Jin doesn't use phabricator, outlining all the steps above:

https://docs.google.com/document/d/1s2_ALpvDT9xTGihYE8BIXo41dSR9T11sZNUFs1mMrFM/edit?usp=sharing

I'll be emailing Jin shortly and cc'ing willy and arzhel on the email.

RobH mentioned this in Unknown Object (Task).Jun 18 2020, 5:22 PM
gerritbot added a comment.Jul 1 2020, 1:50 PM

Change 608882 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/dns@master] Depool eqsin for cr3-eqsin setup

https://gerrit.wikimedia.org/r/c/operations/dns/ /608882

gerritbot added a comment.Jul 1 2020, 3:00 PM

Change 608882 merged by Ayounsi:
[operations/dns@master] Depool eqsin for cr3-eqsin setup

https://gerrit.wikimedia.org/r/c/operations/dns/ /608882

Stashbot added a comment.Jul 1 2020, 3:00 PM

Mentioned in SAL (#wikimedia-operations) [2020-07-01T15:00:35Z] <XioNoX> depool eqsin for routers work - T255766

Stashbot added a comment.Jul 1 2020, 3:03 PM

Mentioned in SAL (#wikimedia-operations) [2020-07-01T15:03:09Z] <XioNoX> move vrrp master to cr2-eqsin - T255766

Stashbot added a comment.Jul 1 2020, 3:09 PM

Mentioned in SAL (#wikimedia-operations) [2020-07-01T15:09:48Z] <XioNoX> bump eqsin-codfw ospf link cost - T255766

Stashbot added a comment.Jul 1 2020, 3:13 PM

Mentioned in SAL (#wikimedia-operations) [2020-07-01T15:13:02Z] <XioNoX> disable cr1-eqsin transit/peering BGP - T255766

Stashbot added a comment.Jul 1 2020, 3:15 PM

Mentioned in SAL (#wikimedia-operations) [2020-07-01T15:15:23Z] <XioNoX> disable BGP to pybal on cr1-eqsin - T255766

Stashbot added a comment.Jul 1 2020, 3:16 PM

Mentioned in SAL (#wikimedia-operations) [2020-07-01T15:16:58Z] <XioNoX> re0.cr1-eqsin> request system power-off both-routing-engines - T255766

gerritbot added a comment.Jul 1 2020, 3:56 PM

Change 606425 merged by Ayounsi:
[operations/puppet@production] cr1-eqsin -> cr3-eqsin

https://gerrit.wikimedia.org/r/c/operations/puppet/ /606425

Stashbot added a comment.Jul 1 2020, 4:00 PM

Mentioned in SAL (#wikimedia-operations) [2020-07-01T16:00:13Z] <XioNoX> updating eqsin LVS BGP neighbors IPs - T255766

gerritbot added a comment.Jul 1 2020, 5:23 PM

Change 606423 merged by Ayounsi:
[operations/dns@master] Remove cr1-eqsin loopback & rename relevant links (cr1->cr3)

https://gerrit.wikimedia.org/r/c/operations/dns/ /606423

• ayounsi added a comment.Jul 1 2020, 5:37 PM

Replacement went smooth! Last step is to update Netbox.

gerritbot added a comment.Jul 1 2020, 5:49 PM

Change 606419 merged by Ayounsi:
[operations/homer/public@master] Replace cr1-eqsin with cr3-eqsin

https://gerrit.wikimedia.org/r/c/operations/homer/public/ /606419

• ayounsi mentioned this in rOHPUa35d39fee0f9: Replace cr1-eqsin with cr3-eqsin.Jul 1 2020, 5:52 PM
Maintenance_bot removed a project: Patch-For-Review.Jul 1 2020, 8:19 PM
• ayounsi closed this task as Resolved.Jul 2 2020, 8:13 AM
• ayounsi claimed this task.

Netbox updated.

• ayounsi closed subtask T253246: (Need By: TBD) rack/setup/install cr3-eqsin.wikimedia.org as Resolved.Jul 2 2020, 8:14 AM
• ayounsi mentioned this in T253246: (Need By: TBD) rack/setup/install cr3-eqsin.wikimedia.org.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits