Page MenuHomePhabricator

knams equipment move
Closed, ResolvedPublic

Description

Vancis needs to relocate our equipment to room S043 rack Z7.

This has been scheduled for this Thursday 13th at 13:00 local time (11:00 UTC), and is expected to last 4h.

bblack> we'll need to depool user traffic away from esams ahead of the maintenance, preferably ~24h ahead just in case

Event Timeline

ayounsi created this task.Apr 10 2017, 12:59 PM
Restricted Application added a project: Operations. · View Herald TranscriptApr 10 2017, 12:59 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
elukey added a subscriber: elukey.Apr 10 2017, 1:25 PM
ayounsi added a subscriber: BBlack.Apr 10 2017, 2:20 PM

After discussion with @BBlack
As knams going down will not impact connectivity between esams and eqiad, and esams has enough transit capacity to take over knams transits, the following plan has been agreed on:
~1h before: disable esams in DNS
~30m before: disable transit BGP sessions in knams
When the maintenance is done and confirmed successful, re-enable BGP sessions, then enable esams in DNS.

ema moved this task from Triage to General on the Traffic board.Apr 11 2017, 12:32 PM

Change 348056 had a related patch set uploaded (by Ayounsi):
[operations/dns@master] Depooling ESAMS for T162601

https://gerrit.wikimedia.org/r/348056

Change 348056 merged by Ayounsi:
[operations/dns@master] Depooling ESAMS for T162601

https://gerrit.wikimedia.org/r/348056

Mentioned in SAL (#wikimedia-operations) [2017-04-13T10:42:14Z] <XioNoX> disable V6 transit BGP session on cr2-knams for T162601

Mentioned in SAL (#wikimedia-operations) [2017-04-13T10:47:09Z] <XioNoX> Confirmed we can still reach cr2-knams:lo0 via v6 (from esams), disabling IPv4 transit sessions for T162601

Work finished at 14:00 local time, interfaces confirmed up, LACP active.
BGP re-enabled at ~14:30. Everything established, interfaces passing traffic.
Will revert the DNS de-pooling at ~15:00.

ayounsi closed this task as Resolved.Apr 13 2017, 1:21 PM

esams reenabled in DNS, confirmed traffic is properly passing through knams.