Page MenuHomePhabricator

Wikimedia's eqsin datacenter (Asia Pacific) had network connectivity issues
Closed, ResolvedPublicBUG REPORT

Description

Datacenter depooled: https://gerrit.wikimedia.org/r/c/operations/dns/+/699910/

but DNS could not be updated imediately due to it not able to reach git.

Incident Document: https://docs.google.com/document/d/1_rV0RU9wZ0Y1VQUJkOq5L2uDUv-7XgOCuJyR6o5f_BY/edit

Details

Event Timeline

jcrespo renamed this task from Wikimedias eqsin datacenter has network connectivity issues (?) to Wikimedia's eqsin datacenter (Asia Pacific) had network connectivity issues.Jun 15 2021, 9:54 AM

Once telia issues have been resolved we need to repool ESQIN. @ayounsi can you confirm when we are good to repool

Ok @Volans was kind enough to explain how I could just revert the original change instead:

https://gerrit.wikimedia.org/r/c/operations/dns/+/699859

So I will do that. BTW Telia issues looking good, any IPs I check are looking ok right now:

cmooney@alert1001:~$ mtr -c 1000 -b -w --address 208.80.154.88 103.102.166.6 
Start: 2021-06-15T17:49:40+0000
HOST: alert1001                                          Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- ae3-1003.cr2-eqiad.wikimedia.org (208.80.154.67)    0.0%  1000    3.5   1.1   0.2  42.2   2.9
  2.|-- ae0.cr1-eqiad.wikimedia.org (208.80.154.193)        0.0%  1000    0.2   0.6   0.2  18.6   1.5
  3.|-- xe-5-2-1.cr1-codfw.wikimedia.org (208.80.153.221)   0.0%  1000   30.1  31.1  30.1  56.4   3.1
  4.|-- xe-0-1-0.cr3-eqsin.wikimedia.org (103.102.166.138)  0.0%  1000  225.1 227.4 225.0 292.0   9.7
  5.|-- bast5002.wikimedia.org (103.102.166.6)              0.0%  1000  225.4 225.4 225.0 240.1   1.5
cmooney@alert1001:~$ mtr -c 1000 -b -w --address 208.80.154.88 103.102.166.13
Start: 2021-06-15T18:19:41+0000
HOST: alert1001                                          Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- ae3-1003.cr2-eqiad.wikimedia.org (208.80.154.67)    0.0%  1000    0.2   0.6   0.2  16.4   1.7
  2.|-- ae0.cr1-eqiad.wikimedia.org (208.80.154.193)        0.0%  1000    0.2   1.0   0.2  37.9   2.8
  3.|-- xe-5-2-1.cr1-codfw.wikimedia.org (208.80.153.221)   0.0%  1000   30.2  31.2  30.1  60.9   3.3
  4.|-- xe-0-1-0.cr3-eqsin.wikimedia.org (103.102.166.138)  0.1%  1000  225.1 227.2 224.9 291.4   9.2
  5.|-- install5001.wikimedia.org (103.102.166.13)          0.0%  1000  225.3 225.4 225.1 240.7   1.4

CR merged and DNS updated.

All looks good, dns servers are returning the eqsin IPs again and traffic is back to normal levels for this time of day. No errors like we had earlier that I can see.

Will continue to monitor for the next while to make sure we're ok.