Wikimedia's eqsin datacenter (Asia Pacific) had network connectivity issues
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

None

Authored By

	jcrespo
	Jun 15 2021, 9:36 AM

Description

Datacenter depooled: https://gerrit.wikimedia.org/r/c/operations/dns/+/699910/

but DNS could not be updated imediately due to it not able to reach git.

Incident Document: https://docs.google.com/document/d/1_rV0RU9wZ0Y1VQUJkOq5L2uDUv-7XgOCuJyR6o5f_BY/edit

Details

	Project	Branch	Lines +/-	Subject
	operations/dns	master	+0 -1	Repool eqsin

Customize query in gerrit

Event Timeline

jcrespo created this task.Jun 15 2021, 9:36 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 15 2021, 9:36 AM

taavi added a project: Wikimedia-Incident.Jun 15 2021, 9:36 AM

taavi added a subscriber: taavi.

Peachey88 updated the task description. (Show Details)Jun 15 2021, 9:38 AM

SCP-2000 added a subscriber: SCP-2000.Jun 15 2021, 9:39 AM

SunAfterRain added a subscriber: SunAfterRain.Jun 15 2021, 9:43 AM

jcrespo renamed this task from Wikimedias eqsin datacenter has network connectivity issues (?) to Wikimedia's eqsin datacenter (Asia Pacific) had network connectivity issues.Jun 15 2021, 9:54 AM

Once telia issues have been resolved we need to repool ESQIN. @ayounsi can you confirm when we are good to repool

RhinosF1 added a subscriber: RhinosF1.Jun 15 2021, 10:22 AM

LSobanski added a subscriber: LSobanski.Jun 15 2021, 4:54 PM

I made a typo in the commit msg so this didn't link: https://gerrit.wikimedia.org/r/c/operations/dns/+/699957

Ok @Volans was kind enough to explain how I could just revert the original change instead:

https://gerrit.wikimedia.org/r/c/operations/dns/+/699859

So I will do that. BTW Telia issues looking good, any IPs I check are looking ok right now:

cmooney@alert1001:~$ mtr -c 1000 -b -w --address 208.80.154.88 103.102.166.6 
Start: 2021-06-15T17:49:40+0000
HOST: alert1001                                          Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- ae3-1003.cr2-eqiad.wikimedia.org (208.80.154.67)    0.0%  1000    3.5   1.1   0.2  42.2   2.9
  2.|-- ae0.cr1-eqiad.wikimedia.org (208.80.154.193)        0.0%  1000    0.2   0.6   0.2  18.6   1.5
  3.|-- xe-5-2-1.cr1-codfw.wikimedia.org (208.80.153.221)   0.0%  1000   30.1  31.1  30.1  56.4   3.1
  4.|-- xe-0-1-0.cr3-eqsin.wikimedia.org (103.102.166.138)  0.0%  1000  225.1 227.4 225.0 292.0   9.7
  5.|-- bast5002.wikimedia.org (103.102.166.6)              0.0%  1000  225.4 225.4 225.0 240.1   1.5

cmooney@alert1001:~$ mtr -c 1000 -b -w --address 208.80.154.88 103.102.166.13
Start: 2021-06-15T18:19:41+0000
HOST: alert1001                                          Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- ae3-1003.cr2-eqiad.wikimedia.org (208.80.154.67)    0.0%  1000    0.2   0.6   0.2  16.4   1.7
  2.|-- ae0.cr1-eqiad.wikimedia.org (208.80.154.193)        0.0%  1000    0.2   1.0   0.2  37.9   2.8
  3.|-- xe-5-2-1.cr1-codfw.wikimedia.org (208.80.153.221)   0.0%  1000   30.2  31.2  30.1  60.9   3.3
  4.|-- xe-0-1-0.cr3-eqsin.wikimedia.org (103.102.166.138)  0.1%  1000  225.1 227.2 224.9 291.4   9.2
  5.|-- install5001.wikimedia.org (103.102.166.13)          0.0%  1000  225.3 225.4 225.1 240.7   1.4

CR merged and DNS updated.

All looks good, dns servers are returning the eqsin IPs again and traffic is back to normal levels for this time of day. No errors like we had earlier that I can see.

Will continue to monitor for the next while to make sure we're ok.

cmooney closed this task as Resolved.Jun 16 2021, 7:35 AM

Wikimedia's eqsin datacenter (Asia Pacific) had network connectivity issuesClosed, ResolvedPublicBUG REPORTActions

Description

Details

Event Timeline

Wikimedia's eqsin datacenter (Asia Pacific) had network connectivity issues
Closed, ResolvedPublicBUG REPORT
Actions