Please configure the routers for Wikidough's anycasted IP
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ssingh
	May 24 2021, 1:37 PM

Description

T283027 is the main task that tracks the deployment of Wikidough as an anycasted service. One of the tasks there is, "[ ] configure the routers (Arzhel)" and in my discussions with Arzhel, we decided it's better if this is a separate task.

As per https://wikitech.wikimedia.org/wiki/Anycast#How_to_deploy_a_new_service?,

Configure the router side:
set protocols bgp group Anycast4 neighbor <server_IP>

Wikidough's IP is 185.71.138.138 [ see https://netbox.wikimedia.org/ipam/ip-addresses/8539/ ]. The current hosts are on codfw.

Additionally, the link above talks about:

(Optional) if deploying a new type of service, ask Netops to add a backup static route

Do you feel that this should be done for the Wikidough service?

Thank you very much for the help!

Details

Subject	Repo	Branch	Lines +/-
Add doh4002 to BGP anycast in ulsfo	operations/homer/public	master	+1 -0
Add doh5002 to BGP anycast in eqsin	operations/homer/public	master	+1 -0
Add doh1001 and doh1002 to BGP anycast in eqiad	operations/homer/public	master	+2 -0
Add doh4001 to BGP anycast in ulsfo	operations/homer/public	master	+1 -0
Add doh5001 to BGP anycast in eqsin	operations/homer/public	master	+1 -0
Correct IP address for doh3002 BGP peer in esams	operations/homer/public	master	+1 -1
Added doh3001 & doh3002 to Anycast peers in esams	operations/homer/public	master	+2 -0
Added Wikidough VMs to BGP Anycast codfw	operations/homer/public	master	+12 -2

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	ssingh	T252132 Deploy Wikimedia DNS: DNS-over-HTTPS (DoH) and DNS-over-TLS (DoT) public resolver
Resolved	ssingh	T283027 Offer Wikidough as an anycasted service
Resolved	ssingh	T283503 Please configure the routers for Wikidough's anycasted IP

Event Timeline

ssingh created this task.May 24 2021, 1:37 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 24 2021, 1:37 PM

Maintenance_bot added a project: SRE.May 24 2021, 1:45 PM

ssingh added a parent task: T283027: Offer Wikidough as an anycasted service.May 24 2021, 1:48 PM

Hey Sukhbir,

Can you confirm "doh2001" and "doh2002" are the VMs which will Anycast the Wikidough IP?

I'm not sure using a static route is the best idea. It's better if the DoH VMs themselves announce the route via BGP. I believe this should be done with Bird as described in the above link, but a brief check on doh2001 doesn't seem to show this is running there yet?

For the BGP Anycast config on the routers it will need a little more work than described in the link above. Currently the Anycast templates on the core routers in codfw would not allow the Wikidough anycast IP in (as it's from a differnt parent block than existing anycasted IPs). Additionally we need to create the aggregate /24 route and announce it to the internet when the single /32 you send is present. Arzhel and I will need to discuss the best way to do this. Shouldn't be hard.

thanks,

Cathal.

Marostegui triaged this task as Medium priority.May 24 2021, 3:57 PM

Marostegui moved this task from Backlog to Radar on the SRE board.

In T283503#7107926, @cmooney wrote:

Hey Sukhbir,

Can you confirm "doh2001" and "doh2002" are the VMs which will Anycast the Wikidough IP?

Sorry, I should have mentioned. Yes, doh2001, doh2002, and malmok. malmok will be deprecated later but we need to support it for now to allow current Wikidough users using the service via the malmok.wikimedia.org domain. (See: https://office.wikimedia.org/wiki/SRE/Wikidough). The idea is to move to the doh* hosts and wikimedia-dns.org.

I'm not sure using a static route is the best idea. It's better if the DoH VMs themselves announce the route via BGP.

Ah, I read the static route as a backup in addition to the VMs announcing themselves; not sure if it makes sense but the way I interpreted it is that the hosts will still be reachable via their host IPs, in addition to the anycasted IP.

but a brief check on doh2001 doesn't seem to show this is running there yet?

That's correct, the change is ready (https://gerrit.wikimedia.org/r/c/operations/puppet/+/692368) but I didn't merge it yet because I was not sure of the order: should the router configuration happen first, or should this change be merged, or it doesn't matter? :)

For the BGP Anycast config on the routers it will need a little more work than described in the link above. Currently the Anycast templates on the core routers in codfw would not allow the Wikidough anycast IP in (as it's from a differnt parent block than existing anycasted IPs). Additionally we need to create the aggregate /24 route and announce it to the internet when the single /32 you send is present. Arzhel and I will need to discuss the best way to do this. Shouldn't be hard.

Thanks for all the help with this task!

I spoke with Arzhel about it and he's pointed me in the right direction.

I think your change is good to merge, however we don't want to have a huge time gap between merging it and the one for the network side.

I will prepare that and if Arzhel is happy with it we can merge first your one, to add Bird to the VMs, and then shortly after the one I am creating which will cover the network side.

Cathal.

In T283503#7108144, @cmooney wrote:

I spoke with Arzhel about it and he's pointed me in the right direction.

I think your change is good to merge, however we don't want to have a huge time gap between merging it and the one for the network side.

I will prepare that and if Arzhel is happy with it we can merge first your one, to add Bird to the VMs, and then shortly after the one I am creating which will cover the network side.

Cathal.

Thank you, that sounds like a plan!

cmooney added a comment.May 24 2021, 5:15 PM

This comment was removed by cmooney.

Change 694305 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Added Wikidough VMs to BGP Anycast codfw

https://gerrit.wikimedia.org/r/694305

gerritbot added a project: Patch-For-Review.May 25 2021, 9:06 AM

Change 694305 merged by jenkins-bot:

[operations/homer/public@master] Added Wikidough VMs to BGP Anycast codfw

https://gerrit.wikimedia.org/r/694305

jenkins-bot mentioned this in rOHPU37f0bba4fb4b: Added Wikidough VMs to BGP Anycast codfw.May 26 2021, 3:18 PM

Mentioned in SAL (#wikimedia-operations) [2021-05-26T15:23:33Z] <sukhe@cumin1001> START - Cookbook sre.hosts.downtime for 0:45:00 on malmok.wikimedia.org with reason: applying anycast update: T283503

Mentioned in SAL (#wikimedia-operations) [2021-05-26T15:23:36Z] <sukhe@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:45:00 on malmok.wikimedia.org with reason: applying anycast update: T283503

Mentioned in SAL (#wikimedia-operations) [2021-05-26T16:09:47Z] <sukhe@cumin1001> START - Cookbook sre.hosts.downtime for 0:45:00 on malmok.wikimedia.org with reason: [WIP] applying anycast update: T283503

Mentioned in SAL (#wikimedia-operations) [2021-05-26T16:09:52Z] <sukhe@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:45:00 on malmok.wikimedia.org with reason: [WIP] applying anycast update: T283503

Maintenance_bot removed a project: Patch-For-Review.May 26 2021, 4:10 PM

Merged and pushed with homer to cr1-codfw and cr2-codfw, working ok with the first VM (Bird being enabled on others shortly):

cmooney@re0.cr2-codfw> show route receive-protocol bgp 208.80.153.43 terse 

inet.0: 833036 destinations, 2904522 routes (832935 active, 4 holddown, 321 hidden)
Restart Complete
  Prefix		  Nexthop	       MED     Lclpref    AS path
* 185.71.138.138/32       208.80.153.43                           64605 I

cathal@officepc:~$ mtr -b -w -z -c 4 185.71.138.138
Start: 2021-05-26T17:32:45+0100
HOST: officepc                                                                Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS???    _gateway (192.168.240.1)                                         0.0%     4    0.1   0.2   0.1   0.2   0.0
  2. AS6830   46.7.82.1                                                        0.0%     4   11.3  16.0   9.4  26.6   7.7
  3. AS6830   109.255.255.254                                                  0.0%     4   13.1  12.2   8.9  15.9   3.0
  4. AS6830   ie-dub01a-rc1-ae-31-0.aorta.net (84.116.238.42)                  0.0%     4    8.5   8.7   8.0   9.1   0.5
  5. AS6830   ie-dub02a-ri1-ae-73-0.aorta.net (84.116.134.110)                 0.0%     4   10.5  13.5   9.1  19.6   4.8
  6. AS1299   dln-b2-link.ip.twelve99.net (62.115.172.136)                     0.0%     4   14.5  11.1   9.4  14.5   2.4
  7. AS1299   ldn-bb1-link.ip.twelve99.net (62.115.120.100)                    0.0%     4   21.7  20.7  19.2  22.0   1.4
  8. AS1299   adm-bb3-link.ip.twelve99.net (213.155.136.99)                    0.0%     4   32.3  28.7  26.4  32.3   2.6
  9. AS1299   adm-b3-link.ip.twelve99.net (62.115.122.179)                     0.0%     4   31.8  29.4  27.9  31.8   1.7
 10. AS1299   wikimedia-ic316335-adm-b3.ip.twelve99-cust.net (62.115.145.25)   0.0%     4   30.3  29.9  25.3  35.5   4.2
 11. AS14907  ae0.cr2-esams.wikimedia.org (91.198.174.252)                     0.0%     4   28.9  30.0  26.2  36.2   4.3
 12. AS14907  xe-4-1-3.cr2-eqiad.wikimedia.org (91.198.174.248)                0.0%     4  125.4 121.0 116.7 125.4   4.0
 13. AS14907  xe-5-0-2.cr2-codfw.wikimedia.org (208.80.154.215)                0.0%     4  169.6 162.7 160.1 169.6   4.6
 14. AS14907  wikimedia-dns.org (185.71.138.138)                               0.0%     4  146.2 148.6 146.2 151.4   2.2

root@debiantest:~# dig +nsid +https www.ietf.org @wikimedia-dns.org

; <<>> DiG 9.17.13-2+0~20210520.56+debian11~1.gbp96c80e-Debian <<>> +nsid +https www.ietf.org @wikimedia-dns.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50195
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
; NSID: 6d 61 6c 6d 6f 6b ("malmok")
;; QUESTION SECTION:
;www.ietf.org.			IN	A

;; ANSWER SECTION:
www.ietf.org.		1357	IN	CNAME	www.ietf.org.cdn.cloudflare.net.
www.ietf.org.cdn.cloudflare.net. 300 IN	A	104.16.44.99
www.ietf.org.cdn.cloudflare.net. 300 IN	A	104.16.45.99

;; Query time: 155 msec
;; SERVER: 185.71.138.138#443(wikimedia-dns.org) (HTTPS)
;; WHEN: Wed May 26 18:04:18 BST 2021
;; MSG SIZE  rcvd: 128

Change 697970 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Added doh3001 & doh3002 to Anycast peers in esams

https://gerrit.wikimedia.org/r/697970

gerritbot added a project: Patch-For-Review.Jun 3 2021, 1:43 PM

Change 697970 merged by jenkins-bot:

[operations/homer/public@master] Added doh3001 & doh3002 to Anycast peers in esams

https://gerrit.wikimedia.org/r/697970

cmooney mentioned this in rOHPU08cad5343428: Added doh3001 & doh3002 to Anycast peers in esams.Jun 3 2021, 2:39 PM

Peerings to doh3001 and doh3002 added on cr1-esams and cr2-esams now.

Anycast range is being announced and from here in Ireland I'm hitting doh3001 :)

BEFORE:

cathal@officepc:~$ mtr -z -b -w -c 5 wikimedia-dns.org
Start: 2021-06-03T13:34:07+0100
HOST: officepc                                                                  Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS???    _gateway (192.168.240.1)                                           0.0%     5    0.1   0.1   0.1   0.2   0.0
  2. AS6830   46.7.82.1                                                          0.0%     5   11.1  16.3   6.5  34.2  10.7
  3. AS6830   109.255.255.254                                                    0.0%     5   12.2  10.8   9.3  12.2   1.2
  4. AS6830   ie-dub01a-rc1-ae-31-0.aorta.net (84.116.238.42)                    0.0%     5   11.2  11.4   8.5  14.1   2.2
  5. AS6830   ie-dub02a-ri1-ae-73-0.aorta.net (84.116.134.110)                   0.0%     5    9.7  10.1   9.6  11.1   0.7
  6. AS1299   dln-b2-link.ip.twelve99.net (62.115.172.136)                       0.0%     5    9.2  10.0   9.0  11.1   0.9
  7. AS1299   ldn-bb1-link.ip.twelve99.net (62.115.120.100)                      0.0%     5   21.7  21.2  19.8  22.3   1.1
  8. AS1299   nyk-bb2-link.ip.twelve99.net (62.115.113.20)                       0.0%     5   97.8  92.4  89.8  97.8   3.1
  9. AS1299   ash-bb2-link.ip.twelve99.net (62.115.136.201)                     60.0%     5   97.3  99.2  97.3 101.2   2.8
 10. AS1299   atl-b24-link.ip.twelve99.net (62.115.125.128)                      0.0%     5  107.2 109.8 106.9 114.6   3.1
 11. AS1299   dls-b24-link.ip.twelve99.net (62.115.120.113)                     20.0%     5  130.5 130.1 128.7 131.4   1.1
 12. AS1299   dls-b2-link.ip.twelve99.net (62.115.136.144)                       0.0%     5  127.8 126.6 124.2 127.9   1.8
 13. AS1299   wikimedia-ic308846-dls-b22.ip.twelve99-cust.net (80.239.192.102)   0.0%     5  131.3 132.4 131.3 133.4   0.9
 14. AS14907  wikimedia-dns.org (185.71.138.138)                                 0.0%     5  127.5 129.1 125.6 137.1   4.6

root@debiantest:~# dig +nsid +https www.ietf.org @wikimedia-dns.org
; <<>> DiG 9.17.13-2+0~20210520.56+debian11~1.gbp96c80e-Debian <<>> +nsid +https www.ietf.org @wikimedia-dns.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54401
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
; NSID: 64 6f 68 32 30 30 31 ("doh2001")
;; QUESTION SECTION:
;www.ietf.org.			IN	A

;; ANSWER SECTION:
www.ietf.org.		1371	IN	CNAME	www.ietf.org.cdn.cloudflare.net.
www.ietf.org.cdn.cloudflare.net. 174 IN	A	104.16.45.99
www.ietf.org.cdn.cloudflare.net. 174 IN	A	104.16.44.99

;; Query time: 140 msec
;; SERVER: 185.71.138.138#443(wikimedia-dns.org) (HTTPS)
;; WHEN: Thu Jun 03 15:54:34 BST 2021
;; MSG SIZE  rcvd: 129

AFTER:

cathal@officepc:~$ mtr -z -b -w -c 5 wikimedia-dns.org
Start: 2021-06-03T15:58:37+0100
HOST: officepc                                                                Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS???    _gateway (192.168.240.1)                                         0.0%     5    0.1   0.2   0.1   0.3   0.1
  2. AS6830   46.7.82.1                                                        0.0%     5    8.2  13.0   8.2  25.4   7.1
  3. AS6830   109.255.255.254                                                  0.0%     5   11.1   9.0   6.5  11.1   2.0
  4. AS6830   ie-dub01a-rc1-ae-31-0.aorta.net (84.116.238.42)                  0.0%     5   16.1  13.1  10.4  16.1   2.6
  5. AS6830   ie-dub02a-ri1-ae-73-0.aorta.net (84.116.134.110)                 0.0%     5   10.4  13.5   9.0  22.9   5.5
  6. AS1299   dln-b2-link.ip.twelve99.net (62.115.172.136)                     0.0%     5   12.4  10.9   9.0  12.4   1.2
  7. AS1299   ldn-bb1-link.ip.twelve99.net (62.115.120.100)                    0.0%     5   18.4  27.2  18.4  51.5  13.8
  8. AS1299   adm-bb3-link.ip.twelve99.net (213.155.136.99)                    0.0%     5   33.7  28.9  27.2  33.7   2.7
  9. AS1299   adm-b3-link.ip.twelve99.net (62.115.122.179)                     0.0%     5   32.4  30.4  26.6  33.5   2.7
 10. AS1299   wikimedia-ic316335-adm-b3.ip.twelve99-cust.net (62.115.145.25)   0.0%     5   30.3  34.9  28.6  48.1   7.8
 11. AS14907  wikimedia-dns.org (185.71.138.138)                               0.0%     5   26.1  29.2  26.1  35.7   3.9

root@debiantest:~# dig +nsid +https www.ietf.org @wikimedia-dns.org
; <<>> DiG 9.17.13-2+0~20210520.56+debian11~1.gbp96c80e-Debian <<>> +nsid +https www.ietf.org @wikimedia-dns.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27637
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
; NSID: 64 6f 68 33 30 30 31 ("doh3001")
;; QUESTION SECTION:
;www.ietf.org.			IN	A

;; ANSWER SECTION:
www.ietf.org.		1709	IN	CNAME	www.ietf.org.cdn.cloudflare.net.
www.ietf.org.cdn.cloudflare.net. 209 IN	A	104.16.45.99
www.ietf.org.cdn.cloudflare.net. 209 IN	A	104.16.44.99

;; Query time: 32 msec
;; SERVER: 185.71.138.138#443(wikimedia-dns.org) (HTTPS)
;; WHEN: Thu Jun 03 15:57:42 BST 2021
;; MSG SIZE  rcvd: 129

Maintenance_bot removed a project: Patch-For-Review.Jun 3 2021, 3:10 PM

Change 697993 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Correct IP address for doh3002 BGP peer in esams

https://gerrit.wikimedia.org/r/697993

gerritbot added a project: Patch-For-Review.Jun 3 2021, 3:37 PM

Change 697993 merged by jenkins-bot:

[operations/homer/public@master] Correct IP address for doh3002 BGP peer in esams

https://gerrit.wikimedia.org/r/697993

cmooney mentioned this in rOHPU7662e9ffdf45: Correct IP address for doh3002 BGP peer in esams.Jun 3 2021, 3:48 PM

There was an issue with peering to doh3002 due to a problem that occurred with Netbox automation, triggered by the VM creation running twice I believe.

Above change has been pushed and now all looks good, peering to doh3002 is also up.

cmooney@re0.cr3-esams> show bgp summary | match 64605    
91.198.174.7          64605        129        122       0       0       55:29 Establ
91.198.174.8          64605          8          5       0       0        2:05 Establ

root@debiantest:~# dig +nsid +https www.ietf.org @wikimedia-dns.org

; <<>> DiG 9.17.13-2+0~20210520.56+debian11~1.gbp96c80e-Debian <<>> +nsid +https www.ietf.org @wikimedia-dns.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59923
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
; NSID: 64 6f 68 33 30 30 32 ("doh3002")
;; QUESTION SECTION:
;www.ietf.org.			IN	A

;; ANSWER SECTION:
www.ietf.org.		1800	IN	CNAME	www.ietf.org.cdn.cloudflare.net.
www.ietf.org.cdn.cloudflare.net. 300 IN	A	104.16.44.99
www.ietf.org.cdn.cloudflare.net. 300 IN	A	104.16.45.99

;; Query time: 2964 msec
;; SERVER: 185.71.138.138#443(wikimedia-dns.org) (HTTPS)
;; WHEN: Thu Jun 03 16:50:11 BST 2021
;; MSG SIZE  rcvd: 129

Maintenance_bot removed a project: Patch-For-Review.Jun 3 2021, 4:11 PM

Change 698162 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/homer/public@master] Add doh5001 to BGP anycast in eqsin

https://gerrit.wikimedia.org/r/698162

gerritbot added a project: Patch-For-Review.Jun 4 2021, 9:46 AM

Change 698162 merged by jenkins-bot:

[operations/homer/public@master] Add doh5001 to BGP anycast in eqsin

https://gerrit.wikimedia.org/r/698162

ssingh mentioned this in rOHPU6f7b4ee23d04: Add doh5001 to BGP anycast in eqsin.Jun 4 2021, 10:31 AM

Maintenance_bot removed a project: Patch-For-Review.Jun 4 2021, 11:11 AM

doh5001 is also up; from Mumbai, we are reaching eqsin as desired:

$ kdig @wikimedia-dns.org +nsid +tls-ca wikipedia.org
;; TLS session (TLS1.3)-(ECDHE-SECP256R1)-(ECDSA-SECP256R1-SHA256)-(AES-256-GCM)
;; ->>HEADER<<- opcode: QUERY; status: NOERROR; id: 51983
;; Flags: qr rd ra; QUERY: 1; ANSWER: 1; AUTHORITY: 0; ADDITIONAL: 1

;; EDNS PSEUDOSECTION:
;; Version: 0; flags: ; UDP size: 512 B; ext-rcode: NOERROR
;; NSID: 646F6835303031 "doh5001"

;; QUESTION SECTION:
;; wikipedia.org.      		IN	A

;; ANSWER SECTION:
wikipedia.org.      	600	IN	A	103.102.166.224

;; Received 69 B
;; Time 2021-06-04 11:24:52 UTC
;; From 185.71.138.138@853(TCP) in 562

We just have one Wikidough host (doh5001) in eqsin for now; see T284246#7132717 for more information on what blocks doh5002.

Thanks for the continued help @cmooney!

Change 698971 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/homer/public@master] Add doh4001 to BGP anycast in eqsin

https://gerrit.wikimedia.org/r/698971

gerritbot added a project: Patch-For-Review.Jun 9 2021, 12:00 PM

Change 698971 merged by jenkins-bot:

[operations/homer/public@master] Add doh4001 to BGP anycast in ulsfo

https://gerrit.wikimedia.org/r/698971

ssingh mentioned this in rOHPU27935eb46397: Add doh4001 to BGP anycast in ulsfo.Jun 10 2021, 1:34 PM

INFO:homer.transports.junos:Committing the configuration on cr4-ulsfo.wikimedia.org
INFO:homer:Homer run completed successfully on 2 devices: ['cr3-ulsfo.wikimedia.org', 'cr4-ulsfo.wikimedia.org']

doh4001 and ulsfo configured.

I can confirm the 185.71.138.0/24 prefix is now being announced to peers from ulsfo, for example:

cmooney@cr4-ulsfo> show route advertising-protocol bgp 198.32.176.31 

inet.0: 837944 destinations, 1862385 routes (833756 active, 1 holddown, 4984 hidden)
Restart Complete
  Prefix		  Nexthop	       MED     Lclpref    AS path
* 185.15.56.0/24          Self                                    I
* 185.71.138.0/24         Self                                    I
* 198.35.26.0/24          Self                                    I
* 198.35.27.0/24          Self                                    I
* 198.73.209.0/24         Self                                    11820 ?

Change 699217 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/homer/public@master] Add doh1001 and doh1002 to BGP anycast in eqiad

https://gerrit.wikimedia.org/r/699217

Change 699217 merged by jenkins-bot:

[operations/homer/public@master] Add doh1001 and doh1002 to BGP anycast in eqiad

https://gerrit.wikimedia.org/r/699217

ssingh mentioned this in rOHPUb1f013210378: Add doh1001 and doh1002 to BGP anycast in eqiad.Jun 10 2021, 2:28 PM

INFO:homer.transports.junos:Committing the configuration on cr2-eqiad.wikimedia.org
INFO:homer:Homer run completed successfully on 2 devices: ['cr1-eqiad.wikimedia.org', 'cr2-eqiad.wikimedia.org']

doh1001 and doh1002 configured and 185.71.138.0/24 seems to be announced from eqiad.

Additional confirmation, since I am enjoying the reduced latency of the new Toronto -> eqiad route instead of the old Toronto -> codfw :)

kdig @wikimedia-dns.org +nsid +tls-ca wikipedia.org 
;; TLS session (TLS1.3)-(ECDHE-SECP256R1)-(ECDSA-SECP256R1-SHA256)-(AES-256-GCM)
;; ->>HEADER<<- opcode: QUERY; status: NOERROR; id: 28304
;; Flags: qr rd ra; QUERY: 1; ANSWER: 1; AUTHORITY: 0; ADDITIONAL: 1

;; EDNS PSEUDOSECTION:
;; Version: 0; flags: ; UDP size: 512 B; ext-rcode: NOERROR
;; NSID: 646F6831303031 "doh1001"

;; QUESTION SECTION:
;; wikipedia.org.      		IN	A

;; ANSWER SECTION:
wikipedia.org.      	600	IN	A	208.80.154.224

;; Received 69 B
;; Time 2021-06-10 10:35:18 EDT
;; From 185.71.138.138@853(TCP) in 175.5 ms

Yep! Seeing very nice latency from NY to wikidough now :)

root@nyc2:~# mtr -b -w -z -c 5 185.71.138.138
Start: 2021-06-10T16:35:02+0200
HOST: nyc2                                                                             Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS26548  162.249.171.225                                                           0.0%     5    0.6   0.5   0.4   0.6   0.1
  2. AS26548  169.197.81.66                                                             0.0%     5    1.4   0.7   0.4   1.4   0.4
  3. AS6939   100ge0-25.core1.nyc9.he.net (216.66.2.105)                                0.0%     5    1.4   1.3   1.1   1.4   0.1
  4. AS6939   100ge15-1.core1.nyc5.he.net (184.104.194.249)                             0.0%     5    0.9   1.1   0.9   1.3   0.2
  5. AS6939   100ge4-2.core1.nyc4.he.net (184.105.213.217)                              0.0%     5    1.3   3.2   1.2  10.7   4.2
  6. AS6939   100ge16-1.core1.ash1.he.net (184.105.223.165)                             0.0%     5   24.2  18.1   6.4  27.3  10.6
  7. AS6939   wikimedia-as14907.10gigabitethernet5.switch4.ash1.he.net (216.66.30.90)   0.0%     5    6.4   6.5   6.4   6.7   0.1
  8. AS14907  wikimedia-dns.org (185.71.138.138)                                        0.0%     5    6.5   6.6   6.4   6.7   0.1

Maintenance_bot removed a project: Patch-For-Review.Jun 10 2021, 3:10 PM

Marking this as resolved as we have completed all the intended tasks for now and the routers have been configured.

On our (Traffic's) end, what remains is the doh4002 and doh5002 hosts that block on the lack of additional IPs in ulsfo and eqsin. But given that we have completed the deployment on other hosts, those are just a matter of bringing up the VMs and doing the homer configuration.

@cmooney and @ayounsi: thanks very much for all your help in making this task easy!