Hi all. I have brought this up and are discussing this internally in SRE. I will update the ticket when I have more information. Thank you.

@Jgiannelos: Thanks for the additional confirmation; the SSH key has been updated. Marking this as resolved, please feel free to reopen if there are any issues.

In T284987#7158017, @ChristineDeKock wrote:

Hi @ssingh. Thank you for your trouble. I have signed the L3 form.

Hi @ChristineDeKock: Can you please read through and sign the L3 (Acknowledgement of Wikimedia Server Access Responsibilities) document? Thank you!

Marc has been added to VictorOps and the SRE team; resolving this task. Thanks, all.

@mepps: You have been added to the restricted group. Please let us know if there are any questions, thanks!

In T281344#7140212, @ssingh wrote:

In T281344#7139967, @Volans wrote:

Anything still pending here on the SRE-Access-Requests side?

Thanks for checking! I updated pwstore as that was completed; I am not sure about the remaining item so I will let @MMandere comment and we can work towards completing that.

Marking this as resolved as we have completed all the intended tasks for now and the routers have been configured.

Additional confirmation, since I am enjoying the reduced latency of the new Toronto -> eqiad route instead of the old Toronto -> codfw :)

INFO:homer.transports.junos:Committing the configuration on cr2-eqiad.wikimedia.org
INFO:homer:Homer run completed successfully on 2 devices: ['cr1-eqiad.wikimedia.org', 'cr2-eqiad.wikimedia.org']

doh1001 and doh1002 have been created; closing this task. Thanks for the help!

INFO:homer.transports.junos:Committing the configuration on cr4-ulsfo.wikimedia.org
INFO:homer:Homer run completed successfully on 2 devices: ['cr3-ulsfo.wikimedia.org', 'cr4-ulsfo.wikimedia.org']

In T284349#7135796, @colewhite wrote:

Cookbook ran successfully. Currently unprovisioned.

In T281344#7139967, @Volans wrote:

Anything still pending here on the SRE-Access-Requests side?

doh5001 is also up; from Mumbai, we are reaching eqsin as desired:

In T284246#7133295, @Dzahn wrote:

@ssingh doh5001.wikimedia.org is ready for you now. doh5002 on hold for lack of IP in that subnet.

Thanks for all the help and sorry it took a while! I think 10G should be fine for now given the current usage on the other Wikidough hosts.

Puppet roles/profiles/hiera configurations have not been removed as they will be needed for the cescout VM that will be provisioned later.

In T283614#7116994, @cmooney wrote:

This service is now live in codfw, answering DoH, DoT and plain-old DNS.

I believe dnsdist is terminating the DoH/DoT, but regular queries on UDP/TCP 53 go directly to Power DNS Recursor. No problem with that, but worth noting we only get something back for NSID when we talk to dnsdist, and I assume with the Atlas probes we can only make regular queries.

root@debiantest:~# dig +nsid +https www.ietf.org @wikimedia-dns.org

; <<>> DiG 9.17.13-2+0~20210520.56+debian11~1.gbp96c80e-Debian <<>> +nsid +https www.ietf.org @wikimedia-dns.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39806
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
; NSID: 6d 61 6c 6d 6f 6b ("malmok")
;; QUESTION SECTION:
;www.ietf.org.			IN	A

;; ANSWER SECTION:
www.ietf.org.		944	IN	CNAME	www.ietf.org.cdn.cloudflare.net.
www.ietf.org.cdn.cloudflare.net. 300 IN	A	104.16.44.99
www.ietf.org.cdn.cloudflare.net. 300 IN	A	104.16.45.99

;; Query time: 159 msec
;; SERVER: 185.71.138.138#443(wikimedia-dns.org) (HTTPS)
;; WHEN: Wed May 26 18:11:11 BST 2021
;; MSG SIZE  rcvd: 128

root@debiantest:~# dig +nsid +tls www.ietf.org @wikimedia-dns.org

; <<>> DiG 9.17.13-2+0~20210520.56+debian11~1.gbp96c80e-Debian <<>> +nsid +tls www.ietf.org @wikimedia-dns.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50986
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
; NSID: 6d 61 6c 6d 6f 6b ("malmok")
;; QUESTION SECTION:
;www.ietf.org.			IN	A

;; ANSWER SECTION:
www.ietf.org.		275	IN	CNAME	www.ietf.org.cdn.cloudflare.net.
www.ietf.org.cdn.cloudflare.net. 300 IN	A	104.16.44.99
www.ietf.org.cdn.cloudflare.net. 300 IN	A	104.16.45.99

;; Query time: 348 msec
;; SERVER: 185.71.138.138#853(wikimedia-dns.org) (TLS)
;; WHEN: Wed May 26 18:22:20 BST 2021
;; MSG SIZE  rcvd: 128

root@debiantest:~# dig +nsid www.ietf.org @wikimedia-dns.org

; <<>> DiG 9.17.13-2+0~20210520.56+debian11~1.gbp96c80e-Debian <<>> +nsid www.ietf.org @wikimedia-dns.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35293
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1220
; COOKIE: bd8627f07957ae320100000060ae81eb55bd3b21e9c23314 (good)
;; QUESTION SECTION:
;www.ietf.org.			IN	A

;; ANSWER SECTION:
www.ietf.org.		1800	IN	CNAME	www.ietf.org.cdn.cloudflare.net.
www.ietf.org.cdn.cloudflare.net. 300 IN	A	104.16.44.99
www.ietf.org.cdn.cloudflare.net. 300 IN	A	104.16.45.99

;; Query time: 2467 msec
;; SERVER: 185.71.138.138#53(wikimedia-dns.org) (UDP)
;; WHEN: Wed May 26 18:14:19 BST 2021
;; MSG SIZE  rcvd: 146

In T283503#7108144, @cmooney wrote:

I spoke with Arzhel about it and he's pointed me in the right direction.

I think your change is good to merge, however we don't want to have a huge time gap between merging it and the one for the network side.

I will prepare that and if Arzhel is happy with it we can merge first your one, to add Bird to the VMs, and then shortly after the one I am creating which will cover the network side.

Cathal.

In T283503#7107926, @cmooney wrote:

Hey Sukhbir,

Can you confirm "doh2001" and "doh2002" are the VMs which will Anycast the Wikidough IP?

In T283192#7099749, @Dzahn wrote:

VMs have been created, added to site.pp with "insetup", added to DHCP and partma.

OS has been installed (buster) and puppet certs signed. You can now SSH to them.

In T283192#7099503, @Dzahn wrote:

@ssingh Please add "doh" to the list of clusters on https://wikitech.wikimedia.org/wiki/Infrastructure_naming_conventions

@ayounsi, @Aklapper: I forgot to update this ticket, apologies! Last I checked, all reports were in the green and there wasn't any debugging information. But since we have had no user updates and it's been a while since I looked, I will check again and update this ticket accordingly. Thanks!

@MMandere: Added to "ops" LDAP group. https://wikitech.wikimedia.org/wiki/LDAP/Groups#ops_group

Hi @JAllemandou: thanks, I think proceeding through Wikimedia Taiwan as @Shizhao mentioned is probably a good idea. In the meantime, I will ask for a contact in the community and also if the partnerships team has some idea (we have asked for their help in the recent past with a similar issue).

We decided to file https://github.com/citizenlab/test-lists/pull/730 so that we can get some test data.

In T275409#6849975, @Legoktm wrote:

For reference, https://wiki.mozilla.org/Security/DOH-resolver-policy has links to the privacy policies of the Mozilla-approved resolvers.

I edited the task but I wanted to add here as well that this happens when I run the Hive query from Spark.

Thanks for opening this task, Marcel.

Thank you for the interest in this task! Like we shared yesterday, we have identified that the traffic is coming from a popular mobile app in India. We have initiated contact with the app developers, and are waiting to hear back from them. In the meantime, given the volume of requests, we have decided to ban those specific requests until the issue is resolved. While we will refrain from naming the app at this time, we can share that it is not on the list of apps mentioned in this task. Nevertheless, we thank you for your comments and suggestions on how to debug this!

Thank you everyone for the comments and suggestions. I just wanted to share that we have identified the app and will update this task tomorrow. (And yes, it is a mobile app.)

In T273741#6813536, @Daniel.gayo wrote:

Could it be this app?

https://apps.apple.com/hk/app/iclass-corporate/id1439400748?l=en

The picture appears in a screenshot...

In T273741#6813531, @Michaelrhanson wrote:

I found several places where this URL is being used in sample code, which might have been picked up by somebody and built into an app:

https://stackoverflow.com/questions/18586466/foursqaure-photo-add-against-checkin
https://stackoverflow.com/questions/18232898/node-js-http-get-with-node-js-step-module
https://html.developreference.com/article/14455997/Downloading+image+from+the+web+with+imagemagick+and+saving+to+parse

seems like this particular flower has been kicking around as a sample image for quite a few years.