Hi @JAllemandou: thanks, I think proceeding through Wikimedia Taiwan as @Shizhao mentioned is probably a good idea. In the meantime, I will ask for a contact in the community and also if the partnerships team has some idea (we have asked for their help in the recent past with a similar issue).

We decided to file https://github.com/citizenlab/test-lists/pull/730 so that we can get some test data.

In T275409#6849975, @Legoktm wrote:

For reference, https://wiki.mozilla.org/Security/DOH-resolver-policy has links to the privacy policies of the Mozilla-approved resolvers.

I edited the task but I wanted to add here as well that this happens when I run the Hive query from Spark.

Thanks for opening this task, Marcel.

Thank you for the interest in this task! Like we shared yesterday, we have identified that the traffic is coming from a popular mobile app in India. We have initiated contact with the app developers, and are waiting to hear back from them. In the meantime, given the volume of requests, we have decided to ban those specific requests until the issue is resolved. While we will refrain from naming the app at this time, we can share that it is not on the list of apps mentioned in this task. Nevertheless, we thank you for your comments and suggestions on how to debug this!

Thank you everyone for the comments and suggestions. I just wanted to share that we have identified the app and will update this task tomorrow. (And yes, it is a mobile app.)

In T273741#6813536, @Daniel.gayo wrote:

Could it be this app?

https://apps.apple.com/hk/app/iclass-corporate/id1439400748?l=en

The picture appears in a screenshot...

In T273741#6813531, @Michaelrhanson wrote:

I found several places where this URL is being used in sample code, which might have been picked up by somebody and built into an app:

https://stackoverflow.com/questions/18586466/foursqaure-photo-add-against-checkin
https://stackoverflow.com/questions/18232898/node-js-http-get-with-node-js-step-module
https://html.developreference.com/article/14455997/Downloading+image+from+the+web+with+imagemagick+and+saving+to+parse

seems like this particular flower has been kicking around as a sample image for quite a few years.

Change merged and tested; broken security polling is now disabled.

In T269256#6751931, @srodlund wrote:

Thanks, @ssingh! I've addressed these issues. I did have to replace the formula with an actual image, as the formatting was not copying over from the doc, and I wasn't able to format it correctly as text in the blog. Let me know if this looks okay to you!

In T269256#6751807, @srodlund wrote:

@Nuria This is published! https://techblog.wikimedia.org/2021/01/15/censorship-outages-and-internet-shutdowns-monitoring-wikipedias-accessibility-around-the-world/

Can you take a look and let me know if everything looks good to you and if there is anything that needs correction?

Once I have your go ahead, I'll announce it more widely.

@ssingh @elukey

I've been looking into this a bit and have had some second thoughts.

With https://gerrit.wikimedia.org/r/639838 merged, I am going to mark this as resolved as the first iteration of the test suite for Wikidough is now complete.

Hi, thanks for your work on this task! Is my understanding correct that this blocks on T241195?

Thanks @Dzahn for th review and the additional context.

Hi @Jakob_WMDE: This request has been merged and you should have received an email with the Kerberos password. Please let us know if there are any issues. Thanks!

Hi @Swagoel: You should have received an email with the Kerberos password. Please let us know if there are any issues, thanks!

(Additional context: T267314).

Request has been merged and user has been added to the wmf group. Please reopen if there are any issues. Thanks!

In T269444#6669321, @Ottomata wrote:

APPROVED!

The requested mailing list has been created. Marking this as resolved but please reopen if there are any issues. Thanks!

The list testing-infrastructure has been created and you should have received an email.

@gengh: Access request merged; please let me know if there are any issues. Welcome!

Thanks for confirming!

Key confirmed out-of-band; request merged. Thanks! Feel free to reopen if there are any issues.

@WMDE-leszek: This request has been merged, you should have access! I will leave this task open for another day or so but please let me know if there are any issues. Thanks!

Change has been requested by a WMDE manager themselves (https://wikitech.wikimedia.org/wiki/SRE_Clinic_Duty#wmde_access) and approved by Andrew.

@thcipriani: Adding you to this task to see if you have any possible concerns about this request as it involves the use of the release servers. Thanks!

Thanks for the confirmation Katie!

Hi, just adding to this ticket here as I have a CI job failing that would be alleviated by fixing this issue (by specifying Python 3.8 as the interpreter): https://integration.wikimedia.org/ci/job/tox-docker/15602/console.

Hi! Following up on these tickets now; sorry for the delay.

Another important change in 1.5.0 is https://github.com/PowerDNS/pdns/pull/7138 [dnsdist/rec: Drop remaining capabilities after startup]. For our dnsdist instance, this is handled for dnsdist.conf and the TLS certs by the following:

For the DoH endpoints, we were already setting the paths explicitly (at https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/dnsdist/templates/dnsdist.conf.erb#55) and also have /dns-query hardcoded (at https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/dnsdist/manifests/init.pp#64), so no change is required for the upgrade.

I am marking this as "WMF-NDA" in case something like wikipedia20.org is required (discussed with Hang on Slack).

Yes, sure! I think I can take care of the systemd timer part.

In T257527#6437113, @CDanis wrote:

Rollout planning braindump

There's three degrees of freedom to play with here:

1. The set of domains for which we request reports
2. The sampling fraction we set for all of/each of those (when a user agent sees an error, how often does it create a report for that error?)
3. The TTL we set for how long user agents will persist the above

I'm thinking the set of domains should roughly follow wiki deployment groups, as they're intended for phased rollouts that roughly follow wiki 'importance' / size of userbase affected, and also they make intuitive sense & are easy to explain).

I'm also thinking that TTLs should initially be short: let's say 1 hour to start with, and later on increasing up to something on the order of a week? The tradeoff to be made with TTLs is:

• on a long TTL, if we also set the sampling fraction too high in a record that then persists for a long time, then eventgate and/or logstash could be overwhelmed in the event of a large outage.

• on a too-short TTL, we won't get reports at all when infrequent users experience errors

We could probably do some analysis to figure out the per-user distribution of pageview inter-arrival times (or maybe that's already known? # of 1-day active users vs 7da vs 30da gives you enough of an idea), but I was thinking to keep it simple and just say "1 hour TTL to start, 1 week TTL when we're confident". The sampling fraction matters much more, anyway.

As for setting a sampling fraction, that's where things get really tricky.

• Some domains are vastly more popular than other domains. There are also per-domain concerns, like vastly different geographical and geopolitical distributions of users, etc. In the long run, we very likely want different sampling fractions for different domains.

In T259816#6400068, @jbond wrote:

So this means that they treat the DO bit to not only return the DNSSEC records but also to validate them? I can check this in the code but I just wanted to confirm if I am understanding this correctly. There is a table here https://docs.powerdns.com/recursor/dnssec.html#what-when that explains what happens and when re: DNSSEC validations.

DO or AD but yes i agree however i believe this is a PDNS implementation detail and not something specified in the RFC's

My understanding was that the role of the AD flag was redefined can now also be used by the client in the DNS query to explicitly ask for validation, meaning that it signals that it can understand and is interested in the response of the AD bit. (https://tools.ietf.org/html/rfc6840#section-5.7)

Indeed it was redefined however its new definition does not ask for validation it just asks for the AD bit to be returned in the answer. This could be used for instance to indicate in the browser tool bar that a domain is DNSSEC validated. i.e. if i do dig +noad ns ripe.net @1.1.1.1 dnssec validation is still preformed by the cloudflare servers but the AD bit is not set in the answer so the client has no way of knowing that.

In T259816#6399814, @jbond wrote:

Given that outages due to misconfigured DNSSEC domains are all too common (see https://ianix.com/pub/dnssec-outages.html for a list)

Im not sure i would agree that they are "all to common". The list referenced shows failures in less the 30 "Major sites"*. Further the TLD's shown, other then ru and pl, seem pretty small.

Firefox has no way of distinguishing between a SERVFAIL response that resulted from a misconfigured auth server or from an actual bogus response.

the client can ask the question again with +CD to rule out validation or to try and preform it themself, however i doubt FF/Chrome do this

The only remaining item on this task is the "cloud (labs) groups: deployment-prep", that is better suited for the releng team. I am marking this as resolved from the SRE side and assigning it to them but please feel to reopen if there are any issues. Thank you.