This happened today again, starting at 20:05 and resulted in basically the same issues we observed over the weekend and in the previous incident. I have not identified the query and I leave that to people who know better, but the steps we took to resolve this were:

In T366360#9914657, @Dwisehaupt wrote:

Frack config has been updated to use the new ntp-[abc].anycast.wmnet servers. The previous dnsXXXX and ntp.anycast.wmnet entries have been removed.

Moving the links working out well (which I think this is the first time?) is a big take away from this task; glad to hear it went nicely!

I went ahead and created silence ID 71789606-ffdc-4244-a12e-d6344b1f1ab9. Sorry for being bold; please remove if required.

In T364931#9803735, @Clement_Goubert wrote:

Created silence d57b75c8-7b05-4651-8ec5-4bdb13d464f7 until Monday May 27th

On investigation, we found that (cp7001):

Thanks for the great analysis, Chris!

In T366360#9855483, @ayounsi wrote:

Last time we rolled out this change, it was simply updating modules/install_server/files/autoinstall/common.cfg. Do you have any other place in mind where this might need to be reconfigured? I am personally for removing this completely but it's not a big deal and we can keep it around as well.

https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/Opengear_Serial_Consoles
https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/ServerTech

They actually use the older naming/endpoint, which we might need to keep as CNAME as reconfiguring PDUs is manual.

In T366360#9854526, @ayounsi wrote:

Moving the dynamic nature of NTP definition to some automated system instead of human or Puppet is a great idea :)
Human as in right now for network devices, the list is hard-coded https://github.com/wikimedia/operations-homer-public/blob/master/config/common.yaml#L365

To clarify, there is no change to the configuration of the DNS hosts themselves and the peer list there. This is only for the consumers of P:systemd::timesyncd (which we don't use on the DNS hosts).

In T366193#9849495, @ayounsi wrote:

That's quite interesting seeing the variation of tradeoffs, and can be quite (an important) rabbithole. Is the goal to figure it out before anycasting ns1, or first anycast ns1 from anywhere then figure out how to modify the setup for possible better redundancy.
It could be useful to list all the failure scenarios, and if we need to mitigate them or not. (server or network missconfig, Bird bug, etc). In other words are we putting too many of our eggs in the same basket ?

In T366193#9845869, @ayounsi wrote:

assign a /24 from https://netbox.wikimedia.org/ipam/aggregates/ to be used for this

Hi @jhathaway: I wanted to get your input about this. The request here is to add a DKIM record for wikimedia.org so that learn.wiki can allow sending email from comdevteam@wikimedia.org. The CNAMEs above look fine so there are no concerns with that.

If you want to send from the wikimedia.org domain, then yes, that's what you will need. There is no concern with the records themselves.

Some numbers from cp7001, with the usual caveats around measuring this with openssl speed and commenting on this simply to understand the above-mentioned performance differences:

The CNAME here specifies wikimedia.org but I think it should be learn.wiki here. So instead of:

Host is depooled:

Thanks to @cmooney for rolling the above out. For further context, we (Traffic and netops) decided to try out the anycast range in magru for the Wikidough service before doing it for more critical things, such as announcing the ns2.wikimedia.org IP from magru. The timeline on that still is a TBD but this is a nice test to make sure that the configuration was correct.

This has been alerting for a while and in general has many alerts on the -ops channel. If this expected in some way (I have no idea and I haven't looked!) can it at least be silenced? This is not paging or anything just to be clear so that's fine.

In T364126#9785855, @akosiaris wrote:

Hi, @OSefu-WMF, @ssingh

A quick question:

Media (images, video, etc) are served from upload.wikimedia.org and are requested without Cookies. Is it possible to enable CP3 just for upload.wikimedia.org? I am not sure. Since CP3 is pretty new and technical details are a bit unclear, I don't know whether Chrome honors the CP3 disabling for e.g. en.wikipedia.org for subrequests as well or not. If it is possible to enable CP3 just for media, there is a chance (some?) Chrome end-users will see performance improvements without impacting the metrics mentioned in the linked GDoc

We have merged this change today and are serving the /.well-known/traffic-advice file with the content noted in the documentation to disable this feature.

On mr1-magru, I see 10.140.1.18 (prometheus7001) and denied by policy, which makes me wonder if we need to run https://netbox.wikimedia.org/extras/scripts/capirca.GetHosts/ for Capirca to generate the ACL?

On https://librenms.wikimedia.org/alerts, I see the following for mr1-magru:

OK, so I finally found why this is failing. For a reason that I don't fully understand, hieradata/magru/ directory actually needs to exist for the lookup() against hieradata/magru.yaml to work. See the commit that fixes this (and notice the commit message!):

Sorry, I forgot to reply to this!

The failure above is:

Thanks @cmooney, looks good! One small update to the above since we will most likely transpose these to hieradata/common/lvs/interfaces.yaml: 10.140.1.3/24 is private1-b4-magru like 10.140.0.2/24 is private1-b3-magru (and not just private-b4-magru).

@Lina_Farid_WMDE: to speed up things, you can also send an email to @KFrancis ( https://meta.wikimedia.org/wiki/User:KFrancis_(WMF) ; kfrancis@wikimedia.org) from your email, requesting an NDA. Thanks!

Hi @KFrancis: @Lina_Farid_WMDE will require an NDA as well as I don't see their name on the spreadsheet. Thank you as always!

Update is that we will need to add a DKIM record for MailChimp so a patch will follow. Rest everything seems to be in order.

We fixed it but forgot to close this task so resolving. Thanks @Dzahn!

Discussed a bit with @EdErhart-WMF on what the goal is here on Slack and will update this task later when there is more clarity.

This request is now merged; please re-open if there are any issues, thanks!

@Kgraessle: The request has been merged; please let us know if there is any issue. Thanks!

Thanks @jcrespo! I should have silenced the alert or restarted the service; both of those are in progress now so we should see this resolve soon.

Thanks for the task! At least for now, I restarted haproxy so that we don't get this alert and we also don't leave it silenced in case the initial restart (below) was nothing more than a transient one.

@DMburugu: this requires your approval, thanks!

@NBaca-WMF: This needs your approval, thanks!

@Aitolkyn I am marking this as resolved but if that's not the case, please re-open it again thanks!

@Papaul deserves a lot of love for fixing this persistent issue. The 21.x firmware (specifically, Network_Firmware_YK81Y_WN64_21.60.22.11_03) worked in the first attempt when reimaging cp1114. I think we can consider this closed given we have observed the fix on two hosts now.

@Papaul: Thanks for the update! Looks promising indeed and to actually close this, we should downgrade another host in eqiad and then try it out. Because what happens sometimes is that if a given host reimaged successfully once then it continues to reimage successfully for some more period of time (we don't know what that is but at least the same day :).

Thanks for the task @RobH! As in the previous runs, please feel free to leave these for Traffic:

Thanks @Muehlenhoff! And good to know @Michael that this is resolved; closing this task.

Added to wmf LDAP group (as well as Phabricator). Please try to access Logstash and let us know if there are any issues.

Marking this as resolved; if kinit doesn't work for you or if there are any issues, please re-open this. Thanks!

In T362602#9717940, @BTullis wrote:

I have created the principal for Surbhi.

btullis@krb1001:~$ sudo sudo manage_principals.py get sg912
get_principal: Principal does not exist while retrieving "sg912@WIKIMEDIA".
btullis@krb1001:~$ sudo manage_principals.py create sg912 --email_address=sgupta@wikimedia.org
Principal successfully created. Make sure to update data.yaml in Puppet.
Successfully sent email to sgupta@wikimedia.org

In T328457#9714365, @ssingh wrote:

@BTullis: Sorry if you are not the right person for this but it seems like we are having a related permissions issue in T362533 where @Aitolkyn is unable to use SQL Lab and I suspect that this is related. If not, please let me know. Thanks!

Thanks indeed @Urbanecm_WMF! Nice catch. @Aitolkyn: the contract expiry and date have been updated. If this has been resolved for you, please feel free to the task.

Can we open this task up now or we should still keep it private? The subtasks will still be private if this is public -- can someone confirm? If yes, we should open this up so that we can use it for tracking and if not, we should create a new public task.

@BTullis: Sorry if you are not the right person for this but it seems like we are having a related permissions issue in T362533 where @Aitolkyn is unable to use SQL Lab and I suspect that this is related. If not, please let me know. Thanks!

In T362533#9713602, @Aitolkyn wrote:

@ssingh Thank you for checking! I get the following error when trying to access my tables:

mysql error: SELECT command denied to user 'research'@'10.67.30.187' for table `aitolkyn`.`domain_reverted_added_2019_2023`

Maybe I need to set anything before I can access those?

Hi @FNavas-foundation: @Aitolkyn already should already have access to Superset as they are part of the analytics_privatedata_users group. @Aitolkyn, can you please confirm?