What are the downsides to using iptables rules for this?

I think we're in good shape here now.

This would have been helpful in troubleshooting T264504

In T264504#6532332, @Xqt wrote:

I tried to get a confirmation mail again and got it. It worked as expected now. Thanks a lot.

wiki-mail-codfw.wikimedia.org has been delisted. This should resolve the issue outlined in the description here.

Actually, after some further manual testing I think we have a reason:

With regard to why mail from eqiad seemed to be working while codfw was not -- part of this is because the working email examples are gerrit mails, which in addition to having different message contents are also sent outward via the main mx host interface instead of the wiki-mail-site.wikimedia.org bulk mail interface.

Hi @Sbailey, the updated SSH key has been deployed to servers by now. Please re-open if any follow-up is needed. Thanks!

New key has been confirmed via google chat and email

This has been done, I'll transition to resolved now

Hi @Sbailey I've reached out to you via google chat and by email to verify. Thanks!

The file bast2002:/home/urbanecm/id_ed25519_wmnet_20201001.pub.sig does indeed match the key in the description, and on the patch

Is there another host in production where you have working access? Placing a file there would work too, just let me know where to check. Otherwise we can figure out another method. Thanks!

The requested access has been enabled and will become active within the next 30 minutes. I'll transition this task to resolved now, but please don't hesitate to re-open if any follow-up is needed. Thanks!

Since this is somewhat of an atypical access request (in that the account and group membership are pre-existing, but attributes are changing) please have a close look at https://gerrit.wikimedia.org/r/631455 to ensure it matches the expected outcome. Thanks in advance!

Hi @Sbailey as a security precaution, could you please use your existing shell access to upload the desired new ssh key onto one of the bastions (let's say bast1002) as a file in your home directory called sbailey_new_ssh_key? Once done and confirmed we'll be ready to move forward with the above patch. Thanks in advance!

Removing the SRE-Access-Requests tag for now, please re-add when ready to proceed with this. Thanks!

In T263662#6488383, @jcrespo wrote:

This should be fixed now in alert1001 and alert2001. The issue was that my puppet only had ensure_package(wmfbackups-check) but I didn't upgrade to 0.2 on alert* hosts, only the icinga ones.

In T263662#6488341, @jcrespo wrote:

I see the issue, after apt upgrade, the 0.2 version was available, which was the one that worked. It must have been caught in the small window between first deploy (0.1) and 0.2 fix, and I didn't notice alert1001 was about to be made the primary host.

0.1 had missing dependencies. Those are fixed on 0.2. Now it works. Apologies.

In T263662#6488323, @jcrespo wrote:

Was there an OS upgrade on the change from icinga1001 to alert1001? I must not have uploaded all versions for all os versions, or something.

Alert1001 is now the active Icinga server. Meta monitoring for alert[12]001 has been enabled as well.

Thanks @JMeybohm, ok I think we should defer to your expertise with regard to the optimal way to output these logs from the Kubernetes environment.

In T262675#6459313, @JMeybohm wrote:

What I think I need from your side is mainly the "okay" to push those events to the logstash-* indices of the elasticsearch cluster (I can try to figure out what that would mean in terms of documents per day, size etc. - as you might need some numbers there I guess) and probably some support in how to access it (set up needed credentials/accounts etc.). But if you have any objections in general or better ideas on how to do this, please let me know.

The buster kafkamon hosts are now live. Will let them settle for a bit before moving on to cleanup/teardown of the old hosts.

Icinga/alerts certificate issue has been fixed and meta monitoring is now working against the new alert[12]001 hosts.

Thanks @Volans, sync_check_icinga_contacts is happy now on alert[12]001

Hey @elukey, prep work is done for the new hosts. Will be performing cut-over in the near future, will keep you on the cc.

In T234854#6344456, @jcrespo wrote:

I am getting a lot of 500 internal server errors on logstash-next instance. I am guessing that is expected/WIP?

Currently the sync_check_icinga_contacts unit is failed on alert1001. I've armed the keyholder, but am not sure if there's an additional step to carry out on the wikitech-static host to permit the key from a new host. Or even if the sync should be running from multiple places at the same time.

Thanks! These Galera checks are now green in the new icinga instance

This came up again this morning in codfw, cluster went yellow due to shard allocation failure on logstash-2020.08.18 index.

lists.wikimedia.org is now running from the buster host lists1001.wikimedia.org.

Saw this same Data too large, data for... error also affecting shard allocation on the HDD hosts yesterday. Bumping the heap on the eqiad HDD hosts manually from 24G to 26G and issuing a /_cluster/reroute?retry_failed=true cleared it. Uploaded https://gerrit.wikimedia.org/r/619032 to persist the setting (and for deploy to codfw)

I've updated the description to outline the two auto-retrigger and auto-resolve options as available by VO today.

@Nuria could you please review and give a thumbs up/down on the request for analytics-privatedata-users membership?