In T230236#5627261, @BBlack wrote:

https://grafana.wikimedia.org/d/B9JpocKZz/ipsec-tunnel-status probably needs some cleanup (some of the graphs are empty, there's a note there to ignore icinga errors, etc). Also fix missing doc link on the alert?

In T236497#5644446, @BBlack wrote:

Sorry I missed that you already had a patch! But in any case, we only need commenting from cache::nodes to fix up this case (there's no good reason to e.g. churn it out of conftool or the various iptables rules defined from the other stuff).

Since it looks like cp3056 might be down for some time could we remove it from the config until fixed? It would be good to let the ipsec checks in icinga return to green.

In terms of “what” should be escalated, so far we discussed

Yes, I think we're in good shape here

Host monitoring for SCS systems has been added to icinga

Essentially a duplicate of T220390 where audit work has gone into documenting clusters and use cases

The document https://docs.google.com/document/d/1mr217D6eyoGvGUG31M-FVMve9MCOCRKZxUWFZOZirQw/edit#heading=h.mbousz3hsm22 was created & shared via the goal meetings during Q4. Time permitting this probably could use another round of comments to extend/finalize and remove any info that's gone stale by now.

To circle back on this, we moved forward with option 2 and are using task T225005 to track the migration effort

Hello! @gsingers, as a last step could you please review and sign the L3 document? Once that's done (and the related patch has a +1 from a peer within SRE) we'll be ready to merge and deploy analytics-privatedata-users group membership.

Transitioning this resolved as all subtasks have now been resolved. If additional follow-up is needed, please don't hesitate to re-open. Thanks!

The requested group memberships have been provisioned. I'll transition this to resolved now, but please don't hesitate to re-open if any follow up is necessary. Thanks!

Great, thank you!

Access has been granted. Transitioning this to resolved now, but if any follow-up is needed please don't hesitate to re-open. Thanks!

Regarding chat I'd encourage them to reach out with any questions via IRC. Details about available channels and their associated topics can be fount at https://meta.wikimedia.org/wiki/IRC/Channels

Hi @Nuria could you please review this group request for approval?

! In T232417#5567208, @aezell wrote:
tl:dr; Contacting someone in the abuse department at Yahoo/AOL is probably the best bet to figure this out.

! In T234564#5565056, @Krinkle wrote:
Would it be possible to give type:mediawiki channel:(error OR exception OR fatal) a separate index as well? These are the only critical ones involved in deployment and should not suffer due to spam from random info/debug channels.
We might want to include type:syslog program:php72-fpm and type:scap in there as well.

Hello, could you please expand on this request? What resources are meant to be accessed, and do you know specifically what LDAP group?

Hello, @CorinnaHillebrand_WMDE, could you please review and sign the L3 Acknowledgement of Wikimedia Server Access Responsibilities Document, update the task description with your desired shell username and SSH public key (must be a unique key only to be used in wmf production), and coordinate a comment of approval on this task from your manager?

@kevinbazira could you please review and sign the L3 Acknowledgement of Wikimedia Server Access Responsibilities Document, add details to the task description outlining high level reasoning for the access, and coordinate a comment of approval from your manager?

Hi @Nuria could you please review this for approval?

@EYener in the task description it looks like the ssh key fingerprint was provided, instead of the ssh public key itself. Could you please update with your ssh public key? It should begin with "ssh-ed25519" or "ssh-rsa" . Thanks in advance!

Hello, I've uploaded a patch set for this access. Typically yes @Nuria approves additions to analytics groups. Once that's done we should be in good shape to move forward. Thanks!

Hi @Varnent, the old list address is disabled and messages sent there will held in moderation indefinitely. The communication mail that was sent out about this IMO is clear that the old list address has been replaced with the new address. But to help with usability I've added an auto-response to remind users who email the old address of the change, just in case. Also, if desired, current list admins can still log in and update the description that users see via the web, and tend to any moderated messages during the transition time.

Actually pulling an example raw message from kafka should answer my own question about an example problem message

Looking backwards through logs I see:

Hello, the WMFSF list has been disabled and archives will remain in place. I'll transition to resolved now. Thanks!

This occurred in prod as well after host reboots, and a fix has been deployed in puppet. The fix moves include ::profile::rsyslog::udp_json_logback_compat from profile::elasticsearch to profile::elasticsearch::cirrus which prevents the rsyslog udp 11514 listener from being deployed to logstash collectors where it conflicts with logstash. I think we're in good shape here now, setting this to resolved.

In T233636#5536946, @MoritzMuehlenhoff wrote:

There's two issues with the patch merged for Erin Yener: (1) If contractors have a @wikimedia.org address, they should be added to cn=wmf, not cn=nda. (2) Contractors need an entry in data.yaml with the contract end and a person of contact (expiry_date, expiry_contact fields). Otherwise we'll miss dropping their credentials when the contract expires (we ping the point of contact one week before the contract expires and will extend access if the contract is contuining)

done!

The resolution of parent task T233636 should address this ask as well. Please re-open if any follow up is needed. Thanks!

jkumalah has been added to ldap group wmf, eyener has been added to ldap group nda, and both added to ldap_only_users via puppet.

Copying excepetion.message to message looks to have made an improvement here. I see no results in the past 15 minutes when querying for normalized_message:%{messsage}

A few ideas to address this:

These logs appear to be nesting the message field inside the exception field, and the message field at the root is not present. Which should explain why normalized_message is containing a literal %{message}.

Hello, yes generally speaking based upon the production mail logs I am seeing mail from lawroom.com being accepted and sent onwards to google for final delivery. Messages from this domain appears to occur in bursts, with the majority of recent activity in the logs on Sept 17th, 23rd, and 25th.

Thanks for the ping/reminder! Basic aliasing for w.wiki has been deployed and successfully tested.