I think this recursion in logrotate config was the culprit https://gerrit.wikimedia.org/r/c/operations/puppet/+/1239441

Logrotate config changed and restarted. I hope this is fixed now. Currently /var/log/account is empty.

I think we see these weird file names because of a bug in the logrotate config. It uses a wildcard /var/log/account/* which makes it rotate already rotated logs and recurses.

Does this have any relation to T417458 , T116224 over in cloud VPSes?

(since it affects other confd's in other projects) is this related to work on T356296 ?

lists.discovery.wmnet exists in DNS now

Feel free to reopen if you run into any issues or did not receive that email.

@santhosh Hi! I have created your Kerberos principal. You should have received an email just now with a temporary password and instructions how to change that.

I was going to say something similar on Gerrit.. I was starting to wonder if all service names (gerrit-spare, gerrit-replica) should also be load-balanced before answering the review request for the DNS change. +1

https://lists.wikimedia.org/hyperkitty/list/listadmins-announce@lists.wikimedia.org/thread/C4JEQDKLSC5V6RTRI3QZLFVIRWAVU3NA/

@Nicholusmuwonge_wmde Katie Francis of WMF Legal will need your email address to start the NDA process. You can mail her directly from your WMDE address to kick-start that or just mention your own email address here please.

for the record; if someone wants to test or generally push via https if ssh ever has a problem

I did the remove all access lists and reparent on All-Archived-Projects and checked off that box.

Thank you @tappof

In T417156#11608400, @CDanis wrote:

This should be fixed now -- I did some quick checks but further confirmation appreciated :)

@KSiebert @HMonroy Hi! Could you let us know a bit more about the dashboard?

In an attempt to answer your question how many and which are still affected now:

In T415237#11603737, @Audiodude wrote:

My pleasure. How about we take another 24 hours for technical/ops folks on this thread to comment on that meta page and then I'll disseminate a final copy on the Village Pump?

A practical question: What even is a practical way and the right place to communicate with non-technical editors / users of Etherpad? As it stands that would be the general public, even outside of Wikimedia.

@Tchanders please attach your new public key here. someone will pick it up soon.

I added the description "This project is INACTIVE (see https://phabricator.wikimedia.org/T376982 if you have questions)." to the 3 repos and set them to READ-ONLY.

fwiw, on tcp-proxy machines (serving gerrit-ssh now) we are using haproxy 3.0.11-1+deb13u1 on trixie and it's LTS

Wondering how feasible it would be to export each pad as a simple text file and then compress and backup THOSE instead of the database.

This has happened. The DNS switch has been made.

let's start with the "WMF group" part of your request. This has actually moved to a self-service workflow, the Wikimedia Identity Management System.

per IRC discussion: The (existing, pre-CDN) Gerrit IPs have been removed from cloudgw "dmz_cidr" list because we think they are not needed and then we don't have to think about them again in the future.

Or we should have some other mechanism to ensure that Gerrit is restarted following a replication config change.

Yea, we will do that, Manuel.

Yes, https://gerrit.wikimedia.org/r/c/operations/puppet/+/1215684/6/hieradata/common/profile/trafficserver/backend.yaml was needed

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1237543

per team meeting: let's truncate the table

19:34 < hashar> !log restarting Gerrit to fix broken replication to GitHub # T416912 T416912#11598660

T416912#11598660

19:34 < hashar> !log restarting Gerrit to fix broken replication to GitHub # T416912

deployed cert addition to admin_ng per https://wikitech.wikimedia.org/wiki/Kubernetes/Remove_a_service#Deploy_changes_to_helmfile.d/admin_ng

I am going to manually link T414234 here then so that I can see the status.

calling it enough testing based on "one of three 5xx we've served for Gerrit via the CDN in the past month. The other two were healthchecks from Liberica"

What Xqt said. It's helpful to see what the status of the actual wiki creation task is.. because that determines whether the subtask can be resolved.

publish image pipeline job worked:

Yay:) I was going to ask about this (comments over on gitlab) and you guys already fixed it, thanks!

you have the groups. feel free to reopen if you run into any issues.

you have been added to the requested LDAP groups "wmde" and "nda".

If all the affected lists are private anyways (based on Andre's comment from 2021) - they probably don't need admins or are not used. Is there a real problem being solved by keeping this open?

Hi, could you link to the dashboard? Thanks!

WIP in T405119

In T416501#11592078, @sbassett wrote:

Ok, thanks. Do you know if they have a triage clinic or queue for these kinds of requests?

@sbassett Let's just add the global SRE-access-requests tag (in addition to the team tag).

The repo for this that was already created:

Hi, collaboration-services is not particularly related to logstash. I added Observability team which I think is a better fit. Cheers

I think the misunderstanding here is that the existing developer account / LDAP user is called "jacobthwaites".

@tappof I don't think this is asking for shell access. It's just for LDAP groups "wmde" and "nda" which is usually the standard for WMDE staff.

The usual follow-up after a security upgrade happened.