Hello DC-Ops team, I can be your o11y point of contact for this task as it's easier for us to coordinate timezone wise. Cheers.

It seems like thanos-swift is not using CFSSL.

I also don't think that the overarching goal is for o11y but to service owners as that would benefit all of the SRE team. :)

I'm removing the observability tag because directly addressing emails generated by services owned by other teams falls outside the scope of the observability team.

This issue is temporary, the culprit was fixed with https://gerrit.wikimedia.org/r/c/operations/puppet/+/1005949

I was wondering if the filesystem where /srv is mounted had any mount flags that would prevent the deletion of those files/directories such as ro or noexec but looking at the mount options of /dev/mapper/vg0-srv they are rw,relatime,stripe=256, none of them seem to be the culprit so the issue resides somewhere else.

I've been testing this on a Pontoon host but the arguments and actions specified in the unit are working as expected. I'm still debugging the issue.

Closing as resolved because Karma is upgraded, and deployed in production. In addition to this IRC Echo was migrated from Python 2 to Python 3 in https://gerrit.wikimedia.org/r/c/operations/puppet/+/972929

In T360414#9703751, @MoritzMuehlenhoff wrote:

In T360414#9702570, @andrea.denisse wrote:

I've documented the migration process on Wikitech: https://wikitech.wikimedia.org/wiki/Cergen#Migrating_to_CFSSL

Please take a look at it if you can.

I fixed the title to reflect that this is specifically for moving a service based on Envoy, otherwise LGTM, thanks!

I've documented the migration process on Wikitech: https://wikitech.wikimedia.org/wiki/Cergen#Migrating_to_CFSSL

@MoritzMuehlenhoff @fgiunchedi Thank you both, I'll make sure to remove it.

Hello, we've been receiving email alerts from this host since March 27th. I opened T361604 to track it.

Special thanks to @fgiunchedi for their help and support.

I'm closing this task, I regret documenting the issue to try to solve it in order to help unclutter the inbox of other SREs with noise.

Hi @Aklapper , thanks for your comment.

Hi @bd808 thanks for kindly sharing your insights.

In T361262#9671474, @bd808 wrote:

I will be the jerk to ask why we should choose a Google Group rather than a Mailman list. Is this sensitive data that needs to be hidden from the community? Will there ever be a potential for non-WMF staff subscribers? Is there another compelling business need like an associated Google Drive shared folder or easy Google Calendar invite management that makes a Google Group a more relevant solution than a FOSS-backed mailing list?

In T358506#9666364, @Volans wrote:

The premise seems to mix different things. PuppetDB is a totally separated service from the PuppetMaster/PuppetServer ones and runs on their own hosts. Are you saying that naggen fails to connect to PuppetDB?

What kind of queries does it do? Could it use the public proxy that is exposed in the intranet? If so connect to puppetdb-api.discovery.wmnet:8090

I'm removing the observability tag because directly addressing emails generated by services owned by other teams falls outside the scope of the observability team.

Hi Infrastructure Foundations Team,

Hi @dcaro , I've upgraded karma to 0.119 and it's now live in production.
Please let me know if there's anything else I can help with.
Best.

The production-elk7-codfw.log file was present in the system. After verifying the contents of the file looked correct I manually started the service and the unit is healthy again.

We received a similar alert today: (SystemdUnitFailed) firing: logrotate.service on logstash2003:9100

Hello, I see several email alerts regarding logstash1011: