Definitely a different problem.

Logstash was crashlooping because it was attempting to load a template that did not exist on the host anymore. Now that it is using the right template, logs are flowing again.

Injecting statslib into the legacy statsd code path introduces significant complexity and risk.

@jcrespo roll-restarted swift proxies today using sre.swift.roll-restart-reboot-swift-ms-proxies cookbook in response to high 502s and 504s from ATS.

In T279112#9328548, @Krinkle wrote:

This example in the task description is still broken today. E.g. meta.domain:wikipedia.org has either no results, or results that do not contain en.wikipedia.org, en.m.wikipedia.org, zh.m.wikipedia.org, and the other 200+ subdomains.

Instead, it sometimes matches spurious records from third-party proxy sites like fr-wikipedia.org.

Trying option 1 seems like a good start to try handling memory size issues. Note that we may want to adjust logstash tuning as well afterwards.

In T350366#9301288, @Joe wrote:

The error in the loki build is due to its dependency on the fact it depends on golang-1.13 which has been dismissed years ago. @colewhite do you think we can just remove the loki image instead?

There are other tasks filed doing what this task intended.

In T302373#9275162, @fgiunchedi wrote:

Fresh off the presses I think we can upgrade to statsd_exporter 0.25, which would allow us to drop our custom patch to relay statsd metrics

Thanks, @aaron!

Summarizing highlights from my IRC conversation with @fgiunchedi:

Thanks for the report!

In T344751#9255685, @herron wrote:

Since these 2 buckets were requested specifically I'd be inclined to either append them to the default set, or document the rationale for omitting them. [.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10, 30, 60, (+Inf)] seems fair IMO

I have no problem keeping the 30s and 60s buckets. I was under the impression from the meeting that @Krinkle preferred to omit them?

Per meeting with @Krinkle today:

It doesn't look like there is an option to change the timeout parameter. We'll need to patch curator. 😕

I propose we implement a mixed approach.

OpenSearch replies with HTTP 200 {"acknowledged":false} indicating the operation hasn't failed, but has hit the 30s "explicit operation timeout". This is different than master_timeout (which curator is providing) specifying the "timeout for connection to master".

In T348508#9242529, @fgiunchedi wrote:

How long the operation took is suspiciously close to 30s, which the default timeout parameter value, maybe we should bump that in addition to (or instead of) master_timeout ? upstream docs

Done!

Stopped and removed units for coal, uwsgi-coal, wmf_auto_restart_coal and wmf_auto_restart_uwsgi-coal.

Delete actions took a few milliseconds over 30s today. Optimistically resolving.

Optimistically resolving now that WAL is enabled. Will watch the logs for new instances.

I concur. Let's bump the timeout.

There's an outstanding Icinga alert that seems related: CRITICAL - degraded: The following units failed: dispatch-scheduler.service,docker-image-prune-old.service

Discovered some more evidence of this in logs this morning. There is another recommendation to enable WAL on the sqlite db (new in Grafana 9.4).

Error logs have gone away and the prometheus view looks good.

In T346796#9192790, @AKhatun_WMF wrote:

I am getting this error when I kinit
kinit: Client 'akhatun@WIKIMEDIA' not found in Kerberos database while getting initial credentials
Am I supposed to get a temporary password though email?

The group membership change has been deployed.

Restored the level of access held before last contract expired.

The group membership change has been deployed.

Migrated to nda ldap group.

ping: @thcipriani as approver for deployment group membership

@MGerlach is there an expiry date for this contract renewal?

The group membership change has been deployed.

mw2381:

$ ulimit -Hn
1048576
$ ulimit -Sn
1024

I see logs in logstash! \o/

In T288624#9138410, @dancy wrote:

Extra problem:
I cannot find a way to get visibility into these log messages in https://logstash.wikimedia.org/. Assistance would be appreciated. @colewhite can you provide advice?

It's been more than a week and I can see no more instances of this in the logs.

In T341792#9179651, @gerritbot wrote:

Change 958991 merged by Btullis:

[operations/puppet@production] Add the analytics and search-platform teams to flink zk contacts

https://gerrit.wikimedia.org/r/958991

Considering how much time has passed, it's probably safe to say this is complete. If not, please reach out :)

Interestingly, if StatsLib creates executeTiming_seconds_bucket as a counter and executeTiming_seconds as a timer and sends them to the exporter, this renders statsd-exporter inoperable until a restart is commanded. It appears we have to be careful not to step on the metric names as statsd-exporter would generate them for summaries and histograms.

In T344751#9175304, @Krinkle wrote:

The handful of timing metrics we have (and actively make use of) vary a lot in their range. I'm not sure a single set can be of much use. The size and range of timing measures vary a lot throughout the platform, from job measures in the range between whole seconds and hours, to WANCache callbacks that are measures between 0.1 and 100 milliseconds.

It appears there is some problem with the kafka-jumbo nodes in deployment prep.

In T345362#9161058, @Jdlrobson wrote:

@colewhite I'm getting DatasourceError rweb email alerts. Is that covered by this task or T344961 ?

Grafana is updated and silence is removed.

Linking my comment here for visibility: T345337#9150551

9.4.14 is live on grafana-next. Will do some testing there before rolling to production early next week. Reinstalled the silence until we can complete the upgrade.

In T345337#9150415, @jbond wrote:

@colewhite from your comment on on the elastic search restart cookbook can i assume that moving to the opensearch python packages would also allow us to drop this dependency from spicerack. i.e. we could use that package for the current search cookbooks and this new ones?

Ran into this today trying to pip install wikimedia-spicerack (Python 3.11).

Related: T255864: Use/adopt search cluster ES management cookbooks for logging ES too