This alert has not triggered since 2026-09-21, so I'm optimistically marking this as Resolved.

there's no task needed anymore

Oh we already have one, just found it: T433870: [logs-cli] add a `--first` flag, or replace `--latest` with `--lines`

this is mostly a UX issue that would fit there.

I think adding the --direction option is useful, but is not covering a common use case: printing the last X lines in the log, oldest to newest.

Note: we are experimenting with cross-posting to Phabricator the weekly report that previously was only posted to Asana, where only few people have access.

This is live and working both in toolsbeta and tools! 🎉

The HTML version of the templates, along with a txt version containing the same text, is now merged and deployed in alerts-api (https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/7).

One alternative that we discussed today with @dcaro and @Raymond_Ndibe could be to use the publish: field to determine if a component is a web service, e.g. if a component has publish: / in the config, then it should be treated as a web service and not a continuous job.

Also see the upstream discussion at https://github.com/grafana-community/helm-charts/issues/830

@dcaro and myself found a quick fix, forcing the minio subchart in Loki to pull the images from the open source fork https://github.com/pgsty/silo (implemented in https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1440).

If we can avoid migrating all existing configs, I think it would be nice.

I +1d https://gerrit.wikimedia.org/r/c/cloud/wmcs-cookbooks/+/1341237, anything left to review on this one?

Added the line above and a few other tweaks. This is a preview of both "firing" and "resolved" alerts for the 3 types of alerts we are supporting at the moment:

All the review comments in https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/5 and https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/6 were addressed. Those two MRs are pending a re-review before they can be deployed.

Note: we are experimenting with cross-posting to Phabricator the weekly report that previously was only posted to Asana, where only few people have access.

@Andrew looks like you applied this change, can it be marked as Resolved?

This was briefly discussed today during a Discord call in the Wikimedia community Discord.

Iterated on it a bit more:

+1 to the new versions that were just added by @dcaro to the task description.

I did some small updates:

+1

Then please try to insert an empty line between Bug: and Signed-off-by: :)

They are on separate lines, but are rendered as Markdown and the line break is swallowed. :/

toolforge_run_ansible.sh -D --tags ldap

Today the workaround did not work, I ran toolforge_load_users_to_ldap.sh twice, and both times it failed with:

The same alert was triggering in 2024: T365164: [wikireplicas] clouddb* free memory decreases over time and at the time I added a note about it in https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting#Wikireplicas_memory_issues_(clouddb*)

@Arcstur thanks for the additional context. Unfortunately bots and scrapers are currently an issue for many Toolforge tools, we are working at solutions to mitigate this (T433395: [infra] Allow some opt-in scraping solution for tools).

Processlist for s3 (the instance at 72.7% mem):

Current memory usage on clouddb1022:

Interestingly, clouddb1023 is a analytics host, while clouddb1022 is a web host so the workloads should be different:

clouddb1022 is now alerting :/

Index created successfully:

s53685 (tools.editgroups) could be related to this spike, as it logged a few slow queries on 2026-09-12 right when the history length started to grow.

+1

@Milimetric the view definition is in maintain-views.yaml and was last changed in January 2026:

Thanks @Marostegui for looking into this. I will mark it as Resolved for now, we can reopen it if the issue re-occurs.

@Magnus thanks for having a look. I can confirm that I can see some moments where no SQL queries are running from mixnmatch, so maybe it is not your tool that is causing the issue this time. Differently from last month, I'm struggling to identify any tool that is constantly running long queries, these were the long transactions from a few minutes ago, and the users were different:

The MRs above are now merged and deployed to both toolsbeta and tools. They are working as expected, but no email is being sent out yet, that will need a few more patches that I'm still refining are not ready for review yet.

The MR above looks good and should be all that we need for this task. I didn't have time to complete my code review but I'll do it on Monday.

Note: we are experimenting with cross-posting to Phabricator the weekly report that previously was only posted to Asana, where only few people have access.

Note: we are experimenting with cross-posting to Phabricator the weekly report that previously was only posted to Asana, where only few people have access.

All of the transactions above completed and there's only 3 active transactions now:

@Raymond_Ndibe and myself added some comments on the patches, but we would like to have some input from @dcaro as well.

Here's a quick Claude-generated example email, based on the alertmanager template mentioned in the task description.