Update: I am reconsidering the idea of doing the Alloy schema change all at once along with two logs-api deployments, because that requires adding non-trivial stream-reconciliation code to logs-api, especially in the case of follow=True (tailing). From the point of view of (even temporary) code complexity, it is instead much simpler to:

1. Update Alloy’s schema to label all new log entries with _both_ job (as today) and source (desired state).
2. Do not change logs-api, which continues serving logs based on the job label. Wait 14 days (Loki’s retention window).
3. Update logs-api to only serve logs based on the new source label and job_name metadata. This can be a one-line change.
4. (Eventually) update again Alloy’s schema to drop the now-unused job label.

Proposed checklist:

• With our new approach of having a single label (source = run-logs) for all logs (still split by tool tenant), Loki will scan all run-logs of that tool over the requested time range to filter by job_name. This allows for a very chatty job to affect log retrieval latency for all other jobs of that tool. Confirm whether we don’t want to keep job_name as a label.
• Notify users that they may observe duplicated logs (on pod restart, Alloy will re-read log files from their beginning). After the Alloy changes described below are deployed, Loki will not deduplicate these already-ingested logs because they will have different labels and metadata.
• Set allow_structured_metadata: true (docs) explicitly in loki/common.yaml. That setting is true by default in our version of Loki, but it doesn’t hurt to be explicit given the layers of indirection (Helm). Deploy to production.
• Update the Alloy config as below.
• Test in lima-kilo: locally deploy the updated Alloy config, and run queries against the Loki API from a kind pod.
• In logs-api, duplicate Loki queries: keep the current job-selector query, and concurrently add the equivalent query with selector source = run-logs and metadata filters. Deploy logs-api to production.
• Deploy the Alloy changes to production.
• Keep the logs-api backward-compatibility code (duplicated queries) for at least 14 days (Loki’s log retention window).
• Remove the query duplication from logs-api, deploy to production.

Thank you for your contribution @Mahveotm!

Fixed via T428253.

@DamianZaremba thank you for reporting this, please let us know if you notice anything wrong! Changelog updated as well.

All MRs merged, and verified that toolforge jobs logs is working as expected on toolforge.org and returning more than 500 entries.

Thank you, I have regained access to my SUL account!

I believe this is caused by calling datetime.now on two separate occasions (1, 2), computing the difference at the precision of a second, and asserting it should always be 0. Computers are fast, but we’re bound to have the same encoding 2026-06-29T14:48:34Z for the first "now" sometimes mean …34.002 and sometimes …34.997. When it’s the latter, the second "now" is likely to fall on something like …35.001 and _get_duration will return 1.

Thank you! I have followed your suggestion and sent an email, subject line "Account recovery help needed for SUL: TLepage-WMF". Happy to close this, and sorry for the noise here!

I believe you are right, apologies for the lack of precision, still getting my footing among the various account types! I indeed have the same problem when trying to log in to mediawiki.org.