I fixed the credentials a few hours ago yep, I'll close, but let me know if you still see issues (maybe one worker did not pick them up or something).

@Niinarh7102026 that query shows as running right now, are you still having issues?

In T440966#12425046, @Urbanecm wrote:

Confirmed, it's not working for some reason :).

In T369800#10490208, @Raymond_Ndibe wrote:

yeaaaa I think there is a task somewhere about moving replica_cnf out of puppet. maybe it's time to work on that

Weekly asana report:

We can copy the logs service dashboard as template (https://grafana.wmcloud.org/d/kcAb-KUSe/logs-service-overview?from=now-2d&to=now&timezone=utc&var-datasource=P8433460076D33992), same for the fastapi exporter and such.

In T440294#12409183, @Raymond_Ndibe wrote:

Currently we use page_size=-1 but that does not give all the results, only 100 (according to the running version)

I think this is not correct. To be certain I did some experiments and so far I couldn't find a case where this is true.
If you are curious about the things I checked see:

T440294_page_limit_test.md1 KBDownload

@dcaro just out of curiosity are there other endpoints where this is true?

That being said, I'd say we are prematurely optimizing things, but we already have a patch so might as well get it in.

Did a pass and made some changes, you can check the diff here: https://wikitech.wikimedia.org/w/index.php?title=Help%3AToolforge%2FMigrate_build_service_job_to_push_to_deploy&diff=2465003&oldid=2462321

Ran this from my laptop on toolsbeta harbor, freed ~40G:

All mrs are in, I'll update the ci images so they have the newer pre-commit versions and such

Happened again, it did not help much xd

In T439496#12400755, @Raymond_Ndibe wrote:

In T439496#12399361, @dcaro wrote:

so I played with this even more and while the publish approach is nice, it does prevent you from ever defining a continuous published job in components-api.

Do you have the code? Seems fixable imo.

what do you suggest?

so I played with this even more and while the publish approach is nice, it does prevent you from ever defining a continuous published job in components-api.

I have enabled set -x right before the failing command, that should show some extra logs next time it happens.

This has been fixed and deployed now :), let me know if you still find issues

In T431340#12397043, @Andrew wrote:

I /think/ this is covered by the 'minesweeper' process which was accidentally not running when I opened this.

The issue is the limit, we are sending 'limit=0' to loki webservice point and it does not like it, running this works:

local.tf-test2@toolslocal:~$ toolforge jobs logs -f testoneoff1 -l 100
2026-10-05T16:43:12.443676Z [testoneoff1-kw9kx] [job] Mon Oct  5 04:43:12 PM UTC 2026

In T440103#12396760, @dcaro wrote:

Version tools-harbor.wmcloud.org/toolforge/logs-api:image-0.0.48-20260911084153-e8e7be4a seems to work on lima-kilo, looking

Version tools-harbor.wmcloud.org/toolforge/logs-api:image-0.0.48-20260911084153-e8e7be4a seems to work on lima-kilo, looking