Page MenuHomePhabricator

Loki usage
Closed, ResolvedPublic

Description

I am in the process of (slowly) removing NFS dependencies from tools; this is directly related to resource constraints on the nfs worker nodes but also is a desire to have a more standardised/supported environment.

I have no data on what the largest log producers are, but I can state that ClueBot NG (production and staging) write a notable amount of logs (1+ lines per change on enwiki).

Prior to switching to loki logs (there are some other dependencies to work though) I am raising this task for 2 primary purposes:

  1. The tools produce > 10G of logs per day, which will need to be forwarded/ingested. I have previously seem assumptions in the M per tool so checking higher volume is ok.
  1. It would be very useful to be able to query more specifically, to avoid having to download and grep 100s of GB of data.

At a minimum it would be useful to have:

  • since time
  • until time

(Usually debugging is related to either near current time or a window when an edit occurred).

Perhaps more complicated of a request:
I emit logs with fields providing linking to other log records (1 tool is string based, 1 is json based). Having the ability to declare "my tool emits json logs please parse them as such" and then being able to limit queried logs on specific fields would be highly beneficial.

If I'm troubleshooting an edit my current flow is to grep the diff= (revision id), grab the internal uuid for the execution, then grep out all entries matching that uuid.

Being able to perform something similar with loki would ensure minimal resource usage while providing responsive investigation.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Adding taavi as they seem to be doing a lot of work on logging/loki

The task implementing that is T400917: [jobs-api] Allow customizing time to request Loki logs for, leaving this task open for the volume request info.

Realistic historical usage on staging.

11G	botng-20250723.log
11G	botng-20250724.log
11G	botng-20250725.log
11G	botng-20250726.log
11G	botng-20250727.log
11G	botng-20250728.log
11G	botng-20250729.log
11G	botng-20250730.log
11G	botng-20250731.log
11G	botng-20250801.log
11G	botng-20250802.log
11G	botng-20250803.log
11G	botng-20250804.log
4.6G	botng-20250805.log
141G	total

I've got pretty much everything running under pack/build service now (some deployed as jobs, some as components, some in testing).

Related issues found:
T401190
T401422
T401552
T401553

(Some/all of these might be resolved with the new log api)

Leaving this open for a comment re usage, if no issues come up or there are no major concerns then I think this can be closed off.

@taavi do you think that the expected usage is fine? Do we have quotas/limits in Loki for how many GB of logs can be stored by a single tool?

fnegri triaged this task as High priority.Aug 20 2025, 2:35 PM

In general we have rate limiting in place so a single tool shouldn't be able to cause problems for other tools. That being said, while direct comparisons between Loki and NFS storage sizes are difficult (due to compression and different type of metadata stored), my math indicates that the amount of logs this tool is producing is at least double or triple compared to the everything currently using Loki combined. The cluster is designed for higher volumes (as we migrate stuff from NFS to Loki) but I'd still like to be around and aware when you flick the switch. Also if there is any low-hanging ways to reduce the log volume, fixing those would be very appreciated by us.

DamianZaremba claimed this task.

This has been migrated over for a while (https://phabricator.wikimedia.org/T401151#11084273), some logs have been reduced (debug -> info), still some areas to cleanup (irc messages are almost gone etc).

I've heard no screams as things where gradually moved into components/loki, so closing this.