I am in the process of (slowly) removing NFS dependencies from tools; this is directly related to resource constraints on the nfs worker nodes but also is a desire to have a more standardised/supported environment.
I have no data on what the largest log producers are, but I can state that ClueBot NG (production and staging) write a notable amount of logs (1+ lines per change on enwiki).
Prior to switching to loki logs (there are some other dependencies to work though) I am raising this task for 2 primary purposes:
- The tools produce > 10G of logs per day, which will need to be forwarded/ingested. I have previously seem assumptions in the M per tool so checking higher volume is ok.
- It would be very useful to be able to query more specifically, to avoid having to download and grep 100s of GB of data.
At a minimum it would be useful to have:
- since time
- until time
(Usually debugging is related to either near current time or a window when an edit occurred).
Perhaps more complicated of a request:
I emit logs with fields providing linking to other log records (1 tool is string based, 1 is json based). Having the ability to declare "my tool emits json logs please parse them as such" and then being able to limit queried logs on specific fields would be highly beneficial.
If I'm troubleshooting an edit my current flow is to grep the diff= (revision id), grab the internal uuid for the execution, then grep out all entries matching that uuid.
Being able to perform something similar with loki would ensure minimal resource usage while providing responsive investigation.