Or we could make a wrapper around qacct so it, by default, only reads last * bytes, an example implementation is

@madhuvishy has done some testing on moving the Grid Engine control files off of NFS and we talked about rotating the accounting file more often as part of the eventual implementation work for that. We have a few use cases for 14-30 days of audit data, but that would be much easier to manage in 24 hour chunks. The current "file grows until we manually truncate it" model is pretty horrible.

Maybe throw it into some kind of more database like system like logstash so we can just query that?

In T182451#3824888, @Multichill wrote:

Maybe throw it into some kind of more database like system like logstash so we can just query that?

I don't see investing in custom Grid Engine accounting tooling having a useful long term return. I certainly would not stop anyone from working on it, but today we have many more pressing issues to deal with from the cloud-services-team side. I would be much more interested in examining slurm as replacement for the grid engine use cases that we can't move to Kubernetes.

It’s also possible to read the file in reverse, using tac (which uses a bunch of seeks under the hood). For example, I used this successfully to get the submission, start and end time of a recently terminated job:

tac /var/lib/gridengine/default/common/accounting |
while IFS=: read -r qname hostname group owner job_name job_number account priority submission_time start_time end_time rest; do
    if [[ $job_name == wd-shex-infer-6 ]]; then
        printf "Submission time: %(%Y-%m-%d %H:%M:%S)T\nStart time: %(%Y-%m-%d %H:%M:%S)T\nEnd time: %(%Y-%m-%d %H:%M:%S)T\n" "$submission_time" "$start_time" "$end_time";
        break;
    fi;
done

By breaking from the loop as soon as the first match is found, we effectively kill the tac with SIGPIPE, so it doesn’t read more data than necessary. (I wrapped the whole thing in timeout 10s bash -c '…' just to be sure, though, and I recommend anyone else experimenting with this to do the same.)

Edit: you can also achieve this with qacct -j JOB -f <(tac /var/lib/gridengine/default/common/accounting), but you have to terminate qacct manually once it’s printed the information you want, otherwise it’ll keep reading the accounting file (in reverse) forever.

Last year, we started rotating the accounting file. Is this still a problem?