Page MenuHomePhabricator

gerrit: scoring/ores/editquality takes a long time to git gc
Closed, ResolvedPublic

Description

Nothing urgent, but something I wanted to document somewhere.

I've noticed that jgit gc running inside of gerrit takes a long time to garbage collect the scoring/ores/editquality repo. I wrote a script to parse the gc logs to verify my assumptions:

#!/usr/bin/env python

import datetime
import re
import time

with open('/var/log/gerrit/gc_log') as f:
    gc_log = f.readlines()

BEFORE = 'before'
AFTER = 'after'

def format_time(timestamp):
    time_obj = time.strptime(timestamp, r'%Y-%m-%d %H:%M:%S,%f')
    return time.mktime(time_obj)

logs = {}
for log in gc_log:
    log_line = re.match(
        r'^\[([\d-]{10} [\d:,]{12})\].*?: \[(.*?)\] (before|after):',
        log
    )

    if not log_line:
        continue

    timestamp = format_time(log_line.group(1))
    repo = log_line.group(2)

    if not logs.get(repo):
        logs[repo] = {}

    if log_line.group(3) == BEFORE:
        logs[repo]['start'] = timestamp
    if log_line.group(3) == AFTER:
        logs[repo]['end'] = timestamp

total_seconds = 0
for log in logs:
    elapsed = logs[log]['end'] - logs[log]['start']
    total_seconds += elapsed
    print('{}\t{}'.format(datetime.timedelta(seconds=elapsed), log))

print('{}\tTOTAL'.format(datetime.timedelta(seconds=total_seconds)))

It seems like more than half of the total git gc time is spent on that repo:

thcipriani@gerrit1001:~$ python elapsed_gc_time.py | sort -rn | head
7:01:27 TOTAL
4:49:00 scoring/ores/editquality
0:30:36 mediawiki/services/ores/editquality
0:07:19 All-Users
0:03:06 operations/debs/pkg-php/php
0:02:51 mediawiki/vagrant
0:02:48 mediawiki/core
0:02:23 analytics/superset/deploy
0:02:16 research/ores/wheels
0:02:16 apps/android/wikipedia

Some discussion on that topic happened during the October 2022 Gerrit community meeting. Some links:

gc log for scoring/ores/editquality:

[2022-10-01 04:12:20,719] INFO  : [scoring/ores/editquality] gc config: gc.aggressive=true; 
[2022-10-01 04:12:20,719] INFO  : [scoring/ores/editquality] pack config: maxDeltaDepth=50, deltaSearchWindowSize=10, deltaSearchMemoryLimit=0, deltaCacheSize=52428800, deltaCacheLimit=100, compressionLevel=-1, indexVersion=2, bigFileThreshold=52428800, threads=0, reuseDeltas=true, reuseObjects=true, deltaCompress=true, buildBitmaps=true, bitmapContiguousCommitCount=100, bitmapRecentCommitCount=20000, bitmapRecentCommitSpan=100, bitmapDistantCommitSpan=5000, bitmapExcessiveBranchCount=100, bitmapInactiveBranchAge=90, searchForReuseTimeoutPT596523H14M7S, singlePack=false
[2022-10-01 04:12:20,719] INFO  : [scoring/ores/editquality] before: numberOfPackedRefs=10, numberOfPackFiles=2, sizeOfLooseObjects=0, numberOfLooseObjects=0, numberOfPackedObjects=8580, numberOfLooseRefs=1, sizeOfPackedObjects=2617021456
[2022-10-01 09:08:43,350] INFO  : [scoring/ores/editquality] after:  numberOfPackedRefs=10, numberOfPackFiles=2, sizeOfLooseObjects=0, numberOfLooseObjects=0, numberOfPackedObjects=8580, numberOfLooseRefs=1, sizeOfPackedObjects=2617021456

Event Timeline

hashar triaged this task as Medium priority.Nov 10 2019, 10:45 AM
hashar subscribed.
$ git count-objects -vH
count: 0
size: 0 bytes
in-pack: 9589
packs: 3
size-pack: 2.54 GiB
prune-packable: 0
garbage: 0
size-garbage: 0 bytes

So slightly less than 10k objects but a pack of 2.54GiB.

Not sure what jgit gc does exactly on the weekly run. But if it goes to repack everything and recompute the delta, on huge files that is surely taking a while.

In the C implementation, there are a lot of configuration settings for garbage collection, notably the depth and window parameters for the delta, there are also setting about how objects are mapped in memory which might affect performances if the buffer size is too small. Some examples:

  • core.packedGitWindowSize, core.packedGitLimit
  • core.bigFileThreshold, store files deflatted without any delta compression. C git defaults to 512MiB.

I guess theoretically we could tweak settings to handle those huge files. That would need a lot of knowledge about git delta/packing. Maybe it would be better to instead migrate those files to git-lfs / git-fat (I can never remember which one we want to aim for)?

FWIW, I ran git-sizer on a mirror of the repo. It's certainly a large repo with some large blobs:

~/projects
(/^ヮ^)/*:・゚✧ git clone --mirror https://gerrit.wikimedia.org/r/scoring/ores/editquality && cd editquality.git

editquality.git (BARE:master)
(/^ヮ^)/*:・゚✧ git sizer --verbose
Processing blobs: 4392
Processing trees: 2856
Processing commits: 2288
Matching commits to trees: 2288
Processing annotated tags: 0
Processing references: 215
NameValueLevel of concern
Overall repository size
* Commits
** Count2.29 k
** Total size745 KiB
* Trees
** Count2.86 k
** Total size2.46 MiB
** Total tree entries57.7 k
* Blobs
** Count4.39 k
** Total size7.17 GiB
* Annotated tags
** Count0
* References
** Count215
Biggest objects
* Commits
** Maximum size [1]1.98 KiB
** Maximum parents [2]2
* Trees
** Maximum entries [3]75
* Blobs
** Maximum size [4]35.3 MiB***
History structure
* Maximum history depth521
* Maximum tag depth0
Biggest checkouts
* Number of directories [5]25
* Maximum path depth [5]6
* Maximum path length [5]69 B
* Number of files [5]408
* Total size of files [6]676 MiB
* Number of symlinks0
* Number of submodules0

[1] rOEQcce3c51f97b8
[2] a8b047f222c691e789616b07ba1c715ddcc8f1c2 (refs/heads/master)
[3] 3f9267f9c3da569755f0b7aad038b9461bdb0213 (refs/heads/enwiki_2016:tuning_reports)
[4] 3a668149daa9ac95b50e9f98bf6e4d4c502244c8 (rOEQdd1d36c2db0c:models/enwiktionary.reverted.rf.model)
[5] 61b38329abaf55c64aa8350e4ef1254ca8178c09 (refs/heads/enwiki_2016^{tree})
[6] 49c2fe00f97ebda6a3131503dcb78b0bdce83ebe (rOEQ945ccd1e4932^{tree})

This could be an old repo that used before we had git LFS. We should be able to find out and get it cleaned up if I'm right in the process of addressing T238746: Clean up ORES and related gerrit repos.

find /srv/gerrit/git -name '*.pack' -printf '%s\t%P\n'|sort -nr|head -n8
2724552421	scoring/ores/editquality.git/objects/pack/pack-412939905cc0116439ca5c9a81b23c813f2aedcb.pack
2068418144	mediawiki/services/ores/editquality.git/objects/incoming_4614717884657486412.pack
2067790090	mediawiki/services/ores/editquality.git/objects/incoming_2738121913207968148.pack
1789259557	analytics/superset/deploy.git/objects/pack/pack-266cbfed8aca8380f0639c43500136c8c26db491.pack
1068618992	analytics/superset/deploy.git/objects/pack/pack-3d897ed5f7773e24c3bf4f8830f46c9fee6c106d.pack
963549161	operations/software/netbox-deploy.git/objects/pack/pack-6f67f333539eee1390f78c5e3c2cb5ebc7694154.pack
824505089	operations/debs/spark2.git/objects/pack/pack-08974eeeaff2f91c55c8f013cfa4880dc273f26f.pack
672600423	mediawiki/services/ores/editquality.git/objects/pack/pack-e005323c520f90e9d5f7a61f1a93a378b5d2ec3d.pack

The couple files with incoming_ prefix are from Jun 25 2019 and July 30 2019. They seem to be temporary jgit files:

org.eclipse.jgit/src/org/eclipse/jgit/internal/storage/file/ObjectDirectoryPackParser.java
public PackLock parse(ProgressMonitor receiving, ProgressMonitor resolving)
    throws IOException {
  tmpPack = File.createTempFile("incoming_", ".pack", db.getDirectory()); //$NON-NLS-1$ //$NON-NLS-2$
  tmpIdx = new File(db.getDirectory(), baseName(tmpPack) + ".idx"); //$NON-NLS-1$
  try {
    out = new RandomAccessFile(tmpPack, "rw"); //$NON-NLS-1$

    super.parse(receiving, resolving);

    out.seek(packEnd);
    out.write(packHash);
    out.getChannel().force(true);
    out.close();

    writeIdx();

    tmpPack.setReadOnly();
    tmpIdx.setReadOnly();

    return renameAndOpenPack(getLockMessage());
  } finally {
    if (def != null)
      def.end();
    try {
      if (out != null && out.getChannel().isOpen())
        out.close();
    } catch (IOException closeError) {
      // Ignored. We want to delete the file.
    }
    cleanupTemporaryFiles();
  }
}

The actual pack file is <700MB (672 600 423 bytes).

For scoring/ores/editquality I thought about gc.bigPackThreshold which keep packs over that threshold unfortunately it is apparently not available in jgit. We run jgit gc --aggressive once per week and from the doc it explicitly does NOT reuse delta from packfiles so we essentially recompute all the deltas which is definitely an expensive operation for large binary files.

I ran the script today and it is still a thing:

$ python /home/thcipriani/elapsed_gc_time.py |sort -rn|head
7:14:43	TOTAL
4:56:23	scoring/ores/editquality
0:33:57	mediawiki/services/ores/editquality
0:06:01	mediawiki/core
0:05:36	analytics/superset/deploy
0:04:35	operations/puppet
0:03:20	operations/debs/pkg-php/php
0:03:08	operations/software/netbox-deploy
0:03:07	operations/container/miscweb
0:02:47	operations/software/swift-ring

Really sorry for the issue :( We are actively working on Lift Wing and we hope to deprecate any git-lfs usage in favor of swift during the next 4/6 months, if it is not viable and blocking something please let us know and we'll try to work on it.

@elukey scoring/ores/editquality.git has a bunch of a large objects which somehow cause jGit gc to take four hours. I think cause old models are still in git while the newer one would be in git-lfs.

The only issue is once per week the server is CPU busy for a few hours, but beside that there is no operational issue :) I have updated this task since jGit GC performances is being discussed in the Gerrit community meeting. So essentially you can ignore it.

Happy to see you will move the models to Swift which sounds nicer than storing large files in plain git ;-]

From Luca:

You don't need to run the aggressive GC, ever. Unless the repo has massively changed its structure

The gc can also be tuned via .git/config which I guess we could provision via Puppet.

Change 854514 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] gerrit: remove git gc aggressive

https://gerrit.wikimedia.org/r/854514

Change 854514 merged by Dzahn:

[operations/puppet@production] gerrit: remove git gc aggressive

https://gerrit.wikimedia.org/r/854514

Mentioned in SAL (#wikimedia-operations) [2022-11-09T20:35:02Z] <mutante> gerrit1001 (gerrit) - restarting gerrit service to disable aggressive garbage collection. gerrit:854514 - T237807

I ran Tyler's script from /home/thcipriani/elapsed_gc_time.py

gerrit1001:~$ python /home/thcipriani/elapsed_gc_time.py|sort -g|tail -n 10
0:00:27	operations/software/netbox-deploy
0:00:30	scoring/ores/editquality
0:00:35	analytics/superset/deploy
0:00:35	wikimedia/production
0:00:42	mediawiki/extensions/Wikibase
0:01:17	All-Users
0:01:31	mediawiki/extensions
0:03:49	operations/puppet
0:04:01	mediawiki/core
0:31:26	TOTAL

I ran Tyler's script from /home/thcipriani/elapsed_gc_time.py

gerrit1001:~$ python /home/thcipriani/elapsed_gc_time.py|sort -g|tail -n 10
0:00:27	operations/software/netbox-deploy
0:00:30	scoring/ores/editquality
0:00:35	analytics/superset/deploy
0:00:35	wikimedia/production
0:00:42	mediawiki/extensions/Wikibase
0:01:17	All-Users
0:01:31	mediawiki/extensions
0:03:49	operations/puppet
0:04:01	mediawiki/core
0:31:26	TOTAL

Wow! That's a world of difference. Let's keep an eye on what this does to disk space. But reducing GC time by 6½ hours is a win :)

Change 856601 had a related patch set uploaded (by Hashar; author: Thcipriani):

[operations/puppet@production] gerrit: script to report on git gc durations

https://gerrit.wikimedia.org/r/856601

Change 856601 merged by Dzahn:

[operations/puppet@production] gerrit: script to report on git gc durations

https://gerrit.wikimedia.org/r/856601

@thcipriani script is now in Puppet and available via /usr/local/bin/gerrit-git-gc-timing

I have documented the script on our Gerrit administration doc https://wikitech.wikimedia.org/w/index.php?title=Gerrit/Administration&diff=2040334&oldid=2017077&diffmode=source