Page MenuHomePhabricator

zuul-merger takes a while to recreate repository branches
Open, Needs TriagePublic

Description

In zuul/merger/merger.py, Repo.reset takes a while when it is creating heads from the upstream repository. The code with debug statements:

def reset(self):
    self.log.debug("Resetting repository %s" % self.local_path)
    self.update()
    repo = self.createRepoObject()
    origin = repo.remotes.origin
    self.log.debug("Creating heads...")
    for ref in origin.refs:
        if ref.remote_head == 'HEAD':
            continue
        self.log.debug("Creating head %s" % ref)
        repo.create_head(ref.remote_head, ref, force=True)
    self.log.debug("Done creating heads...")

    # try reset to remote HEAD (usually origin/master)
    # If it fails, pick the first reference
    try:
        repo.head.reference = origin.refs['HEAD']
    except IndexError:
        repo.head.reference = origin.refs[0]
    self.log.debug("Resetting repo to head %s" % self.local_path)
    reset_repo_to_head(repo)
    self.log.debug("Done resetting repo to head %s" % self.local_path)
    self.log.debug("Cleaning repo %s" % self.local_path)
    repo.git.clean('-x', '-f', '-d')
    self.log.debug("Done cleaning repo %s" % self.local_path)
    self.log.debug("Done resetting repository %s" % self.local_path)

Gives the following log:

2019-04-10 13:07:49,970 DEBUG zuul.MergeServer: Waiting for server
2019-04-10 13:07:50,034 DEBUG zuul.MergeServer: Registering
2019-04-10 13:07:50,035 DEBUG zuul.MergeServer: Starting worker
2019-04-10 13:07:50,035 DEBUG zuul.MergeServer: Starting merge listener
2019-04-10 13:07:50,081 DEBUG zuul.MergeServer: Got merge job: 391d6781e2594912b64a97589007f7d5
2019-04-10 13:07:50,081 DEBUG zuul.Merger: Merging for change 450433,33.
2019-04-10 13:07:50,081 DEBUG zuul.Merger: Processing refspec refs/changes/33/450433/33 for project mediawiki/core / master ref Ze7bc05e2bdb145f092e1ac1320856272
2019-04-10 13:07:51,205 DEBUG zuul.Merger: Unable to find commit for ref master/Ze7bc05e2bdb145f092e1ac1320856272
2019-04-10 13:07:51,205 DEBUG zuul.Merger: No base commit found for (u'mediawiki/core', u'master')
2019-04-10 13:07:51,205 DEBUG zuul.Repo: Resetting repository /srv/zuul/git/mediawiki/core
2019-04-10 13:07:51,206 DEBUG zuul.Repo: Updating repository /srv/zuul/git/mediawiki/core
2019-04-10 13:07:51,211 DEBUG zuul.Repo: Fetching repository (with tags) /srv/zuul/git/mediawiki/core
2019-04-10 13:07:54,077 DEBUG zuul.Repo: Done updating repository /srv/zuul/git/mediawiki/core
2019-04-10 13:07:54,079 DEBUG zuul.Repo: Creating heads...
2019-04-10 13:07:54,196 DEBUG zuul.Repo: Creating head origin/REL1_1
2019-04-10 13:07:54,559 DEBUG zuul.Repo: Creating head origin/REL1_10
2019-04-10 13:07:54,892 DEBUG zuul.Repo: Creating head origin/REL1_11
2019-04-10 13:07:55,166 DEBUG zuul.Repo: Creating head origin/REL1_12
2019-04-10 13:07:55,458 DEBUG zuul.Repo: Creating head origin/REL1_13
2019-04-10 13:07:55,714 DEBUG zuul.Repo: Creating head origin/REL1_14
2019-04-10 13:07:56,047 DEBUG zuul.Repo: Creating head origin/REL1_15
2019-04-10 13:07:56,333 DEBUG zuul.Repo: Creating head origin/REL1_16
2019-04-10 13:07:56,622 DEBUG zuul.Repo: Creating head origin/REL1_17
2019-04-10 13:07:56,913 DEBUG zuul.Repo: Creating head origin/REL1_18
2019-04-10 13:07:57,163 DEBUG zuul.Repo: Creating head origin/REL1_19
2019-04-10 13:07:57,415 DEBUG zuul.Repo: Creating head origin/REL1_2
2019-04-10 13:07:57,629 DEBUG zuul.Repo: Creating head origin/REL1_20
2019-04-10 13:07:57,840 DEBUG zuul.Repo: Creating head origin/REL1_21
2019-04-10 13:07:58,052 DEBUG zuul.Repo: Creating head origin/REL1_22
2019-04-10 13:07:58,304 DEBUG zuul.Repo: Creating head origin/REL1_23
2019-04-10 13:07:58,615 DEBUG zuul.Repo: Creating head origin/REL1_24
2019-04-10 13:07:58,966 DEBUG zuul.Repo: Creating head origin/REL1_25
2019-04-10 13:07:59,306 DEBUG zuul.Repo: Creating head origin/REL1_26
2019-04-10 13:07:59,598 DEBUG zuul.Repo: Creating head origin/REL1_27
2019-04-10 13:07:59,939 DEBUG zuul.Repo: Creating head origin/REL1_28
2019-04-10 13:08:00,261 DEBUG zuul.Repo: Creating head origin/REL1_29
2019-04-10 13:08:00,553 DEBUG zuul.Repo: Creating head origin/REL1_3
2019-04-10 13:08:00,839 DEBUG zuul.Repo: Creating head origin/REL1_30
2019-04-10 13:08:01,128 DEBUG zuul.Repo: Creating head origin/REL1_31
2019-04-10 13:08:01,463 DEBUG zuul.Repo: Creating head origin/REL1_32
2019-04-10 13:08:01,761 DEBUG zuul.Repo: Creating head origin/REL1_4
2019-04-10 13:08:02,046 DEBUG zuul.Repo: Creating head origin/REL1_5
2019-04-10 13:08:02,335 DEBUG zuul.Repo: Creating head origin/REL1_6
2019-04-10 13:08:02,542 DEBUG zuul.Repo: Creating head origin/REL1_7
2019-04-10 13:08:02,788 DEBUG zuul.Repo: Creating head origin/REL1_8
2019-04-10 13:08:03,087 DEBUG zuul.Repo: Creating head origin/REL1_9
2019-04-10 13:08:03,299 DEBUG zuul.Repo: Creating head origin/fundraising/REL1_22
2019-04-10 13:08:03,554 DEBUG zuul.Repo: Creating head origin/fundraising/REL1_23
2019-04-10 13:08:03,887 DEBUG zuul.Repo: Creating head origin/fundraising/REL1_25
2019-04-10 13:08:04,238 DEBUG zuul.Repo: Creating head origin/fundraising/REL1_26
2019-04-10 13:08:04,570 DEBUG zuul.Repo: Creating head origin/fundraising/REL1_27
2019-04-10 13:08:04,853 DEBUG zuul.Repo: Creating head origin/fundraising/REL1_31
2019-04-10 13:08:05,186 DEBUG zuul.Repo: Creating head origin/master
2019-04-10 13:08:05,458 DEBUG zuul.Repo: Creating head origin/sandbox/jdlrobson/design
2019-04-10 13:08:05,742 DEBUG zuul.Repo: Creating head origin/sandbox/twentyafterfour/group0
2019-04-10 13:08:06,011 DEBUG zuul.Repo: Creating head origin/wmf/1.29.0-wmf.1
2019-04-10 13:08:06,227 DEBUG zuul.Repo: Creating head origin/wmf/1.29.0-wmf.10
2019-04-10 13:08:06,434 DEBUG zuul.Repo: Creating head origin/wmf/1.29.0-wmf.11
2019-04-10 13:08:06,691 DEBUG zuul.Repo: Creating head origin/wmf/1.29.0-wmf.12
2019-04-10 13:08:07,000 DEBUG zuul.Repo: Creating head origin/wmf/1.29.0-wmf.13
2019-04-10 13:08:07,335 DEBUG zuul.Repo: Creating head origin/wmf/1.29.0-wmf.14
2019-04-10 13:08:07,624 DEBUG zuul.Repo: Creating head origin/wmf/1.29.0-wmf.15
2019-04-10 13:08:07,957 DEBUG zuul.Repo: Creating head origin/wmf/1.29.0-wmf.16
2019-04-10 13:08:08,243 DEBUG zuul.Repo: Creating head origin/wmf/1.29.0-wmf.17
2019-04-10 13:08:08,576 DEBUG zuul.Repo: Creating head origin/wmf/1.29.0-wmf.18
2019-04-10 13:08:08,865 DEBUG zuul.Repo: Creating head origin/wmf/1.29.0-wmf.19
2019-04-10 13:08:09,148 DEBUG zuul.Repo: Creating head origin/wmf/1.29.0-wmf.2
2019-04-10 13:08:09,387 DEBUG zuul.Repo: Creating head origin/wmf/1.29.0-wmf.20
2019-04-10 13:08:09,723 DEBUG zuul.Repo: Creating head origin/wmf/1.29.0-wmf.21
2019-04-10 13:08:10,009 DEBUG zuul.Repo: Creating head origin/wmf/1.29.0-wmf.3
2019-04-10 13:08:10,351 DEBUG zuul.Repo: Creating head origin/wmf/1.29.0-wmf.4
2019-04-10 13:08:10,684 DEBUG zuul.Repo: Creating head origin/wmf/1.29.0-wmf.5
2019-04-10 13:08:10,972 DEBUG zuul.Repo: Creating head origin/wmf/1.29.0-wmf.6
2019-04-10 13:08:11,289 DEBUG zuul.Repo: Creating head origin/wmf/1.29.0-wmf.7
2019-04-10 13:08:11,600 DEBUG zuul.Repo: Creating head origin/wmf/1.29.0-wmf.8
2019-04-10 13:08:11,933 DEBUG zuul.Repo: Creating head origin/wmf/1.29.0-wmf.9
2019-04-10 13:08:12,222 DEBUG zuul.Repo: Creating head origin/wmf/1.30.0-wmf.1
2019-04-10 13:08:12,508 DEBUG zuul.Repo: Creating head origin/wmf/1.30.0-wmf.10
2019-04-10 13:08:12,758 DEBUG zuul.Repo: Creating head origin/wmf/1.30.0-wmf.11
2019-04-10 13:08:13,091 DEBUG zuul.Repo: Creating head origin/wmf/1.30.0-wmf.12
2019-04-10 13:08:13,380 DEBUG zuul.Repo: Creating head origin/wmf/1.30.0-wmf.13
2019-04-10 13:08:13,690 DEBUG zuul.Repo: Creating head origin/wmf/1.30.0-wmf.14
2019-04-10 13:08:14,024 DEBUG zuul.Repo: Creating head origin/wmf/1.30.0-wmf.15
2019-04-10 13:08:14,323 DEBUG zuul.Repo: Creating head origin/wmf/1.30.0-wmf.16
2019-04-10 13:08:14,596 DEBUG zuul.Repo: Creating head origin/wmf/1.30.0-wmf.17
2019-04-10 13:08:14,879 DEBUG zuul.Repo: Creating head origin/wmf/1.30.0-wmf.18
2019-04-10 13:08:15,190 DEBUG zuul.Repo: Creating head origin/wmf/1.30.0-wmf.19
2019-04-10 13:08:15,523 DEBUG zuul.Repo: Creating head origin/wmf/1.30.0-wmf.2
2019-04-10 13:08:15,812 DEBUG zuul.Repo: Creating head origin/wmf/1.30.0-wmf.3
2019-04-10 13:08:16,092 DEBUG zuul.Repo: Creating head origin/wmf/1.30.0-wmf.4
2019-04-10 13:08:16,381 DEBUG zuul.Repo: Creating head origin/wmf/1.30.0-wmf.5
2019-04-10 13:08:16,639 DEBUG zuul.Repo: Creating head origin/wmf/1.30.0-wmf.6
2019-04-10 13:08:16,972 DEBUG zuul.Repo: Creating head origin/wmf/1.30.0-wmf.7
2019-04-10 13:08:17,261 DEBUG zuul.Repo: Creating head origin/wmf/1.30.0-wmf.9
2019-04-10 13:08:17,547 DEBUG zuul.Repo: Creating head origin/wmf/1.31.0-wmf.1
2019-04-10 13:08:17,880 DEBUG zuul.Repo: Creating head origin/wmf/1.31.0-wmf.10
2019-04-10 13:08:18,163 DEBUG zuul.Repo: Creating head origin/wmf/1.31.0-wmf.11
2019-04-10 13:08:18,496 DEBUG zuul.Repo: Creating head origin/wmf/1.31.0-wmf.12
2019-04-10 13:08:18,785 DEBUG zuul.Repo: Creating head origin/wmf/1.31.0-wmf.15
2019-04-10 13:08:19,071 DEBUG zuul.Repo: Creating head origin/wmf/1.31.0-wmf.16
2019-04-10 13:08:19,404 DEBUG zuul.Repo: Creating head origin/wmf/1.31.0-wmf.17
2019-04-10 13:08:19,717 DEBUG zuul.Repo: Creating head origin/wmf/1.31.0-wmf.2
2019-04-10 13:08:20,046 DEBUG zuul.Repo: Creating head origin/wmf/1.31.0-wmf.20
2019-04-10 13:08:20,335 DEBUG zuul.Repo: Creating head origin/wmf/1.31.0-wmf.21
2019-04-10 13:08:20,620 DEBUG zuul.Repo: Creating head origin/wmf/1.31.0-wmf.22
2019-04-10 13:08:20,962 DEBUG zuul.Repo: Creating head origin/wmf/1.31.0-wmf.23
2019-04-10 13:08:21,279 DEBUG zuul.Repo: Creating head origin/wmf/1.31.0-wmf.24
2019-04-10 13:08:21,570 DEBUG zuul.Repo: Creating head origin/wmf/1.31.0-wmf.25
2019-04-10 13:08:21,861 DEBUG zuul.Repo: Creating head origin/wmf/1.31.0-wmf.26
2019-04-10 13:08:22,142 DEBUG zuul.Repo: Creating head origin/wmf/1.31.0-wmf.27
2019-04-10 13:08:22,428 DEBUG zuul.Repo: Creating head origin/wmf/1.31.0-wmf.28
2019-04-10 13:08:22,677 DEBUG zuul.Repo: Creating head origin/wmf/1.31.0-wmf.29
2019-04-10 13:08:23,011 DEBUG zuul.Repo: Creating head origin/wmf/1.31.0-wmf.3
2019-04-10 13:08:23,350 DEBUG zuul.Repo: Creating head origin/wmf/1.31.0-wmf.30
2019-04-10 13:08:23,685 DEBUG zuul.Repo: Creating head origin/wmf/1.31.0-wmf.4
2019-04-10 13:08:23,974 DEBUG zuul.Repo: Creating head origin/wmf/1.31.0-wmf.5
2019-04-10 13:08:24,260 DEBUG zuul.Repo: Creating head origin/wmf/1.31.0-wmf.6
2019-04-10 13:08:24,593 DEBUG zuul.Repo: Creating head origin/wmf/1.31.0-wmf.7
2019-04-10 13:08:24,948 DEBUG zuul.Repo: Creating head origin/wmf/1.31.0-wmf.8
2019-04-10 13:08:25,282 DEBUG zuul.Repo: Creating head origin/wmf/1.32.0-wmf.1
2019-04-10 13:08:25,568 DEBUG zuul.Repo: Creating head origin/wmf/1.32.0-wmf.10
2019-04-10 13:08:25,857 DEBUG zuul.Repo: Creating head origin/wmf/1.32.0-wmf.12
2019-04-10 13:08:26,081 DEBUG zuul.Repo: Creating head origin/wmf/1.32.0-wmf.13
2019-04-10 13:08:26,331 DEBUG zuul.Repo: Creating head origin/wmf/1.32.0-wmf.14
2019-04-10 13:08:26,623 DEBUG zuul.Repo: Creating head origin/wmf/1.32.0-wmf.15
2019-04-10 13:08:26,950 DEBUG zuul.Repo: Creating head origin/wmf/1.32.0-wmf.16
2019-04-10 13:08:27,283 DEBUG zuul.Repo: Creating head origin/wmf/1.32.0-wmf.18
2019-04-10 13:08:27,573 DEBUG zuul.Repo: Creating head origin/wmf/1.32.0-wmf.19
2019-04-10 13:08:27,908 DEBUG zuul.Repo: Creating head origin/wmf/1.32.0-wmf.2
2019-04-10 13:08:28,241 DEBUG zuul.Repo: Creating head origin/wmf/1.32.0-wmf.20
2019-04-10 13:08:28,530 DEBUG zuul.Repo: Creating head origin/wmf/1.32.0-wmf.22
2019-04-10 13:08:28,863 DEBUG zuul.Repo: Creating head origin/wmf/1.32.0-wmf.23
2019-04-10 13:08:29,149 DEBUG zuul.Repo: Creating head origin/wmf/1.32.0-wmf.24
2019-04-10 13:08:29,482 DEBUG zuul.Repo: Creating head origin/wmf/1.32.0-wmf.26
2019-04-10 13:08:29,771 DEBUG zuul.Repo: Creating head origin/wmf/1.32.0-wmf.3
2019-04-10 13:08:30,072 DEBUG zuul.Repo: Creating head origin/wmf/1.32.0-wmf.4
2019-04-10 13:08:30,404 DEBUG zuul.Repo: Creating head origin/wmf/1.32.0-wmf.5
2019-04-10 13:08:30,690 DEBUG zuul.Repo: Creating head origin/wmf/1.32.0-wmf.6
2019-04-10 13:08:30,965 DEBUG zuul.Repo: Creating head origin/wmf/1.32.0-wmf.7
2019-04-10 13:08:31,223 DEBUG zuul.Repo: Creating head origin/wmf/1.32.0-wmf.8
2019-04-10 13:08:31,444 DEBUG zuul.Repo: Creating head origin/wmf/1.32.0-wmf.999
2019-04-10 13:08:31,661 DEBUG zuul.Repo: Creating head origin/wmf/1.33-wmf.8
2019-04-10 13:08:31,915 DEBUG zuul.Repo: Creating head origin/wmf/1.33.0-wmf.1
2019-04-10 13:08:32,245 DEBUG zuul.Repo: Creating head origin/wmf/1.33.0-wmf.12
2019-04-10 13:08:32,537 DEBUG zuul.Repo: Creating head origin/wmf/1.33.0-wmf.13
2019-04-10 13:08:32,872 DEBUG zuul.Repo: Creating head origin/wmf/1.33.0-wmf.14
2019-04-10 13:08:33,161 DEBUG zuul.Repo: Creating head origin/wmf/1.33.0-wmf.16
2019-04-10 13:08:33,430 DEBUG zuul.Repo: Creating head origin/wmf/1.33.0-wmf.17
2019-04-10 13:08:33,639 DEBUG zuul.Repo: Creating head origin/wmf/1.33.0-wmf.18
2019-04-10 13:08:33,847 DEBUG zuul.Repo: Creating head origin/wmf/1.33.0-wmf.19
2019-04-10 13:08:34,055 DEBUG zuul.Repo: Creating head origin/wmf/1.33.0-wmf.2
2019-04-10 13:08:34,305 DEBUG zuul.Repo: Creating head origin/wmf/1.33.0-wmf.20
2019-04-10 13:08:34,569 DEBUG zuul.Repo: Creating head origin/wmf/1.33.0-wmf.21
2019-04-10 13:08:34,905 DEBUG zuul.Repo: Creating head origin/wmf/1.33.0-wmf.22
2019-04-10 13:08:35,220 DEBUG zuul.Repo: Creating head origin/wmf/1.33.0-wmf.23
2019-04-10 13:08:35,563 DEBUG zuul.Repo: Creating head origin/wmf/1.33.0-wmf.24
2019-04-10 13:08:35,852 DEBUG zuul.Repo: Creating head origin/wmf/1.33.0-wmf.25
2019-04-10 13:08:36,137 DEBUG zuul.Repo: Creating head origin/wmf/1.33.0-wmf.3
2019-04-10 13:08:36,412 DEBUG zuul.Repo: Creating head origin/wmf/1.33.0-wmf.4
2019-04-10 13:08:36,708 DEBUG zuul.Repo: Creating head origin/wmf/1.33.0-wmf.6
2019-04-10 13:08:36,962 DEBUG zuul.Repo: Creating head origin/wmf/1.33.0-wmf.8
2019-04-10 13:08:37,301 DEBUG zuul.Repo: Creating head origin/wmf/1.33.0-wmf.9
2019-04-10 13:08:37,637 DEBUG zuul.Repo: Creating head origin/wmf/php-1.32.0-wmf.15
2019-04-10 13:08:37,970 DEBUG zuul.Repo: Creating head origin/wmf/testing
2019-04-10 13:08:38,259 DEBUG zuul.Repo: Done creating heads...
2019-04-10 13:08:38,380 DEBUG zuul.Repo: Resetting repo to head /srv/zuul/git/mediawiki/core
2019-04-10 13:08:38,530 DEBUG zuul.Repo: Done resetting repo to head /srv/zuul/git/mediawiki/core
2019-04-10 13:08:38,530 DEBUG zuul.Repo: Cleaning repo /srv/zuul/git/mediawiki/core
2019-04-10 13:08:38,561 DEBUG zuul.Repo: Done cleaning repo /srv/zuul/git/mediawiki/core
2019-04-10 13:08:38,561 DEBUG zuul.Repo: Done resetting repository /srv/zuul/git/mediawiki/core
2019-04-10 13:08:38,754 DEBUG zuul.Merger: Merging change...
2019-04-10 13:08:38,756 DEBUG zuul.Repo: Checking out c51e4e24ef4ce5a1ef9933fd20266e75a758027f
2019-04-10 13:08:41,613 DEBUG zuul.Repo: Merging refs/changes/33/450433/33 with args ['-s', 'resolve', 'FETCH_HEAD']
...

Time spend in creating the heads is roughly 45 seconds for mediawiki/core

2019-04-10 13:07:54,079 DEBUG zuul.Repo: Creating heads...
2019-04-10 13:08:38,259 DEBUG zuul.Repo: Done creating heads...

Event Timeline

hashar created this task.Apr 10 2019, 1:12 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 10 2019, 1:12 PM
hashar updated the task description. (Show Details)Apr 10 2019, 1:35 PM

Seems like zuul-merger does not prune stall references:

zuul@contint2001:/srv/zuul/git/mediawiki/core$ git remote prune -n origin
Pruning origin
URL: ssh://jenkins-bot@gerrit.wikimedia.org:29418/mediawiki/core
 * [would prune] origin/REL1_1
 * [would prune] origin/REL1_10
 * [would prune] origin/REL1_11
 * [would prune] origin/REL1_12
 * [would prune] origin/REL1_13
 * [would prune] origin/REL1_14
 * [would prune] origin/REL1_15
 * [would prune] origin/REL1_16
 * [would prune] origin/REL1_17
 * [would prune] origin/REL1_18
 * [would prune] origin/REL1_19
 * [would prune] origin/REL1_2
 * [would prune] origin/REL1_20
 * [would prune] origin/REL1_21
 * [would prune] origin/REL1_22
 * [would prune] origin/REL1_24
 * [would prune] origin/REL1_3
 * [would prune] origin/REL1_4
 * [would prune] origin/REL1_5
 * [would prune] origin/REL1_6
 * [would prune] origin/REL1_7
 * [would prune] origin/REL1_8
 * [would prune] origin/REL1_9
 * [would prune] origin/fundraising/REL1_22
 * [would prune] origin/fundraising/REL1_23
 * [would prune] origin/fundraising/REL1_25
 * [would prune] origin/fundraising/REL1_26
 * [would prune] origin/sandbox/jdlrobson/design
 * [would prune] origin/sandbox/twentyafterfour/group0
 * [would prune] origin/wmf/1.29.0-wmf.1
 * [would prune] origin/wmf/1.29.0-wmf.10
 * [would prune] origin/wmf/1.29.0-wmf.11
 * [would prune] origin/wmf/1.29.0-wmf.12
 * [would prune] origin/wmf/1.29.0-wmf.13
 * [would prune] origin/wmf/1.29.0-wmf.14
 * [would prune] origin/wmf/1.29.0-wmf.15
 * [would prune] origin/wmf/1.29.0-wmf.16
 * [would prune] origin/wmf/1.29.0-wmf.17
 * [would prune] origin/wmf/1.29.0-wmf.18
 * [would prune] origin/wmf/1.29.0-wmf.19
 * [would prune] origin/wmf/1.29.0-wmf.2
 * [would prune] origin/wmf/1.29.0-wmf.20
 * [would prune] origin/wmf/1.29.0-wmf.21
 * [would prune] origin/wmf/1.29.0-wmf.3
 * [would prune] origin/wmf/1.29.0-wmf.4
 * [would prune] origin/wmf/1.29.0-wmf.5
 * [would prune] origin/wmf/1.29.0-wmf.6
 * [would prune] origin/wmf/1.29.0-wmf.7
 * [would prune] origin/wmf/1.29.0-wmf.8
 * [would prune] origin/wmf/1.29.0-wmf.9
 * [would prune] origin/wmf/1.30.0-wmf.1
 * [would prune] origin/wmf/1.30.0-wmf.10
 * [would prune] origin/wmf/1.30.0-wmf.11
 * [would prune] origin/wmf/1.30.0-wmf.12
 * [would prune] origin/wmf/1.30.0-wmf.13
 * [would prune] origin/wmf/1.30.0-wmf.14
 * [would prune] origin/wmf/1.30.0-wmf.15
 * [would prune] origin/wmf/1.30.0-wmf.16
 * [would prune] origin/wmf/1.30.0-wmf.17
 * [would prune] origin/wmf/1.30.0-wmf.18
 * [would prune] origin/wmf/1.30.0-wmf.19
 * [would prune] origin/wmf/1.30.0-wmf.2
 * [would prune] origin/wmf/1.30.0-wmf.3
 * [would prune] origin/wmf/1.30.0-wmf.4
 * [would prune] origin/wmf/1.30.0-wmf.5
 * [would prune] origin/wmf/1.30.0-wmf.6
 * [would prune] origin/wmf/1.30.0-wmf.7
 * [would prune] origin/wmf/1.30.0-wmf.9
 * [would prune] origin/wmf/1.31.0-wmf.1
 * [would prune] origin/wmf/1.31.0-wmf.10
 * [would prune] origin/wmf/1.31.0-wmf.11
 * [would prune] origin/wmf/1.31.0-wmf.12
 * [would prune] origin/wmf/1.31.0-wmf.15
 * [would prune] origin/wmf/1.31.0-wmf.16
 * [would prune] origin/wmf/1.31.0-wmf.17
 * [would prune] origin/wmf/1.31.0-wmf.2
 * [would prune] origin/wmf/1.31.0-wmf.20
 * [would prune] origin/wmf/1.31.0-wmf.21
 * [would prune] origin/wmf/1.31.0-wmf.22
 * [would prune] origin/wmf/1.31.0-wmf.23
 * [would prune] origin/wmf/1.31.0-wmf.24
 * [would prune] origin/wmf/1.31.0-wmf.25
 * [would prune] origin/wmf/1.31.0-wmf.26
 * [would prune] origin/wmf/1.31.0-wmf.27
 * [would prune] origin/wmf/1.31.0-wmf.28
 * [would prune] origin/wmf/1.31.0-wmf.29
 * [would prune] origin/wmf/1.31.0-wmf.3
 * [would prune] origin/wmf/1.31.0-wmf.30
 * [would prune] origin/wmf/1.31.0-wmf.4
 * [would prune] origin/wmf/1.31.0-wmf.5
 * [would prune] origin/wmf/1.31.0-wmf.6
 * [would prune] origin/wmf/1.31.0-wmf.7
 * [would prune] origin/wmf/1.31.0-wmf.8
 * [would prune] origin/wmf/1.32.0-wmf.1
 * [would prune] origin/wmf/1.32.0-wmf.10
 * [would prune] origin/wmf/1.32.0-wmf.12
 * [would prune] origin/wmf/1.32.0-wmf.13
 * [would prune] origin/wmf/1.32.0-wmf.14
 * [would prune] origin/wmf/1.32.0-wmf.15
 * [would prune] origin/wmf/1.32.0-wmf.16
 * [would prune] origin/wmf/1.32.0-wmf.18
 * [would prune] origin/wmf/1.32.0-wmf.19
 * [would prune] origin/wmf/1.32.0-wmf.2
 * [would prune] origin/wmf/1.32.0-wmf.20
 * [would prune] origin/wmf/1.32.0-wmf.22
 * [would prune] origin/wmf/1.32.0-wmf.23
 * [would prune] origin/wmf/1.32.0-wmf.24
 * [would prune] origin/wmf/1.32.0-wmf.26
 * [would prune] origin/wmf/1.32.0-wmf.3
 * [would prune] origin/wmf/1.32.0-wmf.4
 * [would prune] origin/wmf/1.32.0-wmf.5
 * [would prune] origin/wmf/1.32.0-wmf.6
 * [would prune] origin/wmf/1.32.0-wmf.7
 * [would prune] origin/wmf/1.32.0-wmf.8
 * [would prune] origin/wmf/1.32.0-wmf.999
 * [would prune] origin/wmf/1.33-wmf.8
 * [would prune] origin/wmf/1.33.0-wmf.1
 * [would prune] origin/wmf/1.33.0-wmf.12
 * [would prune] origin/wmf/1.33.0-wmf.13
 * [would prune] origin/wmf/1.33.0-wmf.14
 * [would prune] origin/wmf/1.33.0-wmf.16
 * [would prune] origin/wmf/1.33.0-wmf.2
 * [would prune] origin/wmf/1.33.0-wmf.3
 * [would prune] origin/wmf/1.33.0-wmf.4
 * [would prune] origin/wmf/1.33.0-wmf.6
 * [would prune] origin/wmf/1.33.0-wmf.8
 * [would prune] origin/wmf/1.33.0-wmf.9
 * [would prune] origin/wmf/php-1.32.0-wmf.15
 * [would prune] origin/wmf/testing

Mentioned in SAL (#wikimedia-releng) [2019-04-10T14:13:09Z] <hashar> contint2001: sudo -u zuul git -C /srv/zuul/git/mediawiki/core remote prune origin # T220606

Down to just 10 seconds after running git remote prune origin.

2019-04-10 14:12:33,095 DEBUG zuul.Repo: Resetting repository /srv/zuul/git/mediawiki/core
2019-04-10 14:12:33,096 DEBUG zuul.Repo: Updating repository /srv/zuul/git/mediawiki/core
2019-04-10 14:12:43,281 DEBUG zuul.Repo: Checking out 37e13f0c6eb3f8bccc49a646de477c2c7bb0d0c7

Left to do:

  • Run git remote prune origin on all zuul-merger repositories
  • why it is so slow even for the dozen of branches we have
  • why does zuul-merger never prune remote branches? It should from time to time at least?

Mentioned in SAL (#wikimedia-releng) [2019-04-10T14:16:17Z] <hashar> contint1001: sudo -u zuul git -C /srv/zuul/git/mediawiki/core remote prune origin # T220606

Mentioned in SAL (#wikimedia-operations) [2019-04-10T14:20:21Z] <hashar> CI processing was a bit slower than usual over the past couple hours or so. It should be slightly faster now T220606

I could not reproduce on my local machine given a basic test script which creates several heads:

import git
import logging

logging.basicConfig(level=logging.DEBUG)

repo = git.Repo('.')
origin = repo.remotes.origin
for ref in origin.refs:
    if ref.remote_head == 'HEAD':
        print('remote HEAD')
        continue
    print('Creating head for %s' % ref)
    repo.create_head('testor-' + ref.remote_head, ref, force=True)

On my local machine that takes ~ 120 ms.

On contint1001 I copied /srv/zuul/git/mediawiki/core into my homedir and I have run the same script: 8 seconds.

strace -e trace=file for a single creation:

Creating head for origin/master
[pid 1056] 16:57:49.857489 open("/home/hashar/core/.git/origin/master", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 1056] 16:57:49.857541 open("/home/hashar/core/.git/packed-refs", O_RDONLY) = 3
[pid 1056] 16:57:49.964446 open("/home/hashar/core/.git/refs/origin/master", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 1056] 16:57:49.964543 open("/home/hashar/core/.git/packed-refs", O_RDONLY) = 3
[pid 1056] 16:57:50.098010 open("/home/hashar/core/.git/refs/tags/origin/master", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 1056] 16:57:50.098090 open("/home/hashar/core/.git/packed-refs", O_RDONLY) = 3
[pid 1056] 16:57:50.198512 open("/home/hashar/core/.git/refs/heads/origin/master", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 1056] 16:57:50.198556 open("/home/hashar/core/.git/packed-refs", O_RDONLY) = 3
[pid 1056] 16:57:50.296983 open("/home/hashar/core/.git/refs/remotes/origin/master", O_RDONLY) = 3
[pid 1109] 16:57:50.297312 open(".git/objects/pack/pack-4118d6c9a62ecdfff0c74d5f63ea3fdaccccbf34.idx", O_RDONLY|O_NOATIME|O_CLOEXEC) = 3
[pid 1109] 16:57:50.297467 open(".git/objects/08/e27d6d40ed2f6eea8abc509254e11c92212a8d", O_RDONLY|O_NOATIME|O_CLOEXEC) = 3
[pid 1056] 16:57:50.297778 stat("/home/hashar/core/.git/refs/heads", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 1056] 16:57:50.297855 open("/home/hashar/core/.git/refs/heads/testor-master.lock", O_WRONLY|O_CREAT|O_EXCL, 0600) = 3
[pid 1056] 16:57:50.298013 rename("/home/hashar/core/.git/refs/heads/testor-master.lock", "/home/hashar/core/.git/refs/heads/testor-master") = 0
[pid 1056] 16:57:50.348803 chmod("/home/hashar/core/.git/refs/heads/testor-master", 0644) = 0

A note is that `.git/packed-refs` is 8.5MBytes. So might be related to T70481

So I think we can just prune all references from time to time:

rm .git/packed_refs
find .git/refs/ -mindepth 1 -delete
git fetch
git pack-refs
hashar added a comment.EditedApr 10 2019, 7:51 PM

A fast way to delete all Zuul references:

git fetch origin --prune +refs/zuul/*:refs/zuul/*

And to prune stalled local branches:

git fetch origin --update-head-ok --prune +refs/heads/*:refs/heads/*