Page MenuHomePhabricator
Create Task
Maniphest T303828

Delete wmf branches from Gerrit repositories
Closed, ResolvedPublic2 Estimated Story Points
Actions

Description

As we cut more and more wmf branches, we should automatize the garbage collection of the branches from all repositories.

We once started a deletion with T244368 which got us a quick shell script delete-wmf-branches script.

T250095 made make-release to convert wmf branches to tags https://gerrit.wikimedia.org/r/c/mediawiki/tools/release/+/683934. Example usage:

branch.py --core --core-bundle wmf_core --bundle wmf_branch --delete <VERSION>

We have barely run it:

git ls-remote https://gerrit.wikimedia.org/r/mediawiki/core --headsrefs/heads/wmf/*
f12c3f71a3ea02f305770253afef2e4736b5af76	refs/heads/wmf/1.38.0-wmf.10
3eae43afcb2fe3adffa9201db67fd6f706e5ecc9	refs/heads/wmf/1.38.0-wmf.11
3283c039f723ffc7c66e646caa1534fbb5b6a6a7	refs/heads/wmf/1.38.0-wmf.12
be7254dc6ebf8bf8d7c2e81e166c026d0a1e112a	refs/heads/wmf/1.38.0-wmf.13
a9251939ca2b6ab9feadc5b9a4d6bf9c65526d6b	refs/heads/wmf/1.38.0-wmf.14
79173b685a5ad9bb9d4c8af30f7669731a9d6906	refs/heads/wmf/1.38.0-wmf.15
86035d5765bab50e741b58553ab62a40fd9d8033	refs/heads/wmf/1.38.0-wmf.16
a7e96ffb7ddca970d61ed334f62b625284956629	refs/heads/wmf/1.38.0-wmf.17
cd0a7969c8bdbffc35f151ea8023742b2670bbdc	refs/heads/wmf/1.38.0-wmf.18
aaac0d6a13a7c612b0759ee45751f0c5cb83b1dc	refs/heads/wmf/1.38.0-wmf.19
179b099f4da126086f547344f3d8c647acd008d5	refs/heads/wmf/1.38.0-wmf.2
ef9116432d8e7744c218167ad7f9d779f7f201a4	refs/heads/wmf/1.38.0-wmf.20
19ee29788b9445fbef6f643312916a92da580934	refs/heads/wmf/1.38.0-wmf.21
ba204e68a838ef4e11cc146f48e826831f035b32	refs/heads/wmf/1.38.0-wmf.22
2a7b91e8f53c0ffa2d0cec0478333f462303e171	refs/heads/wmf/1.38.0-wmf.23
a2be7506f8c9972baa685ff3d3d477fd19132671	refs/heads/wmf/1.38.0-wmf.24
eefd25b639d72cddec3473ce41a9bf82e11f2658	refs/heads/wmf/1.38.0-wmf.25
d797b02e58609f8ab242a940616d4c7829b9eec6	refs/heads/wmf/1.38.0-wmf.26
f03b0f73a105de57bd15fb49984b258c1a838b87	refs/heads/wmf/1.38.0-wmf.3
b9d44aab22a2668a28d539dc1d813bab994bef4e	refs/heads/wmf/1.38.0-wmf.4
2c7d643a5fa67c005e3dd1f015526bf21942ae9d	refs/heads/wmf/1.38.0-wmf.5
2b2d738ef2863fabea0f078959aee036a9e81230	refs/heads/wmf/1.38.0-wmf.6
85644d6274af2f748e5cabeeff8da7ae9a37febf	refs/heads/wmf/1.38.0-wmf.7
3934f55e6e02eb910ac12da1121b1ab83c2756a3	refs/heads/wmf/1.38.0-wmf.8
9aaa2218d225fb4e77d24b0a0f609791f5a06977	refs/heads/wmf/1.38.0-wmf.9

I suppose they are manually cleaned up from time to time T291501 was for 1.37 though they are not cleaned for all extensions/skins (AbuseFilter has branches since wmf/1.35.0-wmf.1).

Maybe they can be converted to tags (see above make-release branch.py supports that).

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
gerrit: restore IP addresses in ssh_known_hostsoperations/puppetproduction+4 -1
scap: do not show logo when cleaning old versionsoperations/puppetproduction+1 -1
scap: delete wmf branches automaticallyoperations/puppetproduction+1 -1
make-release: hardened branch existence detectionmediawiki/tools/releasemaster+41 -10
make-release: abort branch deletion if tag failedmediawiki/tools/releasemaster+9 -5
make-release: add test for delete_branchmediawiki/tools/releasemaster+52 -0
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
scap clean: hide remote line on pushing deletesrepos/releng/scap!669hasharscap-clean-mute-branch-deletion-pushmaster
scap prep: set pushInsteadOf in the submodules configrepos/releng/scap!607hasharscap-prep-pushInsteadOfmaster
Customize query in GitLab

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Mar 18 2022, 6:03 PM

Change 771955 had a related patch set uploaded (by Hashar; author: Hashar):

[mediawiki/tools/release@master] make-release: add test for delete_branch

https://gerrit.wikimedia.org/r/771955

gerritbot added a project: Patch-For-Review.Mar 18 2022, 6:03 PM

Change 771956 had a related patch set uploaded (by Hashar; author: Hashar):

[mediawiki/tools/release@master] make-release: abort branch deletion if tag failed

https://gerrit.wikimedia.org/r/771956

hashar added a comment.Mar 18 2022, 6:07 PM

I have looked at the make-release/branch.py script and it has support to delete old branches. Unfortunately while reading the code and testing it (with --noop) I found several issues with errors handling. Notably:

  • when the tag fails to be created, we still delete the branch
  • a non existing branch on one of the repo abort the script execution (so it is not rentrant when processing a bundle)

I have added some tests and made a change to prevent deleting a branch when the tag failed to create. That is a start.

hashar added a comment.Mar 18 2022, 6:09 PM

Another one is that repositories which are no more deployed to Wikimedia are unknown to the branch.py script and would thus never have their old wmf branches collected.

There was also a concern about tags piling up, I guess we can move them to refs/attic/tags/* so we still have them around if needed. But that is a different discussion.

gerritbot added a comment.Mar 23 2022, 10:34 PM

Change 771955 merged by jenkins-bot:

[mediawiki/tools/release@master] make-release: add test for delete_branch

https://gerrit.wikimedia.org/r/771955

gerritbot added a comment.Mar 23 2022, 10:42 PM

Change 771956 merged by jenkins-bot:

[mediawiki/tools/release@master] make-release: abort branch deletion if tag failed

https://gerrit.wikimedia.org/r/771956

Maintenance_bot removed a project: Patch-For-Review.Mar 23 2022, 11:10 PM
jnuche reassigned this task from jnuche to hashar.Mar 24 2022, 9:05 AM
jnuche subscribed.
gerritbot added a comment.Mar 24 2022, 1:56 PM

Change 773504 had a related patch set uploaded (by Hashar; author: Hashar):

[mediawiki/tools/release@master] make-release: hardened branch existence detection

https://gerrit.wikimedia.org/r/773504

gerritbot added a project: Patch-For-Review.Mar 24 2022, 1:56 PM
gerritbot added a comment.Mar 25 2022, 3:53 PM

Change 773504 merged by jenkins-bot:

[mediawiki/tools/release@master] make-release: hardened branch existence detection

https://gerrit.wikimedia.org/r/773504

Maintenance_bot removed a project: Patch-For-Review.Mar 25 2022, 4:10 PM
thcipriani edited projects, added Release-Engineering-Team (🌱 Spring Cleaning — April 2022); removed Release-Engineering-Team (🚂🧪 Trainsperiment Week).Apr 4 2022, 9:01 PM
thcipriani moved this task from Backlog to In progress on the Release-Engineering-Team (🌱 Spring Cleaning — April 2022) board.
hashar added a parent task: T220606: zuul-merger takes a while to recreate repository branches.Apr 19 2022, 9:30 AM
hashar mentioned this in T220606: zuul-merger takes a while to recreate repository branches.
hashar removed hashar as the assignee of this task.Apr 19 2022, 9:35 AM

I am unassigning myself since I have way too many tasks to juggle with. To resume I think we would need to:

A) decide whether we delete old wmf branches or convert them to tags. I thought deleting the branch would cause the old refs to be garbage collected, but they are still associated with a change in Gerrit (refs/changes/YY/XXXYY/Z) which is sufficient to hold them in the repos

B) enhance the script to act on a list of branches (possibly a wildcard) we want to act on instead of on a per branch basis

Krinkle subscribed.May 27 2022, 4:35 PM

I currently have the following aliases (dotfiles repo):

	# Wildcard deletion of remote branches
	#
	# Usage: git remote-wildcard-br-d 'wmf/1.36*'
	# Usage: git remote-<tab>
	#
	# Inspired by <https://stackoverflow.com/q/3670355/319266>
	#
	# Note that git-push requires its branch argument to be without the
	# localised "refs/remotes/origin" prefix. As such, we strip that off.
	#
	# $ git branch -rl 'origin/wmf/1.36*' --format '%(refname)'
	# $ git branch -rl 'origin/wmf/1.36*' --format '%(refname:lstrip=3)'
	# $ git push origin --delete
	#
	remote-wildcard-br-d = "!f() { git branch -rl \"origin/$1\" --format '%(refname:lstrip=3)' | xargs git push origin --delete; }; f"

And have removed the old branches again on various repos including mediawiki/core, so as to de-clutter interfaces such as the "Cherry pick" dialog etc

Stashbot added a comment.Aug 16 2022, 7:17 PM

Mentioned in SAL (#wikimedia-releng) [2022-08-16T19:17:38Z] <Krinkle> mediawiki/extensions/Scribunto$ git remote-wildcard-br-d 'wmf/1.35*' # ref T303828

thcipriani edited projects, added Release-Engineering-Team (Bonus Level 🕹️); removed Release-Engineering-Team (🌱 Spring Cleaning — April 2022).Aug 17 2022, 3:29 PM
dancy moved this task from Backlog to Ready on the Release-Engineering-Team (Bonus Level 🕹️) board.Aug 22 2022, 4:08 PM
• demon claimed this task.Aug 22 2022, 4:17 PM
dancy claimed this task.Aug 29 2022, 4:07 PM
dancy added a subscriber: • demon.
Stashbot added a comment.Sep 6 2022, 12:20 AM

Mentioned in SAL (#wikimedia-releng) [2022-09-06T00:20:09Z] <Krinkle> Prune various old mediawiki/core wmf branches for Gerrit usability, ref T303828

dancy removed dancy as the assignee of this task.Sep 19 2022, 4:04 PM
thcipriani edited projects, added Release-Engineering-Team (Priority Backlog 📥); removed Release-Engineering-Team (Bonus Level 🕹️).Nov 14 2022, 1:49 PM
• Cyaj8 updated the task description. (Show Details)Nov 18 2022, 12:24 PM
Dylsss updated the task description. (Show Details)Nov 18 2022, 1:10 PM
TheresNoTime removed a subscriber: RhinosF1.Dec 15 2022, 11:35 PM
thcipriani edited projects, added Release-Engineering-Team (Onboarding 🚀); removed Release-Engineering-Team (Priority Backlog 📥).Jun 9 2023, 9:20 PM
thcipriani merged a task: T281939: Clear out the old branches on mediawiki/core.git that aren't active any more.Nov 15 2023, 4:36 PM
thcipriani added a subscriber: Jdforrester-WMF.
hashar mentioned this in T218783: `scap clean` failure: SSH Too many authentication failures: 7.Feb 15 2024, 7:28 PM

The branch are no more cleaned since T218783 which moved the logic behind a feature flag (scap clean --delete-gerrit-branch) due to some weird breakage related to ssh auth socket. If that T218783 is investigated again and fixed, that would clean out the old branch and I solve this one.

hashar mentioned this in T374717: Speed up git clone operation in wmf-quibble job.Sep 13 2024, 2:02 PM
hashar added a parent task: T374717: Speed up git clone operation in wmf-quibble job.
hashar added a comment.EditedSep 13 2024, 2:05 PM

From T374717 , a recent build of wmf-quibble job cloning 64 jobs ends up taking more than 15 minutes of cumulative time. Most of that time is spent in purging and recreating branches :)

thcipriani moved this task from Onboarding 🚀 to Priority Backlog 📥 on the Release-Engineering-Team board.Oct 30 2024, 3:45 PM
thcipriani edited projects, added Release-Engineering-Team (Priority Backlog 📥); removed Release-Engineering-Team (Onboarding 🚀).
hashar added a comment.Nov 25 2024, 1:38 PM

After having fixed T218783 (via T306425), scap clean --delete-gerrit-branch 1.39.0-wmf.1 fails:

scap clean --delete-gerrit-branch 1.44.0-wmf.2
13:27:06 Cleaning branch: 1.44.0-wmf.2
13:27:06 Clean files owned by www-data
13:27:08 Clean /srv/mediawiki-staging/wmf-config/ExtensionMessages-1.44.0-wmf.2.php
13:27:08 Started prune-git-branches
remote: Processing changes: refs: 1, done    
remote: error: internal error
remote: Processing changes: refs: 1, done    
remote: error: internal error
...

From the Gerrit server side:

[2024-11-25T13:27:09.112Z] [SSH git-receive-pack /mediawiki/extensions/3D (trainbranchbot)] INFO  com.google.gerrit.server.git.MultiProgressMonitor : Processing changes: refs: 1 (\) [CONTEXT ratelimit_period="1 MINUTES [skipped: 2]" ]
[2024-11-25T13:27:09.113Z] [SSH git-receive-pack /mediawiki/extensions/3D (trainbranchbot)] WARN  com.google.gerrit.server.git.MultiProgressMonitor : MultiProgressMonitor worker did not call end() before returning
[2024-11-25T13:27:09.113Z] [SSH git-receive-pack /mediawiki/extensions/3D (trainbranchbot)] ERROR com.google.gerrit.server.git.receive.AsyncReceiveCommits : error while processing push
java.util.concurrent.ExecutionException: com.google.gerrit.exceptions.StorageException: Cannot update gitlinks
        at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
        at com.google.gerrit.server.git.receive.AsyncReceiveCommits.preReceive(AsyncReceiveCommits.java:407)
        at com.google.gerrit.server.git.receive.AsyncReceiveCommits.lambda$asHook$0(AsyncReceiveCommits.java:351)
        at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:2287)
        at org.eclipse.jgit.transport.ReceivePack.receive(ReceivePack.java:2200)
        at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:98)
        at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:109)
        at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:74)
        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:492)
        at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:703)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: com.google.gerrit.exceptions.StorageException: Cannot update gitlinks
        at com.google.gerrit.server.submit.SubmoduleOp.updateSuperProjects(SubmoduleOp.java:123)
        at com.google.gerrit.server.update.SuperprojectUpdateSubmissionListener.afterSubmission(SuperprojectUpdateSubmissionListener.java:78)
        at com.google.gerrit.server.update.SubmissionExecutor.lambda$afterExecutions$2(SubmissionExecutor.java:73)
        at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:423)
        at com.google.gerrit.server.update.SubmissionExecutor.afterExecutions(SubmissionExecutor.java:73)
        at com.google.gerrit.server.git.receive.ReceiveCommits.handleRegularCommands(ReceiveCommits.java:896)
        at com.google.gerrit.server.git.receive.ReceiveCommits.processCommandsUnsafe(ReceiveCommits.java:788)
        at com.google.gerrit.server.git.receive.ReceiveCommits.processCommands(ReceiveCommits.java:698)
        at com.google.gerrit.server.git.receive.AsyncReceiveCommits.lambda$preReceive$2(AsyncReceiveCommits.java:386)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at com.google.gerrit.server.util.RequestScopePropagator.lambda$cleanup$1(RequestScopePropagator.java:187)
        at com.google.gerrit.server.util.RequestScopePropagator.lambda$context$0(RequestScopePropagator.java:174)
        at com.google.gerrit.server.util.ThreadLocalRequestScopePropagator.lambda$wrapImpl$0(ThreadLocalRequestScopePropagator.java:45)
        at com.google.gerrit.server.util.RequestScopePropagator$1.call(RequestScopePropagator.java:85)
        at com.google.gerrit.server.util.RequestScopePropagator$2.run(RequestScopePropagator.java:116)
        ... 8 more
Caused by: com.google.gerrit.server.update.UpdateException: java.lang.NullPointerException
        at com.google.gerrit.server.update.BatchUpdate.executeUpdateRepo(BatchUpdate.java:556)
        at com.google.gerrit.server.update.BatchUpdates.execute(BatchUpdates.java:100)
        at com.google.gerrit.server.submit.SubmoduleOp.updateSuperProjects(SubmoduleOp.java:117)
        ... 23 more
Caused by: java.lang.NullPointerException
        at java.base/java.util.Objects.requireNonNull(Objects.java:209)
        at org.eclipse.jgit.lib.ObjectBuilder.setAuthor(ObjectBuilder.java:61)
        at org.eclipse.jgit.lib.CommitBuilder.setAuthor(CommitBuilder.java:100)
        at com.google.gerrit.server.submit.SubmoduleCommits.composeGitlinksCommit(SubmoduleCommits.java:167)
        at com.google.gerrit.server.submit.GitlinkOp.updateRepo(GitlinkOp.java:57)
        at com.google.gerrit.server.update.BatchUpdate.executeUpdateRepo(BatchUpdate.java:543)
        ... 25 more

The branch has been deleted on mediawiki/extensions/3D, Gerrit then tries to register that deletion in the super project (mediawiki/core) and fails when crafting the commit message since the new object is a null oid. We have to delete the branch from mediawiki/core first.

CodeReviewBot added a project: Patch-For-Review.Nov 25 2024, 1:55 PM

hashar opened https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/598

scap clean: first delete branch from superproject

Stashbot added a comment.Nov 25 2024, 2:52 PM

Mentioned in SAL (#wikimedia-releng) [2024-11-25T14:52:14Z] <hashar> Manually deleting old wmf branches before wmf/1.44.0-wmf.1 # T303828

CodeReviewBot added a comment.Nov 25 2024, 3:40 PM

hashar merged https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/598

scap clean: first delete branch from superproject

Maintenance_bot removed a project: Patch-For-Review.Nov 25 2024, 4:30 PM
Stashbot added a comment.Nov 25 2024, 5:00 PM

Mentioned in SAL (#wikimedia-releng) [2024-11-25T17:00:52Z] <hashar> Deleted wmf/1.44.0-wmf.2 branch using scap clean --delete-gerrit-branch 1.44.0-wmf.2 # T303828

Stashbot added a comment.Nov 25 2024, 5:01 PM

Mentioned in SAL (#wikimedia-releng) [2024-11-25T17:01:10Z] <hashar> Manually deleted wmf/1.44.0-wmf.1 branch from MediaWiki repositories # T303828

gerritbot added a comment.Nov 25 2024, 5:14 PM

Change #1097444 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] scap: delete wmf branches automatically

https://gerrit.wikimedia.org/r/1097444

gerritbot added a project: Patch-For-Review.Nov 25 2024, 5:14 PM
gerritbot added a comment.Nov 25 2024, 8:56 PM

Change #1097444 merged by RLazarus:

[operations/puppet@production] scap: delete wmf branches automatically

https://gerrit.wikimedia.org/r/1097444

Maintenance_bot removed a project: Patch-For-Review.Nov 25 2024, 9:30 PM
hashar closed this task as Resolved.Nov 26 2024, 8:12 AM
hashar claimed this task.

The train-clean timer passed during the night and sent us an email:

Systemd timer ran the following command:

    /usr/bin/scap clean --delete-gerrit-branch auto

Its return value was 0 and emitted the following output:

           ___ ____
         ⎛   ⎛ ,----
          \  //==--'
     _//|,.·//==--'    ____________________________
    _OO≣=-  ︶ ᴹw ⎞_§ ______  ___\ ___\ ,\__ \/ __ \
   (∞)_, )  (     |  ______/__  \/ /__ / /_/ / /_/ /
     ¨--¨|| |- (  / ______\____/ \___/ \__^_/  .__/
         ««_/  «_/ jgs/bd808                /_/

05:00:01 No eligible versions to remove.

It is thus working, it did nothing since yesterday I ran scap clean --delete-gerrit-branch 1.44.0-wmf.2.

I am claiming victory and all old wmf branches have been suppressed.

hashar mentioned this in T380841: Convert old release branches to tags.Nov 26 2024, 9:54 AM
hashar mentioned this in T375664: 1.44.0-wmf.5 deployment blockers.Nov 26 2024, 1:27 PM
hashar reopened this task as Open.Dec 3 2024, 7:34 AM

With 1.44.0-wmf.6, the timer ran scap clean --delete-gerrit-branch auto and that pruned the old branch:

mediawiki/core$ git remote prune origin
Pruning origin
URL: https://gerrit.wikimedia.org/r/mediawiki/core.git
 * [pruned] origin/wmf/1.44.0-wmf.3

But that did not clean the branch from the extensions and skins :-(

hashar added a comment.Dec 3 2024, 7:36 AM

The log is sent by email and shows lot of:

fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
05:00:01 Cleaning branch: 1.44.0-wmf.3
05:00:01 Clean files owned by www-data
05:00:05 Clean /srv/mediawiki-staging/wmf-config/ExtensionMessages-1.44.0-wmf.3.php
05:00:05 Started prune-git-branches
05:00:05 Deleting mediawiki/core branch
RSA host key for IP address '2620:0:861:2:208:80:154:151' not in list of known hosts.
remote: 
remote: Processing changes: refs: 1 (\)
remote: Processing changes: refs: 1 (\)
remote: Processing changes: refs: 1 (\)
remote: Processing changes: refs: 1, done            
05:00:06 Deleting branch from other projects
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
05:00:40 Finished prune-git-branches (duration: 00m 35s)
05:00:40 Clean /srv/mediawiki-staging/php-1.44.0-wmf.3
05:00:42 Clean /srv/patches/1.44.0-wmf.3
05:00:42 Started sync-masters
05:00:42 sync-masters:   0% (ok: 0; fail: 0; left: 1)                           
05:00:52 sync-masters: 100% (in-flight: 0; ok: 1; fail: 0; left: 0)             
05:00:52 Finished sync-masters (duration: 00m 10s)
05:00:52 Started sync-proxies
05:00:52 sync-proxies:   0% (ok: 0; fail: 0; left: 4)                           
05:01:05 sync-proxies: 100% (in-flight: 0; ok: 4; fail: 0; left: 0)             
05:01:05 Per-host sync duration: average 9.4s, median 9.0s
05:01:05 rsync transfer: average 419,089 bytes/host, total 1,676,356 bytes
05:01:05 Finished sync-proxies (duration: 00m 12s)
05:01:05 Started sync-apaches
05:01:05 sync-apaches:   0% (ok: 0; fail: 0; left: 29)                          
05:01:28 sync-apaches: 100% (in-flight: 0; ok: 29; fail: 0; left: 0)            
05:01:28 Per-host sync duration: average 9.9s, median 8.0s
05:01:28 rsync transfer: average 419,089 bytes/host, total 12,153,581 bytes
05:01:28 Finished sync-apaches (duration: 00m 22s)
05:01:28 Started clean-remote-caches
05:01:28 clean-remote-caches:   0% (ok: 0; fail: 0; left: 33)                   
05:01:29 clean-remote-caches: 100% (in-flight: 0; ok: 33; fail: 0; left: 0)     
05:01:29 Finished clean-remote-caches (duration: 00m 01s)
05:01:29 Pruned MediaWiki: 1.44.0-wmf.3 (duration: 01m 27s)
hashar added a comment.Dec 3 2024, 7:59 AM

The issue is for extensions and skins, the remote push url is the https one. git/curl would thus lookup for the username in a netrc which does not exist and fail. For mediawiki/core it works since we have the push url set:

$ git -C /srv/mediawiki-staging/php-1.44.0-wmf.6  config --show-origin --get remote.origin.pushurl
file:.git/config	ssh://gerrit.wikimedia.org:29418/mediawiki/core

Which is not the case for extensions/skins.

The reason it worked for me is that I have url.ssh://hashar@gerrit.wikimedia.org:29418.pushinsteadof=https://gerrit.wikimedia.org/r in my personal config and thus:

$ git -C /srv/mediawiki-staging/php-1.44.0-wmf.6/extensions/Flow remote -v
origin	https://gerrit.wikimedia.org/r/mediawiki/extensions/Flow (fetch)
origin	ssh://hashar@gerrit.wikimedia.org:29418/mediawiki/extensions/Flow (push)
        ^^^^^

But as another user:

$ sudo -u mwpresync git -C /srv/mediawiki-staging/php-1.44.0-wmf.6/extensions/Flow remote -v
origin	https://gerrit.wikimedia.org/r/mediawiki/extensions/Flow (fetch)
origin	https://gerrit.wikimedia.org/r/mediawiki/extensions/Flow (push)

For consistency with mediawiki/core we should set the pushUrl explicitly in the local configuration of each repositories.

hashar added a comment.Dec 3 2024, 8:05 AM

scap prep does set the push url for the repositories it is cloning (via _clone_or_update_repo).

Extensions and skins are cloned using git submodule update --init --recursive.

We need scap prep to set the remote url :)

CodeReviewBot added a project: Patch-For-Review.Dec 3 2024, 5:20 PM

hashar opened https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/607

scap prep: set pushInsteadOf in the submodules config

CodeReviewBot added a comment.Dec 3 2024, 8:18 PM

dancy merged https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/607

scap prep: set pushInsteadOf in the submodules config

Maintenance_bot removed a project: Patch-For-Review.Dec 3 2024, 8:30 PM
hashar added a comment.Dec 17 2024, 9:05 AM

The timer ran at 5:00 UTC and the received email had:

Systemd timer ran the following command:

    /usr/bin/scap clean --delete-gerrit-branch auto

Its return value was 0 and emitted the following output:
...
05:00:01 No eligible versions to remove.

I am now afraid I have might have broken. When I look at the deployment server, I'd assume 1.44.0-wmf.5 from Nov 27 21:44 to be removed?

$ /bin/ls -1d /srv/mediawiki-staging/php-*
/srv/mediawiki-staging/php-1.44.0-wmf.5
/srv/mediawiki-staging/php-1.44.0-wmf.6
/srv/mediawiki-staging/php-1.44.0-wmf.8

With /var/lib/scap/scap/bin/python3:

>>> import scap.tasks
>>> import scap.utils

>>> scap.utils.get_active_wikiversions('/srv/mediawiki-staging', 'production')
['1.44.0-wmf.6', '1.44.0-wmf.8']
>>> scap.tasks.get_wikiversions_ondisk('/srv/mediawiki-staging')
['1.44.0-wmf.5', '1.44.0-wmf.6', '1.44.0-wmf.8']

I note though that the train-presync timer has failed earlier but scap prep 1.44.0-wmf.8 has been completed. So I am a bit lost :/

hashar added a comment.Dec 17 2024, 9:16 AM

That comes from 899ca98e567cd4a9b4cca1aabc340561ff4e9f17

scap clean auto: Tweak old version selection logic

Clean._autoselect_versions_to_remove() now returns a list of old on-disk versions, excluding the most recent old version.

An old version is defined to be a version that precedes all active versions.

I have live debugged it on production, it finds 1.44.0-wmf.5, stops at wmf.6 which is the first live version encountered and thus drop wmf.5 from the list of branches to clean. So the branch will be cleaned when we have a new train (in January?)

hashar added a comment.Dec 27 2024, 10:02 AM

The script ran on Tuesday 24 Dec and failed to delete the branch from extensions and skins:

05:00:02 Cleaning branch: 1.44.0-wmf.5
...
05:00:07 Deleting branch from other projects
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address

Which means my earlier patch did not work:

dancy merged https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/607
scap prep: set pushInsteadOf in the submodules config

Because the old 1.44.0-wmf.5 repositories were not prepared with the version of scap including that change but that will work starting with 1.44.0-wmf.6:

$ git config --file /srv/mediawiki-staging/php-1.44.0-wmf.6/.git/modules/extensions/3D/config --get-regexp ^url
url.ssh://gerrit.wikimedia.org:29418.pushinsteadof https://gerrit.wikimedia.org/r

php-1.44.0-wm.5 (and earliers .4, .3) would have to be manually deleted.

hashar added a comment.Jan 3 2025, 1:22 PM

php-1.44.0-wm.5 (and earliers .4, .3) would have to be manually deleted.

I have done the deletions.

hashar added a comment.Jan 7 2025, 8:47 PM

Subject: [RelEng] PASS: train-clean

...
05:00:01 No eligible versions to remove.

Cause there was no train last week :] See you next week!

hashar added a comment.Jan 14 2025, 8:45 AM
Systemd timer ran the following command:

    /usr/bin/scap clean --delete-gerrit-branch auto

Its return value was 0 and emitted the following output:

           ___ ____
         ⎛   ⎛ ,----
          \  //==--'
     _//|,.·//==--'    ____________________________
    _OO≣=-  ︶ ᴹw ⎞_§ ______  ___\ ___\ ,\__ \/ __ \
   (∞)_, )  (     |  ______/__  \/ /__ / /_/ / /_/ /
     ¨--¨|| |- (  / ______\____/ \___/ \__^_/  .__/
         ««_/  «_/ jgs/bd808                /_/

05:00:01 Cleaning branch: 1.44.0-wmf.6
05:00:01 Clean files owned by www-data
05:00:05 Clean /srv/mediawiki-staging/wmf-config/ExtensionMessages-1.44.0-wmf.6.php
05:00:05 Started prune-git-branches
05:00:05 Deleting mediawiki/core branch
RSA host key for IP address '2620:0:861:2:208:80:154:151' not in list of known hosts.
remote: 
remote: Processing changes: refs: 1
remote: Processing changes: refs: 1, done            
05:00:07 Deleting branch from other projects
RSA host key for IP address '2620:0:861:2:208:80:154:151' not in list of known hosts.
remote: 
remote: Processing changes: refs: 1
remote: Processing changes: refs: 1, done            
RSA host key for IP address '2620:0:861:2:208:80:154:151' not in list of known hosts.
remote: 
remote: Processing changes: refs: 1
remote: Processing changes: refs: 1, done            
...
05:02:21 Finished prune-git-branches (duration: 02m 15s)
05:02:21 Clean /srv/mediawiki-staging/php-1.44.0-wmf.6
05:02:23 Clean /srv/patches/1.44.0-wmf.6
05:02:23 Started sync-masters
05:02:23 sync-masters:   0% (ok: 0; fail: 0; left: 1)                           
05:02:33 sync-masters: 100% (in-flight: 0; ok: 1; fail: 0; left: 0)             
05:02:33 Finished sync-masters (duration: 00m 10s)
05:02:33 Started sync-proxies
05:02:33 sync-proxies:   0% (ok: 0; fail: 0; left: 4)                           
05:02:46 sync-proxies: 100% (in-flight: 0; ok: 4; fail: 0; left: 0)             
05:02:46 Per-host sync duration: average 9.4s, median 9.1s
05:02:46 rsync transfer: average 416,461 bytes/host, total 1,665,844 bytes
05:02:46 Finished sync-proxies (duration: 00m 12s)
05:02:46 Started sync-apaches
05:02:46 sync-apaches:   0% (ok: 0; fail: 0; left: 27)                          
05:03:07 sync-apaches: 100% (in-flight: 0; ok: 27; fail: 0; left: 0)            
05:03:07 Per-host sync duration: average 10.2s, median 8.0s
05:03:07 rsync transfer: average 416,461 bytes/host, total 11,244,447 bytes
05:03:07 Finished sync-apaches (duration: 00m 21s)
05:03:07 Started clean-remote-caches
05:03:07 clean-remote-caches:   0% (ok: 0; fail: 0; left: 31)                   
05:03:08 clean-remote-caches: 100% (in-flight: 0; ok: 31; fail: 0; left: 0)     
05:03:08 Finished clean-remote-caches (duration: 00m 01s)
05:03:08 Pruned MediaWiki: 1.44.0-wmf.6 (duration: 03m 06s)

And locally:

$ git -C mediawiki/core remote prune origin
Pruning origin
URL: https://gerrit.wikimedia.org/r/mediawiki/core.git
 * [pruned] origin/wmf/1.44.0-wmf.6
$ git -C mediawiki/extensions/Flow remote prune origin
Pruning origin
URL: https://gerrit.wikimedia.org/r/mediawiki/extensions/Flow
 * [pruned] origin/wmf/1.44.0-wmf.6

So that is essentially a success. Things I would like to improve:

  • get rid of the banner (scap clean --no-logo)
  • fix the spam of RSA host key for IP address '2620:0:861:2:208:80:154:151' not in list of known hosts.
  • remove the git remote messages (remote: Processing changes: refs: 1, done) (git push --quiet)
gerritbot added a comment.Jan 14 2025, 9:09 AM

Change #1111163 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] gerrit: restore IP addresses in ssh_known_hosts

https://gerrit.wikimedia.org/r/1111163

gerritbot added a project: Patch-For-Review.Jan 14 2025, 9:09 AM
gerritbot added a comment.Jan 14 2025, 2:12 PM

Change #1111233 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] scap: do not show logo when cleaning old versions

https://gerrit.wikimedia.org/r/1111233

hashar added a comment.Jan 14 2025, 2:21 PM

For:

remote: 
remote: Processing changes: refs: 1
remote: Processing changes: refs: 1, done

That is emitted by Gerrit and git push relays it to stderr which scap streams back to the user. scap invokes the commands with subprocess.call and subprocess.check_output with neither stdout nor stderr being set). We'd need to capture them and have them logged/emitted only on failure?

thcipriani edited projects, added Release-Engineering-Team (Doing 😎); removed Release-Engineering-Team (Priority Backlog 📥).Jan 14 2025, 2:50 PM
gerritbot added a comment.Jan 14 2025, 3:21 PM

Change #1111233 merged by Dzahn:

[operations/puppet@production] scap: do not show logo when cleaning old versions

https://gerrit.wikimedia.org/r/1111233

gerritbot added a comment.Jan 15 2025, 6:40 PM

Change #1111163 merged by Dzahn:

[operations/puppet@production] gerrit: restore IP addresses in ssh_known_hosts

https://gerrit.wikimedia.org/r/1111163

Maintenance_bot removed a project: Patch-For-Review.Jan 15 2025, 7:30 PM
CodeReviewBot added a project: Patch-For-Review.Feb 25 2025, 10:35 AM

hashar opened https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/669

scap clean: hide remote line on pushing deletes

Aklapper added a parent task: T387539: Deleting branches results in production error stacktraces that no longer works.Feb 28 2025, 12:20 PM
CodeReviewBot added a comment.Mar 1 2025, 9:27 AM

hashar merged https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/669

scap clean: hide remote line on pushing deletes

Maintenance_bot removed a project: Patch-For-Review.Mar 1 2025, 9:30 AM
Stashbot added a comment.Mar 3 2025, 10:26 AM

Mentioned in SAL (#wikimedia-operations) [2025-03-03T10:26:42Z] <hashar> Upgraded scap to 4.139.0 # T303828

hashar closed this task as Resolved.Mar 4 2025, 7:15 AM

The spammy processing is gone (except for the mediawiki/core deletion). I claim a success

Systemd timer ran the following command:

    /usr/bin/scap clean --delete-gerrit-branch --no-logo auto

Its return value was 0 and emitted the following output:

05:00:01 Cleaning branch: 1.44.0-wmf.16
05:00:01 Clean files owned by www-data
05:00:03 Clean /srv/mediawiki-staging/wmf-config/ExtensionMessages-1.44.0-wmf.16.php
05:00:03 Started prune-git-branches
05:00:03 Deleting mediawiki/core branch
remote: 
remote: Processing changes: refs: 1
remote: Processing changes: refs: 1, done            
05:00:04 Deleting branch from other projects
05:02:17 Finished prune-git-branches (duration: 02m 14s)
05:02:17 Clean /srv/mediawiki-staging/php-1.44.0-wmf.16
05:02:19 Clean /srv/patches/1.44.0-wmf.16
05:02:19 Started sync-masters
05:02:19 sync-masters:   0% (ok: 0; fail: 0; left: 1)                           
05:02:29 sync-masters: 100% (in-flight: 0; ok: 1; fail: 0; left: 0)             
05:02:29 Finished sync-masters (duration: 00m 09s)
05:02:29 Started sync-apaches
05:02:29 sync-apaches:   0% (ok: 0; fail: 0; left: 24)                          
05:05:12 sync-apaches:  20% (in-flight: 19; ok: 5; fail: 0; left: 0)            
05:05:43 sync-apaches:  66% (in-flight: 8; ok: 16; fail: 0; left: 0)            
05:05:56 sync-apaches: 100% (in-flight: 0; ok: 24; fail: 0; left: 0)            
05:05:56 Per-host sync duration: average 157.8s, median 177.2s
05:05:56 rsync transfer: average 3,629,482,836 bytes/host, total 87,107,588,068 bytes
05:05:56 Finished sync-apaches (duration: 03m 27s)
05:05:56 Started clean-remote-caches
05:05:56 clean-remote-caches:   0% (ok: 0; fail: 0; left: 24)                   
05:05:57 clean-remote-caches: 100% (in-flight: 0; ok: 24; fail: 0; left: 0)     
05:05:57 Finished clean-remote-caches (duration: 00m 01s)
05:05:57 Pruned MediaWiki: 1.44.0-wmf.16 (duration: 05m 56s)
Physikerwelt mentioned this in T389998: Allow control over which extra extensions are installed (Math REL1_43 jobs exceed 60min timeout).Mar 27 2025, 1:52 PM
hashar mentioned this in T403854: Enable creation of a patchdemo with all repos rolled back to a specific custom moment in time.Sep 17 2025, 5:04 PM
hashar mentioned this in T413966: Gerrit replication does not replicate tags deletion to GitHub.Jan 7 2026, 3:29 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits