LibUp hasn't successfully run on AutoCreateCategoryPages for 7 weeks
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Umherirrender
	Feb 20 2021, 9:57 PM

Description

When looking at https://libraryupgrader2.wmcloud.org/r/mediawiki/extensions/AutoCreateCategoryPages?branch=master

$ date
Tue Dec 29 03:24:45 UTC 2020

$ git clone file:///srv/git/mediawiki-extensions-AutoCreateCategoryPages.git repo --depth=1 -b master
Cloning into 'repo'...

$ git config user.name libraryupgrader

$ git config user.email tools.libraryupgrader@tools.wmflabs.org

$ git submodule update --init

$ grr init
Installed commit-msg hook.

$ git show-ref refs/heads/master
baeb1cf9c7d37946aa76d2c6be6fdd27f4fd231e refs/heads/master

It shows a date from before 7 weeks
From the log it is not possible to see why it skips this repo.

Is there another log to look at?

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Legoktm	T275292 LibUp hasn't successfully run on AutoCreateCategoryPages for 7 weeks
		Resolved		hashar	T275946 Can't delete weird ref using git in Gerrit

Event Timeline

Umherirrender created this task.Feb 20 2021, 9:57 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 20 2021, 9:57 PM

This means something in the logging code crashed...let me run it manually.

RhinosF1 subscribed.Feb 26 2021, 10:26 PM

Feb 26 22:26:23 upgrader-06 libup-celery[619]: [2021-02-26 22:26:23,924: INFO/MainProcess] Received task: libup.tasks.run_check[cc77926a-1df6-4fc7-be08-cbb74645e355]
Feb 26 22:26:24 upgrader-06 libup-celery[619]: From https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/AutoCreateCategoryPages
Feb 26 22:26:24 upgrader-06 libup-celery[619]:  ! [rejected]        refs/master -> master  (non-fast-forward)
Feb 26 22:26:24 upgrader-06 libup-celery[619]: [2021-02-26 22:26:24,454: ERROR/ForkPoolWorker-1] Task libup.tasks.run_check[cc77926a-1df6-4fc7-be08-cbb74645e355] raised unexpected: CalledProcessError(1, ['git', 'fetch', 'origin', 'master:
Feb 26 22:26:24 upgrader-06 libup-celery[619]: Traceback (most recent call last):
Feb 26 22:26:24 upgrader-06 libup-celery[619]:   File "/srv/libraryupgrader/venv/lib/python3.7/site-packages/celery/app/trace.py", line 405, in trace_task
Feb 26 22:26:24 upgrader-06 libup-celery[619]:     R = retval = fun(*args, **kwargs)
Feb 26 22:26:24 upgrader-06 libup-celery[619]:   File "/srv/libraryupgrader/venv/lib/python3.7/site-packages/celery/app/trace.py", line 697, in __protected_call__
Feb 26 22:26:24 upgrader-06 libup-celery[619]:     return self.run(*args, **kwargs)
Feb 26 22:26:24 upgrader-06 libup-celery[619]:   File "/srv/libraryupgrader/libup/tasks.py", line 40, in run_check
Feb 26 22:26:24 upgrader-06 libup-celery[619]:     gerrit.ensure_clone(repo.name, repo.branch)
Feb 26 22:26:24 upgrader-06 libup-celery[619]:   File "/srv/libraryupgrader/libup/gerrit.py", line 112, in ensure_clone
Feb 26 22:26:24 upgrader-06 libup-celery[619]:     subprocess.check_call(['git', 'fetch', 'origin', f'{branch}:{branch}'], cwd=path)
Feb 26 22:26:24 upgrader-06 libup-celery[619]:   File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
Feb 26 22:26:24 upgrader-06 libup-celery[619]:     raise CalledProcessError(retcode, cmd)
Feb 26 22:26:24 upgrader-06 libup-celery[619]: subprocess.CalledProcessError: Command '['git', 'fetch', 'origin', 'master:master']' returned non-zero exit status 1.

Pretty sure there's something wrong with this repo: T255802: PHPCS dashboard shows that AutoCreateCategoryPages extension haven't phpcs but it have

It was possible to upload a patch set to update the version of codesniffer - https://gerrit.wikimedia.org/r/c/mediawiki/extensions/AutoCreateCategoryPages/+/667283

I ran git ls-remote on AutoCreateCategoryPages:

28db01ebf85c5f94aad8bdb2aa66c8cad2461745	HEAD
...
28db01ebf85c5f94aad8bdb2aa66c8cad2461745	refs/heads/master
2a5f5e7ce5292d9d6d033311a025a8be3f52b069	refs/master <-- what is this???

In comparison MassMessage has no refs/master. I think that bad ref is what's causing the issues.

Legoktm renamed this task from Why libup skips some repos without logging the reason? to LibUp hasn't successfully run on AutoCreateCategoryPages for 7 weeks.Feb 27 2021, 12:15 AM

https://gerrit.wikimedia.org/r/admin/repos/mediawiki/extensions/AutoCreateCategoryPages,branches

REL1_31	5c1543cc20fe078a44ccaba0712553514622abc6	
REL1_32	2a5f5e7ce5292d9d6d033311a025a8be3f52b069	
REL1_33	0b94a462094ea913309e315f75f98ecfe63df1b1	
REL1_34	ad9d9ab9ece4f90397f794fb9baab71be5b5c3a0	
REL1_35	2a5f5e7ce5292d9d6d033311a025a8be3f52b069	
master	28db01ebf85c5f94aad8bdb2aa66c8cad2461745

The hash is showing up twice

The right hash for REL1_35 seems to be https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/AutoCreateCategoryPages/+/3aa113c2016eebae632fd95bff3c82efb703e8cd

As reference for the deletion of the ref: T275946

Legoktm added a subtask: T275946: Can't delete weird ref using git in Gerrit.Feb 28 2021, 12:23 AM

Suggestion from yesterday on how to harden libup against this in the future:

[00:20:55] <thcipriani>	 I guess what I mean is git fetch origin refs/heads/master:refs/remotes/origin/master (or whatever your local patch is) would be the only way to prevent this from happening

hashar closed subtask T275946: Can't delete weird ref using git in Gerrit as Resolved.Mar 4 2021, 4:12 PM

I had to manually delete the clone on the libup server so it did a fresh checkout and now we have a successful run: https://libraryupgrader2.wmcloud.org/logs2/137811 \o/

Umherirrender mentioned this in T279446: LibUp hasn't successfully run on UniversalLanguageSelector for 3 weeks.Apr 6 2021, 3:46 PM

In T275292#6883979, @Legoktm wrote:

I had to manually delete the clone on the libup server so it did a fresh checkout and now we have a successful run: https://libraryupgrader2.wmcloud.org/logs2/137811 \o/

Unfortunately that was the last run. It seems the extension is skipped on the daily run.
Please have another look into the logs. Thanks.

https://libraryupgrader2.wmcloud.org/r/mediawiki/extensions/AutoCreateCategoryPages?branch=master

Apr 09 05:43:01 upgrader-06 libup-celery[619]: From https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/AutoCreateCategoryPages
Apr 09 05:43:01 upgrader-06 libup-celery[619]:  ! [rejected]        refs/master -> master  (non-fast-forward)

root@upgrader-06:/srv/git# rm -rf mediawiki-extensions-AutoCreateCategoryPages.git/

aaand now https://libraryupgrader2.wmcloud.org/logs2/222722

but when you run it again (so it pulls rather than clones):

Apr 09 05:46:20 upgrader-06 libup-celery[619]: From https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/AutoCreateCategoryPages
Apr 09 05:46:20 upgrader-06 libup-celery[619]:  ! [rejected]        refs/master -> master  (non-fast-forward)

Maybe the repo is still broken?? It's calling git fetch origin master:master on a bare clone.

It is possible that the delete of the branch in T275946 does not get published to the replica and pull from there still fails?

Legoktm reopened subtask T275946: Can't delete weird ref using git in Gerrit as Open.Apr 14 2021, 1:19 AM

Legoktm mentioned this in T275946: Can't delete weird ref using git in Gerrit.

hashar closed subtask T275946: Can't delete weird ref using git in Gerrit as Resolved.Apr 14 2021, 2:01 PM

legoktm@upgrader-06:/srv/git$ sudo rm -rf mediawiki-extensions-AutoCreateCategoryPages.git/

And now we wait :)

There is a run today - https://libraryupgrader2.wmcloud.org/logs2/228424

Thanks for the analyze and work on it.

LibUp hasn't successfully run on AutoCreateCategoryPages for 7 weeksClosed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

LibUp hasn't successfully run on AutoCreateCategoryPages for 7 weeks
Closed, ResolvedPublic
Actions

Related Objects
Search...