Page MenuHomePhabricator

https://github.com/wikimedia/mediawiki/ release tags vanished
Closed, ResolvedPublic

Description

I don't know whether it was done on purpose or not but suddenly all our tests are failing because each single release tag for https://github.com/wikimedia/mediawiki/releases did vanished [0] which means we can't download any specific release from github (as it is faster than any WMF node) because of:

wget https://github.com/wikimedia/mediawiki/archive/1.23.5.tar.gz -O 1.23.5.tar.gz
--2015-05-26 09:44:28--  https://github.com/wikimedia/mediawiki/archive/1.23.5.tar.gz
Resolving github.com (github.com)... 192.30.252.130
Connecting to github.com (github.com)|192.30.252.130|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/wikimedia/mediawiki/tar.gz/1.23.5 [following]
--2015-05-26 09:44:28--  https://codeload.github.com/wikimedia/mediawiki/tar.gz/1.23.5
Resolving codeload.github.com (codeload.github.com)... 192.30.252.146
Connecting to codeload.github.com (codeload.github.com)|192.30.252.146|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2015-05-26 09:44:28 ERROR 404: Not Found.

[0] https://cloud.githubusercontent.com/assets/1245473/7809880/cfdc973e-039d-11e5-8a06-b037ec7a7094.png

Event Timeline

mwjames created this task.May 26 2015, 10:07 AM
mwjames assigned this task to demon.
mwjames raised the priority of this task from to Needs Triage.
mwjames updated the task description. (Show Details)
mwjames added a subscriber: mwjames.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 26 2015, 10:07 AM
mwjames updated the task description. (Show Details)May 26 2015, 10:08 AM
mwjames set Security to None.

This breaks the CI for a bunch of extensions, including ones run by WMF

Florian added a subscriber: Florian.EditedMay 26 2015, 1:34 PM

Maybe it's related to this bug, so i will not open a new task, until someone says it's not related :P

The same case for branches: On github.com/wikimedia/mediawiki we have only the master branch, all other branches (all REL1_** and all 1.26wmf branches) are missing, too, while they exists in gerrit and git.wikimedia.org

demon removed demon as the assignee of this task.May 27 2015, 6:38 PM
demon added a subscriber: demon.

Unassigning from myself, I'm on vacation until the end of the month.

demon triaged this task as Unbreak Now! priority.May 27 2015, 6:40 PM

Also, this needs fixing ASAP so moving to UBN.

Seems the replication from Gerrit to github broke somehow. The replication is defined in operations/puppet.git in manifests/role/gerrit.git which hasn't been changed recently.

We sync:

refs/heads/*:refs/heads/*
refs/tags/*:refs/tags/*

I have no idea how to force the replication / push missing references. There is some doc though at https://gerrit.wikimedia.org/r/plugins/replication/Documentation/index.html

Change 214254 had a related patch set uploaded (by Foxtrott):
Work around T100409 (mediawiki release tags vanished)

https://gerrit.wikimedia.org/r/214254

Change 214254 merged by jenkins-bot:
Work around T100409 (mediawiki release tags vanished)

https://gerrit.wikimedia.org/r/214254

So in short, one need to figure out the credentials to push from Gerrit reference repositories to the github mirror machine. That might be on wikitech.

The access to the server hosting Gerrit (antimony) are rather limited. Gotta extend them to more people which is T100565: Add all Release-Engineering team as Gerrit admins.

QChris added a subscriber: QChris.May 27 2015, 9:55 PM

I forced replication of mediawiki/core, and that seems to have done the trick (see below).
Seeing if I can find anything in the logs.

_________________________________________________________________
christian@spencer // jobs: 0 // time: 23:45:24 // exit code: 0
cwd: ~
ssh gerrit.wikimedia.org replication start mediawiki/core --wait
Replicate mediawiki/core to gallium.wikimedia.org, Succeeded!
Replicate mediawiki/core to lanthanum.eqiad.wmnet, Succeeded!
Replicate mediawiki/core to antimony.wikimedia.org, Succeeded!
Replicate mediawiki/core to github.com, Succeeded!
----------------------------------------------
Replication completed successfully!

_________________________________________________________________
christian@spencer // jobs: 0 // time: 23:50:16 // exit code: 0
cwd: ~
LC_ALL=C wget https://github.com/wikimedia/mediawiki/archive/1.23.5.tar.gz -O 1.23.5.tar.gz
--2015-05-27 23:50:42--  https://github.com/wikimedia/mediawiki/archive/1.23.5.tar.gz
Resolving github.com... 192.30.252.130
Connecting to github.com|192.30.252.130|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/wikimedia/mediawiki/tar.gz/1.23.5 [following]
--2015-05-27 23:50:43--  https://codeload.github.com/wikimedia/mediawiki/tar.gz/1.23.5
Resolving codeload.github.com... 192.30.252.146
Connecting to codeload.github.com|192.30.252.146|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 18254766 (17M) [application/x-gzip]
Saving to: `1.23.5.tar.gz'

100%[=====================================================================================================================>] 18,254,766   912K/s   in 20s     

2015-05-27 23:51:04 (897 KB/s) - `1.23.5.tar.gz' saved [18254766/18254766]

This seems to have only recreated the WMF specific tags (e.g. wmf/1.25wmf1). General version tags (e.g. 1.24.2) seem to be missing still.

This seems to have only recreated the WMF specific tags (e.g. wmf/1.25wmf1). General version tags (e.g. 1.24.2) seem to be missing still.

Github shows a 1.24.2 tag for me:

_________________________________________________________________
christian@spencer // jobs: 0 // time: 00:47:07 // exit code: 0
cwd: ~/sources/Wikimedia/git/mediawiki/core
git ls-remote https://github.com/wikimedia/mediawiki.git refs/tags/1.24.2
13efd2a4b47abd5ed0290662597ee8eb8d692ffc        refs/tags/1.24.2

Also https://github.com/wikimedia/mediawiki/releases/tag/1.24.2 is working for me and showing the links to the artifacts.

Does the issue still persist for you?

QChris closed this task as Resolved.May 27 2015, 10:55 PM
QChris claimed this task.

Seeing if I can find anything in the logs.

It seems the root cause was a forced replication of all gerrit repositories, which breaks renamed github's repositories.
Added a warning against --all to wikitech: https://wikitech.wikimedia.org/w/index.php?title=Gerrit&diff=160816&oldid=153784

Also https://github.com/wikimedia/mediawiki/releases/tag/1.24.2 is working for me and showing the links to the artifacts.
Does the issue still persist for you?

I can access the tags by manipulating the URL, so downloads should work again (i.e. original issue solved), but it does not show up in the selector on the github webpage. Probably more a problem of github, not of wikimedia.

[...], but it does not show up in the selector on the github webpage. Probably more a problem of github, not of wikimedia.

Thanks for the screenshot.
I can reproduce what you describe.
For me the box on github (currently) shows

  • 24 wmf/1.25* tags
  • 21 wmf/1.24* tags (no wmf/1.24wmf13, but that one does not exist in gerrit either)
  • 22 wmf/1.23* tags
  • 22 wmf/1.22* tags
  • 11 wmf/1.21* tags

That sums up to exactly 100 shown tags, which looks suspiciously like
an github UI issue. Evenmore so, because (as you wrote) the tags that
are not listed can be reached by mangling URLs. So those tags exist on
the github repo, but simply are not shown in the box on github.

I guess you could file a bug with them, but there is little we can
do on the gerrit side to fix it.

Thanks for fixing this issue!

greg moved this task from INBOX to Done on the Release-Engineering-Team board.Jun 2 2015, 3:53 PM

Change 264462 had a related patch set uploaded (by Foxtrott):
Revert "Work around T100409 (mediawiki release tags vanished)"

https://gerrit.wikimedia.org/r/264462

Change 264462 merged by Foxtrott:
Revert "Work around T100409 (mediawiki release tags vanished)"

https://gerrit.wikimedia.org/r/264462