Page MenuHomePhabricator

Gerrit replication to GitHub is broken
Closed, ResolvedPublic

Description

$ ssh -p 29418 gerrit.wikimedia.org gerrit show-queue --by-queue --wide
Task     State        StartTime         Command
------------------------------------------------------------------------------

Queue: ReplicateTo-github
3f5a6c70              09:16:13.802      (retry 41) [8db15ba9] push git@github.com:wikimedia/mediawiki-extensions-BlueSpiceAbout
9f6a7805 09:17:14.441 09:16:14.440      (retry 560) [61d219dc] push git@github.com:wikimedia/wikimedia-fundraising-SmashPig
7f02245a 09:17:14.715 09:16:14.714      (retry 43) [36a4371a] push git@github.com:wikimedia/mediawiki-extensions-Thanks
1f31a850 09:17:14.981 09:16:14.980      (retry 481) [9550f8be] push git@github.com:wikimedia/wikimedia-fundraising-crm-vendor
ffe054c9 09:17:15.250 09:16:15.250      (retry 42) [018f749a] push git@github.com:wikimedia/mediawiki-extensions-BlueSpiceChecklist
bffadc7d 09:17:15.543 09:16:15.542      (retry 41) [6d618725] push git@github.com:wikimedia/mediawiki-extensions-BlueSpiceUserManager
1fc3e83c 09:17:15.823 09:16:15.822      (retry 99) [1f4c4828] push git@github.com:wikimedia/pywikibot-core
dfb790e9 09:17:16.456 09:16:16.456      (retry 40) [5d6eba7c] push git@github.com:wikimedia/mediawiki-extensions-BlueSpicePageAssignments
9fa91894 09:17:16.722 09:16:16.721      (retry 479) [61a4a7d1] push git@github.com:wikimedia/mediawiki-extensions-GrowthExperiments
7f8d443d 09:17:17.006 09:16:17.005      (retry 479) [e16f3777] push git@github.com:wikimedia/mediawiki-extensions
e479a916 09:17:17.302 09:16:17.301      (retry 418) [2bf1e0d6] push git@github.com:wikimedia/mediawiki-extensions-MachineVision
049cfd7e 09:17:17.587 09:16:17.586      (retry 41) [4db123f4] push git@github.com:wikimedia/mediawiki-extensions-BlueSpiceWhoIsOnline
...

Event Timeline

hashar created this task.Dec 10 2019, 9:18 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 10 2019, 9:18 AM
hashar triaged this task as Unbreak Now! priority.Dec 10 2019, 9:19 AM
Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptDec 10 2019, 9:19 AM

From Gerrit replication logs on gerrit1001.eqiad.wmnet in /var/log/gerrit/replication* logs, the first occurences:

[2019-12-09 23:45:43,026] [defcff46] Cannot replicate to git@github.com:wikimedia/operations-puppet
[2019-12-09 23:46:43,694] [defcff46] Cannot replicate to git@github.com:wikimedia/operations-puppet
[2019-12-09 23:47:44,587] [defcff46] Cannot replicate to git@github.com:wikimedia/operations-puppet
[2019-12-09 23:48:42,932] [61d219dc] Cannot replicate to git@github.com:wikimedia/wikimedia-fundraising-SmashPig
...

The top messages:

    569 wikimedia/operations-puppet
    567 wikimedia/wikimedia-fundraising-SmashPig
    494 wikimedia/oojs-ui
    488 wikimedia/wikimedia-fundraising-crm-vendor
    486 wikimedia/mediawiki-extensions-GrowthExperiments
    486 wikimedia/mediawiki-extensions
    457 wikimedia/wikimedia-fundraising-crm
    425 wikimedia/mediawiki-extensions-MachineVision
    339 wikimedia/labs-tools-facebook-messenger-chatbot
    296 wikimedia/mediawiki-core
    106 wikimedia/pywikibot-core
     97 wikimedia/mediawiki-skins-webplatform
     97 wikimedia/mediawiki-skins-Timeless
     97 wikimedia/mediawiki-skins-MinervaNeue
     97 wikimedia/mediawiki-skins
     95 wikimedia/labs-tools-watch-translations
     86 wikimedia/mediawiki-extensions-Cite
     79 wikimedia/translatewiki
     63 wikimedia/operations-deployment-charts
     56 wikimedia/mediawiki-extensions-Jade
<rest are bunch of mw extensions>

I guess it is safe to say that the GitHub replication is entirely broken.

hashar added a subscriber: greg.Dec 10 2019, 9:34 AM

The first error was on 2019-12-09 23:45:43 UTC.

From https://github.com/organizations/wikimedia/settings/audit-log that would match:

ActionActorOrgRepoTSDate
org.enable_oauth_app_restrictions@gregwikimedia-15759347846572019-12-09 23:39:44 UTC

Looking at recent activity in Wikimedia-GitHub that would match {T234991}.

hashar closed this task as Resolved.Dec 10 2019, 9:48 AM
hashar claimed this task.

Fixed by removing OAuthApp restrictions in Github: T234991#5727344