https://gerrit.wikimedia.org/r/scoring/ores/editquality
Should be mirrored from: https://phabricator.wikimedia.org/source/editquality
See https://phabricator.wikimedia.org/source/editquality/uri/view/19762/
It looks like the last commit to be mirrored was on April 24th. There should be commits as recent as June 4th.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T227094 Update RC Filters for new ORES capacities (July, 2019) | |||
| Resolved | SBisson | T225561 Update ORES thresholds for nlwiki | |||
| Open | None | T223273 Update srwiki thresholds for goodfaith model | |||
| Resolved | SBisson | T225562 Deploy ORES filters for zhwiki | |||
| Open | None | T225563 Deploy ORES filters for jawiki | |||
| Resolved | Halfak | T224484 ORES deployment: Early June | |||
| Resolved | elukey | T224996 Gerrit repo scoring/ores/editquality not mirroring |
I think that @mmodell helped us with a credential issue that manifested in a similar way back in April.
Worked on this today with Tyler in our pairing session.
@Halfak: This should be resolved now. Can you confirm?
@thcipriani: Thanks for your help.
I've just tried to pull new changes from https://gerrit.wikimedia.org/r/scoring/ores/editquality and I'm still stuck on the April 23rd commit.
commit 373b4c143ec4591eaed474a57ad06333af19fc16
Merge: c8f1aa8 40d32e0
Author: Aaron Halfaker <aaron.halfaker@gmail.com>
Date: Tue Apr 23 10:00:17 2019 -0500
Merge pull request #193 from wikimedia/fiwiki_v2When I try to directly clone https://phabricator.wikimedia.org/source/editquality, I get:
Cloning into 'test-eq2'... remote: Counting objects: 4495, done. remote: Compressing objects: 100% (3280/3280), done. remote: Total 4495 (delta 2407), reused 2883 (delta 1162) Receiving objects: 100% (4495/4495), 4.04 MiB | 0 bytes/s, done. Resolving deltas: 100% (2407/2407), done. Checking connectivity... done. Downloading models/arwiki.damaging.gradient_boosting.model (340 KB) Error downloading object: models/arwiki.damaging.gradient_boosting.model (e57c41a): Smudge error: Error downloading models/arwiki.damaging.gradient_boosting.model (e57c41a95a60e05ec821ebbf54ca283f25569c601b62bc368abea2d1dfe4ca98): batch response: Authorization error: https://phabricator.wikimedia.org/source/editquality.git/info/lfs/objects/batch Check that you have proper access to the repository Errors logged to /home/halfak/projects/test-eq/test-eq2/.git/lfs/logs/20190606T101215.719509111.log Use `git lfs logs last` to view the log. error: external filter git-lfs smudge -- %f failed 2 error: external filter git-lfs smudge -- %f failed fatal: models/arwiki.damaging.gradient_boosting.model: smudge filter lfs failed warning: Clone succeeded, but checkout failed. You can inspect what was checked out with 'git status' and retry the checkout with 'git checkout -f HEAD'
It looks like that might be related.
Looks like master is showing a newer version:
even with:
https://github.com/wikimedia/editquality/commit/83f186b6ae3be6334668cfa64516cd5ad1a420e8
Was that a manual update?
Right. I did a manual update with @20after4's help to get unblocked for our upcoming deployment. I actually just did another manual push this morning.
gotcha.
My theory here is that there is some ssh timeout on the phab side for pushing to gerrit (or maybe some error on the phab side(?)). I don't see anything in the gerrit error logs related to phab/editquality.
I do see there are push attempts in the sshd_log:
[2019-06-11 00:05:53,217 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 1ms 714885ms 0 git/2.11.0 [2019-06-11 00:29:56,860 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 688116ms 0 git/2.11.0 [2019-06-11 00:53:34,070 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 1ms 662753ms 0 git/2.11.0 [2019-06-11 01:17:41,123 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 689700ms 0 git/2.11.0 [2019-06-11 01:40:55,275 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 649596ms 0 git/2.11.0 [2019-06-11 02:05:17,911 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 681857ms 0 git/2.11.0 [2019-06-11 02:29:32,952 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 668541ms 0 git/2.11.0 [2019-06-11 02:53:15,044 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 643558ms 0 git/2.11.0 [2019-06-11 03:17:07,385 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 649665ms 0 git/2.11.0 [2019-06-11 03:40:22,483 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 630413ms 0 git/2.11.0 [2019-06-11 04:03:47,598 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 628741ms 0 git/2.11.0 [2019-06-11 04:26:11,145 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 595719ms 0 git/2.11.0 [2019-06-11 04:48:58,620 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 626444ms 0 git/2.11.0 [2019-06-11 05:26:09,319 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 603253ms 0 git/2.11.0 [2019-06-11 06:03:28,095 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 616873ms 0 git/2.11.0 [2019-06-11 06:41:04,985 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 3ms 616671ms 0 git/2.11.0 [2019-06-11 07:18:27,634 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 3ms 605659ms 0 git/2.11.0 [2019-06-11 07:56:53,115 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 653584ms 0 git/2.11.0 [2019-06-11 08:36:29,855 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 641999ms 0 git/2.11.0 [2019-06-11 09:14:54,715 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 623062ms 0 git/2.11.0 [2019-06-11 09:53:41,939 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 624528ms 0 git/2.11.0 [2019-06-11 10:32:31,111 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 1ms 653837ms 0 git/2.11.0 [2019-06-11 11:12:13,556 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 672296ms 0 git/2.11.0 [2019-06-11 11:53:09,105 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 3ms 691439ms 0 git/2.11.0 [2019-06-11 12:33:31,065 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 705240ms 0 git/2.11.0 [2019-06-11 13:14:04,107 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 706536ms 0 git/2.11.0 [2019-06-11 13:56:27,538 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 766926ms 0 git/2.11.0 [2019-06-11 14:37:24,241 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 1ms 701814ms 0 git/2.11.0 [2019-06-11 15:19:12,353 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 768491ms 0 git/2.11.0 [2019-06-11 16:02:47,257 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 807261ms 0 git/2.11.0 [2019-06-11 16:44:30,999 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 688184ms 0 git/2.11.0 [2019-06-11 17:26:05,610 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 690340ms 0 git/2.11.0 [2019-06-11 18:08:48,914 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 1ms 656581ms 0 git/2.11.0 [2019-06-11 18:49:54,052 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 1ms 656431ms 0 git/2.11.0
We could create an HTTPs user that isn't affected by T225308: Users with a different name in the cn field compared to uid field cannot use http auth (unlike phab/phabricator) to try to push of HTTPs rather than SSH if we suspect it is an SSH timeout somewhere along the way causing the problem.
Hi everybody,
I found this task while investigating on the gerrit editquality repository, since it seems not being updated and we are trying to deploy new code. On phab1001's daemon.log I see a lot of Exceptions occurred while mirroring the "editquality" repository. but I don't get what it is failing.
Edit: I was grepping for editquality and I missed the whole stacktrace:
Pushing to ssh://********@gerrit.wikimedia.org:29418/scoring/ores/editquality
Warning: Permanently added '[gerrit.wikimedia.org]:29418,[2620:0:861:2:208:80:154:137]:29418' (RSA) to the list of known hosts.
phabricator@gerrit.wikimedia.org: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryMirrorEngine.php:49]
arcanist(), ...FWIW, I can confirm from the Gerrit side that the project config and group membership stuff hasn't changed. I don't see any ssh pushes by the phabricator user in the logs on Gerrit.
My guess is that we changed the phab user key and need to update the info in gerrit. @mmodell do you hold the keys to that kingdom?
@thcipriani: I suspect you are right, broadly speaking. We have been moving away from having phab mirroring into gerrit since gerrit is supposed to be the canonical repository host. There are just a couple of special-cases like this one. I'll see what I can do.
I confirm that it is working now, thanks a lot! Long term we are going to store models on Swift, so we'll hopefully not need this special use case anymore. Going to close this task since it lists too many different problems over time, and editquality is now mirroring correctly.