Page MenuHomePhabricator

Gerrit repo scoring/ores/editquality not mirroing
Open, HighPublic

Description

Event Timeline

Halfak created this task.Jun 4 2019, 2:57 PM
Restricted Application added a project: artificial-intelligence. · View Herald TranscriptJun 4 2019, 2:57 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Halfak added a subscriber: mmodell.Jun 4 2019, 2:57 PM

I think that @mmodell helped us with a credential issue that manifested in a similar way back in April.

Halfak triaged this task as High priority.Jun 4 2019, 7:02 PM

This is blocking an ORES deployment.

hashar removed a project: Gerrit.Jun 5 2019, 6:16 AM

Worked on this today with Tyler in our pairing session.

@Halfak: This should be resolved now. Can you confirm?

@thcipriani: Thanks for your help.

mmodell added a subtask: Restricted Task.Jun 5 2019, 8:07 PM
Halfak added a comment.Jun 6 2019, 3:15 PM

I've just tried to pull new changes from https://gerrit.wikimedia.org/r/scoring/ores/editquality and I'm still stuck on the April 23rd commit.

commit 373b4c143ec4591eaed474a57ad06333af19fc16
Merge: c8f1aa8 40d32e0
Author: Aaron Halfaker <aaron.halfaker@gmail.com>
Date:   Tue Apr 23 10:00:17 2019 -0500

    Merge pull request #193 from wikimedia/fiwiki_v2

When I try to directly clone https://phabricator.wikimedia.org/source/editquality, I get:

Cloning into 'test-eq2'...
remote: Counting objects: 4495, done.
remote: Compressing objects: 100% (3280/3280), done.
remote: Total 4495 (delta 2407), reused 2883 (delta 1162)
Receiving objects: 100% (4495/4495), 4.04 MiB | 0 bytes/s, done.
Resolving deltas: 100% (2407/2407), done.
Checking connectivity... done.
Downloading models/arwiki.damaging.gradient_boosting.model (340 KB)
Error downloading object: models/arwiki.damaging.gradient_boosting.model (e57c41a): Smudge error: Error downloading models/arwiki.damaging.gradient_boosting.model (e57c41a95a60e05ec821ebbf54ca283f25569c601b62bc368abea2d1dfe4ca98): batch response: Authorization error: https://phabricator.wikimedia.org/source/editquality.git/info/lfs/objects/batch
Check that you have proper access to the repository

Errors logged to /home/halfak/projects/test-eq/test-eq2/.git/lfs/logs/20190606T101215.719509111.log
Use `git lfs logs last` to view the log.
error: external filter git-lfs smudge -- %f failed 2
error: external filter git-lfs smudge -- %f failed
fatal: models/arwiki.damaging.gradient_boosting.model: smudge filter lfs failed
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry the checkout with 'git checkout -f HEAD'

It looks like that might be related.

Right. I did a manual update with @20after4's help to get unblocked for our upcoming deployment. I actually just did another manual push this morning.

gotcha.

My theory here is that there is some ssh timeout on the phab side for pushing to gerrit (or maybe some error on the phab side(?)). I don't see anything in the gerrit error logs related to phab/editquality.

I do see there are push attempts in the sshd_log:

ssh_log
[2019-06-11 00:05:53,217 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 1ms 714885ms 0 git/2.11.0
[2019-06-11 00:29:56,860 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 688116ms 0 git/2.11.0
[2019-06-11 00:53:34,070 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 1ms 662753ms 0 git/2.11.0
[2019-06-11 01:17:41,123 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 689700ms 0 git/2.11.0
[2019-06-11 01:40:55,275 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 649596ms 0 git/2.11.0
[2019-06-11 02:05:17,911 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 681857ms 0 git/2.11.0
[2019-06-11 02:29:32,952 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 668541ms 0 git/2.11.0
[2019-06-11 02:53:15,044 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 643558ms 0 git/2.11.0
[2019-06-11 03:17:07,385 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 649665ms 0 git/2.11.0
[2019-06-11 03:40:22,483 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 630413ms 0 git/2.11.0
[2019-06-11 04:03:47,598 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 628741ms 0 git/2.11.0
[2019-06-11 04:26:11,145 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 595719ms 0 git/2.11.0
[2019-06-11 04:48:58,620 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 626444ms 0 git/2.11.0
[2019-06-11 05:26:09,319 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 603253ms 0 git/2.11.0
[2019-06-11 06:03:28,095 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 616873ms 0 git/2.11.0
[2019-06-11 06:41:04,985 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 3ms 616671ms 0 git/2.11.0
[2019-06-11 07:18:27,634 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 3ms 605659ms 0 git/2.11.0
[2019-06-11 07:56:53,115 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 653584ms 0 git/2.11.0
[2019-06-11 08:36:29,855 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 641999ms 0 git/2.11.0
[2019-06-11 09:14:54,715 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 623062ms 0 git/2.11.0
[2019-06-11 09:53:41,939 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 624528ms 0 git/2.11.0
[2019-06-11 10:32:31,111 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 1ms 653837ms 0 git/2.11.0
[2019-06-11 11:12:13,556 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 672296ms 0 git/2.11.0
[2019-06-11 11:53:09,105 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 3ms 691439ms 0 git/2.11.0
[2019-06-11 12:33:31,065 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 705240ms 0 git/2.11.0
[2019-06-11 13:14:04,107 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 706536ms 0 git/2.11.0
[2019-06-11 13:56:27,538 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 766926ms 0 git/2.11.0
[2019-06-11 14:37:24,241 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 1ms 701814ms 0 git/2.11.0
[2019-06-11 15:19:12,353 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 768491ms 0 git/2.11.0
[2019-06-11 16:02:47,257 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 807261ms 0 git/2.11.0
[2019-06-11 16:44:30,999 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 688184ms 0 git/2.11.0
[2019-06-11 17:26:05,610 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 2ms 690340ms 0 git/2.11.0
[2019-06-11 18:08:48,914 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 1ms 656581ms 0 git/2.11.0
[2019-06-11 18:49:54,052 +0000] xxxxxxxx phab a/xxxx git-receive-pack./mediawiki/services/ores/editquality 1ms 656431ms 0 git/2.11.0

We could create an HTTPs user that isn't affected by T225308: Users with a different name in the cn field compared to uid field cannot use http auth (unlike phab/phabricator) to try to push of HTTPs rather than SSH if we suspect it is an SSH timeout somewhere along the way causing the problem.