Page MenuHomePhabricator
Create Task
Maniphest T362524

Diffusion fails to push three ORES repositories into Gerrit
Closed, ResolvedPublic
Actions

Description

Yesterday and today we've seen a bunch of mirroring issues in /var/log/phd/daemons.log on phab1004.eqiad.wmnet when Diffusion tries to push into Gerrit (via K18 or K19, cf the table in T213246).

Potentially related to T354886: Upgrade to Gerrit 3.8?

STDERR
Pushing to ssh://gerrit.wikimedia.org:29418/[...]
RSA host key for IP address '2620:0:861:2:208:80:154:151' not in list of known hosts.
remote: error: branch refs/meta/config:        
remote: Cannot delete project configuration from 'refs/meta/config'        
remote: error: branch refs/meta/config:        
remote: You need 'Delete Reference' rights or 'Push' rights with the         
remote: 'Force Push' flag set to delete references.        
remote: User: phab        
remote: Contact an administrator to fix the permissions

Related Objects

Event Timeline

Aklapper created this task.Mon, Apr 15, 12:22 PM
hashar subscribed.Tue, Apr 16, 7:37 AM

refs/meta/config holds the Gerrit configuration for the repository, it indeed can not be deleted. The ORES repositories being synced from GitHub to Gerrit using Phabricator is a terrible case which kept causing troubles here and there and we should remove that system (that is 5 years old T213246).

Which repository(ies) has(ve) the issue?

Aklapper added a comment.Tue, Apr 16, 10:01 AM

Which repository(ies) has(ve) the issue?

According to aklapper@phab1004:~$ grep -r "RSA host key for IP address '2620:0:861:2:208:80:154:151' not in list of known hosts" -B1 /var/log/phd/daemons.log* seems to be three repos:
ssh://gerrit.wikimedia.org:29418/scoring/ores/articlequality, ssh://gerrit.wikimedia.org:29418/scoring/ores/draftquality, ssh://gerrit.wikimedia.org:29418/scoring/ores/ores

Aklapper added a comment.Tue, Apr 16, 10:14 AM

...which means K19 fails.

Aklapper added a comment.Wed, Apr 17, 9:13 AM

Switched the three repos from using K19 for an ssh:// URI to using K18 for an https:// URI. Let's see if that helps, if not I may just switch this off completely

Aklapper renamed this task from Diffusion fails to push into Gerrit to Diffusion fails to push three ORES repositories into Gerrit.Sat, Apr 20, 3:35 PM
Aklapper closed this task as Resolved.
Aklapper claimed this task.

No errors anymore today or yesterday, assuming this is fixed.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL