Page MenuHomePhabricator

Reviewer-bot not adding reviewers due to SSH key change
Closed, ResolvedPublic

Description

It looks like reviewer-bot is no longer adding reviewers to new changes because of the SSH key for gerrit having changed. From the logs:

Wed 05 Aug 2026 01:35:03 PM UTC: Running as task gerrit-reviewer-bot-29765615-8r4wf
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:njCmWMsshq3MqQxyIFO36UNwCwzTamXERqylF1XJhd8.
Please contact your system administrator.
Add correct host key in /data/project/gerrit-reviewer-bot/.ssh/known_hosts to get rid of this message.
Offending RSA key in /data/project/gerrit-reviewer-bot/.ssh/known_hosts:1
  remove with:
  ssh-keygen -f "/data/project/gerrit-reviewer-bot/.ssh/known_hosts" -R "[gerrit.wikimedia.org]:29418"
ED25519 host key for [gerrit.wikimedia.org]:29418 has changed and you have requested strict checking.
Host key verification failed.
Traceback (most recent call last):
  File "/data/project/gerrit-reviewer-bot/src/gerrit-reviewer-bot/pop3bot.py", line 122, in main
    add_reviewers(changeset['id'], reviewers)
  File "/data/project/gerrit-reviewer-bot/src/gerrit-reviewer-bot/add_reviewer.py", line 150, in add_reviewers
    raise Exception(command + ' was not executed successfully (code %i)' % retval)
Exception: gerrit set-reviewers --add ejegg --add XenoRyet --add jgleeson --add Eileen --add cstone --add damilare --add Wfan --add lsandergreen --add dmiranda 'wikimedia%2Ffundraising%2Fcrm~1321027' was not executed successfully (code 255)
{'id': 'wikimedia%2Ffundraising%2Fcrm~1321027', 'triplet_id': 'wikimedia%2Ffundraising%2Fcrm~master~I130988081a57793a2bed368900ef62d745359ab1', 'project': 'wikimedia/fundraising/crm', 'branch': 'master', 'hashtags': [], 'change_id': 'I130988081a57793a2bed368900ef62d745359ab1', 'subject': 'Update date_of_largest_donation for donors whose largest is endowment.', 'status': 'NEW', 'created': '2026-08-04 17:27:47.000000000', 'updated': '2026-08-04 23:50:47.000000000', 'submit_type': 'REBASE_IF_NECESSARY', 'mergeable': True, 'insertions': 34, 'deletions': 0, 'total_comment_count': 0, 'unresolved_comment_count': 0, 'has_review_started': True, 'meta_rev_id': 'b4ab08efbb40a6353de53beb6bef0ced6d999e8e', '_number': 1321027, 'virtual_id_number': 1321027, 'owner': {'_account_id': 12685, 'name': 'Lars SG', 'display_name': 'lsandergreen', 'email': 'lsandergreen@wikimedia.org', 'username': 'lsandergreen'}, 'current_revision_number': 2, 'current_revision': '85f9874ee5559c8c60858e5db72e2289f8b688d0', 'revisions': {'85f9874ee5559c8c60858e5db72e2289f8b688d0': {'kind': 'NO_CODE_CHANGE', '_number': 2, 'created': '2026-08-04 23:25:56.000000000', 'uploader': {'_account_id': 12685, 'name': 'Lars SG', 'display_name': 'lsandergreen', 'email': 'lsandergreen@wikimedia.org', 'username': 'lsandergreen'}, 'ref': 'refs/changes/27/1321027/2', 'fetch': {'anonymous http': {'url': 'https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm', 'ref': 'refs/changes/27/1321027/2'}}, 'branch': 'refs/heads/master', 'files': {'ext/wmf-civicrm/CRM/Wmf/Upgrader.php': {'old_mode': 33188, 'new_mode': 33188, 'lines_inserted': 34, 'size_delta': 1138, 'size': 186849}}, 'description': 'Edit commit message'}}, 'requirements': [], 'submit_records': [{'rule_name': 'gerrit~DefaultSubmitRule', 'status': 'NOT_READY', 'labels': [{'label': 'Verified', 'status': 'OK', 'applied_by': {'_account_id': 75, 'name': 'jenkins-bot', 'username': 'jenkins-bot', 'tags': ['SERVICE_USER']}}, {'label': 'Code-Review', 'status': 'NEED'}]}]}
 caused exception:Traceback (most recent call last):
  File "/data/project/gerrit-reviewer-bot/src/gerrit-reviewer-bot/pop3bot.py", line 134, in 
    main()
  File "/data/project/gerrit-reviewer-bot/src/gerrit-reviewer-bot/pop3bot.py", line 122, in main
    add_reviewers(changeset['id'], reviewers)
  File "/data/project/gerrit-reviewer-bot/src/gerrit-reviewer-bot/add_reviewer.py", line 150, in add_reviewers
    raise Exception(command + ' was not executed successfully (code %i)' % retval)
Exception: gerrit set-reviewers --add ejegg --add XenoRyet --add jgleeson --add Eileen --add cstone --add damilare --add Wfan --add lsandergreen --add dmiranda 'wikimedia%2Ffundraising%2Fcrm~1321027' was not executed successfully (code 255)

Event Timeline

bd808 changed the task status from Open to In Progress.Wed, Aug 5, 4:40 PM
bd808 subscribed.
tools.gerrit-reviewer-bot@tools-bastion-14:~/.ssh$ ssh gerrit
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:njCmWMsshq3MqQxyIFO36UNwCwzTamXERqylF1XJhd8.
Please contact your system administrator.
Add correct host key in /data/project/gerrit-reviewer-bot/.ssh/known_hosts to get rid of this message.
Offending RSA key in /data/project/gerrit-reviewer-bot/.ssh/known_hosts:1
  remove with:
  ssh-keygen -f '/data/project/gerrit-reviewer-bot/.ssh/known_hosts' -R '[gerrit.wikimedia.org]:29418'
Host key for [gerrit.wikimedia.org]:29418 has changed and you have requested strict checking.
Host key verification failed.
tools.gerrit-reviewer-bot@tools-bastion-14:~/.ssh$ ssh-keygen -f '/data/project/gerrit-reviewer-bot/.ssh/known_hosts' -R '[gerrit.wikimedia.org]:29418'
# Host [gerrit.wikimedia.org]:29418 found: line 1
/data/project/gerrit-reviewer-bot/.ssh/known_hosts updated.
Original contents retained as /data/project/gerrit-reviewer-bot/.ssh/known_hosts.old
tools.gerrit-reviewer-bot@tools-bastion-14:~/.ssh$ ssh -o StrictHostKeyChecking=no  gerrit
Warning: Permanently added '[gerrit.wikimedia.org]:29418' (ED25519) to the list of known hosts.

  ****    Welcome to Gerrit Code Review    ****

  Hi Reviewer-bot, you have successfully connected over SSH.

  Unfortunately, interactive shells are disabled.
  To clone a hosted Git repository, use:

  git clone ssh://reviewer-bot@gerrit.wikimedia.org:29418/REPOSITORY_NAME.git

Connection to gerrit.wikimedia.org closed.

The SHA256:njCmWMsshq3MqQxyIFO36UNwCwzTamXERqylF1XJhd8 presented in the first error message matches https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/gerrit.wikimedia.org:29418

You can also trust https://config-master.wikimedia.org/known_hosts which lists it. This source is auto-generated.

Another error was reported in the job log following my update of the keys on the shared NFS mount. Investigating.

The bot's ssh hits another key I did not remove:

tools.gerrit-reviewer-bot@shell-1785949307:~/src/gerrit-reviewer-bot$ ./ssh -i id_rsa -p 29418 reviewer-bot@gerrit.wikimedia.org
Warning: the ED25519 host key for '[gerrit.wikimedia.org]:29418' differs from the key for the IP address '[208.80.154.225]:29418'
Offending key for IP in /data/project/gerrit-reviewer-bot/.ssh/known_hosts:25
Matching host key in /data/project/gerrit-reviewer-bot/.ssh/known_hosts:26
Are you sure you want to continue connecting (yes/no)? no
Host key verification failed.
tools.gerrit-reviewer-bot@shell-1785949307:~/src/gerrit-reviewer-bot$ ssh-keygen -f '/data/project/gerrit-reviewer-bot/.ssh/known_hosts' -R '[208.80.154.225]:29418'
# Host [208.80.154.225]:29418 found: line 25
/data/project/gerrit-reviewer-bot/.ssh/known_hosts updated.
Original contents retained as /data/project/gerrit-reviewer-bot/.ssh/known_hosts.old
tools.gerrit-reviewer-bot@shell-1785949307:~/src/gerrit-reviewer-bot$ ./ssh -i id_rsa -p 29418 reviewer-bot@gerrit.wikimedia.org
Warning: Permanently added the ED25519 host key for IP address '[208.80.154.225]:29418' to the list of known hosts.

  ****    Welcome to Gerrit Code Review    ****

  Hi Reviewer-bot, you have successfully connected over SSH.

  Unfortunately, interactive shells are disabled.
  To clone a hosted Git repository, use:

  git clone ssh://reviewer-bot@gerrit.wikimedia.org:29418/REPOSITORY_NAME.git

Connection to gerrit.wikimedia.org closed.
bd808 claimed this task.

Manual runs are now working:

tools.gerrit-reviewer-bot@tools-bastion-14:~$ webservice python3.9 shell -- src/gerrit-reviewer-bot/gerrit_reviewer_bot_tf-python39.sh
[MediaWiki-commits] [Gerrit] ...puppet[production]: mediawiki: Disable legacy
 `short_urls` on vhosts where it does not wo... "Krinkle (Code Review) via MediaWiki-commits"
 <mediawiki-commits@lists.wikimedia.org> I92f7b85114c7e9cabec62c83959410d04cfea992
(getting  eeeb3f0ce635ea448aca3fc683b09c2a8cac7ea6 )

Processing changeset  I92f7b85114c7e9cabec62c83959410d04cfea992 mediawiki: Disable legacy `short_urls` on vhosts where it does not work (take 2) by Krinkle
u hieradata/common/mediawiki.yaml
Done.
tools.gerrit-reviewer-bot@tools-bastion-14:~$

The first one I did after the fix in T434075#12188593 processed a large backlog.

Thank you so much!

I added a feature request for reviewer-bot as a follow-up to prevent these in the future.