Move management routers ssh port
Open, LowPublic
Actions

Assigned To

None

Authored By

	ayounsi
	Mar 15 2021, 8:54 AM

Description

The management routers have ssh open to the world for OOB access.
As it listens on the standard port 22/tcp, it's subject to a lot of bruteforce attempts
They're all unsuccessful as we use key auth, which has been improved with T163969, but those are still:
1/ flooding the logs
2/ occasionally overwhelming the CPU

The next step is to move the ssh port to a different port (eg. 2222) to reduce most of those attempts.

The .ssh/config file will need to be updated accordingly so normal and emergency access stays smooth.

Details

Subject	Repo	Branch	Lines +/-
Management routers: move ssh port to 2222	operations/homer/public	master	+18 -3
Allow different port than default 22	operations/software/homer	master	+23 -12
Management routers: move ssh port to 2222	operations/homer/public	master	+8 -4
Rancid: use port 2222 for mgmt routers	operations/puppet	production	+6 -6

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T277438 Move management routers ssh port
		Unknown Object (Task)
		Unknown Object (Task)
		Unknown Object (Task)
Resolved	RobH	T294872 Q2(Need By: TBD) rack/setup/install new mr1-eqsin
		Unknown Object (Task)
Resolved	• Cmjohnson	T294474 Q2:(Need By: TBD) replace mr1-eqiad

Event Timeline

ayounsi triaged this task as Low priority.Mar 15 2021, 8:54 AM

ayounsi created this task.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 15 2021, 8:54 AM

Maintenance_bot added a project: SRE.Mar 15 2021, 8:57 AM

• crusnov subscribed.Mar 17 2021, 4:12 PM

This is not supported by 3/5 of our management routers. They will need to be replaced by more recent gear anyway.

ayounsi mentioned this in Unknown Object (Task).Mar 24 2021, 8:53 AM

ayounsi added a subtask: Unknown Object (Task).

RhinosF1 subscribed.Mar 24 2021, 8:54 AM

Aklapper added a project: Infrastructure-Foundations.Jun 21 2021, 9:00 PM

I changed the LibreNMS check to exclude management routers.
To revert once this task is done:

Processor usage over 85% EXCEPT Management routers

cmooney mentioned this in T289820: 2021-08-26 Primary inbound port utilisation over 80% page for mr1-esams.wikimedia.org.Aug 27 2021, 4:41 PM

RobH closed subtask Unknown Object (Task) as Resolved.Nov 5 2021, 7:56 PM

Change 785274 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] Management routers: move ssh port to 2222

https://gerrit.wikimedia.org/r/785274

gerritbot added a project: Patch-For-Review.Apr 22 2022, 7:51 AM

Before we merge/deploy any of those changes, Rancid and Homer use 22 as default SSH port.
Rancid supports adding :22 after the hostname according to this comment.

Homer will need a patch to support custom ports.

Change 890394 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/software/homer@master] Allow different port than default 22

https://gerrit.wikimedia.org/r/890394

Change 890402 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Rancid: use port 2222 for mgmt routers

https://gerrit.wikimedia.org/r/890402

Change 890811 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] Use port 2222 for management router ssh

https://gerrit.wikimedia.org/r/890811

Change 785274 abandoned by Ayounsi:

[operations/homer/public@master] Management routers: move ssh port to 2222