The review has one LOW severity finding, no MEDIUM, HIGH or CRITICAL and 15 INFORMATIONAL. Informational meaning the test did not find any vulnerability.

From the report:

3.1 LOW - Scan of external networks using untrusted runners

Description:
It is possible to scan external devices using the gitlab runners.

Proof of concept:
A job is generated to be executed in an untrusted runner and we verified that it is possible to scan external devices without restrictions. We performed a scan to an external website from an untrusted runner using nikto, a known web scanner. A scan was performed to TCP ports 1 through 1000 to a server that has ports 80, 443 and 483
explicitly open. We observed the server could be scanned from the untrusted runner without restrictions. According to the test performed in 3, the outgoing bandwith from the runner is limited. This somewhat mitigates the impact of these scans.

Hosts/Affected endpoints:
GitLab: https://gitlab.devtools.wmcloud.org/

Notes / Recommendation:
We recommend evaluating the implementation of certain measures, such as outgoing firewall rules, to prevent a job executing inside an untrusted runner from performing massive connections to devices in the Internet.

Additional Context

The review revealed that outgoing traffic from Shared/Untrusted Runners is not filtered or restricted. Docker containers in CI jobs can access arbitrary services, including (not only) port scanning. So the finding is not only about scanning external networks but also general access of external services (http, tcp, ssh ...).

Limitations: Access to arbitrary services in not possible from the Trusted Runners, see chapter 3.14 in the report. Trusted Runners have more restrictive firewall settings (profile::gitlab::runner::restrict_firewall).

It should be discussed if Shared Runners need more restrictive settings as well or if the should be more open by design. If we decided to also restrict Shared Runners, the firewall rules from Protected Runners could be adapted and re-used.

Details

Risk Rating: Medium
Author Affiliation: WMF Technology

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	gitlab_runner: enable restrict_firewall for Shared Runners	Jelto	operations/puppet	production	+36 -5

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Mstyles	T304514 Security Readiness Reviews of Trusted GitLab Runners
		Resolved	Security	Jelto	T317341 Findings in Security Readiness Reviews of Trusted GitLab Runners

Event Timeline

Jelto created this task.Sep 8 2022, 4:25 PM

Restricted Application added a project: serviceops-deprecated. · View Herald TranscriptSep 8 2022, 4:25 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Jelto changed the edit policy from "All Users" to "Custom Policy".Sep 8 2022, 4:29 PM

Jelto changed the visibility from "Public (No Login Required)" to "Custom Policy".Sep 8 2022, 4:31 PM

Jelto changed the edit policy from "Custom Policy" to "Custom Policy".

Jelto changed the visibility from "Custom Policy" to "Custom Policy".Sep 8 2022, 4:33 PM

Jelto changed the edit policy from "Custom Policy" to "Custom Policy".

Mstyles changed the edit policy from "Custom Policy" to "acl*security (Project)".Sep 8 2022, 4:34 PM

@Jelto you should be good to update the task description now

Jelto updated the task description. (Show Details)Sep 9 2022, 7:27 AM

dduvall mentioned this in T308271: Deploy buildkitd to trusted GitLab runners.Sep 9 2022, 10:42 PM

Mstyles moved this task from Incoming to In Progress on the Security-Team board.Sep 12 2022, 5:48 PM

jijiki moved this task from Incoming 🐫 to 🙈🙉🙊Backlog on the serviceops-deprecated board.Sep 28 2022, 2:17 PM

jijiki moved this task from 🙈🙉🙊Backlog to Production Errors 🚜 on the serviceops-deprecated board.Sep 28 2022, 4:21 PM

Jelto claimed this task.Oct 17 2022, 3:00 PM

Jelto moved this task from Incoming to Work in Progress on the Collaboration-Services board.

I think we should differentiate between security risks and risks for just abusing the Runners. Shared Runners live inside the WMCS project gitlab-runners. This project contains other Shared Runners and a puppet master only. No other non-production or production services are hosted in the same WMCS project.

Access to other WMCS projects

Access to other WMCS is restricted by security groups. The current default security groups seem to allow ICMP traffic between WMCS projects. A quick test from a CI container on one of the Shared Runners shows that ICMP traffic to other WMCS projects is possible:

root@38ee51b443c0:/# ping -c4 pcc-worker1003.puppet-diffs.eqiad1.wikimedia.cloud
PING pcc-worker1003.puppet-diffs.eqiad1.wikimedia.cloud (172.16.6.165) 56(84) bytes of data.
64 bytes from pcc-worker1003.puppet-diffs.eqiad1.wikimedia.cloud (172.16.6.165): icmp_seq=1 ttl=63 time=0.612 ms
64 bytes from pcc-worker1003.puppet-diffs.eqiad1.wikimedia.cloud (172.16.6.165): icmp_seq=2 ttl=63 time=0.633 ms
64 bytes from pcc-worker1003.puppet-diffs.eqiad1.wikimedia.cloud (172.16.6.165): icmp_seq=3 ttl=63 time=0.580 ms
64 bytes from pcc-worker1003.puppet-diffs.eqiad1.wikimedia.cloud (172.16.6.165): icmp_seq=4 ttl=63 time=0.589 ms

--- pcc-worker1003.puppet-diffs.eqiad1.wikimedia.cloud ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 41ms
rtt min/avg/max/mdev = 0.580/0.603/0.633/0.032 ms

That could be abused to explore community services and other WMCS projects.

Depending on the other projects security groups additional access might be possible too. Here is a quick example of loose security groups in the puppet-diffs WMCS project (running the PCC tool). SSH from the CI container to one of the pcc-workers is possible:

root@38ee51b443c0:/# ssh pcc-worker1003.puppet-diffs.eqiad1.wikimedia.cloud
The authenticity of host 'pcc-worker1003.puppet-diffs.eqiad1.wikimedia.cloud (172.16.6.165)' can't be established.
ECDSA key fingerprint is SHA256:V41umJhibGYPME/lEcu2jx1XJPeK4uZD62Oh43y0+3o.
Are you sure you want to continue connecting (yes/no)?

So we don't have full isolation between WMCS projects. I have some security concerns here and would like to restrict egress traffic from the gitlab-runner project to other WMCS projects (172.16.0.0/21). I'm not sure if WMCS/openstack security groups allow block/drop rules. Otherwise we may have to add ferm also to Shared Runners in WMCS (ferm is installed on Trusted Runners only at the moment).

Access to the internet

Here I don't see direct security issues for our infrastructure. But this kind of access could be abused like described in the security review. Portscans to public hosts or other malicious activity are possible. We could lock Shared Runners down more and allow http/https/other known services only. But I don't think that's needed at the moment and makes usage of CI more complicated. We can implement this if we face cases of abuse at some point.

That would mean we partially accept the finding and risk (at least for public traffic). For traffic to other WMCS I'd like to create some kind of firewall rules (ferm similar to Trusted Runners or security groups).

LSobanski removed a project: serviceops-deprecated.Oct 18 2022, 3:44 PM

sbassett added a project: Vuln-Misconfiguration.Oct 18 2022, 5:01 PM

We discussed the above issue in the IC Sync meeting. We were not able to decide if the above risk should be accepted or if we need further firewall hardening for the Shared Runners. We agreed that RelEng will discuss this also further in their team meeting.

The shared view and expectation of security and access in WMCS was not fully clear. It was not clear if WMCS is seen as fully untrusted, like the internet and any party should expect connection and access from unknown sources. Thinking in this theme, unwanted access needs to be restricted by service owners using Security Groups and firewalls.
On the other hand some projects could assume access to their instances and services is limited by default and they may not be aware of misconfigured Security Groups. So Shared Runners could proactively restrict access to those other projects.

Shared Runners don't have ferm firewall installed currently. Furthermore Horizon Security Groups seem to not support deny rules (for denying egress traffic to other WMCS projects). So if we decide to restrict the outgoing traffic that would cause some more changes to Shared Runners, because ferm needs to be installed and configured too.

I've created a patch to enable the firewall on Shared Runners too and block access to WMCS internal network: https://gerrit.wikimedia.org/r/c/849499

I've done some tests by cherry picking this to the devtools project. The change installs ferm and ferm rules to the Shared Runners, similar to Trusted Runners. Traffic to public addresses and GitLab works as expected and is unrestricted. Access to other WMCS machines is blocked with the new change (similar test as above but from gitlab-runner-1002.devtools.eqiad1.wikimedia.cloud):

root@701ad3f435d6:/# ping -c4 pcc-worker1003.puppet-diffs.eqiad1.wikimedia.cloud
PING pcc-worker1003.puppet-diffs.eqiad1.wikimedia.cloud (172.16.6.165) 56(84) bytes of data.
From 172.17.0.1 (172.17.0.1) icmp_seq=1 Destination Port Unreachable
From 172.17.0.1 (172.17.0.1) icmp_seq=2 Destination Port Unreachable
From 172.17.0.1 (172.17.0.1) icmp_seq=3 Destination Port Unreachable
From 172.17.0.1 (172.17.0.1) icmp_seq=4 Destination Port Unreachable

SSH:

root@701ad3f435d6:/# ssh pcc-worker1003.puppet-diffs.eqiad1.wikimedia.cloud
ssh: connect to host pcc-worker1003.puppet-diffs.eqiad1.wikimedia.cloud port 22: Connection refused

I have to fix one small subnet misalignment in the ferm::conf { 'docker-ferm': ...} default rule, so https://gerrit.wikimedia.org/r/c/849499 is marked as WIP currently. But I think it's should be quite easy to enable this for Shared Runners without disrupting any of the existing CI jobs we have on Shared Runners. With the more restrictive firewall rule we don't have to worry about unauthorized access to other WMCS projects and can close this task/finding. Furthermore we have a working firewall setup. This can be used for blocking other services. This could be useful if we see some abuse and we want to block access to some malicious services (also public ones).

I fixed the last bug in https://gerrit.wikimedia.org/r/c/849499 and tested it again in the devtools project. With the added firewall the Shared test Runner can no longer access other WMCS projects. Outgoing access to public resources still works, similar as before. Existing CI jobs should not be affected (only if specific services in WMCS are used).

So I'd be happy to deploy this firewall to all Shared Runners. We could close this security finding and prevent any attacks on other WMCS projects from Shared Runners. Furthermore Shared and Protected Runners use the same firewall tooling. One additional benefit is that we are able to properly block services, in case any abuse happens.

@brennen @dduvall @dancy I'd like to discuss adding the firewall rules to all Shared Runners on next IC Sync, if possible. I know that our last discussion was not fully clear about who is responsible for securing WMCS hosts. But from my point of view the firewall is adding very little complexity and we already use the same setup on Trusted Runners.

The change to add firewall rules to Shared Runners was merged. However for Shared Runners in WMCS this is noop and no firewall rules are added. WMCS Runners use profile::gitlab_runner instead of role::gitlab_runner (like the test runners in devtools). So we need to migrate those Shared Runners to role::gitlab_runner or refactor the puppet code.

I opened T322409 to discuss this.

In the IC sync we also talked about the future of Shared Runners in WMCS and it seems they are somewhat end-of-life. So it could also make sense to just close T322409 and this task depending on the expected time frame.

All Shared Runners have ferm/firewall enabled now (see T322409). The DOCKER-ISOLATION chain contains :

Chain DOCKER-ISOLATION (1 references)
target     prot opt source               destination         
[...]
REJECT     all  --  anywhere             172.16.0.0/21        reject-with icmp-port-unreachable

which blocks all outgoing traffic from CI job container to other WMCS hosts/projects. So also outgoing port scans as mentioned in the finding 3.1 are blocked. I verified this with the same commands as above.

The risk of scanning public networks (internet) will be accepted for now. If we see such activities in CI jobs this can also be blocked with a additional firewall rule.

I'm closing this task, as the risk involving WMF infrastructure is fixed now.

Dzahn awarded a token.Nov 17 2022, 7:34 PM

@Jelto Can we make this task public now so it is easier to document the firewall restrictions on the WMCS hosted runners?

It should be fine to make the task public.

sbassett changed Author Affiliation from N/A to WMF Technology.Dec 3 2024, 5:45 PM

sbassett added a project: SecTeam-Processed.

sbassett changed the visibility from "Custom Policy" to "Public (No Login Required)".

sbassett changed Risk Rating from N/A to Medium.

LucasWerkmeister mentioned this in T397591: Unable to connect to Beta Cluster from WMCS GitLab CI runners.Jun 24 2025, 3:11 PM

Findings in Security Readiness Reviews of Trusted GitLab RunnersClosed, ResolvedPublicSecurityActions

Description