Page MenuHomePhabricator
Create Task
Maniphest T366882

implement anti-abuse features for GitLab (Move GitLab behind the CDN)
Closed, ResolvedPublic
Actions

Description

In T365259 it was discussed to move Gerrit behind the CDN/loadbalancer for better anti-abuse handling. It was discussed that GitLab may be a better first candidate because the setup is quite similar (one web service and one ssh service) but GitLab is less production-critical and has more test instances available.

Most discussion already happened in T365259 which also holds up for GitLab. So this task is mostly for discussing and tracking the actual technical implementation of anti-abuse hanlding for GitLab.

GitLab consits of multiple machines and services:

This services are not related or distributed in any way. The replicas are standby machines which can be used for emergency switchovers and testing. They run an actual GitLab instance with old (12h) data but this instances are not used for the production GitLab.

Technical exploration

A first exploration was done with LVS. During this exploration, we considered placing GitLab behind the load balancing infrastructure. However, it was determined that this approach would involve significant refactoring and implementation work and overhead, especially since our primary interest is in throttling capabilities. Consequently, this idea was set aside.

We then explored alternative methods of throttling and traffic shaping. Utilizing local tools, such as a separate HAProxy or a firewall, seemed the most promising. Currently, we are testing and implementing throttling using firewall rules. The newer tool nftables offers built-in features that are particularly useful. Therefore, our initial step is to migrate to nftables and then verify potential configurations with it:

  • Upgrade GitLab hosts to nftables
  • Verify that throttling and dynamic IP sets for a denylist are possible in nftables
  • Puppetize a basic set of rules to throttle external HTTP traffic
  • Adjust thresholds
  • enable rules on all instances
  • (Repeat for SSH traffic)?
  • update docs and publish tech news article

Details

Other Assignee
Dzahn
Related Changes in Gerrit:
SubjectRepoBranchLines +/-
gitlab: set throttling policy to accept againoperations/puppetproduction+1 -1
Revert "gitlab: test defs_from_etcd on the replica"operations/puppetproduction+1 -4
gitlab: test defs_from_etcd on the replicaoperations/puppetproduction+4 -1
gitlab: enable nftables throttling (drop)operations/puppetproduction+1 -1
gitlab: adjust throttling threshold for GitLaboperations/puppetproduction+3 -3
gitlab: enable throttling for all GitLab instancesoperations/puppetproduction+1 -1
profile::firewall::nftables_throttling: fix issue of global meteringoperations/puppetproduction+18 -2
gitlab: add profile::prometheus::nft_throttling_denylistoperations/puppetproduction+1 -0
gitlab: use burst parameter with nftables_throttlingoperations/puppetproduction+1 -0
profile::firewall::nftables_throttling: add option for burst packetsoperations/puppetproduction+15 -18
gitlab: enable nft throttling on role level, but just logoperations/puppetproduction+3 -1
gitlab: fix port definition in firewall:serviceoperations/puppetproduction+1 -1
gitlab: set nft throttling policy to drop on replicaoperations/puppetproduction+1 -0
firewall/gitlab: add option to throttle and drop traffic using nftablesoperations/puppetproduction+81 -0
conftool-data: add gitlab and replicasoperations/puppetproduction+11 -0
gitlab: switch gitlab from iptables to nftablesoperations/puppetproduction+7 -12
add LVS service IPs for gitlab and gitlab-sshoperations/dnsmaster+8 -4
gitlab: replace ferm::service with firewall::serviceoperations/puppetproduction+9 -9
gitlab: switch gitlab-replica-b from iptables to nftablesoperations/puppetproduction+1 -0
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Restricted Task
Restricted Task
 ResolvedJeltoT366882 implement anti-abuse features for GitLab (Move GitLab behind the CDN)

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Jun 7 2024, 8:54 AM

Change #1040094 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] conftool-data: add gitlab and replicas

https://gerrit.wikimedia.org/r/1040094

gerritbot added a project: Patch-For-Review.Jun 7 2024, 8:54 AM
Jelto updated the task description. (Show Details)Jun 7 2024, 8:59 AM
Fabfur subscribed.Jun 7 2024, 9:01 AM
Peachey88 removed a project: WMF-NDA.Jun 7 2024, 9:37 AM
gerritbot added a comment.Jun 7 2024, 8:23 PM

Change #1040261 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/dns@master] add LVS service IPs for gitlab and gitlab-ssh

https://gerrit.wikimedia.org/r/1040261

LSobanski triaged this task as High priority.Jun 10 2024, 3:33 PM
LSobanski moved this task from Incoming to Work in Progress on the collaboration-services board.
RhinosF1 subscribed.Jun 21 2024, 9:55 PM
gerritbot added a comment.Jul 10 2024, 12:55 PM

Change #1053306 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: switch gitlab-replica-b from iptables to nftables

https://gerrit.wikimedia.org/r/1053306

gerritbot added a comment.Jul 12 2024, 8:12 AM

Change #1053306 merged by Jelto:

[operations/puppet@production] gitlab: switch gitlab-replica-b from iptables to nftables

https://gerrit.wikimedia.org/r/1053306

gerritbot added a comment.Jul 12 2024, 8:57 AM

Change #1053877 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: replace ferm::service with firewall::service

https://gerrit.wikimedia.org/r/1053877

gerritbot added a comment.Jul 12 2024, 9:12 AM

Change #1053879 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: switch gitlab-replica-b from iptables to nftables

https://gerrit.wikimedia.org/r/1053879

gerritbot added a comment.Jul 15 2024, 9:42 AM

Change #1053877 merged by Jelto:

[operations/puppet@production] gitlab: replace ferm::service with firewall::service

https://gerrit.wikimedia.org/r/1053877

gerritbot added a comment.Jul 15 2024, 7:46 PM

Change #1040261 abandoned by Dzahn:

[operations/dns@master] add LVS service IPs for gitlab and gitlab-ssh

Reason:

not likely anymore that we are going to do this

https://gerrit.wikimedia.org/r/1040261

gerritbot added a comment.Jul 16 2024, 7:52 AM

Change #1053879 merged by Jelto:

[operations/puppet@production] gitlab: switch gitlab from iptables to nftables

https://gerrit.wikimedia.org/r/1053879

Jelto renamed this task from Move GitLab behind the CDN to implement anti-abuse features for GitLAb (Move GitLab behind the CDN).Jul 16 2024, 8:51 AM

I migrated the GitLab hosts to nftables which unblocks us using nftables built-in features

Jelto renamed this task from implement anti-abuse features for GitLAb (Move GitLab behind the CDN) to implement anti-abuse features for GitLab (Move GitLab behind the CDN).Jul 16 2024, 8:53 AM
Jelto updated Other Assignee, added: Dzahn.Jul 16 2024, 8:55 AM
gerritbot added a comment.Jul 16 2024, 9:08 AM

Change #1040094 abandoned by Jelto:

[operations/puppet@production] conftool-data: add gitlab and replicas

Reason:

a different implementation will be tested

https://gerrit.wikimedia.org/r/1040094

brennen moved this task from INBOX to Radar on the Release-Engineering-Team board.Jul 16 2024, 6:37 PM
brennen edited projects, added Release-Engineering-Team (Radar); removed Release-Engineering-Team.
Maintenance_bot removed a project: Patch-For-Review.Jul 16 2024, 7:30 PM
Jelto updated the task description. (Show Details)Jul 19 2024, 1:40 PM
gerritbot added a comment.Jul 22 2024, 10:00 AM

Change #1055886 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: add option to throttle and drop traffic using nftables

https://gerrit.wikimedia.org/r/1055886

gerritbot added a project: Patch-For-Review.Jul 22 2024, 10:00 AM
gerritbot added a comment.Jul 24 2024, 5:50 PM

Change #1055886 merged by Dzahn:

[operations/puppet@production] firewall/gitlab: add option to throttle and drop traffic using nftables

https://gerrit.wikimedia.org/r/1055886

Dzahn added a comment.Jul 24 2024, 6:10 PM

gitlab1003 and gitlab1004 are unchanged.

gitlab2002 now has the new file /etc/nftables/099_throttling-chain_puppet.nft created by puppet through the change above.

For now it is logging but still accepting and not dropping packets.

Next we can change the policy to DROP through a simple Hiera change.

Maintenance_bot removed a project: Patch-For-Review.Jul 24 2024, 6:30 PM
gerritbot added a comment.Jul 24 2024, 6:36 PM

Change #1056581 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] gitlab: set nft throttling policy to drop on replica

https://gerrit.wikimedia.org/r/1056581

gerritbot added a project: Patch-For-Review.Jul 24 2024, 6:36 PM
gerritbot added a comment.Jul 26 2024, 8:52 AM

Change #1056581 merged by Jelto:

[operations/puppet@production] gitlab: set nft throttling policy to drop on replica

https://gerrit.wikimedia.org/r/1056581

Jelto updated the task description. (Show Details)Jul 26 2024, 8:58 AM
Jelto added a comment.EditedJul 26 2024, 9:07 AM
In T366882#10012178, @Dzahn wrote:

gitlab1003 and gitlab1004 are unchanged.

gitlab2002 now has the new file /etc/nftables/099_throttling-chain_puppet.nft created by puppet through the change above.

For now it is logging but still accepting and not dropping packets.

Next we can change the policy to DROP through a simple Hiera change.

Thanks for merging and deploying this change! I enabled the more restrictive drop rule on gitlab-replica-b gitlab1003. I was able to trigger the throttling rule with the usual curl loop and the current limit of 1RPS. Manual browsing of the website works as expected (after waiting 5 minutes until the droplist entry is removed). So I'd suggest to leave this running over the weekend and then we can discuss the next steps on Tuesday (enabling it gitlab-wide, RPS and communication for RelEng and the community).

Maintenance_bot removed a project: Patch-For-Review.Jul 26 2024, 9:31 AM
gerritbot added a comment.Jul 26 2024, 10:20 AM

Change #1057190 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: fix port definition in firewall:service

https://gerrit.wikimedia.org/r/1057190

gerritbot added a project: Patch-For-Review.Jul 26 2024, 10:20 AM
gerritbot added a comment.Jul 26 2024, 9:07 PM

Change #1057190 merged by Dzahn:

[operations/puppet@production] gitlab: fix port definition in firewall:service

https://gerrit.wikimedia.org/r/1057190

Maintenance_bot removed a project: Patch-For-Review.Jul 26 2024, 9:30 PM
gerritbot added a comment.Jul 31 2024, 1:50 PM

Change #1058608 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: enable throttling for all GitLab instances

https://gerrit.wikimedia.org/r/1058608

gerritbot added a project: Patch-For-Review.Jul 31 2024, 1:50 PM
gerritbot added a comment.Aug 6 2024, 3:33 PM

Change #1060131 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: enable nft throttling on role level, but just log

https://gerrit.wikimedia.org/r/1060131

Jelto added a parent task: Restricted Task.Aug 8 2024, 2:40 PM
gerritbot added a comment.Aug 12 2024, 12:09 PM

Change #1060131 merged by Jelto:

[operations/puppet@production] gitlab: enable nft throttling on role level, but just log

https://gerrit.wikimedia.org/r/1060131

Jelto added a comment.Aug 12 2024, 12:16 PM

I enabled the throttling rule for the production GitLab instance in logging-mode (so offending traffic will just be logged and not throttled). @Dzahn let's review the logs and the blocklist tomorrow in our office hours. After the office hours we can disable logging again to prevent issues like T371951.

Dzahn added a comment.Aug 12 2024, 2:55 PM

ACK, sounds like a plan. Same for Gerrit, as you will have seen I upped the threshold 2 times because I was still seeing IPs in the DENYLIST that appeared like legit users. Now though with the higher value I only see googlebot/googleproxy IPs left, at least as of right now. Let's also review this one last time tomorrow and then consider switching it to DROP.

gerritbot added a comment.Aug 15 2024, 1:08 PM

Change #1063004 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] profile::firewall::nftables_throttling: add option for burst packets

https://gerrit.wikimedia.org/r/1063004

gerritbot added a comment.Aug 19 2024, 8:33 AM

Change #1063004 merged by Jelto:

[operations/puppet@production] profile::firewall::nftables_throttling: add option for burst packets

https://gerrit.wikimedia.org/r/1063004

gerritbot added a comment.Aug 21 2024, 8:23 AM

Change #1064328 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: use burst parameter with nftables_throttling

https://gerrit.wikimedia.org/r/1064328

gerritbot added a comment.Aug 21 2024, 9:40 AM

Change #1064328 merged by Jelto:

[operations/puppet@production] gitlab: use burst parameter with nftables_throttling

https://gerrit.wikimedia.org/r/1064328

Jelto added a comment.Aug 21 2024, 2:17 PM

After reviewing the DENYLIST and the nftables logs, we noticed that some GitLab runners outside our infrastructure were getting throttled occasionally (mostly the Digital Ocean Kubernetes Runners). To address this, we added a burst setting to the nftables throttling, similar to what we did with Gerrit. The current setting is up to 300 packets/minute with a burst of 1000.

Since applying the new burst setting, there are no more throttling logs, and the DENYLIST is empty. I'll check the logs again before our next office hours, and we can plan to enable throttling next Tuesday.

Dzahn awarded a token.Aug 21 2024, 3:29 PM
Jelto mentioned this in T373317: SystemdUnitFailed (gitlab1003 nftables).Aug 26 2024, 10:42 AM
gerritbot added a comment.Aug 26 2024, 2:12 PM

Change #1066782 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] profile::firewall::nftables_throttling: fix issue of global metering

https://gerrit.wikimedia.org/r/1066782

gerritbot added a comment.Aug 27 2024, 12:11 PM

Change #1066782 merged by Jelto:

[operations/puppet@production] profile::firewall::nftables_throttling: fix issue of global metering

https://gerrit.wikimedia.org/r/1066782

gerritbot added a comment.Aug 27 2024, 12:32 PM

Change #1067337 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: add profile::prometheus::nft_throttling_denylist

https://gerrit.wikimedia.org/r/1067337

gerritbot added a comment.Aug 27 2024, 2:08 PM

Change #1067337 merged by Jelto:

[operations/puppet@production] gitlab: add profile::prometheus::nft_throttling_denylist

https://gerrit.wikimedia.org/r/1067337

gerritbot added a comment.Sep 2 2024, 12:56 PM

Change #1070025 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: adjust throttling threshold for GitLab

https://gerrit.wikimedia.org/r/1070025

gerritbot added a comment.Sep 3 2024, 7:07 AM

Change #1070025 merged by Jelto:

[operations/puppet@production] gitlab: adjust throttling threshold for GitLab

https://gerrit.wikimedia.org/r/1070025

gerritbot added a comment.Sep 4 2024, 8:43 AM

Change #1058608 merged by Jelto:

[operations/puppet@production] gitlab: enable throttling for all GitLab instances

https://gerrit.wikimedia.org/r/1058608

Jelto added a comment.Sep 4 2024, 9:15 AM

Throttling on all GitLab instances was enabled today. We are dropping packets if there are too many new TCP connections from the same source IP. After 5 minutes they are allowed again. Our own production and cloud networks are excluded from it.

The current threshold is quite high and we have monitored this for a while, so we don't expect it to affect legit users. IPs which are doing more than 300 new TCP connections per minute are blocked for 5 minutes.

In case legitimate users are impacted the following change can be reverted: https://gerrit.wikimedia.org/r/1058608

Jelto updated the task description. (Show Details)Sep 4 2024, 9:16 AM
Maintenance_bot removed a project: Patch-For-Review.Sep 4 2024, 9:31 AM
gerritbot added a comment.Sep 10 2024, 3:28 PM

Change #1071900 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: enable nftables throttling (drop)

https://gerrit.wikimedia.org/r/1071900

gerritbot added a project: Patch-For-Review.Sep 10 2024, 3:28 PM
gerritbot added a comment.Sep 10 2024, 3:35 PM

Change #1071900 merged by Jelto:

[operations/puppet@production] gitlab: enable nftables throttling (drop)

https://gerrit.wikimedia.org/r/1071900

Stashbot added a comment.Sep 10 2024, 3:39 PM

Mentioned in SAL (#wikimedia-operations) [2024-09-10T15:39:00Z] <jelto> enabling throttling on GitLab hosts - T366882

Maintenance_bot removed a project: Patch-For-Review.Sep 10 2024, 4:31 PM
Jelto mentioned this in T374550: Exclude Bitergia IP from Phabricator ratelimiting.Sep 13 2024, 12:14 PM
CodeReviewBot added a project: Patch-For-Review.Sep 16 2024, 12:15 PM

jelto opened https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/67

update replica names

Jelto added a subscriber: dcaro.Sep 18 2024, 8:54 AM

In wikimedia-gitlab, there have been some reports of failing jobs (cc @dcaro), such as:

https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/jobs/363466

fatal: unable to access 'https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api.git/': Failed to connect to gitlab.wikimedia.org port 443 after 4 ms: Could not connect to server

https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/jobs/363488

ERROR: Uploading artifacts as "dotenv" to coordinator... error  error=couldn't execute POST against https://gitlab.wikimedia.org/api/v4/jobs/363488/artifacts?artifact_format=gzip&artifact_type=dotenv: Post "https://gitlab.wikimedia.org/api/v4/jobs/363488/artifacts?artifact_format=gzip&artifact_type=dotenv": dial tcp 208.80.153.8:443: connect: connection refused id=363488 token=glcbt-64
FATAL: invalid argument

https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/jobs/363560

fatal: unable to access 'https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api.git/': Failed to connect to gitlab.wikimedia.org port 443 after 4 ms: Could not connect to server

Interestingly, the metrics and logs don't indicate any throttling at that time:
https://grafana-rw.wikimedia.org/d/R_1IvBZnz/gitlab-omnibus-overview?forceLogin&from=1726531512350&orgId=1&to=1726617571233&viewPanel=51

There was only one instance of throttling, which occurred at 03:23 UTC, but it was for an IP unrelated to Digital Ocean.

The errors seem to occur immediately (failed to connect, connection refused), whereas rate limiting should usually drop packets and cause a timeout instead.

I'll dig a bit further, but we can try disabling throttling again and see if the errors stop happening. Maybe this is also a networking issue in the Digital Ocean Kubernetes cluster.

gerritbot added a comment.Sep 18 2024, 9:09 AM

Change #1073740 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: set throttling policy to accept again

https://gerrit.wikimedia.org/r/1073740

LSobanski mentioned this in T374926: [EPIC][Infra] Move Wikibase and WikibaseLexeme Git submodules to suitable Git host.Sep 18 2024, 9:11 AM
CodeReviewBot added a comment.Sep 19 2024, 9:28 AM

jelto merged https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/67

update replica names

Jelto added a comment.Sep 24 2024, 8:55 AM

I reviewed the throttling in the past 7 days, and noticed that eight unique IPv4 addresses and one IPv6 address were blocked.

The IPs come from smaller hosting providers, VPNs, bots, and two are from Internet Archive. I think this is acceptable for now, even though I’d prefer to get archive.org unblocked. That said, we probably shouldn't invest more time on this until we identify thresholds in a more systematic way: see T374909. Having a few weeks or even months of gaps for archive.org should be fine, in my opinion.

I'll write a short piece for Tech News and make sure the Wikitech docs are up to date. Once that’s done, we should be good to resolve this task.

Jelto updated the task description. (Show Details)Sep 24 2024, 9:56 AM
gerritbot added a comment.Sep 25 2024, 7:52 AM

Change #1075504 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: test defs_from_etcd on the replica

https://gerrit.wikimedia.org/r/1075504

gerritbot added a comment.Sep 25 2024, 11:17 AM

Change #1075504 merged by Jelto:

[operations/puppet@production] gitlab: test defs_from_etcd on the replica

https://gerrit.wikimedia.org/r/1075504

gerritbot added a comment.Sep 25 2024, 11:23 AM

Change #1075533 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] Revert "gitlab: test defs_from_etcd on the replica"

https://gerrit.wikimedia.org/r/1075533

gerritbot added a comment.Sep 25 2024, 11:34 AM

Change #1075533 merged by Jelto:

[operations/puppet@production] Revert "gitlab: test defs_from_etcd on the replica"

https://gerrit.wikimedia.org/r/1075533

Jelto mentioned this in T375613: SystemdUnitFailed (gitlab1003 nftables).Sep 25 2024, 2:44 PM
Jelto added a comment.Sep 27 2024, 12:02 PM

I added a summary of the rate limiting and abuse tooling (including nftables throttling) in Wikitech: https://wikitech.wikimedia.org/wiki/GitLab/Abuse_and_rate_limiting.

Unfortunately the migration to nftables broke the blocked_nets requestctl feature for GitLab. But T348734 can be used to track this issue. I documented it in the wikitech page above as well.

The last thing is to write a small tech news article.

Jelto updated the task description. (Show Details)Oct 1 2024, 3:46 PM
Jelto closed this task as Resolved.Oct 1 2024, 3:52 PM

Throttling is active for around one month on all GitLab instances and works as expected. Documentation is updated and announcements sent. If additional tweaking of thresholds should be needed this task can be reopened or in other tasks. So I'm resolving the task.

gerritbot added a comment.Oct 16 2024, 9:06 AM

Change #1073740 abandoned by Jelto:

[operations/puppet@production] gitlab: set throttling policy to accept again

Reason:

issue did not happen again, change is not needed anymore

https://gerrit.wikimedia.org/r/1073740

Maintenance_bot removed a project: Patch-For-Review.Oct 16 2024, 9:31 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits