Page MenuHomePhabricator
Create Task
Maniphest T400971

Troubleshoot GitLab nftables throttling after switchover
Closed, ResolvedPublic
Actions

Description

After the switchover in T400252: Gitlab switchover (gitlab2002 → gitlab1004) there were reports of legitimate users being blocked by the nftables throttling, see T400252#11046794 for example. Also when comparing the size of the deny list on the old host and on the new host there are more IPs blocked on the new host.

This could be related to the new IP and DNS entry and invalidated browser caches. So clients have to request more resource which would be cached normally. But more troubleshooting and a more reasonable threshold is needed.

It could make sense to switch the throttling to monitoring/logging instead of blocking to review the size of the deny list. Currently throttling is disabled to unblock users (see this change).

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
gitlab: throttling policy toggleoperations/puppetproduction+1 -1
gitlab: nftables monitoring new thresholdsoperations/puppetproduction+3 -3
gitlab: nftables monitoring new thresholdsoperations/puppetproduction+3 -3
nftables: throttle debuggingoperations/puppetproduction+48 -18
nftables: throttle debuggingoperations/puppetproduction+24 -6
gitlab: disable nftables throttling temporarilyoperations/puppetproduction+1 -1
gitlab: raise throttling thresholdsoperations/puppetproduction+2 -2
gitlab: adjust nftables throttling thresholdsoperations/puppetproduction+4 -4
gitlab: enable nftables throttling again in monitoring modeoperations/puppetproduction+2 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

Jelto created this task.Aug 1 2025, 7:31 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 1 2025, 7:31 AM
gerritbot added a comment.Aug 1 2025, 8:31 AM

Change #1175043 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: enable nftables throttling again in monitoring mode

https://gerrit.wikimedia.org/r/1175043

gerritbot added a project: Patch-For-Review.Aug 1 2025, 8:31 AM
gerritbot added a comment.Aug 1 2025, 8:37 AM

Change #1175043 merged by Jelto:

[operations/puppet@production] gitlab: enable nftables throttling again in monitoring mode

https://gerrit.wikimedia.org/r/1175043

Jelto moved this task from Incoming to Work in Progress on the collaboration-services board.Aug 1 2025, 9:00 AM
Maintenance_bot removed a project: Patch-For-Review.Aug 1 2025, 9:30 AM
Jelto triaged this task as Medium priority.Aug 4 2025, 1:10 PM

I spot-checked the IPs which would currently be blocked on gitlab1004 and all of them are legitimate users or runners as far as I can tell.

Also the version of nftables changed from bullseye to buster from 0.9 to 1.0 which might have changed how the metering works. So we have to find new thresholds for 1.0 on bookworm.

This is probably also relevant for Gerrit in T392464: Upgrade Gerrit hosts from Bullseye to Bookworm.

Jelto mentioned this in T387833: Gerrit switchover process.Aug 4 2025, 2:40 PM
Dzahn awarded a token.Aug 4 2025, 4:29 PM
gerritbot added a comment.Aug 6 2025, 1:47 PM

Change #1176246 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: adjust nftables throttling thresholds

https://gerrit.wikimedia.org/r/1176246

gerritbot added a project: Patch-For-Review.Aug 6 2025, 1:47 PM
gerritbot added a comment.Aug 7 2025, 12:56 PM

Change #1176246 merged by Jelto:

[operations/puppet@production] gitlab: adjust nftables throttling thresholds

https://gerrit.wikimedia.org/r/1176246

Maintenance_bot removed a project: Patch-For-Review.Aug 7 2025, 1:31 PM
Jelto added a comment.Aug 11 2025, 7:30 AM

Unfortunately one of the Digital Ocean Runner IPs is throttled quite often. I'll test higher thresholds.

gerritbot added a comment.Aug 11 2025, 12:17 PM

Change #1177363 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: raise throttling thresholds

https://gerrit.wikimedia.org/r/1177363

gerritbot added a project: Patch-For-Review.Aug 11 2025, 12:17 PM
gerritbot added a comment.Aug 11 2025, 12:39 PM

Change #1177363 merged by Jelto:

[operations/puppet@production] gitlab: raise throttling thresholds

https://gerrit.wikimedia.org/r/1177363

Maintenance_bot removed a project: Patch-For-Review.Aug 11 2025, 1:31 PM
gerritbot added a comment.Aug 11 2025, 2:03 PM

Change #1177405 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: disable nftables throttling temporarily

https://gerrit.wikimedia.org/r/1177405

gerritbot added a project: Patch-For-Review.Aug 11 2025, 2:03 PM
gerritbot added a comment.Aug 11 2025, 2:06 PM

Change #1177405 merged by Jelto:

[operations/puppet@production] gitlab: disable nftables throttling temporarily

https://gerrit.wikimedia.org/r/1177405

Maintenance_bot removed a project: Patch-For-Review.Aug 11 2025, 2:31 PM
Milimetric subscribed.Aug 11 2025, 3:23 PM

I know of a few examples of gitlab packaging jobs failing but here's one that I tried to run and know more about: https://gitlab.wikimedia.org/repos/product-analytics/experimentation-lab/experiment-analytics-jobs/-/jobs/583088

The error was

curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)

Jelto added a subscriber: dcaro.Aug 12 2025, 6:37 AM
In T400971#11074365, @Milimetric wrote:

I know of a few examples of gitlab packaging jobs failing but here's one that I tried to run and know more about: https://gitlab.wikimedia.org/repos/product-analytics/experimentation-lab/experiment-analytics-jobs/-/jobs/583088

The error was

curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)

Thanks for reporting this! Similar issues have been reported in wikimedia-gitlab IRC channel by @dcaro. The throttling has been disabled a few minutes after the job failed. Could you retry and let me know when the error appears again?

ABran-WMF subscribed.Aug 12 2025, 9:21 AM
Milimetric added a comment.Aug 12 2025, 9:39 AM

@Jelto : sorry, should've mentioned, yes we retried and it worked with the throttling disabled.

gerritbot added a comment.Aug 12 2025, 9:42 AM

Change #1177956 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] nftables: throttle debugging

https://gerrit.wikimedia.org/r/1177956

gerritbot added a project: Patch-For-Review.Aug 12 2025, 9:43 AM
Milimetric unsubscribed.Aug 12 2025, 9:43 AM
Jelto mentioned this in T401535: Uploading artifacts does not work anymore.Aug 13 2025, 6:32 AM
Jelto added a subtask: T401535: Uploading artifacts does not work anymore.
Don-vip subscribed.Aug 13 2025, 11:09 AM
gerritbot added a comment.Aug 14 2025, 8:29 AM

Change #1177956 merged by Arnaudb:

[operations/puppet@production] nftables: throttle debugging

https://gerrit.wikimedia.org/r/1177956

Maintenance_bot removed a project: Patch-For-Review.Aug 14 2025, 8:31 AM
gerritbot added a comment.Aug 14 2025, 1:40 PM

Change #1178880 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] nftables: throttle debugging

https://gerrit.wikimedia.org/r/1178880

gerritbot added a project: Patch-For-Review.Aug 14 2025, 1:40 PM
Jelto reassigned this task from Jelto to ABran-WMF.Aug 14 2025, 1:49 PM

I'll hand this task over to @ABran-WMF while I'm out. Arnaud already started with troubleshooting this and finding a new set of rules and thresholds for the new bookworm hosts.

gerritbot added a comment.Aug 20 2025, 8:54 AM

Change #1178880 merged by Arnaudb:

[operations/puppet@production] nftables: throttle debugging

https://gerrit.wikimedia.org/r/1178880

ABran-WMF added a comment.Aug 20 2025, 9:15 AM

the last iteration does not seem to have a negative impact on Gerrit and has reduced gitlab1004 DENYLIST to a normal threshold. Graph of the moment where the new configuration is applied: https://grafana.wikimedia.org/goto/hVSdMVuNg?orgId=1
I'll monitore Gitlab's thresholds efficiency, and revert the policy to drop.

Maintenance_bot removed a project: Patch-For-Review.Aug 20 2025, 9:31 AM
gerritbot added a comment.Aug 20 2025, 1:54 PM

Change #1180569 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gitlab: nftables monitoring new thresholds

https://gerrit.wikimedia.org/r/1180569

gerritbot added a project: Patch-For-Review.Aug 20 2025, 1:54 PM
gerritbot added a comment.Aug 20 2025, 1:59 PM

Change #1180569 merged by Arnaudb:

[operations/puppet@production] gitlab: nftables monitoring new thresholds

https://gerrit.wikimedia.org/r/1180569

Maintenance_bot removed a project: Patch-For-Review.Aug 20 2025, 2:31 PM
gerritbot added a comment.Aug 20 2025, 2:55 PM

Change #1180580 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gitlab: nftables monitoring new thresholds

https://gerrit.wikimedia.org/r/1180580

gerritbot added a project: Patch-For-Review.Aug 20 2025, 2:55 PM
gerritbot added a comment.Aug 20 2025, 3:01 PM

Change #1180580 merged by Arnaudb:

[operations/puppet@production] gitlab: nftables monitoring new thresholds

https://gerrit.wikimedia.org/r/1180580

Maintenance_bot removed a project: Patch-For-Review.Aug 20 2025, 3:31 PM
gerritbot added a comment.Aug 21 2025, 5:02 AM

Change #1180703 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gitlab: throttling policy toggle

https://gerrit.wikimedia.org/r/1180703

gerritbot added a project: Patch-For-Review.Aug 21 2025, 5:02 AM
gerritbot added a comment.Aug 21 2025, 5:04 AM

Change #1180703 merged by Arnaudb:

[operations/puppet@production] gitlab: throttling policy toggle

https://gerrit.wikimedia.org/r/1180703

ABran-WMF closed this task as Resolved.Aug 21 2025, 5:09 AM

this troubleshooting can be considered done, I'll keep on monitoring tracking/denylist for the next few days to adjust thresholds as needed

Maintenance_bot removed a project: Patch-For-Review.Aug 21 2025, 5:30 AM
ABran-WMF added a subtask: T402847: ProbeDown - gerrit1003.Aug 26 2025, 6:46 AM
ABran-WMF added a subtask: Restricted Task.
ABran-WMF closed subtask Restricted Task as Resolved.Sep 18 2025, 12:21 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits