Page MenuHomePhabricator
Create Task
Maniphest T399212

nf_conntrack_max is not set at boot in cloudvirts
Closed, ResolvedPublicBUG REPORT
Actions

Description

In cloudvirts, we set some custom values via puppet in /etc/sysctl.d:

cat /etc/sysctl.d/70-nova_conntrack.conf
# sysctl parameters managed by Puppet.
net.netfilter.nf_conntrack_buckets = 8388608
net.netfilter.nf_conntrack_max = 33554432
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 65

These values fail to apply at boot:

root@cloudvirt1067:~# journalctl -t systemd-sysctl
Jun 26 02:54:47 cloudvirt1067 systemd-sysctl[951]: Couldn't write '8388608' to 'net/netfilter/nf_conntrack_buckets', ignoring: No such file or directory
Jun 26 02:54:47 cloudvirt1067 systemd-sysctl[951]: Couldn't write '33554432' to 'net/netfilter/nf_conntrack_max', ignoring: No such file or directory
Jun 26 02:54:47 cloudvirt1067 systemd-sysctl[951]: Couldn't write '65' to 'net/netfilter/nf_conntrack_tcp_timeout_time_wait', ignoring: No such file or directory

It looks like a race condition where systemd tries to apply the values before the nf_conntrack kernel module is loaded. Related upstream issue: https://github.com/systemd/systemd/issues/1113

This is the same issue as T136094: Race condition in setting net.netfilter.nf_conntrack_tcp_timeout_time_wait but cloudvirts don't have the ferm package installed, so the fix from that old task does not work for cloudvirts.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
openstack: nova: Load nf_conntrack module at bootoperations/puppetproduction+11 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

fnegri created this task.Jul 10 2025, 2:53 PM
fnegri mentioned this in T399050: MaxConntrack Max conntrack at 95.11% on cloudvirt1067:9100.
fnegri added a comment.Jul 10 2025, 2:57 PM

I'm not sure what is loading the nf_conntrack module, because the module is loaded eventually, and I can apply the values with sysctl --system:

root@cloudvirt1067:~# cat /proc/sys/net/nf_conntrack_max
524288

root@cloudvirt1067:~# sysctl --system

root@cloudvirt1067:~# cat /proc/sys/net/nf_conntrack_max
33554432
fnegri added a comment.Jul 10 2025, 3:00 PM

Maybe it's loaded by openvswitch:

root@cloudvirt1067:~# lsmod |grep nf_conntrack
nf_conntrack_netlink    57344  0
nfnetlink              20480  3 nfnetlink_cttimeout,nf_conntrack_netlink
nf_conntrack          188416  5 nf_nat,nfnetlink_cttimeout,openvswitch,nf_conntrack_netlink,nf_conncount
nf_defrag_ipv6         24576  2 nf_conntrack,openvswitch
nf_defrag_ipv4         16384  1 nf_conntrack
libcrc32c              16384  4 nf_conntrack,nf_nat,openvswitch,raid456
gerritbot added a comment.Jul 10 2025, 4:11 PM

Change #1167899 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] openstack: nova: Load nf_conntrack module at boot

https://gerrit.wikimedia.org/r/1167899

gerritbot added a project: Patch-For-Review.Jul 10 2025, 4:11 PM
fnegri changed the task status from Open to In Progress.Jul 15 2025, 9:47 AM
fnegri claimed this task.
fnegri triaged this task as Medium priority.
fnegri edited projects, added cloud-services-team (FY2024/2025-Q3-Q4); removed cloud-services-team.
fnegri moved this task from Backlog to In progress on the cloud-services-team (FY2024/2025-Q3-Q4) board.Jul 15 2025, 1:34 PM
fnegri edited projects, added cloud-services-team (FY2025/2026-Q1-Q2); removed cloud-services-team (FY2024/2025-Q3-Q4).Jul 15 2025, 1:38 PM
fnegri moved this task from Backlog to In progress on the cloud-services-team (FY2025/2026-Q1-Q2) board.Jul 15 2025, 1:47 PM
gerritbot added a comment.Jul 16 2025, 9:28 AM

Change #1167899 merged by FNegri:

[operations/puppet@production] openstack: nova: Load nf_conntrack module at boot

https://gerrit.wikimedia.org/r/1167899

Maintenance_bot removed a project: Patch-For-Review.Jul 16 2025, 9:31 AM
Stashbot added a comment.Jul 16 2025, 9:44 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-07-16T09:44:04Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1073.eqiad.wmnet}' (T399212)

Stashbot added a comment.Jul 16 2025, 9:47 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-07-16T09:47:50Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1073.eqiad.wmnet}' (T399212)

fnegri added a comment.Jul 16 2025, 9:48 AM

Merged the patch above and tested that it works by rebooting cloudvirt1073.

Before reboot:

fnegri@cloudvirt1073:~$ sudo cat /proc/sys/net/nf_conntrack_max
524288

After reboot:

fnegri@cloudvirt1073:~$ sudo cat /proc/sys/net/nf_conntrack_max
33554432
fnegri closed this task as Resolved.Jul 16 2025, 9:52 AM
fnegri moved this task from In progress to Done on the cloud-services-team (FY2025/2026-Q1-Q2) board.

I applied the setting on all other cloudvirts without reboot, by running:

sudo cumin 'cloudvirt*' 'sysctl --system'

Verified that it worked with:

fnegri@cloudcumin1001:~$ sudo cumin 'cloudvirt*' 'cat /proc/sys/net/nf_conntrack_max'
43 hosts will be targeted:
cloudvirt[2004-2006]-dev.codfw.wmnet,cloudvirt[1040-1076].eqiad.wmnet,cloudvirtlocal[1001-1003].eqiad.wmnet
OK to proceed on 43 hosts? Enter the number of affected hosts to confirm or "q" to quit: 43
===== NODE GROUP =====
(43) cloudvirt[2004-2006]-dev.codfw.wmnet,cloudvirt[1040-1076].eqiad.wmnet,cloudvirtlocal[1001-1003].eqiad.wmnet
----- OUTPUT of 'cat /proc/sys/net/nf_conntrack_max' -----
33554432
================
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits