Page MenuHomePhabricator

ML Serve controller vms show a slowly increasing resource usage leak over time
Open, MediumPublic

Description

I have recently noticed the following weird CPU pattern for ml-serve-ctrl hosts (ganeti vms) in response to some high latency alerts for the k8s api:

https://grafana.wikimedia.org/d/000000377/host-overview?viewPanel=3&orgId=1&from=1625911186115&to=1626923457301&var-server=ml-serve-ctrl1001&var-datasource=eqiad%20prometheus%2Fops&var-cluster=ml_serve

The increase seems to have started when we deployed kubelets on the controller nodes, to allow routing between workers and them (for webhooks etc..).

From https://grafana.wikimedia.org/d/000000435/kubernetes-api, it seems that several things match the cpu usage trend:

  • k8s api latency increasing
  • etcd latency increasing

From https://grafana.wikimedia.org/d/G8zPL7-Wz/kubernetes-node other things match, mostly metrics related to sync operations for kube-* daemons (missing metrics should become available after https://gerrit.wikimedia.org/r/c/operations/puppet/+/707235.

Last but not the least, on the etcd side I noticed logs like:

Jul 22 15:46:37 ml-etcd1003 etcd[9164]: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:7" took too long (321.038398ms) to execute

The etcd metrics have been added only yesterday (we were missing the config to collect them), and now are available at https://grafana-rw.wikimedia.org/d/Ku6V7QYGz/jayme-etcd3?orgId=1&var-site=codfw&var-cluster=ml_etcd&var-instance_prefix=ml-etcd&from=now-24h&to=now

Things done so far:

  • roll restart of kube daemons
  • removal of istio from eqiad (but codfw follows the same pattern and we haven't deployed istio there)
  • add more ganeti vcores (2->4)
  • moved from DRBD disk template to plain for ml-serve-ctrl1002

The last one is still in progress, I'll check latency/cpu after the weekend to see if anything improved. The rationale for the choice was T224556, basically what we did for the etcd nodes as well. To add the Kubelets I had to mount a /dev/vdb disk (10G) for Docker to all the ml-serve-ctrl vms. What I want to see is if Docker's underlying disk causes lags when replicating somehow, even if what runs on top shouldn't do a ton of fsync-like calls.

The other thing to check is why etcd shows a regression in latency as well. From vm-level metrics I don't see any overhead or clear bottleneck, so I am not sure if it is a client side problem (namely being extremely slow in communicating with etcd) or something else.

Event Timeline

elukey triaged this task as High priority.Jul 23 2021, 10:34 AM
elukey created this task.

Mentioned in SAL (#wikimedia-operations) [2021-07-23T15:11:41Z] <elukey> stop ml-serve-ctrl1001 + gnt-instance modify -t plain ml-serve-ctrl1001.eqiad.wmnet on ganeti1009 + start instance back - T287238

The disk-template change didn't really stop the slow-cpu-leak in eqiad, so I assume that it wasn't the culprit. At this point we should go one level deeper and analyze exactly what causes the cpu increase, otherwise we'll keep trying to shoot in the dark.

Overall, I would proceed in this way:

  • the codfw cluster is still brand new (namely without any istio/knative/etc..) and showing the problem, we could concentrate tests on it.
  • the eqiad cluster shows the issue as well, but we should use it to test the kubeflow step in the meantime.

Possible ways forward:

  • concentrate on cpu usage for kube* daemons on ml-serve-ctrl and see if anything comes up. A quick run of strace for kube-apiserver shows, for example, a lot of <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) and the overall counters for few second of execution of its threads is:
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 75.77    9.036747         923      9787      1390 futex
  7.77    0.926562        2040       454           epoll_pwait
  7.61    0.907891        7092       128           write
  6.45    0.769184         163      4697           nanosleep
  2.33    0.278459         807       345       166 read
  0.04    0.004352          48        89           sched_yield
  0.03    0.003730         746         5         4 restart_syscall
------ ----------- ----------- --------- --------- ----------------
100.00   11.926925                 15505      1560 total
  • Check if etcd is the problem, since from various dashboards the etcd latencies registered by the controller nodes are increasing as well (more or less the same pattern as the cpu)
  • Check if kubelet or what runs on top of it (calico/felix/bird/etc..) are causing this problem. We have kubelets running on Ganeti nodes on other clusters as well (like kubernetes1015) but they are not showing up these issues.
elukey@ml-serve-ctrl2001:~$ ping ml-etcd2001.codfw.wmnet -6
PING ml-etcd2001.codfw.wmnet(ml-etcd2001.codfw.wmnet (2620:0:860:102:10:192:16:44)) 56 data bytes
64 bytes from ml-etcd2001.codfw.wmnet (2620:0:860:102:10:192:16:44): icmp_seq=1 ttl=63 time=0.401 ms
64 bytes from ml-etcd2001.codfw.wmnet (2620:0:860:102:10:192:16:44): icmp_seq=2 ttl=63 time=0.470 ms
64 bytes from ml-etcd2001.codfw.wmnet (2620:0:860:102:10:192:16:44): icmp_seq=3 ttl=63 time=0.433 ms
64 bytes from ml-etcd2001.codfw.wmnet (2620:0:860:102:10:192:16:44): icmp_seq=4 ttl=63 time=0.663 ms
64 bytes from ml-etcd2001.codfw.wmnet (2620:0:860:102:10:192:16:44): icmp_seq=5 ttl=63 time=0.348 ms
^C
--- ml-etcd2001.codfw.wmnet ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 113ms
rtt min/avg/max/mdev = 0.348/0.463/0.663/0.107 ms
elukey@ml-serve-ctrl2001:~$ ping ml-etcd2001.codfw.wmnet -4
PING ml-etcd2001.codfw.wmnet (10.192.16.44) 56(84) bytes of data.
64 bytes from ml-etcd2001.codfw.wmnet (10.192.16.44): icmp_seq=1 ttl=63 time=5.11 ms
64 bytes from ml-etcd2001.codfw.wmnet (10.192.16.44): icmp_seq=2 ttl=63 time=5.14 ms
64 bytes from ml-etcd2001.codfw.wmnet (10.192.16.44): icmp_seq=3 ttl=63 time=4.99 ms
64 bytes from ml-etcd2001.codfw.wmnet (10.192.16.44): icmp_seq=4 ttl=63 time=7.50 ms
64 bytes from ml-etcd2001.codfw.wmnet (10.192.16.44): icmp_seq=5 ttl=63 time=5.54 ms
64 bytes from ml-etcd2001.codfw.wmnet (10.192.16.44): icmp_seq=6 ttl=63 time=4.97 ms

I can reproduce this reliably on multiple nodes (etcd <-> controllers), there is definitely something going on.

Indeed that is odd. I see in a traceroute the latency is high even to the first hop (cr2-codfw Juniper router,) when testing with v4:

cmooney@ml-serve-ctrl2001:~$ sudo traceroute -I 2620:0:860:102:10:192:16:44
traceroute to 2620:0:860:102:10:192:16:44 (2620:0:860:102:10:192:16:44), 30 hops max, 80 byte packets
 1  ae3-2019.cr2-codfw.wikimedia.org (2620:0:860:103:fe00::2)  0.683 ms  0.415 ms  0.437 ms
 2  ml-etcd2001.codfw.wmnet (2620:0:860:102:10:192:16:44)  0.416 ms  0.367 ms  0.340 ms
cmooney@ml-serve-ctrl2001:~$ sudo traceroute -I 10.192.16.44
traceroute to 10.192.16.44 (10.192.16.44), 30 hops max, 60 byte packets
 1  ae3-2019.cr2-codfw.wikimedia.org (10.192.32.3)  5.085 ms  4.486 ms  4.301 ms
 2  ml-etcd2001.codfw.wmnet (10.192.16.44)  3.768 ms  3.449 ms  3.699 ms

Looking at the VM it is running on ganeti2010:

cmooney@ganeti2019:~$ sudo gnt-instance list  | grep ml-serve
ml-serve-ctrl2001.codfw.wmnet   kvm        debootstrap+default ganeti2010.codfw.wmnet running      4.0

The Ganeti instance is connected directly to the private subnet for codfw row C (10.192.132.0/22), same as the VM. It has a Linux bridge defined which captures the physical device (eno1 - connected to Juniper asw in that rack), as well as several tap devices which is I assume how the VMs are connected:

cmooney@ganeti2010:/etc/network$ sudo brctl show private
bridge name	bridge id		STP enabled	interfaces
private		8000.4cd98f6da085	no		eno1
							tap0
							tap1
							tap2

The bridge device has an IP address configured directly on it, so the Ganeti host itself is part of the subnet:

cmooney@ganeti2010:/etc/network$ ip -br -4 addr show private
private          UP             10.192.32.139/22

The "ml-serve-ctrl2001" VM is connected via the "tap1" interface (brctl lists 'port 3', equivalent to the third port it lists above):

cmooney@ganeti2010:/etc/network$ sudo brctl showmacs private | egrep aa:00:00:b7:68:43\|^port
port no	mac addr		is local?	ageing timer
  3	aa:00:00:b7:68:43	no		   0.01

Anyway, if I perform the same traceroute, direct from the Ganeti host, rather than the VM, the latency problem is not there:

cmooney@ganeti2010:/etc/network$ sudo traceroute -I 10.192.16.44
traceroute to 10.192.16.44 (10.192.16.44), 30 hops max, 60 byte packets
 1  ae3-2019.cr2-codfw.wikimedia.org (10.192.32.3)  0.380 ms  0.343 ms  0.357 ms
 2  ml-etcd2001.codfw.wmnet (10.192.16.44)  0.353 ms  0.403 ms  0.403 ms

I do, however, get a really long RTT when pinging the "ml-serve-ctrl2001" VM, directly from the Ganeti machine it's actually running on:

cmooney@ganeti2010:/etc/network$ ping 10.192.32.33
PING 10.192.32.33 (10.192.32.33) 56(84) bytes of data.
64 bytes from 10.192.32.33: icmp_seq=1 ttl=64 time=9.44 ms
64 bytes from 10.192.32.33: icmp_seq=2 ttl=64 time=11.8 ms
64 bytes from 10.192.32.33: icmp_seq=3 ttl=64 time=5.19 ms
64 bytes from 10.192.32.33: icmp_seq=4 ttl=64 time=8.89 ms
64 bytes from 10.192.32.33: icmp_seq=5 ttl=64 time=9.30 ms

Pinging it's IPv6 IP is fine though:

cmooney@ganeti2010:/etc/network$ ping6 2620:0:860:103:10:192:32:33
PING 2620:0:860:103:10:192:32:33(2620:0:860:103:10:192:32:33) 56 data bytes
64 bytes from 2620:0:860:103:10:192:32:33: icmp_seq=1 ttl=64 time=0.325 ms
64 bytes from 2620:0:860:103:10:192:32:33: icmp_seq=2 ttl=64 time=0.196 ms
64 bytes from 2620:0:860:103:10:192:32:33: icmp_seq=3 ttl=64 time=0.238 ms
64 bytes from 2620:0:860:103:10:192:32:33: icmp_seq=4 ttl=64 time=0.357 ms

Both of these show the same MAC address in terms of ARP/ND:

cmooney@ganeti2010:/etc/network$ ip neigh show | grep aa:00:00:b7:68:43
10.192.32.33 dev private lladdr aa:00:00:b7:68:43 STALE
2620:0:860:103:10:192:32:33 dev private lladdr aa:00:00:b7:68:43 router STALE

As seen above this MAC is reachable via tap1, and given it is the same for both I'm not sure why they would be different.

If you do it the other way around, ping the Ganeti host from the VM, the RTT with IPv4 is also bad:

cmooney@ml-serve-ctrl2001:~$ ping 10.192.32.139 -c 2
PING 10.192.32.139 (10.192.32.139) 56(84) bytes of data.
64 bytes from 10.192.32.139: icmp_seq=1 ttl=64 time=7.99 ms
64 bytes from 10.192.32.139: icmp_seq=2 ttl=64 time=5.07 ms

But interestingly if you do a tcpdump on the Ganeti host at that time, the timestamp difference between the request and reply are all less than 1ms (diff is in the nanosecond range):

cmooney@ganeti2010:/etc$ sudo tcpdump -i tap1 -l -nn icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap1, link-type EN10MB (Ethernet), capture size 262144 bytes
18:47:47.234161 IP 10.192.32.33 > 10.192.32.139: ICMP echo request, id 29647, seq 1, length 64
18:47:47.234229 IP 10.192.32.139 > 10.192.32.33: ICMP echo reply, id 29647, seq 1, length 64
18:47:48.233840 IP 10.192.32.33 > 10.192.32.139: ICMP echo request, id 29647, seq 2, length 64
18:47:48.233908 IP 10.192.32.139 > 10.192.32.33: ICMP echo reply, id 29647, seq 2, length 64

So unfortunately I can't say much other than the issue seems to be within the Ganeti host. Might need to see if someone with better knowledge on that setup can help us, I'm not sure how those "tap" interfaces get created. Certainly odd that the IPv6 is so much better though - you'd expect most low-level problems to affect both.

Sorry to clog this up with junk, but this test is relevant. When pinging from the Ganeti host to the VM, if you do a TCPdump on the VM, you can see it is the one that takes the long time to respond (i.e. VM gets request packet, takes 10ms or so to send response):

cmooney@ml-serve-ctrl2001:~$ sudo tcpdump -i ens6 -l -p -nn host 10.192.32.139
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens6, link-type EN10MB (Ethernet), capture size 262144 bytes
19:06:32.617356 IP 10.192.32.139 > 10.192.32.33: ICMP echo request, id 2506, seq 1, length 64
19:06:32.626786 IP 10.192.32.33 > 10.192.32.139: ICMP echo reply, id 2506, seq 1, length 64
19:06:33.619208 IP 10.192.32.139 > 10.192.32.33: ICMP echo request, id 2506, seq 2, length 64
19:06:33.629997 IP 10.192.32.33 > 10.192.32.139: ICMP echo reply, id 2506, seq 2, length 64

My guess is the difference between IPv6 and IPv4 is the difference in the number of netfilter rules on this end VM for each protocol:

cmooney@ml-serve-ctrl2001:~$ sudo ip6tables -L -v --line -n | wc -l
148
cmooney@ml-serve-ctrl2001:~$ sudo iptables -L -v --line -n | wc -l
17626

17,000 rules is rather a lot! I think the issue is that the "KUBE-FIREWALL" chain has the same rule over and over and over again:

cmooney@ml-serve-ctrl2001:~$ sudo iptables -L KUBE-FIREWALL -v -n | wc -l
17498
cmooney@ml-serve-ctrl2001:~$ sudo iptables -L KUBE-FIREWALL -v -n | sort | uniq | wc -l
4

These two lines show over and over again:

0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x8000/0x8000 /* kubernetes firewall for dropping marked packets */
0     0 DROP       all  --  *      *      !127.0.0.0/8          127.0.0.0/8          ! ctstate RELATED,ESTABLISHED,DNAT /* block incoming localnet connections */

@cmooney awesome analysis, thanks a lot!

The issue seems to be https://github.com/kubernetes/kubernetes/issues/82361. It is hitting only Debian Buster nodes:

elukey@cumin1001:~$ sudo cumin 'c:profile::kubernetes::node' 'cat /etc/debian_version'
50 hosts will be targeted:
kubernetes[2001-2017].codfw.wmnet,kubernetes[1001-1017].eqiad.wmnet,kubestage[2001-2002].codfw.wmnet,kubestage[1001-1002].eqiad.wmnet,ml-serve[2001-2004].codfw.wmnet,ml-serve[1001-1004].eqiad.wmnet,ml-serve-ctrl[2001-2002].codfw.wmnet,ml-serve-ctrl[1001-1002].eqiad.wmnet
Ok to proceed on 50 hosts? Enter the number of affected hosts to confirm or "q" to quit 50
===== NODE GROUP =====                                                                                                                   
(12) ml-serve[2001-2004].codfw.wmnet,ml-serve[1001-1004].eqiad.wmnet,ml-serve-ctrl[2001-2002].codfw.wmnet,ml-serve-ctrl[1001-1002].eqiad.wmnet                                                                                                                                    
----- OUTPUT of 'cat /etc/debian_version' -----                                                                                          
10.10                                                                                                                                    
===== NODE GROUP =====                                                                                                                   
(38) kubernetes[2001-2017].codfw.wmnet,kubernetes[1001-1017].eqiad.wmnet,kubestage[2001-2002].codfw.wmnet,kubestage[1001-1002].eqiad.wmnet                                                                                                                                        
----- OUTPUT of 'cat /etc/debian_version' -----                                                                                          
9.13

And the ML team is the only one running kubelets on Debian Buster afaics! The version of iptables matches the one mentioned in the github issue:

elukey@cumin1001:~$ sudo cumin 'c:profile::kubernetes::node' 'dpkg -l  iptables'
50 hosts will be targeted:
kubernetes[2001-2017].codfw.wmnet,kubernetes[1001-1017].eqiad.wmnet,kubestage[2001-2002].codfw.wmnet,kubestage[1001-1002].eqiad.wmnet,ml-serve[2001-2004].codfw.wmnet,ml-serve[1001-1004].eqiad.wmnet,ml-serve-ctrl[2001-2002].codfw.wmnet,ml-serve-ctrl[1001-1002].eqiad.wmnet
Ok to proceed on 50 hosts? Enter the number of affected hosts to confirm or "q" to quit 50
===== NODE GROUP =====                                                                                                                   
(12) ml-serve[2001-2004].codfw.wmnet,ml-serve[1001-1004].eqiad.wmnet,ml-serve-ctrl[2001-2002].codfw.wmnet,ml-serve-ctrl[1001-1002].eqiad.wmnet                                                                                                                                    
----- OUTPUT of 'dpkg -l  iptables' -----                                                                                                
Desired=Unknown/Install/Remove/Purge/Hold                                                                                                
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend                                                           
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-=================================================
ii  iptables       1.8.2-4      amd64        administration tools for packet filtering and NAT
===== NODE GROUP =====                                                                                                                   
(38) kubernetes[2001-2017].codfw.wmnet,kubernetes[1001-1017].eqiad.wmnet,kubestage[2001-2002].codfw.wmnet,kubestage[1001-1002].eqiad.wmnet                                                                                                                                        
----- OUTPUT of 'dpkg -l  iptables' -----                                                                                                
Desired=Unknown/Install/Remove/Purge/Hold                                                                                                
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend                                                           
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version                  Architecture Description
+++-==============-========================-============-=================================================
ii  iptables       1.6.0+snapshot20161117-6 amd64        administration tools for packet filtering and NAT

I am going to test iptables from buster backports: https://packages.debian.org/buster-backports/i386/iptables

Mentioned in SAL (#wikimedia-operations) [2021-07-27T06:50:18Z] <elukey> install iptables from buster-backports (manually) on ml-serve-ctrl200[1,2] as test (+ reboot the nodes for a clean start) - T287238

The issue is also present on worker nodes:

elukey@ml-serve2001:~$ sudo iptables -L KUBE-FIREWALL -v -n | wc -l
38866

I didn't notice any weird cpu pattern though, it seems affecting more Ganeti VMs. We should fix the problem there anyway :)

Change 708258 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] WIP profile::kubernetes::node: deploy iptables from buster-backports

https://gerrit.wikimedia.org/r/708258

Change 708265 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] aptrepo: add a component for iptables to wikimedia-buster

https://gerrit.wikimedia.org/r/708265

Change 708265 merged by Elukey:

[operations/puppet@production] aptrepo: add a component for iptables to wikimedia-buster

https://gerrit.wikimedia.org/r/708265

Change 708258 merged by Elukey:

[operations/puppet@production] profile::kubernetes::node: add component/iptables for Buster

https://gerrit.wikimedia.org/r/708258

Change 708297 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::kubernetes::node: add netbase to iptables dependencies

https://gerrit.wikimedia.org/r/708297

Change 708297 merged by Elukey:

[operations/puppet@production] profile::kubernetes::node: add netbase to iptables dependencies

https://gerrit.wikimedia.org/r/708297

Mentioned in SAL (#wikimedia-operations) [2021-07-27T14:29:18Z] <elukey> reduce vcores for ml-serve-ctrl[12]00[12] after performance testing - T287238

Mentioned in SAL (#wikimedia-operations) [2021-07-27T15:42:25Z] <elukey> add disk_template drbd back to ml-serve-ctrl100[12] vms after performance testing - T287238

Deployed the new iptables to all ML buster clusters, preliminary results look really good. Will wait for a day before calling this a victory.

Deployed the new iptables to all ML buster clusters, preliminary results look really good. Will wait for a day before calling this a victory.

Latency and cpu usage look stable after the change, I think that we are good!

One nit - I had to manually install iptables and related dependencies manually even after https://gerrit.wikimedia.org/r/c/operations/puppet/+/708258/8/modules/profile/manifests/kubernetes/node.pp, since the iptables packages are shipped (IIUC) from the base debian install and ensure_packages do not try to upgrade them to the new version. This is not great when we reimage nodes, I am pretty sure that we'll forget to run apt-get manually, it would be great to find a way in puppet to do it transparently (namely to force it to install iptables packages from the component/iptables185 regardless of packages already installed on the host).

Deployed the new iptables to all ML buster clusters, preliminary results look really good. Will wait for a day before calling this a victory.

Latency and cpu usage look stable after the change, I think that we are good!

One nit - I had to manually install iptables and related dependencies manually even after https://gerrit.wikimedia.org/r/c/operations/puppet/+/708258/8/modules/profile/manifests/kubernetes/node.pp, since the iptables packages are shipped (IIUC) from the base debian install and ensure_packages do not try to upgrade them to the new version. This is not great when we reimage nodes, I am pretty sure that we'll forget to run apt-get manually, it would be great to find a way in puppet to do it transparently (namely to force it to install iptables packages from the component/iptables185 regardless of packages already installed on the host).

We can amend the reimage scripts that it runs apt-get upgrade before the final reboot, this would fix it in generic manner.

But for this specific issue it would be best anyway to get it fixed in Debian proper, we only need to isolated the fix from 1.8.3, backport it for 1.8.2 and submit it for inclusion in the next point release (or we add it to apt.wikimedia.org in the mean time).

I had to manually install iptables and related dependencies manually even after https://gerrit.wikimedia.org/r/c/operations/puppet/+/708258/8/modules/profile/manifests/kubernetes/node.pp,

I also notice that not all libraries from the the packages array have been upgraded

$ apt-cache policy iptables libip4tc0 libip6tc0 libiptc0 libxtables12 libmnl0 libnetfilter-conntrack3 libnfnetlink0 libnftnl11 netbase
iptables:
  Installed: 1.8.5-3~bpo10+1
  Candidate: 1.8.5-3~bpo10+1
  Version table:
 *** 1.8.5-3~bpo10+1 1001
        100 http://mirrors.wikimedia.org/debian buster-backports/main amd64 Packages
       1001 http://apt.wikimedia.org/wikimedia buster-wikimedia/component/iptables185 amd64 Packages
        100 /var/lib/dpkg/status
     1.8.2-4 500
        500 http://mirrors.wikimedia.org/debian buster/main amd64 Packages
libip4tc0:
  Installed: 1.8.2-4
  Candidate: 1.8.2-4
  Version table:
 *** 1.8.2-4 500
        500 http://mirrors.wikimedia.org/debian buster/main amd64 Packages
        100 /var/lib/dpkg/status
libip6tc0:
  Installed: 1.8.2-4
  Candidate: 1.8.2-4
  Version table:
 *** 1.8.2-4 500
        500 http://mirrors.wikimedia.org/debian buster/main amd64 Packages
        100 /var/lib/dpkg/status
libiptc0:
  Installed: 1.8.2-4
  Candidate: 1.8.5-3~bpo10+1
  Version table:
     1.8.5-3~bpo10+1 1001
        100 http://mirrors.wikimedia.org/debian buster-backports/main amd64 Packages
       1001 http://apt.wikimedia.org/wikimedia buster-wikimedia/component/iptables185 amd64 Packages
 *** 1.8.2-4 500
        500 http://mirrors.wikimedia.org/debian buster/main amd64 Packages
        100 /var/lib/dpkg/status
libxtables12:
  Installed: 1.8.5-3~bpo10+1
  Candidate: 1.8.5-3~bpo10+1
  Version table:
 *** 1.8.5-3~bpo10+1 1001
        100 http://mirrors.wikimedia.org/debian buster-backports/main amd64 Packages
       1001 http://apt.wikimedia.org/wikimedia buster-wikimedia/component/iptables185 amd64 Packages
        100 /var/lib/dpkg/status
     1.8.2-4 500
        500 http://mirrors.wikimedia.org/debian buster/main amd64 Packages
libmnl0:
  Installed: 1.0.4-2
  Candidate: 1.0.4-2
  Version table:
 *** 1.0.4-2 500
        500 http://mirrors.wikimedia.org/debian buster/main amd64 Packages
        100 /var/lib/dpkg/status
libnetfilter-conntrack3:
  Installed: 1.0.7-1
  Candidate: 1.0.7-1
  Version table:
 *** 1.0.7-1 500
        500 http://mirrors.wikimedia.org/debian buster/main amd64 Packages
        100 /var/lib/dpkg/status
libnfnetlink0:
  Installed: 1.0.1-3+b1
  Candidate: 1.0.1-3+b1
  Version table:
 *** 1.0.1-3+b1 500
        500 http://mirrors.wikimedia.org/debian buster/main amd64 Packages
        100 /var/lib/dpkg/status
libnftnl11:
  Installed: 1.1.7-1~bpo10+1
  Candidate: 1.1.7-1~bpo10+1
  Version table:
 *** 1.1.7-1~bpo10+1 1001
        100 http://mirrors.wikimedia.org/debian buster-backports/main amd64 Packages
       1001 http://apt.wikimedia.org/wikimedia buster-wikimedia/component/iptables185 amd64 Packages
        100 /var/lib/dpkg/status
     1.1.2-2 500
        500 http://mirrors.wikimedia.org/debian buster/main amd64 Packages
netbase:
  Installed: 6.1~bpo10+1
  Candidate: 6.1~bpo10+1
  Version table:
 *** 6.1~bpo10+1 1001
        100 http://mirrors.wikimedia.org/debian buster-backports/main amd64 Packages
       1001 http://apt.wikimedia.org/wikimedia buster-wikimedia/component/iptables185 amd64 Packages
        100 /var/lib/dpkg/status
     5.6 500
        500 http://mirrors.wikimedia.org/debian buster/main amd64 Packages

not sure if this is a problem but thought i should mention it. In relation to the fix aside from moritz suggestion. in puppet you can pass a specific version to the ensure parameter. so we could potentially oiverload how the packages array works and allow users to also specify a version. We could do this in two ways:

  1. update the packages array to accept entries like [$package_default_ensure, $package_with_version==1.1.7-1~bpo10+1]
  2. update the packages parameter to accept Variant[Array,Hash[string, String]]. which would allow both the following:
apt::package_from_component { 'foobar':
  component => 'component/foobar',
  packages => ['foo', 'bar']
}
apt::package_from_component { 'foobar':
  component => 'component/foobar',
  packages => { 'foo' => 'present', 'bar' => '1.1.7-1~bpo10+1'}
}

I had to manually install iptables and related dependencies manually even after https://gerrit.wikimedia.org/r/c/operations/puppet/+/708258/8/modules/profile/manifests/kubernetes/node.pp,

I also notice that not all libraries from the the packages array have been upgraded

$ apt-cache policy iptables libip4tc0 libip6tc0 libiptc0 libxtables12 libmnl0 libnetfilter-conntrack3 libnfnetlink0 libnftnl11 netbase
iptables:
  Installed: 1.8.5-3~bpo10+1
  Candidate: 1.8.5-3~bpo10+1
  Version table:
 *** 1.8.5-3~bpo10+1 1001
        100 http://mirrors.wikimedia.org/debian buster-backports/main amd64 Packages
       1001 http://apt.wikimedia.org/wikimedia buster-wikimedia/component/iptables185 amd64 Packages
        100 /var/lib/dpkg/status
     1.8.2-4 500
        500 http://mirrors.wikimedia.org/debian buster/main amd64 Packages
libip4tc0:
  Installed: 1.8.2-4
  Candidate: 1.8.2-4
  Version table:
 *** 1.8.2-4 500
        500 http://mirrors.wikimedia.org/debian buster/main amd64 Packages
        100 /var/lib/dpkg/status
libip6tc0:
  Installed: 1.8.2-4
  Candidate: 1.8.2-4
  Version table:
 *** 1.8.2-4 500
        500 http://mirrors.wikimedia.org/debian buster/main amd64 Packages
        100 /var/lib/dpkg/status
libiptc0:
  Installed: 1.8.2-4
  Candidate: 1.8.5-3~bpo10+1
  Version table:
     1.8.5-3~bpo10+1 1001
        100 http://mirrors.wikimedia.org/debian buster-backports/main amd64 Packages
       1001 http://apt.wikimedia.org/wikimedia buster-wikimedia/component/iptables185 amd64 Packages
 *** 1.8.2-4 500
        500 http://mirrors.wikimedia.org/debian buster/main amd64 Packages
        100 /var/lib/dpkg/status
libxtables12:
  Installed: 1.8.5-3~bpo10+1
  Candidate: 1.8.5-3~bpo10+1
  Version table:
 *** 1.8.5-3~bpo10+1 1001
        100 http://mirrors.wikimedia.org/debian buster-backports/main amd64 Packages
       1001 http://apt.wikimedia.org/wikimedia buster-wikimedia/component/iptables185 amd64 Packages
        100 /var/lib/dpkg/status
     1.8.2-4 500
        500 http://mirrors.wikimedia.org/debian buster/main amd64 Packages
libmnl0:
  Installed: 1.0.4-2
  Candidate: 1.0.4-2
  Version table:
 *** 1.0.4-2 500
        500 http://mirrors.wikimedia.org/debian buster/main amd64 Packages
        100 /var/lib/dpkg/status
libnetfilter-conntrack3:
  Installed: 1.0.7-1
  Candidate: 1.0.7-1
  Version table:
 *** 1.0.7-1 500
        500 http://mirrors.wikimedia.org/debian buster/main amd64 Packages
        100 /var/lib/dpkg/status
libnfnetlink0:
  Installed: 1.0.1-3+b1
  Candidate: 1.0.1-3+b1
  Version table:
 *** 1.0.1-3+b1 500
        500 http://mirrors.wikimedia.org/debian buster/main amd64 Packages
        100 /var/lib/dpkg/status
libnftnl11:
  Installed: 1.1.7-1~bpo10+1
  Candidate: 1.1.7-1~bpo10+1
  Version table:
 *** 1.1.7-1~bpo10+1 1001
        100 http://mirrors.wikimedia.org/debian buster-backports/main amd64 Packages
       1001 http://apt.wikimedia.org/wikimedia buster-wikimedia/component/iptables185 amd64 Packages
        100 /var/lib/dpkg/status
     1.1.2-2 500
        500 http://mirrors.wikimedia.org/debian buster/main amd64 Packages
netbase:
  Installed: 6.1~bpo10+1
  Candidate: 6.1~bpo10+1
  Version table:
 *** 6.1~bpo10+1 1001
        100 http://mirrors.wikimedia.org/debian buster-backports/main amd64 Packages
       1001 http://apt.wikimedia.org/wikimedia buster-wikimedia/component/iptables185 amd64 Packages
        100 /var/lib/dpkg/status
     5.6 500
        500 http://mirrors.wikimedia.org/debian buster/main amd64 Packages

not sure if this is a problem but thought i should mention it.

Thanks! I found only one package not upgraded, the rest of the versions seem the same between buster main and backports (lemme know if you find something weird).

In relation to the fix aside from moritz suggestion. in puppet you can pass a specific version to the ensure parameter. so we could potentially oiverload how the packages array works and allow users to also specify a version. We could do this in two ways:

  1. update the packages array to accept entries like [$package_default_ensure, $package_with_version==1.1.7-1~bpo10+1]
  2. update the packages parameter to accept Variant[Array,Hash[string, String]]. which would allow both the following:
apt::package_from_component { 'foobar':
  component => 'component/foobar',
  packages => ['foo', 'bar']
}
apt::package_from_component { 'foobar':
  component => 'component/foobar',
  packages => { 'foo' => 'present', 'bar' => '1.1.7-1~bpo10+1'}
}

This could be an option yes, as interim solution!

This variant looks good to me, but it should be used rarely and with caution (since it effectively ties software updates to a git commit bumping the version):

apt::package_from_component { 'foobar':
  component => 'component/foobar',
  packages => { 'foo' => 'present', 'bar' => '1.1.7-1~bpo10+1'}
}

Change 708746 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] R:apt::package_from_component: add more flexible packages param

https://gerrit.wikimedia.org/r/708746

Change 708746 merged by Jbond:

[operations/puppet@production] R:apt::package_from_component: add more flexible packages param

https://gerrit.wikimedia.org/r/708746

This variant looks good to me, but it should be used rarely and with caution (since it effectively ties software updates to a git commit bumping the version):

apt::package_from_component { 'foobar':
  component => 'component/foobar',
  packages => { 'foo' => 'present', 'bar' => '1.1.7-1~bpo10+1'}
}

This is now merged

elukey lowered the priority of this task from High to Medium.Aug 6 2021, 9:36 AM