Mentioned in SAL (#wikimedia-cloud) [2026-07-17T19:53:18Z] <andrewbogott> hard rebooted logging-logstash-03 and logging-logstash-04 for T432426

Mentioned in SAL (#wikimedia-cloud) [2026-07-17T19:54:56Z] <andrewbogott> hard rebooting demo-patrollers and reader-embedding, reporting as down. Cleanup for T432426

Mentioned in SAL (#wikimedia-cloud) [2026-07-17T19:56:00Z] <andrewbogott> hard rebooting wsexport-calibre01, T432426

There are lots of reasons for a host to become unreachable, but falling off the network is one of them. This dashboard is a good place to start:

https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown

Let's keep an eye on that and see if we detect more 'fell of network' examples. I acked all the false positives currently on that dash, and will continue to do so for hosts that are showing as down but still have network connectivity.

This just happened again for gitlab-1002.devtools.eqiad1.wikimedia.cloud

first sign of 'network unreachable' is at 2026-07-21T08:12:50.345128+00:00

It seems to have no ipv4 assigned currently:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:08:7d:8a brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    inet6 2a02:ec80:a000:1::285/64 scope global deprecated 
       valid_lft forever preferred_lft 0sec
    inet6 fe80::f816:3eff:fe08:7d8a/64 scope link 
       valid_lft forever preferred_lft forever

dhclient was not running, no evidence of a crash or other things in syslog. this is normal, dhclient is not supposed to run on bookworm or trixie

Starting dhclient caused a full recovery on the host.

This may be recurring on logging-logstash-04. Taavi (and now I) suspect that the issue is with systemd-networkd being unable to reach the dhcp server when renewing leases.

Now running in debug mode (systemctl edit systemd-networkd.service; add a 'Environment=SYSTEMD_LOG_LEVEL=debug' in the [Service] block)

This is a close fit for what we're seeing: https://github.com/systemd/systemd/issues/25441 (h/t to @taavi who found that)

I asked @Jelto about one of his affected hosts and he says "But occasionally some of our WMCS instance lose network connectivity and I just "soft reboot" them from the horizon menu. This fixes the issue most of the time"

So this may have been happening for a while :(

Change #1314020 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloud-vps: override SYSTEMD_NETLINK_DEFAULT_TIMEOUT on all instances

https://gerrit.wikimedia.org/r/1314020

In T432426#12143509, @Andrew wrote:

This may be recurring on logging-logstash-04. Taavi (and now I) suspect that the issue is with systemd-networkd being unable to reach the dhcp server when renewing leases.

Now running in debug mode (systemctl edit systemd-networkd.service; add a 'Environment=SYSTEMD_LOG_LEVEL=debug' in the [Service] block)

Related:

In T432254#12148225, @Stashbot wrote:

Mentioned in SAL (#wikimedia-cloud) [2026-07-22T22:24:24Z] <bd808> logging-logstash-04 hard reboot via Horizon (T432254)

Change #1314020 merged by Andrew Bogott:

[operations/puppet@production] cloud-vps: override SYSTEMD_NETLINK_DEFAULT_TIMEOUT on all instances

https://gerrit.wikimedia.org/r/1314020

Change #1318256 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloud-vps: add OS version guard around systemd-networkd override

https://gerrit.wikimedia.org/r/1318256

Change #1318256 merged by Andrew Bogott:

[operations/puppet@production] cloud-vps: add OS version guard around systemd-networkd override

https://gerrit.wikimedia.org/r/1318256

Change #1318779 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloud-vps: kick networkd if network breaks

https://gerrit.wikimedia.org/r/1318779

@Andrew has logging-logstash-04 remained stable or do we have logging data of a new failure?

additionally were there any more instances of this happening on Trixie hosts after the patch you pushed?

The latest:

• I just now found a Trixie host without an IP. I don't know for certain that it broke post-patch but I fear the worst. We should keep an eye out for more examples.

• T433173 makes it very annoying to watch for broken VMs. I've spent some time fixing things there but there are still a bunch of false positives -- I've at least ack'd all the false positives for the moment.

• Both logging-logstash-04 and logging-logstash-05 have been up for > 5 days but they are Bookworm so they aren't especially useful datapoints about the mitigation attempts.

• There's the fairly dumb https://gerrit.wikimedia.org/r/c/operations/puppet/+/1318779 awaiting review. It's the only real idea I have for mitigation on Bookworm. We could apply it on Trixie as well... it's a bit of a rug-sweep but a lot better than doing nothing.

Reading through the task it strikes me that this might be an issue with DHCP lease renewal. A set of operations which could lead to this scenario is:

1. Host comes online, the usual DHCP DISCOVER / OFFER / REQUEST / ACK packets flow between VM and server, and it gets an IP
2. At the end of the VMs allocated DHCP lease time the renewal process fails
3. Restarting systemd-networkd kicks off a new exchange as per step 1 which fixes it
   1. Or indeed manually running dhclient does

So I'd focus on trying to confirm if this is the case or not, and if it does appear to be right try to work out why the DHCP renewal fails. As a start check if systemd-networkd is logging anything to the journal relating to DHCP

journalctl -u systemd-networkd -g DHCP

sudo networkctl status <interface>

The lease time is only shown in the above on newer systemd release, if it's not there we can check here:

grep . /run/systemd/netif/leases/*

The "LIFETIME" there will tell you how often the host has to renew. systemd should try to rewnew half way through that ('t1' time). What we can do is run a tcpdump on a host we might expect this in the background and try to see if its sending the renewal requests:

sudo tcpdump -i <int> -l -s0 -w /tmp/dhcp_capture.pcap udp port 67 or udp port 68

There won't be any other traffic on those ports so we can leave it running for an extended period and review. It might also help to increase the systemd log level for systemd-networkd to debug as by default it doesn't log renewals. But I'd say start with the tcpdump.

What is acting as the DHCP server here, is there anything in its logs that might point to a reason?

This morning there are no new Trixie hosts down. Three new bookworm hosts show as down -- two of them have fallen off the network and one of them is frozen.

Trixie host f78be803-6d05-43cc-9b40-a8748ff56204 statanalyser.statanalyser.eqiad1.wikimedia.cloud just fell off the network.

This does not seem to be the same issue; it's an old bullseye server that was recently upgraded in place.

In T432426#12187143, @Andrew wrote:

Trixie host f78be803-6d05-43cc-9b40-a8748ff56204 statanalyser.statanalyser.eqiad1.wikimedia.cloud just fell off the network.

Further digging reveals this to be an in-place upgrade from Bullseye still relying on dhclient. So... this host should not be taken as an example for anyone.

Bookworm example today: demo-patrollers.wmf-research-tools.eqiad1.wikimedia.cloud