urldownloader1003 (VM on ganeti1027) has no network connectivity, causing problems to any service using it to fetch data from the outside world
Description
Description
Related Objects
Related Objects
Event Timeline
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-02-27T15:45:52Z] <effie> reboot urldownloader1003 - T358597
Comment Actions
Looking into the issue, we found that around 26th Feb @ ~21:45 UTC, the urldownloader1003 (ganeti VM running on ganeti1027 ie cluster master) lost network connectivity
I was able to login to the VM via ganeti, where nothing seemed out of order, from the machine's perspective. Running ip a yielded that it believed its network interface was UP. After issuing a systemctl restart networking, it found its network interface as DOWN.
From the outside, we were getting
PING urldownloader1003.wikimedia.org(urldownloader1003.wikimedia.org (2620:0:861:3:208:80:154:75)) 56 data bytes From ae4-1020.cr2-eqiad.wikimedia.org (2620:0:861:107:fe00::3) icmp_seq=1 Destination unreachable: Address unreachable
We found nothing odd in the ganeti logs, or the machine's logs. We will see if such a thing resurfaces.
tx @akosiaris for offering a hand