Page MenuHomePhabricator
Create Task
Maniphest T272555

releases2002 ganeti VM not getting IP after reboot
Closed, ResolvedPublic
Actions

Description

I rebooted releases2002, a ganeti VM.

It did not come back from reboot. I connected to console and saw it was up though, logged in with root password.

Then saw ferm failed to start and no iptables rules. When trying to start ferm.. failure to do a DNS lookup.

Then saw.. it did not get any IP on the interface at all and networking is just down .. rebooted it one more time.. no change.

still have to find out why what is happening here.. until then we should be careful rebooting any ganeti VMs.. maybe it is a global issue

Details

SubjectRepoBranchLines +/-
bacula: Ignore releases2002 backup errors until vm issues are fixedoperations/puppetproduction+2 -1
Customize query in gerrit

Related Objects

Event Timeline

Dzahn created this task.Jan 21 2021, 2:19 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 21 2021, 2:19 AM
Dzahn mentioned this in T272092: Request volume for Docker images and container filesystems on releases machines.Jan 21 2021, 6:08 PM
• dancy subscribed.Jan 21 2021, 6:14 PM
jcrespo subscribed.Jan 22 2021, 10:27 AM

I reached this through alerts of backups of releases2002 not working since 2021-01-21. I will disable alerts for this host until fixed (please ping me to re enabling them when fixed).

gerritbot added a comment.Jan 22 2021, 10:30 AM

Change 657783 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] bacula: Ignore releases2002 backup errors until vm issues are fixed

https://gerrit.wikimedia.org/r/657783

gerritbot added a project: Patch-For-Review.Jan 22 2021, 10:30 AM
gerritbot added a comment.Jan 22 2021, 10:44 AM

Change 657783 merged by Jcrespo:
[operations/puppet@production] bacula: Ignore releases2002 backup errors until vm issues are fixed

https://gerrit.wikimedia.org/r/657783

Maintenance_bot removed a project: Patch-For-Review.Jan 22 2021, 11:10 AM
akosiaris subscribed.Jan 22 2021, 11:59 AM

I think the following explains it:

root@releases2002:~# ip addr ls
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether aa:00:00:c1:bf:5d brd ff:ff:ff:ff:ff:ff

and

root@releases2002:~# cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
allow-hotplug ens5
iface ens5 inet static
	address 10.192.16.180/22
	gateway 10.192.16.1
	# dns-* options are implemented by the resolvconf package, if installed
	dns-nameservers 10.3.0.1
	dns-search codfw.wmnet
   pre-up /sbin/ip token set ::10:192:16:180 dev ens5
   up ip addr add 2620:0:860:102:10:192:16:180/64 dev ens5

So, somehow this PCI slot for the network card changed, thus renaming the card cause systemd implements now persistent interface devices names. [1]

Now as to why, my guess is: https://phabricator.wikimedia.org/T272092#6762844 which

akosiaris@releases2002:~$ sudo lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 VGA compatible controller: Device 1234:1111 (rev 02)
00:03.0 Unclassified device [00ff]: Red Hat, Inc Virtio memory balloon
00:04.0 SCSI storage controller: Red Hat, Inc Virtio block device
00:05.0 SCSI storage controller: Red Hat, Inc Virtio block device
00:06.0 Ethernet controller: Red Hat, Inc Virtio network device

points out.

[1] https://www.freedesktop.org/software/systemd/man/systemd.net-naming-scheme.html

akosiaris closed this task as Resolved.Jan 22 2021, 12:00 PM
akosiaris claimed this task.

Anyway, s/ens5/ens6/ in /etc/network/interfaces and the issue has been fixed. I was wondering whether it makes sense to invest time to "fix" this but having met 1 instance of it in 5-6 years that we have ganeti around, I am gonna say it's not worth it. That being said, let's document this.

jcrespo reopened this task as Open.Jan 22 2021, 12:32 PM

Let me reopen for reenabling backup monitoring (even if main issue has been fixed).

Thanks @akosiaris for the help here.

jcrespo claimed this task.Jan 22 2021, 12:33 PM
jcrespo closed this task as Resolved.Jan 22 2021, 12:47 PM
jcrespo reassigned this task from jcrespo to akosiaris.

Backing up 0 bytes was unsurprisingly fast :-) Thanks again to both of you.

Dzahn added a comment.Jan 22 2021, 5:19 PM

@akosiaris Thank you!:)

Stashbot added a comment.Jan 22 2021, 5:57 PM

Mentioned in SAL (#wikimedia-operations) [2021-01-22T17:57:43Z] <mutante> releases1002 (releases.wm.org active backend) - rebooting - hopefully it does not run into T272555 but if it does now it's known how to fix

Dzahn added a comment.Jan 22 2021, 6:27 PM

releases1002 had the exact same issue.. so confirmed it was caused by adding the new disk.

The same fix (ens5->ens6) also resolved it again.

Dzahn mentioned this in T208529: Install docker on releases-jenkins.Jan 22 2021, 6:56 PM
Dzahn added a comment.Jan 22 2021, 7:57 PM
In T272555#6768676, @akosiaris wrote:

let's document this.

added this https://wikitech.wikimedia.org/w/index.php?title=Ganeti&type=revision&diff=1894909&oldid=1893790

Dzahn mentioned this in T243057: Move Prometheus off eqsin/ulsfo/esams bastions.Jun 11 2021, 6:56 PM
Stashbot added a comment.Mar 29 2022, 5:23 PM

Mentioned in SAL (#wikimedia-operations) [2022-03-29T17:23:06Z] <mutante> gitlab2001 - did not come back from reboot via cookbook. logged in via console. then "s/ens5/ens13" in /etc/network/interfaces ; reboot ; issue was like T272555 and others

Jelto mentioned this in T304622: GitLab minor version upgrade: 14.9.x.Mar 30 2022, 5:05 PM
Stashbot added a comment.Mar 31 2022, 11:27 PM

Mentioned in SAL (#wikimedia-operations) [2022-03-31T23:27:31Z] <mutante> gitlab2001 - rebooted on ganeti level (needed when adding new virtual hardware), then ran into the usual bug T272555 where you have to manually fix the interface in /etc/network/interfaces T274463

Stashbot mentioned this in T274463: Backups for GitLab.Mar 31 2022, 11:27 PM
cmooney mentioned this in T273026: Errors for ifup@ens5.service after rebooting Ganeti VMs.Mar 28 2023, 10:32 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits