Page MenuHomePhabricator
Create Task
Maniphest T105152

diskimage cloud-init does not bring up network
Closed, ResolvedPublic
Actions

Description

Since upgrading python-diskimage-builder from 1.30 to 1.46, the instances booted by Nodepool do not have any network connection attached. Seems caused by cloud-init, maybe it can not access some metadata service.

Boot log at P922:

P922 boot of a 1.46 disk image
1[H[J[1;1H[?25l[m[H[J[1;1H[2;25HGNU GRUB version 2.02~beta2-22
2
3[m[4;2H+----------------------------------------------------------------------------+[5;2H|[5;79H|[6;2H|[6;79H|[7;2H|[7;79H|[8;2H|[8;79H|[9;2H|[9;79H|[10;2H|[10;79H|[11;2H|[11;79H|[12;2H|[12;79H|[13;2H|[13;79H|[14;2H|[14;79H|[15;2H|[15;79H|[16;2H|[16;79H|[17;2H+----------------------------------------------------------------------------+[m[18;2H[19;2H[m Use the ^ and v keys to select which entry is highlighted.
4 Press enter to boot the selected OS, `e' to edit the commands
5 before booting or `c' for a command-line. [5;80H [7m[5;3H*Debian GNU/Linux [m[5;78H[m[m[6;3H Advanced options for Debian GNU/Linux [m[6;78H[m[m[7;3H [m[7;78H[m[m[8;3H [m[8;78H[m[m[9;3H [m[9;78H[m[m[10;3H [m[10;78H[m[m[11;3H [m[11;78H[m[m[12;3H [m[12;78H[m[m[13;3H [m[13;78H[m[m[14;3H [m[14;78H[m[m[15;3H [m[15;78H[m[m[16;3H [m[16;78H[m[16;80H [5;78H[22;1H The highlighted entry will be executed automatically in 5s. [5;78H[22;1H The highlighted entry will be executed automatically in 4s. [5;78H[22;1H The highlighted entry will be executed automatically in 3s. [5;78H[22;1H The highlighted entry will be executed automatically in 2s. [5;78H[22;1H The highlighted entry will be executed automatically in 1s. [5;78H[22;1H The highlighted entry will be executed automatically in 0s. [5;78H[?25h[H[J[1;1H[H[J[1;1HLoading Linux 3.16.0-4-amd64 ...
6text is deprecated. Use set gfxpayload=vga=normal before linux command
7instead.
8text is deprecated. Use set gfxpayload=vga=normal before linux command
9instead.
10Loading initial ramdisk ...
11[ 0.000000] Initializing cgroup subsys cpuset
12[ 0.000000] Initializing cgroup subsys cpu
13[ 0.000000] Initializing cgroup subsys cpuacct
14[ 0.000000] Linux version 3.16.0-4-amd64 (debian-kernel@lists.debian.org) (gcc version 4.8.4 (Debian 4.8.4-1) ) #1 SMP Debian 3.16.7-ckt11-1 (2015-05-24)
15[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.16.0-4-amd64 root=LABEL=cloudimg-rootfs ro nofb nomodeset vga=normal console=tty0 console=ttyS0,115200 no_timer_check nofb nomodeset vga=normal
16[ 0.000000] e820: BIOS-provided physical RAM map:
17[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
18[ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
19[ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
20[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bfffdfff] usable
21[ 0.000000] BIOS-e820: [mem 0x00000000bfffe000-0x00000000bfffffff] reserved
22[ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
23[ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
24[ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013fffffff] usable
25[ 0.000000] NX (Execute Disable) protection: active
26[ 0.000000] SMBIOS 2.4 present.
27[ 0.000000] Hypervisor detected: KVM
28[ 0.000000] AGP: No AGP bridge found
29[ 0.000000] e820: last_pfn = 0x140000 max_arch_pfn = 0x400000000
30[ 0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
31[ 0.000000] e820: last_pfn = 0xbfffe max_arch_pfn = 0x400000000
32[ 0.000000] found SMP MP-table at [mem 0x000f0b40-0x000f0b4f] mapped at [ffff8800000f0b40]
33[ 0.000000] Using GB pages for direct mapping
34[ 0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
35[ 0.000000] init_memory_mapping: [mem 0x13fe00000-0x13fffffff]
36[ 0.000000] init_memory_mapping: [mem 0x13c000000-0x13fdfffff]
37[ 0.000000] init_memory_mapping: [mem 0x100000000-0x13bffffff]
38[ 0.000000] init_memory_mapping: [mem 0x00100000-0xbfffdfff]
39[ 0.000000] RAMDISK: [mem 0x3637e000-0x371b6fff]
40[ 0.000000] ACPI: Early table checksum verification disabled
41[ 0.000000] ACPI: RSDP 0x00000000000F0950 000014 (v00 BOCHS )
42[ 0.000000] ACPI: RSDT 0x00000000BFFFFBF2 000034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001)
43[ 0.000000] ACPI: FACP 0x00000000BFFFF1C0 000074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001)
44[ 0.000000] ACPI: DSDT 0x00000000BFFFE040 001180 (v01 BOCHS BXPCDSDT 00000001 BXPC 00000001)
45[ 0.000000] ACPI: FACS 0x00000000BFFFE000 000040
46[ 0.000000] ACPI: SSDT 0x00000000BFFFF234 00093E (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001)
47[ 0.000000] ACPI: APIC 0x00000000BFFFFB72 000080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001)
48[ 0.000000] ACPI: RSDT 0x00000000BFFFFBF2 000034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001)
49[ 0.000000] No NUMA configuration found
50[ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013fffffff]
51[ 0.000000] Initmem setup node 0 [mem 0x00000000-0x13fffffff]
52[ 0.000000] NODE_DATA [mem 0x13fff9000-0x13fffdfff]
53[ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
54[ 0.000000] kvm-clock: cpu 0, msr 1:3fff1001, primary cpu clock
55[ 0.000000] Zone ranges:
56[ 0.000000] DMA [mem 0x00001000-0x00ffffff]
57[ 0.000000] DMA32 [mem 0x01000000-0xffffffff]
58[ 0.000000] Normal [mem 0x100000000-0x13fffffff]
59[ 0.000000] Movable zone start for each node
60[ 0.000000] Early memory node ranges
61[ 0.000000] node 0: [mem 0x00001000-0x0009efff]
62[ 0.000000] node 0: [mem 0x00100000-0xbfffdfff]
63[ 0.000000] node 0: [mem 0x100000000-0x13fffffff]
64[ 0.000000] ACPI: PM-Timer IO Port: 0xb008
65[ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
66[ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
67[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
68[ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
69[ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23
70[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
71[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
72[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
73[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
74[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
75[ 0.000000] Using ACPI (MADT) for SMP configuration information
76[ 0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs
77[ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff]
78[ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff]
79[ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff]
80[ 0.000000] PM: Registered nosave memory: [mem 0xbfffe000-0xbfffffff]
81[ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff]
82[ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff]
83[ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff]
84[ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff]
85[ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices
86[ 0.000000] Booting paravirtualized kernel on KVM
87[ 0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:2 nr_node_ids:1
88[ 0.000000] PERCPU: Embedded 27 pages/cpu @ffff88013fc00000 s80896 r8192 d21504 u1048576
89[ 0.000000] KVM setup async PF for cpu 0
90[ 0.000000] kvm-stealtime: cpu 0, msr 13fc0d080
91[ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1034119
92[ 0.000000] Policy zone: Normal
93[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.16.0-4-amd64 root=LABEL=cloudimg-rootfs ro nofb nomodeset vga=normal console=tty0 console=ttyS0,115200 no_timer_check nofb nomodeset vga=normal
94[ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
95[ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340
96[ 0.000000] AGP: Checking aperture...
97[ 0.000000] AGP: No AGP bridge found
98[ 0.000000] Memory: 4044092K/4193904K available (5209K kernel code, 946K rwdata, 1832K rodata, 1204K init, 840K bss, 149812K reserved)
99[ 0.000000] Hierarchical RCU implementation.
100[ 0.000000] RCU dyntick-idle grace-period acceleration is enabled.
101[ 0.000000] RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=2.
102[ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
103[ 0.000000] NR_IRQS:33024 nr_irqs:512 16
104[ 0.000000] Console: colour VGA+ 80x25
105[ 0.000000] console [tty0] enabled
106[ 0.000000] console [ttyS0] enabled
107[ 0.000000] tsc: Detected 2693.248 MHz processor
108[ 0.008000] Calibrating delay loop (skipped) preset value.. 5386.49 BogoMIPS (lpj=10772992)
109[ 0.008000] pid_max: default: 32768 minimum: 301
110[ 0.008000] ACPI: Core revision 20140424
111[ 0.009005] ACPI: All ACPI Tables successfully acquired
112[ 0.012067] Security Framework initialized
113[ 0.012797] AppArmor: AppArmor disabled by boot time parameter
114[ 0.013727] Yama: disabled by default; enable with sysctl kernel.yama.*
115[ 0.015792] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
116[ 0.017428] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
117[ 0.020407] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes)
118[ 0.021758] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes)
119[ 0.024189] Initializing cgroup subsys memory
120[ 0.024945] Initializing cgroup subsys devices
121[ 0.025701] Initializing cgroup subsys freezer
122[ 0.026779] Initializing cgroup subsys net_cls
123[ 0.028009] Initializing cgroup subsys blkio
124[ 0.028754] Initializing cgroup subsys perf_event
125[ 0.029539] Initializing cgroup subsys net_prio
126[ 0.032757] mce: CPU supports 10 MCE banks
127[ 0.033528] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0
128[ 0.033528] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0
129[ 0.033528] tlb_flushall_shift: 6
130[ 0.036320] Freeing SMP alternatives memory: 20K (ffffffff81a1b000 - ffffffff81a20000)
131[ 0.041412] ftrace: allocating 21623 entries in 85 pages
132[ 0.048561] Enabling x2apic
133[ 0.049167] Enabled x2apic
134[ 0.049886] Switched APIC routing to physical x2apic.
135[ 0.052398] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
136[ 0.053542] smpboot: CPU0: Intel Xeon E312xx (Sandy Bridge) (fam: 06, model: 2a, stepping: 01)
137[ 0.056039] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only.
138[ 0.059497] NMI watchdog: disabled (cpu0): hardware events not enabled
139[ 0.060131] x86: Booting SMP configuration:
140[ 0.060987] .... node #0, CPUs: #1
141[ 0.008000] kvm-clock: cpu 1, msr 1:3fff1041, secondary cpu clock
142[ 0.076066] x86: Booted up 1 node, 2 CPUs
143[ 0.076053] KVM setup async PF for cpu 1
144[ 0.076053] kvm-stealtime: cpu 1, msr 13fd0d080
145[ 0.078716] smpboot: Total of 2 processors activated (10772.99 BogoMIPS)
146[ 0.080095] devtmpfs: initialized
147[ 0.084791] pinctrl core: initialized pinctrl subsystem
148[ 0.085787] NET: Registered protocol family 16
149[ 0.087574] cpuidle: using governor ladder
150[ 0.088008] cpuidle: using governor menu
151[ 0.088813] ACPI: bus type PCI registered
152[ 0.089519] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
153[ 0.090738] PCI: Using configuration type 1 for base access
154[ 0.104139] ACPI: Added _OSI(Module Device)
155[ 0.104940] ACPI: Added _OSI(Processor Device)
156[ 0.106612] ACPI: Added _OSI(3.0 _SCP Extensions)
157[ 0.107381] ACPI: Added _OSI(Processor Aggregator Device)
158[ 0.110638] ACPI: Interpreter enabled
159[ 0.111324] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S1_] (20140424/hwxface-580)
160[ 0.112815] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20140424/hwxface-580)
161[ 0.114438] ACPI: (supports S0 S3 S4 S5)
162[ 0.115123] ACPI: Using IOAPIC for interrupt routing
163[ 0.115981] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
164[ 0.119214] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
165[ 0.120010] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI]
166[ 0.121046] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM
167[ 0.122457] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
168[ 0.124262] acpiphp: Slot [3] registered
169[ 0.124977] acpiphp: Slot [4] registered
170[ 0.127505] acpiphp: Slot [5] registered
171[ 0.128025] acpiphp: Slot [6] registered
172[ 0.128730] acpiphp: Slot [7] registered
173[ 0.130045] acpiphp: Slot [8] registered
174[ 0.130759] acpiphp: Slot [9] registered
175[ 0.131463] acpiphp: Slot [10] registered
176[ 0.132025] acpiphp: Slot [11] registered
177[ 0.132747] acpiphp: Slot [12] registered
178[ 0.133523] acpiphp: Slot [13] registered
179[ 0.134240] acpiphp: Slot [14] registered
180[ 0.134953] acpiphp: Slot [15] registered
181[ 0.135756] acpiphp: Slot [16] registered
182[ 0.136029] acpiphp: Slot [17] registered
183[ 0.138850] acpiphp: Slot [18] registered
184[ 0.139577] acpiphp: Slot [19] registered
185[ 0.140028] acpiphp: Slot [20] registered
186[ 0.141057] acpiphp: Slot [21] registered
187[ 0.143109] acpiphp: Slot [22] registered
188[ 0.144042] acpiphp: Slot [23] registered
189[ 0.147142] acpiphp: Slot [24] registered
190[ 0.148044] acpiphp: Slot [25] registered
191[ 0.149492] acpiphp: Slot [26] registered
192[ 0.150235] acpiphp: Slot [27] registered
193[ 0.150953] acpiphp: Slot [28] registered
194[ 0.151696] acpiphp: Slot [29] registered
195[ 0.152028] acpiphp: Slot [30] registered
196[ 0.154551] acpiphp: Slot [31] registered
197[ 0.155259] PCI host bridge to bus 0000:00
198[ 0.155963] pci_bus 0000:00: root bus resource [bus 00-ff]
199[ 0.156006] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7]
200[ 0.156960] pci_bus 0000:00: root bus resource [io 0x0d00-0xadff]
201[ 0.157908] pci_bus 0000:00: root bus resource [io 0xae0f-0xaeff]
202[ 0.158858] pci_bus 0000:00: root bus resource [io 0xaf20-0xafdf]
203[ 0.160005] pci_bus 0000:00: root bus resource [io 0xafe4-0xffff]
204[ 0.160973] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff]
205[ 0.162006] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff]
206[ 0.171605] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7]
207[ 0.172005] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6]
208[ 0.174182] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177]
209[ 0.175542] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376]
210[ 0.181454] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI
211[ 0.182689] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB
212[ 0.223133] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)
213[ 0.224537] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
214[ 0.225898] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
215[ 0.227253] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)
216[ 0.230028] ACPI: PCI Interrupt Link [LNKS] (IRQs *9)
217[ 0.231587] ACPI: Enabled 16 GPEs in block 00 to 0F
218[ 0.232123] vgaarb: setting as boot device: PCI:0000:00:02.0
219[ 0.232930] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
220[ 0.234268] vgaarb: loaded
221[ 0.234809] vgaarb: bridge control possible 0000:00:02.0
222[ 0.236075] PCI: Using ACPI for IRQ routing
223[ 0.237142] Switched to clocksource kvm-clock
224[ 0.246349] pnp: PnP ACPI init
225[ 0.246987] ACPI: bus type PNP registered
226[ 0.248234] pnp: PnP ACPI: found 6 devices
227[ 0.248954] ACPI: bus type PNP unregistered
228[ 0.260681] NET: Registered protocol family 2
229[ 0.261799] TCP established hash table entries: 32768 (order: 6, 262144 bytes)
230[ 0.263777] TCP bind hash table entries: 32768 (order: 7, 524288 bytes)
231[ 0.264881] TCP: Hash tables configured (established 32768 bind 32768)
232[ 0.266234] TCP: reno registered
233[ 0.267172] UDP hash table entries: 2048 (order: 4, 65536 bytes)
234[ 0.268145] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes)
235[ 0.269258] NET: Registered protocol family 1
236[ 0.270036] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
237[ 0.270975] pci 0000:00:01.0: PIIX3: Enabling Passive Release
238[ 0.271890] pci 0000:00:01.0: Activating ISA DMA hang workarounds
239[ 0.273123] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
240[ 0.275889] Unpacking initramfs...
241[ 0.609086] Freeing initrd memory: 14564K (ffff88003637e000 - ffff8800371b7000)
242[ 0.611384] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
243[ 0.612391] software IO TLB [mem 0xbbffe000-0xbfffe000] (64MB) mapped at [ffff8800bbffe000-ffff8800bfffdfff]
244[ 0.639934] microcode: CPU0 sig=0x206a1, pf=0x1, revision=0x1
245[ 0.641056] microcode: CPU1 sig=0x206a1, pf=0x1, revision=0x1
246[ 0.642363] microcode: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
247[ 0.644094] futex hash table entries: 512 (order: 3, 32768 bytes)
248[ 0.645114] audit: initializing netlink subsys (disabled)
249[ 0.647829] audit: type=2000 audit(1436360163.698:1): initialized
250[ 0.649151] HugeTLB registered 2 MB page size, pre-allocated 0 pages
251[ 0.650818] zbud: loaded
252[ 0.651632] VFS: Disk quotas dquot_6.5.2
253[ 0.652494] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
254[ 0.653557] msgmni has been set to 7927
255[ 0.655098] alg: No test for stdrng (krng)
256[ 0.655893] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
257[ 0.657256] io scheduler noop registered
258[ 0.659800] io scheduler deadline registered
259[ 0.660734] io scheduler cfq registered (default)
260[ 0.661640] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
261[ 0.662890] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
262[ 0.663943] GHES: HEST is not enabled!
263[ 0.664707] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
264[ 0.694888] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
265[ 0.724174] 00:05: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A
266[ 0.725776] Linux agpgart interface v0.103
267[ 0.726974] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
268[ 0.729072] serio: i8042 KBD port at 0x60,0x64 irq 1
269[ 0.731979] serio: i8042 AUX port at 0x60,0x64 irq 12
270[ 0.733227] mousedev: PS/2 mouse device common for all mice
271[ 0.735164] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0
272[ 0.737007] rtc_cmos 00:00: RTC can wake from S4
273[ 0.739483] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0
274[ 0.740544] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram
275[ 0.741527] ledtrig-cpu: registered to indicate activity on CPUs
276[ 0.747784] AMD IOMMUv2 driver by Joerg Roedel <joerg.roedel@amd.com>
277[ 0.748783] AMD IOMMUv2 functionality not available on this system
278[ 0.749826] TCP: cubic registered
279[ 0.753318] NET: Registered protocol family 10
280[ 0.757104] mip6: Mobile IPv6
281[ 0.757730] NET: Registered protocol family 17
282[ 0.759305] mpls_gso: MPLS GSO support
283[ 0.760867] registered taskstats version 1
284[ 0.763358] rtc_cmos 00:00: setting system clock to 2015-07-08 12:56:03 UTC (1436360163)
285[ 0.767424] Freeing unused kernel memory: 1204K (ffffffff818ee000 - ffffffff81a1b000)
286[ 0.768757] Write protecting the kernel read-only data: 8192k
287[ 0.772550] Freeing unused kernel memory: 924K (ffff880001519000 - ffff880001600000)
288[ 0.774348] Freeing unused kernel memory: 216K (ffff8800017ca000 - ffff880001800000)
289Loading, please wait...
290[ 0.786898] systemd-udevd[59]: starting version 215
291[ 0.788113] random: systemd-udevd urandom read with 1 bits of entropy available
292[ 0.799148] ACPI: bus type USB registered
293[ 0.801261] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10
294[ 0.804337] usbcore: registered new interface driver usbfs
295[ 0.805285] usbcore: registered new interface driver hub
296[ 0.807774] usbcore: registered new device driver usb
297[ 0.808881] SCSI subsystem initialized
298[ 0.809684] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
299[ 0.811754] uhci_hcd: USB Universal Host Controller Interface driver
300[ 0.821036] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10
301[ 0.832048] scsi0 : ata_piix
302[ 0.840182] scsi1 : ata_piix
303[ 0.840929] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc0a0 irq 14
304[ 0.843774] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc0a8 irq 15
305[ 0.846927] uhci_hcd 0000:00:01.2: UHCI Host Controller
306[ 0.847926] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1
307[ 0.852075] uhci_hcd 0000:00:01.2: detected 2 ports
308[ 0.853392] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c040
309[ 0.855284] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001
310[ 0.856598] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
311[ 0.857968] usb usb1: Product: UHCI Host Controller
312[ 0.858821] usb usb1: Manufacturer: Linux 3.16.0-4-amd64 uhci_hcd
313[ 0.860278] usb usb1: SerialNumber: 0000:00:01.2
314[ 0.861447] hub 1-0:1.0: USB hub found
315[ 0.863273] hub 1-0:1.0: 2 ports detected
316[ 0.864476] FDC 0 is a S82078B
317[ 0.865228] vda: vda1
318Begin: Loading essential drivers ... done.
319Begin: Running /scripts/init-premount ... done.
320Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
321Begin: Running /scripts/local-premount ... done.
322Begin: Checking root file system ... fsck from util-linux 2.25.2
323fsck: error 2 (No such file or directory) while executing fsck.ext4 for /dev/vda1
324fsck exited with status code 8
325done.
326Warning: File system check failed but did not detect errors
327[ 1.176131] usb 1-1: new full-speed USB device number 2 using uhci_hcd
328[ 1.346374] usb 1-1: New USB device found, idVendor=0627, idProduct=0001
329[ 1.347611] usb 1-1: New USB device strings: Mfr=1, Product=3, SerialNumber=5
330[ 1.349009] usb 1-1: Product: QEMU USB Tablet
331[ 1.350727] usb 1-1: Manufacturer: QEMU
332[ 1.351440] usb 1-1: SerialNumber: 42
333[ 1.357179] hidraw: raw HID events driver (C) Jiri Kosina
334[ 1.362195] usbcore: registered new interface driver usbhid
335[ 1.363198] usbhid: USB HID core driver
336[ 1.365144] input: QEMU QEMU USB Tablet as /devices/pci0000:00/0000:00:01.2/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input2
337[ 1.367999] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v0.01 Pointer [QEMU QEMU USB Tablet] on usb-0000:00:01.2-1/input0
338[ 1.612131] tsc: Refined TSC clocksource calibration: 2693.232 MHz
339[ 6.095833] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null)
340done.
341Begin: Running /scripts/local-bottom ... [ 6.173063] vda: vda1
342GROWROOT: CHANGED: partition=1 start=16065 old: size=3373650 end=3389715 new: size=83859300,end=83875365
343[ 6.227404] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null)
344done.
345Begin: Running /scripts/init-bottom ... done.
346[ 6.440360] systemd[1]: systemd 215 running in system mode. (+PAM +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ -SECCOMP -APPARMOR)
347[ 6.443402] systemd[1]: Detected virtualization 'kvm'.
348[ 6.444548] systemd[1]: Detected architecture 'x86-64'.
349
350Welcome to [1mDebian GNU/Linux 8 (jessie)[0m!
351
352[ 6.496413] systemd[1]: Inserted module 'autofs4'
353[ 6.497859] systemd[1]: Set hostname to <debian>.
354[ 6.548347] systemd-sysv-generator[163]: Ignoring creation of an alias umountiscsi.service for itself
355[ 6.863594] systemd[1]: Cannot add dependency job for unit dbus.socket, ignoring: Unit dbus.socket failed to load: No such file or directory.
356[ 6.865817] systemd[1]: Cannot add dependency job for unit display-manager.service, ignoring: Unit display-manager.service failed to load: No such file or directory.
357[ 6.876945] systemd[1]: Expecting device dev-ttyS0.device...
358 Expecting device dev-ttyS0.device...
359[ 6.879054] systemd[1]: Starting Forward Password Requests to Wall Directory Watch.
360[ 6.880891] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
361[ 6.882661] systemd[1]: Starting Encrypted Volumes.
362[[32m OK [0m] Reached target Encrypted Volumes.
363[ 6.884583] systemd[1]: Reached target Encrypted Volumes.
364[ 6.885555] systemd[1]: Starting Arbitrary Executable File Formats File System Automount Point.
365[[32m OK [0m] Set up automount Arbitrary Executable File Formats F...utomount Point.
366[ 6.889902] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
367[ 6.891535] systemd[1]: Starting Dispatch Password Requests to Console Directory Watch.
368[ 6.893037] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
369[ 6.895420] systemd[1]: Starting Paths.
370[[32m OK [0m] Reached target Paths.
371[ 6.897139] systemd[1]: Reached target Paths.
372[ 6.898937] systemd[1]: Starting Swap.
373[[32m OK [0m] Reached target Swap.
374[ 6.900531] systemd[1]: Reached target Swap.
375[ 6.901360] systemd[1]: Starting Root Slice.
376[[32m OK [0m] Created slice Root Slice.
377[ 6.903408] systemd[1]: Created slice Root Slice.
378[ 6.904262] systemd[1]: Starting User and Session Slice.
379[[32m OK [0m] Created slice User and Session Slice.
380[ 6.906263] systemd[1]: Created slice User and Session Slice.
381[ 6.907263] systemd[1]: Starting Delayed Shutdown Socket.
382[[32m OK [0m] Listening on Delayed Shutdown Socket.
383[ 6.909255] systemd[1]: Listening on Delayed Shutdown Socket.
384[ 6.911838] systemd[1]: Starting /dev/initctl Compatibility Named Pipe.
385[[32m OK [0m] Listening on /dev/initctl Compatibility Named Pipe.
386[ 6.914679] systemd[1]: Listening on /dev/initctl Compatibility Named Pipe.
387[ 6.915867] systemd[1]: Starting Journal Socket (/dev/log).
388[[32m OK [0m] Listening on Journal Socket (/dev/log).
389[ 6.917898] systemd[1]: Listening on Journal Socket (/dev/log).
390[ 6.918936] systemd[1]: Starting udev Control Socket.
391[[32m OK [0m] Listening on udev Control Socket.
392[ 6.920819] systemd[1]: Listening on udev Control Socket.
393[ 6.921774] systemd[1]: Starting udev Kernel Socket.
394[[32m OK [0m] Listening on udev Kernel Socket.
395[ 6.925231] systemd[1]: Listening on udev Kernel Socket.
396[ 6.926488] systemd[1]: Starting Journal Socket.
397[[32m OK [0m] Listening on Journal Socket.
398[ 6.928295] systemd[1]: Listening on Journal Socket.
399[ 6.929205] systemd[1]: Starting Sockets.
400[[32m OK [0m] Reached target Sockets.
401[ 6.930891] systemd[1]: Reached target Sockets.
402[ 6.931714] systemd[1]: Starting System Slice.
403[[32m OK [0m] Created slice System Slice.
404[ 6.933540] systemd[1]: Created slice System Slice.
405[ 6.935986] systemd[1]: Starting File System Check on Root Device...
406 Starting File System Check on Root Device...
407[ 6.938408] systemd[1]: Starting system-getty.slice.
408[[32m OK [0m] Created slice system-getty.slice.
409[ 6.940324] systemd[1]: Created slice system-getty.slice.
410[ 6.941300] systemd[1]: Starting system-serial\x2dgetty.slice.
411[[32m OK [0m] Created slice system-serial\x2dgetty.slice.
412[ 6.944348] systemd[1]: Created slice system-serial\x2dgetty.slice.
413[ 6.945488] systemd[1]: Mounting Huge Pages File System...
414 Mounting Huge Pages File System...
415[ 6.949545] systemd[1]: Started Set Up Additional Binary Formats.
416[ 6.953076] systemd[1]: Starting Load Kernel Modules...
417 Starting Load Kernel Modules...
418[ 6.954907] systemd[1]: Starting udev Coldplug all Devices...
419 Starting udev Coldplug all Devices...
420[ 6.956914] systemd[1]: Mounting Debug File System...
421 Mounting Debug File System...
422[ 6.959764] systemd[1]: Starting Create list of required static device nodes for the current kernel...
423 Starting Create list of required static device nodes...rrent kernel...
424[ 6.964302] systemd[1]: Mounting POSIX Message Queue File System...
425 Mounting POSIX Message Queue File System...
426[ 6.966384] systemd[1]: Starting Journal Service...
427 Starting Journal Service...
428[[32m OK [0m] Started Journal Service.
429[ 6.969636] systemd[1]: Started Journal Service.
430[[32m OK [0m] Reached target Slices.
431[[32m OK [0m] Started Load Kernel Modules.
432[[32m OK [0m] Started Create list of required static device nodes ...current kernel.
433[[32m OK [0m] Started udev Coldplug all Devices.
434 Starting Create Static Device Nodes in /dev...
435 Starting Apply Kernel Variables...
436[[32m OK [0m] Mounted POSIX Message Queue File System.
437[[32m OK [0m] Mounted Debug File System.
438[[32m OK [0m] Mounted Huge Pages File System.
439[[32m OK [0m] Started File System Check on Root Device.
440[[32m OK [0m] Started Create Static Device Nodes in /dev.
441[ 7.020342] systemd-fsck[166]: cloudimg-rootfs: clean, 33320/421824 files, 261207/421706 blocks
442[[32m OK [0m] Started Apply Kernel Variables.
443 Starting udev Kernel Device Manager...
444 Starting Remount Root and Kernel File Systems...
445[ 7.045469] systemd-udevd[192]: starting version 215
446[[32m OK [0m] Started udev Kernel Device Manager.
447 Starting Copy rules generated while the root was[ 7.049613] EXT4-fs (vda1): re-mounted. Opts: errors=remount-ro
448 ro...
449[[32m OK [0m] Started Remount Root and Kernel File Systems.
450[[32m OK [0m] Started Copy rules generated while the root was ro.
451 Starting Various fixups to make systemd work better on Debian...
452 Starting Load/Save Random Seed...
453[[32m OK [0m] Reached target Local File Systems (Pre).
454[[32m OK [0m] Reached target Local File Systems.
455 Starting Create Volatile Files and Directories...
456[[32m OK [0m] Started Various fixups to make systemd work better on Debian.
457[[32m OK [0m] Started Load/Save Random Seed.
458 Starting LSB: Raise network interfaces....
459[[32m OK [0m] Started Create Volatile Files and Directories.
460 Starting Update UTMP about System Boot/Shutdown...
461[[32m OK [0m] Started Update UTMP about System Boot/Shutdown.
462[ 7.177683] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input3
463[ 7.181204] ACPI: Power Button [PWRF]
464[ 7.192154] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0
465[ 7.206435] [drm] Initialized drm 1.1.0 20060810
466[[32m OK [0m] Found device /dev/ttyS0.
467[ 7.239237] AVX version of gcm_enc/dec engaged.
468[ 7.251528] ppdev: user-space parallel port driver
469[ 7.252755] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni)
470[ 7.263659] alg: No test for crc32 (crc32-pclmul)
471[ 7.417700] intel_rapl: no valid rapl domains found in package 0
472[ 7.532383] intel_rapl: no valid rapl domains found in package 0
473[[32m OK [0m] Started LSB: Raise network interfaces..
474[[32m OK [0m] Reached target Network.
475[[32m OK [0m] Reached target Network is Online.
476 Starting LSB: Starts and stops the iSCSI initiator s...ault targets...
477[ 7.676084] Loading iSCSI transport class v2.0-870.
478[ 7.683780] iscsi: registered transport (tcp)
479[ 7.712951] iscsi: registered transport (iser)
480[[32m OK [0m] Started LSB: Starts and stops the iSCSI initiator se...efault targets.
481[[32m OK [0m] Reached target Remote File Systems (Pre).
482[[32m OK [0m] Reached target Remote File Systems.
483 Starting Trigger Flushing of Journal to Persistent Storage...
484 Starting LSB: Prepare console...
485[ 7.767439] systemd-journald[176]: Received request to flush runtime journal from PID 1
486[[32m OK [0m] Started Trigger Flushing of Journal to Persistent Storage.
487[[32m OK [0m] Started LSB: Prepare console.
488[[32m OK [0m] Reached target System Initialization.
489[[32m OK [0m] Reached target Timers.
490[[32m OK [0m] Reached target Basic System.
491 Starting /etc/rc.local Compatibility...
492 Starting getty on tty2-tty6 if dbus and logind are not available...
493 Starting LSB: Cloud init local...
494 Starting Permit User Sessions...
495[[32m OK [0m] Started /etc/rc.local Compatibility.
496[[32m OK [0m] Started Permit User Sessions.
497 Starting Getty on tty2...
498[[32m OK [0m] Started Getty on tty2.
499 Starting Getty on tty1...
500[[32m OK [0m] Started Getty on tty1.
501 Starting Serial Getty on ttyS0...
502[[32m OK [0m] Started Serial Getty on ttyS0.
503 Starting Getty on tty3...
504[[32m OK [0m] Started Getty on tty3.
505 Starting Getty on tty4...
506[[32m OK [0m] Started Getty on tty4.
507 Starting Getty on tty5...
508[[32m OK [0m] Started Getty on tty5.
509 Starting Getty on tty6...
510[[32m OK [0m] Started Getty on tty6.
511[[32m OK [0m] Started getty on tty2-tty6 if dbus and logind are not available.
512[[32m OK [0m] Reached target Login Prompts.
513[ 8.167090] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input4
514[[32m OK [0m] Started LSB: Cloud init local.
515 Starting LSB: Cloud init...
516
517Debian GNU/Linux 8 debian ttyS0
518
519debian login: [ 135.799421] EXT4-fs (vda1): resizing filesystem from 421706 to 10482412 blocks
520[ 135.880377] EXT4-fs (vda1): resized filesystem to 10482412
521ci-info: no authorized ssh keys fingerprints found for user debian.
522ec2:
523ec2: #############################################################
524ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
525ec2: 1024 e7:6e:2d:c0:06:73:9b:43:d4:ad:94:5e:9f:b8:38:3e /etc/ssh/ssh_host_dsa_key.pub (DSA)
526ec2: 256 f2:85:ca:9c:d9:f4:c1:72:6a:31:0e:38:96:3c:18:41 /etc/ssh/ssh_host_ecdsa_key.pub (ECDSA)
527ec2: 2048 aa:b6:b9:63:94:82:e7:ec:cf:b8:9e:ff:cb:8d:90:c2 /etc/ssh/ssh_host_rsa_key.pub (RSA)
528ec2: -----END SSH HOST KEY FINGERPRINTS-----
529ec2: #############################################################
530-----BEGIN SSH HOST KEY KEYS-----
531ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIU+wwhB+sA6K4AOwO9sSkDgH5mNqMxGBLQmInGkn7kiW7MP4DVwY/YFzasZ2nKHBldEK9tIjwmNK4nfaH61ue8= root@debian
532ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHTwgyZ0m9d59TkPEd8BneNZBPc55a+bOYLmKDwAAOy8FiLtPPvHQQwf71WdnMVSsNFBKdDIcMcFLG1nJDujh/ep+DUvukzHAE1h4P3/HySgwqGJT+4ydn4NnZGgH+0eq4B/4XnW6RzouwYmzDrqRfJblDJFyiMbeLrRji8UbwgNuM1w1/yVA+48wGe1e1sc6KIOMtEJjW4ihBcVOoCLm9RBNRkC1nQuNzVdVRnxxPVMo/LmelTxdwgTD8FEjIKBDGEmZWoX4MVYsNAy+JLQM+qsapHRfryvZIaPaaeA7Y8+5WvLDmYafG9da0mSKaTYesnT1wom/3zu+DLu6AMF6/ root@debian
533-----END SSH HOST KEY KEYS-----

Details

SubjectRepoBranchLines +/-
dib: bump to 1.0.0 and remove wikimedia-networkingintegration/configmaster+1 -6
nodepool: element with basic networking packagesoperations/puppetproduction+4 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

hashar created this task.Jul 8 2015, 1:00 PM
hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar added a project: Continuous-Integration-Scaling.
hashar subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 8 2015, 1:00 PM
hashar updated the task description. (Show Details)Jul 8 2015, 1:02 PM
hashar set Security to None.
hashar added a comment.Jul 8 2015, 1:34 PM

I installed libguestfs-tools package on labnodepool1001.eqiad.wmnet to inspect the disk image (puppet is https://gerrit.wikimedia.org/r/223543 , doc added at https://wikitech.wikimedia.org/wiki/Nodepool ).

So one can thus:

mkdir "$USER/mount"
guestmount -a ci-dib-jessie-wikimedia-1436281442.qcow2 -i --ro "$USER/mount"
cd "$USER/mount"

And inspect the image content. Quite handy but I am not sure what to look for to investigate the network start up issue.

hashar added a comment.Jul 8 2015, 2:18 PM

On the same project, a Jessie image provided by labs boots just fine. A difference is that it outputs:

Starting LSB: Cloud init modules --mode config...
Started LSB: Cloud init modules --mode config.
Starting LSB: Cloud init modules final jobs...
// ssh keys output
Started LSB: Cloud init modules final jobs.
// initial puppet runs

Seems the cloud configuration needs to be tweaked. For Jessie labs images that is done via operations/puppet.git , probably via modules/labs_bootstrapvz/files/cloud.cfg. They are very similar though:

--- puppet_bootstrapvz_cloud.cfg	2015-07-08 16:15:54.000000000 +0200
+++ nodepool_diskimage_cloud.cfg	2015-07-08 16:16:21.000000000 +0200
@@ -5,6 +5,7 @@
 # when a 'default' entry is found it will reference the 'default_user'
 # from the distro configuration specified below
 users:
+   - default
 
 # If this is set, 'root' will not be able to ssh in and they 
 # will get a message to login instead as the above $user (ubuntu)
@@ -40,6 +41,7 @@
 # this can be used by upstart jobs for 'start on cloud-config'.
  - emit_upstart
  - mounts
+ - ssh-import-id
  - locale
  - set-passwords
  - grub-dpkg
@@ -48,6 +50,9 @@
  - package-update-upgrade-install
  - landscape
  - timezone
+ - puppet
+ - chef
+ - salt-minion
  - mcollective
  - disable-ec2-metadata
  - runcmd
@@ -73,9 +78,7 @@
    distro: debian
    # Default user name + that default users groups (if added/used)
    default_user:
-     name: admin
-     sudo: ALL=(ALL) NOPASSWD:ALL
-     shell: /bin/bash
+     name: debian
      lock_passwd: True
      gecos: Debian
      groups: [adm, audio, cdrom, dialout, floppy, video, plugdev, dip]
hashar added a comment.Jul 8 2015, 2:28 PM

Then looking at the bootstrapvz files/labs-jessie.manifest.yaml there is a few settings related to cloud init:

plugins:
  cloud_init:
    username: admin
    metadata_sources: ConfigDrive
  ...
  cloud_init:
    meta_data_sources: Ec2
    username: admin

I have thus fetched the /etc/cloud from a booting instance and compare it with the disk image:

diff -u -r jessie-labs/cloud/cloud.cfg dib-instance/cloud/cloud.cfg
--- jessie-labs/cloud/cloud.cfg	2015-07-08 16:24:43.000000000 +0200
+++ dib-instance/cloud/cloud.cfg	2015-07-08 16:25:09.000000000 +0200
@@ -5,6 +5,7 @@
 # when a 'default' entry is found it will reference the 'default_user'
 # from the distro configuration specified below
 users:
+   - default
 
 # If this is set, 'root' will not be able to ssh in and they 
 # will get a message to login instead as the above $user (ubuntu)
@@ -40,6 +41,7 @@
 # this can be used by upstart jobs for 'start on cloud-config'.
  - emit_upstart
  - mounts
+ - ssh-import-id
  - locale
  - set-passwords
  - grub-dpkg
@@ -48,6 +50,9 @@
  - package-update-upgrade-install
  - landscape
  - timezone
+ - puppet
+ - chef
+ - salt-minion
  - mcollective
  - disable-ec2-metadata
  - runcmd
@@ -73,9 +78,7 @@
    distro: debian
    # Default user name + that default users groups (if added/used)
    default_user:
-     name: admin
-     sudo: ALL=(ALL) NOPASSWD:ALL
-     shell: /bin/bash
+     name: debian
      lock_passwd: True
      gecos: Debian
      groups: [adm, audio, cdrom, dialout, floppy, video, plugdev, dip]

Only in dib-instance/cloud/cloud.cfg.d: 01_hostname.cfg
Only in dib-instance/cloud/cloud.cfg.d: 10_etc_hosts.cfg

diff -u -r jessie-labs/cloud/cloud.cfg.d/90_dpkg.cfg dib-instance/cloud/cloud.cfg.d/90_dpkg.cfg
--- jessie-labs/cloud/cloud.cfg.d/90_dpkg.cfg	2015-07-08 16:24:44.000000000 +0200
+++ dib-instance/cloud/cloud.cfg.d/90_dpkg.cfg	2015-07-08 16:25:09.000000000 +0200
@@ -1,2 +1,2 @@
 # to update this file, run dpkg-reconfigure cloud-init
-datasource_list: [ Ec2 ]
+datasource_list: [ NoCloud, AltCloud, CloudStack, ConfigDrive, Ec2, MAAS, OVF, GCE, None ]

Not sure it is anymore relevant :-(

hashar added a comment.Jul 8 2015, 2:47 PM

Maybe the dib image needs to drop the default user being set. I noticed the boot log line:

ci-info: no authorized ssh keys fingerprints found for user debian.

Maybe it aborts the cloud-init for some reason ? :-(

hashar added a comment.Jul 8 2015, 4:11 PM

The dib image has a /etc/cloud/cloud.cfg.d/90_dpkg.cfg with:

datasource_list: [ NoCloud, AltCloud, CloudStack, ConfigDrive, Ec2, MAAS, OVF, GCE, None ]

http://cloudinit.readthedocs.org/en/latest/topics/datasources.html#no-cloud

So maybe that skip the cloud config entirely! How helpful is that Debian?

Gave it a try by updating the disk image config with the element cloud-init-datasources and DIB_CLOUD_INIT_DATASOURCES: Ec2. That should creates a 91_xxx.cfg file which might override the default Debian config.

Nodepool generated the image ci-dib-jessie-wikimedia-1436371638. Doesn't seem to fix the issue though :-\

hashar added subscribers: yuvipanda, Andrew, coren.Jul 8 2015, 4:17 PM

Hi labs!

For some reason when booting the disk image I created, cloud-init fails to configure the network for some reason.

The last image ci-dib-jessie-wikimedia-1436371638 has the issue and P922 gives the boot output.

So my questions are:

  • Would datasource_list: [ NoCloud, AltCloud, CloudStack, ConfigDrive, Ec2, MAAS, OVF, GCE, None ] prevent network access? Maybe NoCloud is always successful and that does not seem to provide network config
  • Can we get console access to a booted an instance? The OpenStack API doesn't seem to provide such functionality :-\
  • Is there a way to turn some cloud-init debug log to the console for investigation? Maybe via a config file or a kernel boot parameter. Havent found anything.
Andrew added a comment.Jul 8 2015, 9:31 PM

Labs images mostly use firstboot.sh and don't rely on cloud-init very much. That means I don't have much to offer about cloud-init failures or debugging.

We don't support console access to instances, although of course you can see the system log both from horizon and wikitech.

Andrew added a comment.Jul 8 2015, 9:34 PM

oh, I'd expect the network to come up before cloud-init does anything, though -- that should be dhcp's job.

hashar added a comment.Jul 8 2015, 10:36 PM

My expectation as well. Maybe we can snapshot the instance and download the image to inspect the cloud debug log somewhere under /var/log ? Will give it a try.

There must be something broken when generating the image. I will try with a previous version of diskimage-builder and see whether that fix it. Wish me an happy bisecting session.

hashar added a comment.EditedJul 9 2015, 10:37 AM

I needed a way to look at the cloud init log file.

So I have booted an instance again, this time with systemd enabled. Once the instance booted I created an image out of it with something like:

openstack server image create --name hashar-snap some-instance

Then I downloaded the image on labnodepool with:

openstack image save --file /var/lib/nodepool/hashar-snap.img hashar-snap

And I mounted it with:

guestmount --ro -i -a /var/lib/nodepool/hashar-snap.img /home/hashar/mount

And success:

$ ll /home/hashar/mount/var/log/cloud-init-output.log
-rw-r--r-- 1 root root 9750 Jul  9 10:06 /home/hashar/mount/var/log/cloud-init-output.log

Attached as

cloud-init not having network9 KBDownload

Abstract:

Cloud-init v. 0.7.6 running 'init-local' at Thu, 09 Jul 2015 10:04:33 +0000. Up 7.72 seconds.
Cloud-init v. 0.7.6 running 'init' at Thu, 09 Jul 2015 10:04:33 +0000. Up 7.98 seconds.
ci-info: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!Net device info failed!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
ci-info: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!Route info failed!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2015-07-09 10:04:33,925 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [0/120s]: request error [('Connection aborted.', error(101, 'Network is unreachable'))]
....
2015-07-09 10:06:33,153 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [119/120s]: request error [('Connection aborted.', error(101, 'Network is unreachable'))]
2015-07-09 10:06:40,160 - DataSourceEc2.py[CRITICAL]: Giving up on md from ['http://169.254.169.254/2009-04-04/meta-data/instance-id'] after 126 seconds 

Generating public/private rsa key pair.
... ssh keys generation output

Cloud-init v. 0.7.6 running 'modules:config' at Thu, 09 Jul 2015 10:06:41 +0000. Up 135.28 seconds.
Generating locales (this might take a while)...
Generation complete.
2015-07-09 10:06:41,895 - util.py[WARNING]: Running apt-configure (<module 'cloudinit.config.cc_apt_configure' from '/usr/lib/python2.7/dist-packages/cloudinit/config/cc_apt_configure.pyc'>) failed
Cloud-init v. 0.7.6 running 'modules:final' at Thu, 09 Jul 2015 10:06:42 +0000. Up 136.23 seconds.
ci-info: no authorized ssh keys fingerprints found for user debian.
Cloud-init v. 0.7.6 finished at Thu, 09 Jul 2015 10:06:42 +0000. Datasource DataSourceNone.  Up 136.36 seconds
2015-07-09 10:06:42,237 - cc_final_message.py[WARNING]: Used fallback datasource

So definitely no network available :-(

Some other bits:

$ cat etc/resolv.conf 
#####################################################################
#### THIS FILE IS MANAGED BY PUPPET
####  as template('base/resolv.conf.erb')
#####################################################################
# Resolver configuration for site eqiad
search eqiad.wmnet
options timeout:1 attempts:3
nameserver 208.80.154.239
nameserver 208.80.153.254
$ find etc|grep network
etc/rc6.d/K05networking
etc/systemd/network
etc/rcS.d/S11networking
etc/network
etc/network/interfaces
etc/network/if-up.d
etc/network/if-up.d/openssh-server
etc/network/if-up.d/upstart
etc/network/if-up.d/mountnfs
etc/network/if-down.d
etc/network/if-down.d/upstart
etc/network/if-pre-up.d
etc/network/interfaces.d
etc/network/interfaces.d/eth0
etc/network/interfaces.d/eth1
etc/network/if-post-down.d
etc/network/run
etc/default/networking
etc/init/network-interface-container.conf
etc/init/network-interface.conf
etc/init/networking.conf
etc/init/network-interface-security.conf
etc/init.d/networking
etc/rc0.d/K05networking
$ cat etc/default/networking 
# Configuration for networking init script being run during
# the boot sequence

# Set to 'no' to skip interfaces configuration on boot
#CONFIGURE_INTERFACES=yes

# Don't configure these interfaces. Shell wildcards supported/
#EXCLUDE_INTERFACES=

# Set to 'yes' to enable additional verbosity
#VERBOSE=no

The eth0 and eth1 files have:

auto eth0
iface eth0 inet dhcp
auto eth1
iface eth1 inet dhcp

But apparently there is no dhcp client available:

$ find . -name dhclient
./usr/share/bash-completion/completions/dhclient
find: `./root': Permission denied
hashar added a comment.Jul 9 2015, 10:41 AM

List of installed packages from /home/hashar/mount/etc/dib-manifests/dib-manifest-dpkg-ci-dib-jessie-wikimedia-1436435961

{"packages": [
{"package": "acl","arch": "amd64","version": "2.2.52-2","status": "ii "    },
{"package": "adduser","arch": "all","version": "3.113+nmu3","status": "ii "    },
{"package": "apt","arch": "amd64","version": "1.0.9.8","status": "ii "    },
{"package": "apt-utils","arch": "amd64","version": "1.0.9.8","status": "ii "    },
{"package": "base-files","arch": "amd64","version": "8+deb8u1","status": "ii "    },
{"package": "base-passwd","arch": "amd64","version": "3.5.37","status": "ii "    },
{"package": "bash","arch": "amd64","version": "4.3-11+b1","status": "ii "    },
{"package": "bash-completion","arch": "all","version": "1:2.1-4","status": "ii "    },
{"package": "binutils","arch": "amd64","version": "2.25-5","status": "ii "    },
{"package": "bsdutils","arch": "amd64","version": "1:2.25.2-6","status": "ii "    },
{"package": "ca-certificates","arch": "all","version": "20141019","status": "ii "    },
{"package": "ca-certificates-java","arch": "all","version": "20140324","status": "ii "    },
{"package": "ccache","arch": "amd64","version": "3.1.10-1","status": "ii "    },
{"package": "cloud-init","arch": "all","version": "0.7.6~bzr976-2","status": "ii "    },
{"package": "cloud-initramfs-growroot","arch": "all","version": "0.18.debian5","status": "ii "    },
{"package": "cloud-utils","arch": "all","version": "0.26-2","status": "ii "    },
{"package": "coreutils","arch": "amd64","version": "8.23-4","status": "ii "    },
{"package": "cpio","arch": "amd64","version": "2.11+dfsg-4.1","status": "ii "    },
{"package": "cpp","arch": "amd64","version": "4:4.9.2-2","status": "ii "    },
{"package": "cpp-4.9","arch": "amd64","version": "4.9.2-10","status": "ii "    },
{"package": "curl","arch": "amd64","version": "7.38.0-4+deb8u2","status": "ii "    },
{"package": "dash","arch": "amd64","version": "0.5.7-4+b1","status": "ii "    },
{"package": "debconf","arch": "all","version": "1.5.56","status": "ii "    },
{"package": "debconf-i18n","arch": "all","version": "1.5.56","status": "ii "    },
{"package": "debian-archive-keyring","arch": "all","version": "2014.3","status": "ii "    },
{"package": "debianutils","arch": "amd64","version": "4.4+b1","status": "ii "    },
{"package": "dh-python","arch": "all","version": "1.20141111-2","status": "ii "    },
{"package": "diffutils","arch": "amd64","version": "1:3.3-1+b1","status": "ii "    },
{"package": "dkms","arch": "all","version": "2.2.0.3-2","status": "ii "    },
{"package": "dmsetup","arch": "amd64","version": "2:1.02.90-2.2","status": "ii "    },
{"package": "dpkg","arch": "amd64","version": "1.17.25","status": "ii "    },
{"package": "e2fslibs:amd64","arch": "amd64","version": "1.42.12-1.1","status": "ii "    },
{"package": "e2fsprogs","arch": "amd64","version": "1.42.12-1.1","status": "ii "    },
{"package": "euca2ools","arch": "all","version": "3.1.0-1","status": "ii "    },
{"package": "file","arch": "amd64","version": "1:5.22+15-2","status": "ii "    },
{"package": "findutils","arch": "amd64","version": "4.4.2-9+b1","status": "ii "    },
{"package": "fontconfig-config","arch": "all","version": "2.11.0-6.3","status": "ii "    },
{"package": "fonts-dejavu-core","arch": "all","version": "2.34-1","status": "ii "    },
{"package": "gcc","arch": "amd64","version": "4:4.9.2-2","status": "ii "    },
{"package": "gcc-4.8-base:amd64","arch": "amd64","version": "4.8.4-1","status": "ii "    },
{"package": "gcc-4.9","arch": "amd64","version": "4.9.2-10","status": "ii "    },
{"package": "gcc-4.9-base:amd64","arch": "amd64","version": "4.9.2-10","status": "ii "    },
{"package": "gettext-base","arch": "amd64","version": "0.19.3-2","status": "ii "    },
{"package": "gnupg","arch": "amd64","version": "1.4.18-7","status": "ii "    },
{"package": "gpgv","arch": "amd64","version": "1.4.18-7","status": "ii "    },
{"package": "grep","arch": "amd64","version": "2.20-4.1","status": "ii "    },
{"package": "grub-common","arch": "amd64","version": "2.02~beta2-22","status": "ii "    },
{"package": "grub-pc","arch": "amd64","version": "2.02~beta2-22","status": "ii "    },
{"package": "grub-pc-bin","arch": "amd64","version": "2.02~beta2-22","status": "ii "    },
{"package": "grub2-common","arch": "amd64","version": "2.02~beta2-22","status": "ii "    },
{"package": "gzip","arch": "amd64","version": "1.6-4","status": "ii "    },
{"package": "hostname","arch": "amd64","version": "3.15","status": "ii "    },
{"package": "ifupdown","arch": "amd64","version": "0.7.53.1","status": "ii "    },
{"package": "init","arch": "amd64","version": "1.22","status": "ii "    },
{"package": "init-system-helpers","arch": "all","version": "1.22","status": "ii "    },
{"package": "initramfs-tools","arch": "all","version": "0.120","status": "ii "    },
{"package": "initscripts","arch": "amd64","version": "2.88dsf-59","status": "ii "    },
{"package": "insserv","arch": "amd64","version": "1.14.0-5","status": "ii "    },
{"package": "iproute2","arch": "amd64","version": "3.16.0-2","status": "ii "    },
{"package": "iso-codes","arch": "all","version": "3.57-1","status": "ii "    },
{"package": "java-common","arch": "all","version": "0.52","status": "ii "    },
{"package": "kbd","arch": "amd64","version": "1.15.5-2","status": "ii "    },
{"package": "klibc-utils","arch": "amd64","version": "2.0.4-2","status": "ii "    },
{"package": "kmod","arch": "amd64","version": "18-3","status": "ii "    },
{"package": "less","arch": "amd64","version": "458-3","status": "ii "    },
{"package": "libacl1:amd64","arch": "amd64","version": "2.2.52-2","status": "ii "    },
{"package": "libapt-inst1.5:amd64","arch": "amd64","version": "1.0.9.8","status": "ii "    },
{"package": "libapt-pkg4.12:amd64","arch": "amd64","version": "1.0.9.8","status": "ii "    },
{"package": "libasan1:amd64","arch": "amd64","version": "4.9.2-10","status": "ii "    },
{"package": "libasprintf0c2:amd64","arch": "amd64","version": "0.19.3-2","status": "ii "    },
{"package": "libasyncns0:amd64","arch": "amd64","version": "0.8-5","status": "ii "    },
{"package": "libatomic1:amd64","arch": "amd64","version": "4.9.2-10","status": "ii "    },
{"package": "libattr1:amd64","arch": "amd64","version": "1:2.4.47-2","status": "ii "    },
{"package": "libaudit-common","arch": "all","version": "1:2.4-1","status": "ii "    },
{"package": "libaudit1:amd64","arch": "amd64","version": "1:2.4-1+b1","status": "ii "    },
{"package": "libavahi-client3:amd64","arch": "amd64","version": "0.6.31-5","status": "ii "    },
{"package": "libavahi-common-data:amd64","arch": "amd64","version": "0.6.31-5","status": "ii "    },
{"package": "libavahi-common3:amd64","arch": "amd64","version": "0.6.31-5","status": "ii "    },
{"package": "libblkid1:amd64","arch": "amd64","version": "2.25.2-6","status": "ii "    },
{"package": "libbsd0:amd64","arch": "amd64","version": "0.7.0-2","status": "ii "    },
{"package": "libbz2-1.0:amd64","arch": "amd64","version": "1.0.6-7+b3","status": "ii "    },
{"package": "libc-bin","arch": "amd64","version": "2.19-18","status": "ii "    },
{"package": "libc6:amd64","arch": "amd64","version": "2.19-18","status": "ii "    },
{"package": "libcap2:amd64","arch": "amd64","version": "1:2.24-8","status": "ii "    },
{"package": "libcap2-bin","arch": "amd64","version": "1:2.24-8","status": "ii "    },
{"package": "libcilkrts5:amd64","arch": "amd64","version": "4.9.2-10","status": "ii "    },
{"package": "libcloog-isl4:amd64","arch": "amd64","version": "0.18.2-1+b2","status": "ii "    },
{"package": "libcomerr2:amd64","arch": "amd64","version": "1.42.12-1.1","status": "ii "    },
{"package": "libcryptsetup4:amd64","arch": "amd64","version": "2:1.6.6-5","status": "ii "    },
{"package": "libcups2:amd64","arch": "amd64","version": "1.7.5-11","status": "ii "    },
{"package": "libcurl3:amd64","arch": "amd64","version": "7.38.0-4+deb8u2","status": "ii "    },
{"package": "libdb5.3:amd64","arch": "amd64","version": "5.3.28-9","status": "ii "    },
{"package": "libdbus-1-3:amd64","arch": "amd64","version": "1.8.18-0+deb8u1","status": "ii "    },
{"package": "libdebconfclient0:amd64","arch": "amd64","version": "0.192","status": "ii "    },
{"package": "libdevmapper1.02.1:amd64","arch": "amd64","version": "2:1.02.90-2.2","status": "ii "    },
{"package": "libedit2:amd64","arch": "amd64","version": "3.1-20140620-2","status": "ii "    },
{"package": "libexpat1:amd64","arch": "amd64","version": "2.1.0-6+b3","status": "ii "    },
{"package": "libffi6:amd64","arch": "amd64","version": "3.1-2+b2","status": "ii "    },
{"package": "libflac8:amd64","arch": "amd64","version": "1.3.0-3","status": "ii "    },
{"package": "libfontconfig1:amd64","arch": "amd64","version": "2.11.0-6.3","status": "ii "    },
{"package": "libfreetype6:amd64","arch": "amd64","version": "2.5.2-3","status": "ii "    },
{"package": "libfuse2:amd64","arch": "amd64","version": "2.9.3-15+deb8u1","status": "ii "    },
{"package": "libgcc-4.9-dev:amd64","arch": "amd64","version": "4.9.2-10","status": "ii "    },
{"package": "libgcc1:amd64","arch": "amd64","version": "1:4.9.2-10","status": "ii "    },
{"package": "libgcrypt20:amd64","arch": "amd64","version": "1.6.3-2","status": "ii "    },
{"package": "libglib2.0-0:amd64","arch": "amd64","version": "2.42.1-1","status": "ii "    },
{"package": "libgmp10:amd64","arch": "amd64","version": "2:6.0.0+dfsg-6","status": "ii "    },
{"package": "libgnutls-deb0-28:amd64","arch": "amd64","version": "3.3.8-6+deb8u1","status": "ii "    },
{"package": "libgomp1:amd64","arch": "amd64","version": "4.9.2-10","status": "ii "    },
{"package": "libgpg-error0:amd64","arch": "amd64","version": "1.17-3","status": "ii "    },
{"package": "libgssapi-krb5-2:amd64","arch": "amd64","version": "1.12.1+dfsg-19","status": "ii "    },
{"package": "libhogweed2:amd64","arch": "amd64","version": "2.7.1-5","status": "ii "    },
{"package": "libice6:amd64","arch": "amd64","version": "2:1.0.9-1+b1","status": "ii "    },
{"package": "libidn11:amd64","arch": "amd64","version": "1.29-1+b2","status": "ii "    },
{"package": "libisl10:amd64","arch": "amd64","version": "0.12.2-2","status": "ii "    },
{"package": "libitm1:amd64","arch": "amd64","version": "4.9.2-10","status": "ii "    },
{"package": "libjpeg62-turbo:amd64","arch": "amd64","version": "1:1.3.1-12","status": "ii "    },
{"package": "libjson-c2:amd64","arch": "amd64","version": "0.11-4","status": "ii "    },
{"package": "libk5crypto3:amd64","arch": "amd64","version": "1.12.1+dfsg-19","status": "ii "    },
{"package": "libkeyutils1:amd64","arch": "amd64","version": "1.5.9-5+b1","status": "ii "    },
{"package": "libklibc","arch": "amd64","version": "2.0.4-2","status": "ii "    },
{"package": "libkmod2:amd64","arch": "amd64","version": "18-3","status": "ii "    },
{"package": "libkrb5-3:amd64","arch": "amd64","version": "1.12.1+dfsg-19","status": "ii "    },
{"package": "libkrb5support0:amd64","arch": "amd64","version": "1.12.1+dfsg-19","status": "ii "    },
{"package": "liblcms2-2:amd64","arch": "amd64","version": "2.6-3+b3","status": "ii "    },
{"package": "libldap-2.4-2:amd64","arch": "amd64","version": "2.4.40+dfsg-1","status": "ii "    },
{"package": "liblocale-gettext-perl","arch": "amd64","version": "1.05-8+b1","status": "ii "    },
{"package": "liblsan0:amd64","arch": "amd64","version": "4.9.2-10","status": "ii "    },
{"package": "liblzma5:amd64","arch": "amd64","version": "5.1.1alpha+20120614-2+b3","status": "ii "    },
{"package": "libmagic1:amd64","arch": "amd64","version": "1:5.22+15-2","status": "ii "    },
{"package": "libmount1:amd64","arch": "amd64","version": "2.25.2-6","status": "ii "    },
{"package": "libmpc3:amd64","arch": "amd64","version": "1.0.2-1","status": "ii "    },
{"package": "libmpdec2:amd64","arch": "amd64","version": "2.4.1-1","status": "ii "    },
{"package": "libmpfr4:amd64","arch": "amd64","version": "3.1.2-2","status": "ii "    },
{"package": "libncurses5:amd64","arch": "amd64","version": "5.9+20140913-1+b1","status": "ii "    },
{"package": "libncursesw5:amd64","arch": "amd64","version": "5.9+20140913-1+b1","status": "ii "    },
{"package": "libnettle4:amd64","arch": "amd64","version": "2.7.1-5","status": "ii "    },
{"package": "libnspr4:amd64","arch": "amd64","version": "2:4.10.7-1","status": "ii "    },
{"package": "libnss3:amd64","arch": "amd64","version": "2:3.17.2-1.1","status": "ii "    },
{"package": "libogg0:amd64","arch": "amd64","version": "1.3.2-1","status": "ii "    },
{"package": "libp11-kit0:amd64","arch": "amd64","version": "0.20.7-1","status": "ii "    },
{"package": "libpam-modules:amd64","arch": "amd64","version": "1.1.8-3.1","status": "ii "    },
{"package": "libpam-modules-bin","arch": "amd64","version": "1.1.8-3.1","status": "ii "    },
{"package": "libpam-runtime","arch": "all","version": "1.1.8-3.1","status": "ii "    },
{"package": "libpam0g:amd64","arch": "amd64","version": "1.1.8-3.1","status": "ii "    },
{"package": "libpcre3:amd64","arch": "amd64","version": "2:8.35-3.3","status": "ii "    },
{"package": "libpcsclite1:amd64","arch": "amd64","version": "1.8.13-1","status": "ii "    },
{"package": "libpng12-0:amd64","arch": "amd64","version": "1.2.50-2+b2","status": "ii "    },
{"package": "libpopt0:amd64","arch": "amd64","version": "1.16-10","status": "ii "    },
{"package": "libprocps3:amd64","arch": "amd64","version": "2:3.3.9-9","status": "ii "    },
{"package": "libpulse0:amd64","arch": "amd64","version": "5.0-13","status": "ii "    },
{"package": "libpython-stdlib:amd64","arch": "amd64","version": "2.7.9-1","status": "ii "    },
{"package": "libpython2.7-minimal:amd64","arch": "amd64","version": "2.7.9-2","status": "ii "    },
{"package": "libpython2.7-stdlib:amd64","arch": "amd64","version": "2.7.9-2","status": "ii "    },
{"package": "libpython3-stdlib:amd64","arch": "amd64","version": "3.4.2-2","status": "ii "    },
{"package": "libpython3.4-minimal:amd64","arch": "amd64","version": "3.4.2-1","status": "ii "    },
{"package": "libpython3.4-stdlib:amd64","arch": "amd64","version": "3.4.2-1","status": "ii "    },
{"package": "libquadmath0:amd64","arch": "amd64","version": "4.9.2-10","status": "ii "    },
{"package": "libreadline6:amd64","arch": "amd64","version": "6.3-8+b3","status": "ii "    },
{"package": "librtmp1:amd64","arch": "amd64","version": "2.4+20150115.gita107cef-1","status": "ii "    },
{"package": "libsasl2-2:amd64","arch": "amd64","version": "2.1.26.dfsg1-13","status": "ii "    },
{"package": "libsasl2-modules-db:amd64","arch": "amd64","version": "2.1.26.dfsg1-13","status": "ii "    },
{"package": "libselinux1:amd64","arch": "amd64","version": "2.3-2","status": "ii "    },
{"package": "libsemanage-common","arch": "all","version": "2.3-1","status": "ii "    },
{"package": "libsemanage1:amd64","arch": "amd64","version": "2.3-1+b1","status": "ii "    },
{"package": "libsepol1:amd64","arch": "amd64","version": "2.3-2","status": "ii "    },
{"package": "libslang2:amd64","arch": "amd64","version": "2.3.0-2","status": "ii "    },
{"package": "libsm6:amd64","arch": "amd64","version": "2:1.2.2-1+b1","status": "ii "    },
{"package": "libsmartcols1:amd64","arch": "amd64","version": "2.25.2-6","status": "ii "    },
{"package": "libsndfile1:amd64","arch": "amd64","version": "1.0.25-9.1","status": "ii "    },
{"package": "libsqlite3-0:amd64","arch": "amd64","version": "3.8.7.1-1+deb8u1","status": "ii "    },
{"package": "libss2:amd64","arch": "amd64","version": "1.42.12-1.1","status": "ii "    },
{"package": "libssh2-1:amd64","arch": "amd64","version": "1.4.3-4.1","status": "ii "    },
{"package": "libssl1.0.0:amd64","arch": "amd64","version": "1.0.1k-3","status": "ii "    },
{"package": "libstdc++6:amd64","arch": "amd64","version": "4.9.2-10","status": "ii "    },
{"package": "libsystemd0:amd64","arch": "amd64","version": "215-17+deb8u1","status": "ii "    },
{"package": "libtasn1-6:amd64","arch": "amd64","version": "4.2-3+deb8u1","status": "ii "    },
{"package": "libtext-charwidth-perl","arch": "amd64","version": "0.04-7+b3","status": "ii "    },
{"package": "libtext-iconv-perl","arch": "amd64","version": "1.7-5+b2","status": "ii "    },
{"package": "libtext-wrapi18n-perl","arch": "all","version": "0.06-7","status": "ii "    },
{"package": "libtinfo5:amd64","arch": "amd64","version": "5.9+20140913-1+b1","status": "ii "    },
{"package": "libtsan0:amd64","arch": "amd64","version": "4.9.2-10","status": "ii "    },
{"package": "libubsan0:amd64","arch": "amd64","version": "4.9.2-10","status": "ii "    },
{"package": "libudev1:amd64","arch": "amd64","version": "215-17+deb8u1","status": "ii "    },
{"package": "libusb-0.1-4:amd64","arch": "amd64","version": "2:0.1.12-25","status": "ii "    },
{"package": "libustr-1.0-1:amd64","arch": "amd64","version": "1.0.4-3+b2","status": "ii "    },
{"package": "libuuid-perl","arch": "amd64","version": "0.05-1+b1","status": "ii "    },
{"package": "libuuid1:amd64","arch": "amd64","version": "2.25.2-6","status": "ii "    },
{"package": "libvorbis0a:amd64","arch": "amd64","version": "1.3.4-2","status": "ii "    },
{"package": "libvorbisenc2:amd64","arch": "amd64","version": "1.3.4-2","status": "ii "    },
{"package": "libwrap0:amd64","arch": "amd64","version": "7.6.q-25","status": "ii "    },
{"package": "libx11-6:amd64","arch": "amd64","version": "2:1.6.2-3","status": "ii "    },
{"package": "libx11-data","arch": "all","version": "2:1.6.2-3","status": "ii "    },
{"package": "libx11-xcb1:amd64","arch": "amd64","version": "2:1.6.2-3","status": "ii "    },
{"package": "libxau6:amd64","arch": "amd64","version": "1:1.0.8-1","status": "ii "    },
{"package": "libxcb1:amd64","arch": "amd64","version": "1.10-3+b1","status": "ii "    },
{"package": "libxdmcp6:amd64","arch": "amd64","version": "1:1.1.1-1+b1","status": "ii "    },
{"package": "libxext6:amd64","arch": "amd64","version": "2:1.3.3-1","status": "ii "    },
{"package": "libxi6:amd64","arch": "amd64","version": "2:1.7.4-1+b2","status": "ii "    },
{"package": "libxml2:amd64","arch": "amd64","version": "2.9.1+dfsg1-5","status": "ii "    },
{"package": "libxslt1.1:amd64","arch": "amd64","version": "1.1.28-2+b2","status": "ii "    },
{"package": "libxtst6:amd64","arch": "amd64","version": "2:1.2.2-1+b1","status": "ii "    },
{"package": "libyaml-0-2:amd64","arch": "amd64","version": "0.1.6-3","status": "ii "    },
{"package": "linux-base","arch": "all","version": "3.5","status": "ii "    },
{"package": "linux-image-3.16.0-4-amd64","arch": "amd64","version": "3.16.7-ckt11-1","status": "ii "    },
{"package": "linux-image-amd64","arch": "amd64","version": "3.16+63","status": "ii "    },
{"package": "locales","arch": "all","version": "2.19-18","status": "ii "    },
{"package": "locales-all","arch": "amd64","version": "2.19-18","status": "ii "    },
{"package": "login","arch": "amd64","version": "1:4.2-3","status": "ii "    },
{"package": "lsb-base","arch": "all","version": "4.1+Debian13+nmu1","status": "ii "    },
{"package": "lsb-release","arch": "all","version": "4.1+Debian13+nmu1","status": "ii "    },
{"package": "make","arch": "amd64","version": "4.0-8.1","status": "ii "    },
{"package": "mawk","arch": "amd64","version": "1.3.3-17","status": "ii "    },
{"package": "mime-support","arch": "all","version": "3.58","status": "ii "    },
{"package": "mount","arch": "amd64","version": "2.25.2-6","status": "ii "    },
{"package": "multiarch-support","arch": "amd64","version": "2.19-18","status": "ii "    },
{"package": "ncurses-base","arch": "all","version": "5.9+20140913-1","status": "ii "    },
{"package": "ncurses-bin","arch": "amd64","version": "5.9+20140913-1+b1","status": "ii "    },
{"package": "open-iscsi","arch": "amd64","version": "2.0.873+git0.3b4b4500-8+deb8u1","status": "ii "    },
{"package": "openjdk-7-jre-headless:amd64","arch": "amd64","version": "7u75-2.5.4-2","status": "ii "    },
{"package": "openssh-client","arch": "amd64","version": "1:6.7p1-5","status": "ii "    },
{"package": "openssh-server","arch": "amd64","version": "1:6.7p1-5","status": "ii "    },
{"package": "openssh-sftp-server","arch": "amd64","version": "1:6.7p1-5","status": "ii "    },
{"package": "openssl","arch": "amd64","version": "1.0.1k-3","status": "ii "    },
{"package": "passwd","arch": "amd64","version": "1:4.2-3","status": "ii "    },
{"package": "patch","arch": "amd64","version": "2.7.5-1","status": "ii "    },
{"package": "perl-base","arch": "amd64","version": "5.20.2-3+deb8u1","status": "ii "    },
{"package": "procps","arch": "amd64","version": "2:3.3.9-9","status": "ii "    },
{"package": "python","arch": "amd64","version": "2.7.9-1","status": "ii "    },
{"package": "python-apt","arch": "amd64","version": "0.9.3.11","status": "ii "    },
{"package": "python-apt-common","arch": "all","version": "0.9.3.11","status": "ii "    },
{"package": "python-boto","arch": "all","version": "2.34.0-2","status": "ii "    },
{"package": "python-chardet","arch": "all","version": "2.3.0-1","status": "ii "    },
{"package": "python-cheetah","arch": "amd64","version": "2.4.4-3","status": "ii "    },
{"package": "python-configobj","arch": "all","version": "5.0.6-1","status": "ii "    },
{"package": "python-crypto","arch": "amd64","version": "2.6.1-5+b2","status": "ii "    },
{"package": "python-ecdsa","arch": "all","version": "0.11-1","status": "ii "    },
{"package": "python-json-pointer","arch": "all","version": "1.0-2","status": "ii "    },
{"package": "python-jsonpatch","arch": "all","version": "1.3-5","status": "ii "    },
{"package": "python-lxml","arch": "amd64","version": "3.4.0-1","status": "ii "    },
{"package": "python-minimal","arch": "amd64","version": "2.7.9-1","status": "ii "    },
{"package": "python-oauth","arch": "all","version": "1.0.1-4","status": "ii "    },
{"package": "python-paramiko","arch": "all","version": "1.15.1-1","status": "ii "    },
{"package": "python-pkg-resources","arch": "all","version": "5.5.1-1","status": "ii "    },
{"package": "python-prettytable","arch": "all","version": "0.7.2-3","status": "ii "    },
{"package": "python-requestbuilder","arch": "all","version": "0.2.3-1","status": "ii "    },
{"package": "python-requests","arch": "all","version": "2.4.3-6","status": "ii "    },
{"package": "python-serial","arch": "all","version": "2.6-1.1","status": "ii "    },
{"package": "python-setuptools","arch": "all","version": "5.5.1-1","status": "ii "    },
{"package": "python-six","arch": "all","version": "1.8.0-1","status": "ii "    },
{"package": "python-software-properties","arch": "all","version": "0.92.25debian1","status": "ii "    },
{"package": "python-urllib3","arch": "all","version": "1.9.1-3","status": "ii "    },
{"package": "python-yaml","arch": "amd64","version": "3.11-2","status": "ii "    },
{"package": "python2.7","arch": "amd64","version": "2.7.9-2","status": "ii "    },
{"package": "python2.7-minimal","arch": "amd64","version": "2.7.9-2","status": "ii "    },
{"package": "python3","arch": "amd64","version": "3.4.2-2","status": "ii "    },
{"package": "python3-apt","arch": "amd64","version": "0.9.3.11","status": "ii "    },
{"package": "python3-minimal","arch": "amd64","version": "3.4.2-2","status": "ii "    },
{"package": "python3.4","arch": "amd64","version": "3.4.2-1","status": "ii "    },
{"package": "python3.4-minimal","arch": "amd64","version": "3.4.2-1","status": "ii "    },
{"package": "readline-common","arch": "all","version": "6.3-8","status": "ii "    },
{"package": "rsync","arch": "amd64","version": "3.1.1-3","status": "ii "    },
{"package": "sed","arch": "amd64","version": "4.2.2-4+b1","status": "ii "    },
{"package": "sensible-utils","arch": "all","version": "0.0.9","status": "ii "    },
{"package": "startpar","arch": "amd64","version": "0.59-3","status": "ii "    },
{"package": "sudo","arch": "amd64","version": "1.8.10p3-1+deb8u2","status": "ii "    },
{"package": "systemd","arch": "amd64","version": "215-17+deb8u1","status": "ii "    },
{"package": "systemd-sysv","arch": "amd64","version": "215-17+deb8u1","status": "ii "    },
{"package": "sysv-rc","arch": "all","version": "2.88dsf-59","status": "ii "    },
{"package": "sysvinit-utils","arch": "amd64","version": "2.88dsf-59","status": "ii "    },
{"package": "tar","arch": "amd64","version": "1.27.1-2+b1","status": "ii "    },
{"package": "tzdata","arch": "all","version": "2015d-0+deb8u1","status": "ii "    },
{"package": "tzdata-java","arch": "all","version": "2015d-0+deb8u1","status": "ii "    },
{"package": "ucf","arch": "all","version": "3.0030","status": "ii "    },
{"package": "udev","arch": "amd64","version": "215-17+deb8u1","status": "ii "    },
{"package": "unattended-upgrades","arch": "all","version": "0.83.3.1","status": "ii "    },
{"package": "util-linux","arch": "amd64","version": "2.25.2-6","status": "ii "    },
{"package": "x11-common","arch": "all","version": "1:7.7+7","status": "ii "    },
{"package": "xz-utils","arch": "amd64","version": "5.1.1alpha+20120614-2+b3","status": "ii "    },
{"package": "zlib1g:amd64","arch": "amd64","version": "1:1.2.8.dfsg-2+b1","status": "ii "}
]}
hashar added a comment.Jul 9 2015, 11:56 AM

cloud-init uses ifconfig to gather network configuration, but it is not installed. We need the package net-tools in the disk image.

debootstrap is run with`--variant=minbase --include=python,sudo,$KERNEL_PACKAGE`

hashar added a project: Upstream.Jul 9 2015, 12:02 PM

I proposed to upstream https://review.openstack.org/200030 which adds the packages isc-dhcp-client and net-tools to the element debian.

hashar claimed this task.Jul 9 2015, 12:02 PM
hashar triaged this task as High priority.
hashar moved this task from Backlog to Patch proposed upstream on the Upstream board.
hashar moved this task from Backlog to In-progress on the Continuous-Integration-Scaling board.
hashar added a parent task: T102878: Create a Jessie image with diskimage-builder suitable for nodepool.
hashar added a comment.Jul 9 2015, 12:11 PM

Adding:

DIB_DEBOOTSTRAP_EXTRA_ARGS: '--include isc-dhcp-client,net-tools'  # T105152

Then disk image uses:

sudo sh -c 'http_proxy= debootstrap --verbose \
        --variant=minbase \
        --include=python,sudo,linux-image-amd64 \
        --components=main         --arch=amd64 \
        --include isc-dhcp-client,net-tools \   <------------------------- there
        jessie         /tmp/image.nfwweRor/mnt \
        http://mirrors.wikimedia.org/debian/         '
hashar added a comment.Jul 9 2015, 12:38 PM

The second --include override the first. Thus the image fails because it lacks python.

gerritbot subscribed.Jul 9 2015, 1:56 PM

Change 223777 had a related patch set uploaded (by Hashar):
nodepool: element with basic networking packages

https://gerrit.wikimedia.org/r/223777

gerritbot added a project: Patch-For-Review.Jul 9 2015, 1:56 PM
hashar added a comment.Jul 9 2015, 2:01 PM

Fixed by introducing a new element wikimedia-networking which installs the dhcp client and ifconfig commands ( https://gerrit.wikimedia.org/r/223777 ).

hashar moved this task from In-progress to Done on the Continuous-Integration-Scaling board.Jul 9 2015, 6:26 PM
gerritbot added a comment.Jul 9 2015, 9:03 PM

Change 223777 merged by Andrew Bogott:
nodepool: element with basic networking packages

https://gerrit.wikimedia.org/r/223777

Andrew mentioned this in rOPUP3c88063ac4d0: nodepool: element with basic networking packages.Jul 9 2015, 9:03 PM
hashar closed this task as Resolved.Jul 9 2015, 9:18 PM

One less problem!

gerritbot added a comment.Aug 31 2015, 1:41 PM

Change 234994 had a related patch set uploaded (by Hashar):
dib: bump to 1.0.0 and remove wikimedia-networking

https://gerrit.wikimedia.org/r/234994

gerritbot added a comment.Aug 31 2015, 2:24 PM

Change 234994 abandoned by Hashar:
dib: bump to 1.0.0 and remove wikimedia-networking

Reason:
Squashed in https://gerrit.wikimedia.org/r/234300 which introduced wikimedia-networking

https://gerrit.wikimedia.org/r/234994

hashar mentioned this in T133203: Attempt to provide a Trusty image for Nodepool.Apr 25 2016, 1:58 PM
hashar added a comment.Apr 25 2016, 4:26 PM

Fixed by upstream change https://review.openstack.org/#/c/200030/ which is in dib since 1.0.0.

• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:49 PM
Restricted Application added a project: Release-Engineering-Team (Kanban). · View Herald TranscriptJun 7 2017, 6:49 PM
• Phabricator_maintenance edited projects, added RelEng-Archive-FY201718-Q1; removed Release-Engineering-Team (Kanban).Sep 26 2017, 11:47 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits