Page MenuHomePhabricator

Latest Trusty kernel boot issues: network and grub
Closed, ResolvedPublic

Description

With the post-meltdown kernel (BOOT_IMAGE=/boot/vmlinuz-4.4.0-109-generic) labvirt1018 can't connect to its network:

 * Starting set console font                                             [ OK ]
 * Stopping set console font                                             [ OK ]
 * Starting userspace bootsplash                                         [ OK ]
 * Starting Send an event to indicate plymouth is up                     [ OK ]
 * Stopping userspace bootsplash                                         [ OK ]
 * Starting configure virtual network devices                            [ OK ]
 * Stopping Send an event to indicate plymouth is up                     [ OK ]
Waiting for network configuration...
Waiting up to 60 more seconds for network configuration...
Booting system without full network configuration...
 * Stopping Failsafe Boot Delay                                          [ OK ]
 * Starting System V initialisation compatibility

We've seen this before, which is why we pinned the lucky kernel version 4.4.0-81-generic. The networking issue occurred sometimes in kernels older and newer depending, although it seems like newer hardware in cases needed the extra package for drivers.


labtestvirt2002 as a guinea pig (3.13 variant too):

linux-image-generic-lts-xenial => linux-image-4.4.0-109-generic{a} linux-image-extra-4.4.0-109-generic{a} linux-image-generic-lts-xenial

apt-get install -y linux-image-4.4.0-109-generic linux-image-extra-4.4.0-109-generic linux-lts-xenial-tools-4.4.0-109 linux-tools-4.4.0-109-generic

...reboots into grub prompt...

grub>

I can get out of it by going back to older 4.4.0-81-generic

aptitude remove --purge linux-image-4.4.0-109-generic

...reboots fine...

I think linux-image-4.4.0-109-generic is right even though https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown?_ga=2.45330468.1075164870.1515518269-114292014.1505251686 notes linux-lts-xenial 4.4.0-108.131~14.04.1 Trusty 14.04 because https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1741934/comments/17 was already filed and fixed bumping the correct version even though the docs are not updated?

Event Timeline

Change 403456 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Revert "labvirts: whitelist the post-meltdown kernel version"

https://gerrit.wikimedia.org/r/403456

Change 403456 merged by Andrew Bogott:
[operations/puppet@production] Revert "labvirts: whitelist the post-meltdown kernel version"

https://gerrit.wikimedia.org/r/403456

labtestvirt2002 came up similarly to 2003 in testing to a grub boot menu post new kernel and reboot

https://phabricator.wikimedia.org/P6572

grub>

I believe previously (at least in some cases https://phabricator.wikimedia.org/T183035#3848319) this was a drivers issue, as I was able to overcome it iirc by installing the extra package where it didn't exist.

But that was where it was booting to a state without nic drivers, the boot into grub prompt is new. labtestvirt2002 post P6572 does

1Configuring memory. Please wait...
2
3
4
5
6 Press the spacebar to pause...
7
8 KEY MAPPING FOR CONSOLE REDIRECTION:
9
10 Use the <ESC><0> key sequence for <F10>
11 Use the <ESC><!> key sequence for <F11>
12 Use the <ESC><@> key sequence for <F12>
13
14 Use the <ESC><Ctrl><M> key sequence for <Ctrl><M>
15 Use the <ESC><Ctrl><H> key sequence for <Ctrl><H>
16 Use the <ESC><Ctrl><I> key sequence for <Ctrl><I>
17 Use the <ESC><Ctrl><J> key sequence for <Ctrl><J>
18
19 Use the <ESC><X><X> key sequence for <Alt><x>, where x is any letter
20 key, and X is the upper case of that key
21
22 Use the <ESC><R><ESC><r><ESC><R> key sequence for <Ctrl><Alt><Del>
23 F2 = System Setup
24Phoenix ROM BIOS PLUS Version 1.10 1.6.3 F10 = System Services
25Copyright 1985-1988 Phoenix Technologies Ltd. F11 = BIOS Boot Manager
26Copyright 1990-2011 Dell Inc. F12 = PXE Boot
27 s
28De l Sys em PowerE ge R410
29www.dell.comPowerEdge R410
30Twsting mcoory. Please wait.
31 ry. Please wait.
32
33
34System running at 2.66 GHz
35System6Memory Size: 8.0 GB,sSystemBMemoryeSpeed: 1333,MHz,LVoltage:11.5VB/12 MB
36
37 S e 0 , System Memory Speed: 1333 MHz, Voltage: 1.5V
38
39Dell SAS 6 Host Bus Adapter BIOS
40MPTBIOS-6.22.03.00 (2008.08.06)
41Copyright 2000-2008 LSI Corporation.
42 (
43 c ..
44Searching for devices at HBA 0...
45
46 HBA ID LUN VENDOR PRODUCT REVISION CAPACITY
47 --- --- --- -------- ---------------- ---------0 ----------
48 0 0 0 Dell VIRTUAL DISK 1028 465 GB
49 8 P BACKPLANE .07
50 LSILogic S S1068E-IR 0 25.47.00 NV 2D:09
51Dell Inc. MPT boot ROM successfully installed!
52
53
54PressiCtrl-Setorenter1ConfigurationrMenution.0.11
55
56
57
58
59iDRAC6 Configuration Utility 1.59
60Copyright 2011 Dell Inc. All Rights Reserved
61
62iDRAC6 Firmware Revision version: 1.70.21
63Primary Backplane Firmware Revision 1.07
64
65 GNU GRUB version 2.02~beta2-9ubuntu1.14
66
67 Minimal BASH-like line editing is supported. For the first word, TAB
68 lists possible command completions. Anywhere else TAB lists possible
69 device or file completions.
70
71
72grub>

I believe previously (at least in some cases https://phabricator.wikimedia.org/T183035#3848319) this was a drives issue, as I was able to overcome it iirc by installing the extra package where it didn't exist.

But that was where it was booting to a state without nic drivers, the boot into grub prompt is new. labtestvirt2002 post P6572 does

1Configuring memory. Please wait...
2
3
4
5
6 Press the spacebar to pause...
7
8 KEY MAPPING FOR CONSOLE REDIRECTION:
9
10 Use the <ESC><0> key sequence for <F10>
11 Use the <ESC><!> key sequence for <F11>
12 Use the <ESC><@> key sequence for <F12>
13
14 Use the <ESC><Ctrl><M> key sequence for <Ctrl><M>
15 Use the <ESC><Ctrl><H> key sequence for <Ctrl><H>
16 Use the <ESC><Ctrl><I> key sequence for <Ctrl><I>
17 Use the <ESC><Ctrl><J> key sequence for <Ctrl><J>
18
19 Use the <ESC><X><X> key sequence for <Alt><x>, where x is any letter
20 key, and X is the upper case of that key
21
22 Use the <ESC><R><ESC><r><ESC><R> key sequence for <Ctrl><Alt><Del>
23 F2 = System Setup
24Phoenix ROM BIOS PLUS Version 1.10 1.6.3 F10 = System Services
25Copyright 1985-1988 Phoenix Technologies Ltd. F11 = BIOS Boot Manager
26Copyright 1990-2011 Dell Inc. F12 = PXE Boot
27 s
28De l Sys em PowerE ge R410
29www.dell.comPowerEdge R410
30Twsting mcoory. Please wait.
31 ry. Please wait.
32
33
34System running at 2.66 GHz
35System6Memory Size: 8.0 GB,sSystemBMemoryeSpeed: 1333,MHz,LVoltage:11.5VB/12 MB
36
37 S e 0 , System Memory Speed: 1333 MHz, Voltage: 1.5V
38
39Dell SAS 6 Host Bus Adapter BIOS
40MPTBIOS-6.22.03.00 (2008.08.06)
41Copyright 2000-2008 LSI Corporation.
42 (
43 c ..
44Searching for devices at HBA 0...
45
46 HBA ID LUN VENDOR PRODUCT REVISION CAPACITY
47 --- --- --- -------- ---------------- ---------0 ----------
48 0 0 0 Dell VIRTUAL DISK 1028 465 GB
49 8 P BACKPLANE .07
50 LSILogic S S1068E-IR 0 25.47.00 NV 2D:09
51Dell Inc. MPT boot ROM successfully installed!
52
53
54PressiCtrl-Setorenter1ConfigurationrMenution.0.11
55
56
57
58
59iDRAC6 Configuration Utility 1.59
60Copyright 2011 Dell Inc. All Rights Reserved
61
62iDRAC6 Firmware Revision version: 1.70.21
63Primary Backplane Firmware Revision 1.07
64
65 GNU GRUB version 2.02~beta2-9ubuntu1.14
66
67 Minimal BASH-like line editing is supported. For the first word, TAB
68 lists possible command completions. Anywhere else TAB lists possible
69 device or file completions.
70
71
72grub>

After much gnashing of teeth w/ grub I think this:

grub> ls

(hd0) (hd0,msdos2) (hd0,msdos1)
grub> cat (hd0,msdos1)/etc/issue
14.04.5 LTS \n \
grub> set root=(hd0,msdos1)
grub> set prefix=(hd0,msdos1)/boot/grub
grub> insmod normal
grub> linux /boot/vmlinuz-4.4.0-81-generic root=/dev/sda1
grub> initrd /boot/initrd.img-4.4.0-81-generic
grub> boot

worked out as it booted me into the old kernel and I can ssh in.

rush@labtestvirt2002>uname -r
4.4.0-81-generic

That seems to speak to this being an issue with 4.4.0-109-generic

After doing this I can reboot safely back into 4.4.0-81-generic:

root@labtestvirt2002:~# aptitude remove --purge linux-image-4.4.0-98-generic

root@labtestvirt2002:~# sudo update-grub
Generating grub configuration file ...
Warning: Setting GRUB_TIMEOUT to a non-zero value when GRUB_HIDDEN_TIMEOUT is set is no longer supported.
Found linux image: /boot/vmlinuz-4.4.0-81-generic
Found initrd image: /boot/initrd.img-4.4.0-81-generic
Found linux image: /boot/vmlinuz-3.13.0-137-generic
Found initrd image: /boot/initrd.img-3.13.0-137-generic
Found memtest86+ image: /boot/memtest86+.elf
Found memtest86+ image: /boot/memtest86+.bin
done

https://www.linux.com/learn/how-rescue-non-booting-grub-2-Linux
https://www.gnu.org/software/grub/manual/grub/grub.html
https://unix.stackexchange.com/questions/329926/grub-starts-in-command-line-after-reboot
https://www.ibm.com/developerworks/library/l-GRUB2-features/index.html
https://askubuntu.com/questions/266429/error-file-grub-i386-pc-normal-mod-not-found

Labtestvirt2002: Dell PowerEdge R410 (HW warranty 2016-01-13)
Labvirt1018: Dell PowerEdge R630 (HW warranty 2020-05-06)

Labtestvirt2001 (which seems to have come up fine) Dell PowerEdge R630 (HW warranty 09/08/2011)

hmmm....throwing stuff at the wall in the past we saw older hardware work just fine and newer need the extra package. I'm pretty confused.

chasemp updated the task description. (Show Details)
chasemp updated the task description. (Show Details)

For fun I installed the 3.13 meltdown variant from https://usn.ubuntu.com/usn/usn-3524-1/ on labtestvirt2002 to see what happens:

root@labtestvirt2002:~# aptitude install linux-image-3.13.0-139-generic
The following NEW packages will be installed:
  linux-image-3.13.0-139-generic
0 packages upgraded, 1 newly installed, 0 to remove and 14 not upgraded.
Need to get 15.4 MB of archives. After unpacking 43.7 MB will be used.
Get: 1 http://mirrors.wikimedia.org/ubuntu/ trusty-updates/main linux-image-3.13.0-139-generic amd64 3.13.0-139.188 [15.4 MB]
Fetched 15.4 MB in 0s (17.5 MB/s)
Selecting previously unselected package linux-image-3.13.0-139-generic.
(Reading database ... 76780 files and directories currently installed.)
Preparing to unpack .../linux-image-3.13.0-139-generic_3.13.0-139.188_amd64.deb ...
Done.
Unpacking linux-image-3.13.0-139-generic (3.13.0-139.188) ...
Setting up linux-image-3.13.0-139-generic (3.13.0-139.188) ...
Running depmod.
update-initramfs: deferring update (hook will be called later)
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 3.13.0-139-generic /boot/vmlinuz-3.13.0-139-generic
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 3.13.0-139-generic /boot/vmlinuz-3.13.0-139-generic
update-initramfs: Generating /boot/initrd.img-3.13.0-139-generic
run-parts: executing /etc/kernel/postinst.d/update-notifier 3.13.0-139-generic /boot/vmlinuz-3.13.0-139-generic
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 3.13.0-139-generic /boot/vmlinuz-3.13.0-139-generic
Generating grub configuration file ...
Warning: Setting GRUB_TIMEOUT to a non-zero value when GRUB_HIDDEN_TIMEOUT is set is no longer supported.
Found linux image: /boot/vmlinuz-4.4.0-81-generic
Found initrd image: /boot/initrd.img-4.4.0-81-generic
Found linux image: /boot/vmlinuz-3.13.0-139-generic
Found initrd image: /boot/initrd.img-3.13.0-139-generic
Found linux image: /boot/vmlinuz-3.13.0-137-generic
Found initrd image: /boot/initrd.img-3.13.0-137-generic
Found memtest86+ image: /boot/memtest86+.elf
Found memtest86+ image: /boot/memtest86+.bin
done

root@labtestvirt2002:~# sudo update-grub
Generating grub configuration file ...
Warning: Setting GRUB_TIMEOUT to a non-zero value when GRUB_HIDDEN_TIMEOUT is set is no longer supported.
Found linux image: /boot/vmlinuz-4.4.0-81-generic
Found initrd image: /boot/initrd.img-4.4.0-81-generic
Found linux image: /boot/vmlinuz-3.13.0-139-generic
Found initrd image: /boot/initrd.img-3.13.0-139-generic
Found linux image: /boot/vmlinuz-3.13.0-137-generic
Found initrd image: /boot/initrd.img-3.13.0-137-generic
Found memtest86+ image: /boot/memtest86+.elf
Found memtest86+ image: /boot/memtest86+.bin
done
root@labtestvirt2002:~# /sbin/reboot

Broadcast message from rush@labtestvirt2002
	(/dev/pts/1) at 20:09 ...

The system is going down for reboot NOW!

... drum roll ...

grub>

labvirt1018

root@labvirt1018:~# ifconfig
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

root@labvirt1018:~# ifconfig -a
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

root@labvirt1018:~# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
root@labvirt1018:~# uname -a
Linux labvirt1018 4.4.0-109-generic #132~14.04.1-Ubuntu SMP Tue Jan 9 21:46:42 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
root@labvirt1018:~#
labvirt1018:~#  lshw -class network
PCI (sysfs)
  *-network:0 UNCLAIMED
       description: Ethernet controller
       product: NetXtreme BCM5720 Gigabit Ethernet PCIe
       vendor: Broadcom Corporation
       physical id: 0
       bus info: pci@0000:02:00.0
       version: 00
       width: 64 bits
       clock: 33MHz
       capabilities: pm vpd msi msix pciexpress bus_master cap_list
       configuration: latency=0
       resources: memory:91a30000-91a3ffff memory:91a40000-91a4ffff memory:91a50000-91a5ffff memory:91f00000-91f3ffff
  *-network:1 UNCLAIMED
       description: Ethernet controller
       product: NetXtreme BCM5720 Gigabit Ethernet PCIe
       vendor: Broadcom Corporation
       physical id: 0.1
       bus info: pci@0000:02:00.1
       version: 00
       width: 64 bits
       clock: 33MHz
       capabilities: pm vpd msi msix pciexpress bus_master cap_list
       configuration: latency=0
       resources: memory:91a00000-91a0ffff memory:91a10000-91a1ffff memory:91a20000-91a2ffff memory:91f40000-91f7ffff
  *-network:0 UNCLAIMED
       description: Ethernet controller
       product: NetXtreme BCM5720 Gigabit Ethernet PCIe
       vendor: Broadcom Corporation
       physical id: 0
       bus info: pci@0000:01:00.0
       version: 00
       width: 64 bits
       clock: 33MHz
       capabilities: pm vpd msi msix pciexpress bus_master cap_list
       configuration: latency=0
       resources: memory:91b30000-91b3ffff memory:91b40000-91b4ffff memory:91b50000-91b5ffff memory:92000000-9203ffff
  *-network:1 UNCLAIMED
       description: Ethernet controller
       product: NetXtreme BCM5720 Gigabit Ethernet PCIe
       vendor: Broadcom Corporation
       physical id: 0.1
       bus info: pci@0000:01:00.1
       version: 00
       width: 64 bits
       clock: 33MHz
       capabilities: pm vpd msi msix pciexpress bus_master cap_list
       configuration: latency=0
       resources: memory:91b00000-91b0ffff memory:91b10000-91b1ffff memory:91b20000-91b2ffff memory:92040000-9207ffff

lshw shows the exact same nic in labvirt1018 and labtestvirt2001 it seems so I wanted to check on the packages on 1018 and it does indeed lack the extra package:

root@labvirt1018:~# dpkg -l | grep linux | grep 4 | grep 109
ii  linux-image-4.4.0-109-generic        4.4.0-109.132~14.04.1                      amd64        Linux kernel image for version 4.4.0 on 64 bit x86 SMP
root@labvirt1018:~#

I'm going to move back to the old known good and install all the needed packages:

apt-get install -y linux-image-4.4.0-109-generic linux-image-extra-4.4.0-109-generic linux-lts-xenial-tools-4.4.0-109 linux-tools-4.4.0-109-generic

chasemp renamed this task from Latest Trusty kernels break networking on labvirts to Latest Trusty kernel boot issues: network and grub.Jan 10 2018, 9:29 PM
root@labvirt1018:~# uname -a
Linux labvirt1018 4.4.0-109-generic #132~14.04.1-Ubuntu SMP Tue Jan 9 21:46:42 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
root@labvirt1018:~# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 18:66:da:fc:9c:58 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 18:66:da:fc:9c:59 brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 18:66:da:fc:9c:5a brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 18:66:da:fc:9c:5b brd ff:ff:ff:ff:ff:ff
6: eth1.1102@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br1102 state UP mode DEFAULT group default qlen 1000
    link/ether 18:66:da:fc:9c:59 brd ff:ff:ff:ff:ff:ff
7: br1102: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 18:66:da:fc:9c:59 brd ff:ff:ff:ff:ff:ff
root@labvirt1018:~#

OK back to labtestvirt2002

root@labtestvirt2002:~# apt-get install -y linux-image-4.4.0-109-generic linux-image-extra-4.4.0-109-generic linux-lts-xenial-tools-4.4.0-109 linux-tools-4.4.0-109-generic
sudo update-grub

Here is grub.cfg => P6577

/sbin/reboot

For labtestvirt2003 there is a difference in newer HP hardware that caused a PXE boot loop when following our docs I updated them here https://wikitech.wikimedia.org/w/index.php?title=Platform-specific_documentation%2FHP_DL3N0&type=revision&diff=1780099&oldid=817967

At the moment the only question I have is what happened w/ labtestvirt2002 the first time around, because once I ran:

aptitude remove --purge linux-image-3.13.0-139-generic
aptitude remove --purge linux-image-4.4.0-109-generic
aptitude remove --purge linux-image-4.4.0-98-generic

and then booted into 4.4.0-81-generic and then:

apt-get install -y linux-image-4.4.0-109-generic linux-image-extra-4.4.0-109-generic linux-lts-xenial-tools-4.4.0-109 linux-tools-4.4.0-109-generic
sudo update-gub

things came up fine. So that's a concern but I can't replicate atm and it's 1 of 4 and it didn't technically break on the new kernel more like grub broke.

Tentatively I think the issues here were:

  • labtestvirt2003 new HP hardware has a different BIOS setting for non-PXE than historical
  • labtestvirt2002 grub ate itself the first time around
  • labvirt1018 was missing the extra package which is a known bad state
chasemp claimed this task.

tentatively closing this until we see something more concrete that is actionable here

238482n375 lowered the priority of this task from High to Lowest.
238482n375 moved this task from Next Up to In Code Review on the Analytics-Kanban board.
238482n375 edited subscribers, added: 238482n375; removed: Aklapper.

SG9tZVBoYWJyaWNhdG9yCk5vIG1lc3NhZ2VzLiBObyBub3RpZmljYXRpb25zLgoKICAgIFNlYXJjaAoKQ3JlYXRlIFRhc2sKTWFuaXBoZXN0ClQxOTcyODEKRml4IGZhaWxpbmcgd2VicmVxdWVzdCBob3VycyAodXBsb2FkIGFuZCB0ZXh0IDIwMTgtMDYtMTQtMTEpCk9wZW4sIE5lZWRzIFRyaWFnZVB1YmxpYwoKICAgIEVkaXQgVGFzawogICAgRWRpdCBSZWxhdGVkIFRhc2tzLi4uCiAgICBFZGl0IFJlbGF0ZWQgT2JqZWN0cy4uLgogICAgUHJvdGVjdCBhcyBzZWN1cml0eSBpc3N1ZQoKICAgIE11dGUgTm90aWZpY2F0aW9ucwogICAgQXdhcmQgVG9rZW4KICAgIEZsYWcgRm9yIExhdGVyCgpUYWdzCgogICAgQW5hbHl0aWNzLUthbmJhbiAoSW4gUHJvZ3Jlc3MpCgpTdWJzY3JpYmVycwpBa2xhcHBlciwgSkFsbGVtYW5kb3UKQXNzaWduZWQgVG8KSkFsbGVtYW5kb3UKQXV0aG9yZWQgQnkKSkFsbGVtYW5kb3UsIEZyaSwgSnVuIDE1CkRlc2NyaXB0aW9uCgpPb3ppZSBqb2JzIGhhdmUgYmVlbiBmYWlsaW5nIGF0IGxlYXN0IGEgZmV3IHRpbWVzIGVhY2guIE1vcmUgaW52ZXN0aWdhdGlvbiBuZWVkZWQuCkpBbGxlbWFuZG91IGNyZWF0ZWQgdGhpcyB0YXNrLkZyaSwgSnVuIDE1LCA3OjIxIEFNCkhlcmFsZCBhZGRlZCBhIHN1YnNjcmliZXI6IEFrbGFwcGVyLiC3IFZpZXcgSGVyYWxkIFRyYW5zY3JpcHRGcmksIEp1biAxNSwgNzoyMSBBTQpKQWxsZW1hbmRvdSBjbGFpbWVkIHRoaXMgdGFzay5GcmksIEp1biAxNSwgNzoyMiBBTQpKQWxsZW1hbmRvdSB1cGRhdGVkIHRoZSB0YXNrIGRlc2NyaXB0aW9uLiAoU2hvdyBEZXRhaWxzKQpKQWxsZW1hbmRvdSBhZGRlZCBhIHByb2plY3Q6IEFuYWx5dGljcy1LYW5iYW4uCkpBbGxlbWFuZG91IG1vdmVkIHRoaXMgdGFzayBmcm9tIE5leHQgVXAgdG8gSW4gUHJvZ3Jlc3Mgb24gdGhlIEFuYWx5dGljcy1LYW5iYW4gYm9hcmQuCkNoYW5nZSBTdWJzY3JpYmVycwpDaGFuZ2UgUHJpb3JpdHkKQXNzaWduIC8gQ2xhaW0KTW92ZSBvbiBXb3JrYm9hcmQKQ2hhbmdlIFByb2plY3QgVGFncwpBbmFseXRpY3MtS2FuYmFuCtcKU2VjdXJpdHkK1wpXaWtpbWVkaWEtVkUtQ2FtcGFpZ25zIChTMi0yMDE4KQrXClNjYXAK1wpTY2FwIChTY2FwMy1BZG9wdGlvbi1QaGFzZTIpCtcKQWJ1c2VGaWx0ZXIK1wpEYXRhLXJlbGVhc2UK1wpIYXNodGFncwrXCkxhYnNEQi1BdWRpdG9yCtcKTGFkaWVzLVRoYXQtRk9TUy1NZWRpYVdpa2kK1wpMYW5ndWFnZS0yMDE4LUFwci1KdW5lCtcKTGFuZ3VhZ2UtMjAxOC1KYW4tTWFyCtcKSEhWTQrXCkhBV2VsY29tZQrXCkJvbGQKSXRhbGljcwpNb25vc3BhY2VkCkxpbmsKQnVsbGV0ZWQgTGlzdApOdW1iZXJlZCBMaXN0CkNvZGUgQmxvY2sKUXVvdGUKVGFibGUKVXBsb2FkIEZpbGUKTWVtZQpQcmV2aWV3CkhlbHAKRnVsbHNjcmVlbiBNb2RlClBpbiBGb3JtIE9uIFNjcmVlbgoyMzg0ODJuMzc1IGFkZGVkIHByb2plY3RzOiBTZWN1cml0eSwgV2lraW1lZGlhLVZFLUNhbXBhaWducyAoUzItMjAxOCksIFNjYXAgKFNjYXAzLUFkb3B0aW9uLVBoYXNlMiksIEFidXNlRmlsdGVyLCBEYXRhLXJlbGVhc2UsIEhhc2h0YWdzLCBMYWJzREItQXVkaXRvciwgTGFkaWVzLVRoYXQtRk9TUy1NZWRpYVdpa2ksIExhbmd1YWdlLTIwMTgtQXByLUp1bmUsIExhbmd1YWdlLTIwMTgtSmFuLU1hciwgSEhWTSwgSEFXZWxjb21lLlBSRVZJRVcKMjM4NDgybjM3NSBtb3ZlZCB0aGlzIHRhc2sgZnJvbSBJbiBQcm9ncmVzcyB0byBJbiBDb2RlIFJldmlldyBvbiB0aGUgQW5hbHl0aWNzLUthbmJhbiBib2FyZC4KMjM4NDgybjM3NSByZW1vdmVkIEpBbGxlbWFuZG91IGFzIHRoZSBhc3NpZ25lZSBvZiB0aGlzIHRhc2suCjIzODQ4Mm4zNzUgdHJpYWdlZCB0aGlzIHRhc2sgYXMgTG93ZXN0IHByaW9yaXR5LgoyMzg0ODJuMzc1IHJlbW92ZWQgc3Vic2NyaWJlcnM6IEFrbGFwcGVyLCBKQWxsZW1hbmRvdS4KQ29udGVudCBsaWNlbnNlZCB1bmRlciBDcmVhdGl2ZSBDb21tb25zIEF0dHJpYnV0aW9uLVNoYXJlQWxpa2UgMy4wIChDQy1CWS1TQSkgdW5sZXNzIG90aGVyd2lzZSBub3RlZDsgY29kZSBsaWNlbnNlZCB1bmRlciBHTlUgR2VuZXJhbCBQdWJsaWMgTGljZW5zZSAoR1BMKSBvciBvdGhlciBvcGVuIHNvdXJjZSBsaWNlbnNlcy4gQnkgdXNpbmcgdGhpcyBzaXRlLCB5b3UgYWdyZWUgdG8gdGhlIFRlcm1zIG9mIFVzZSwgUHJpdmFjeSBQb2xpY3ksIGFuZCBDb2RlIG9mIENvbmR1Y3QuILcgV2lraW1lZGlhIEZvdW5kYXRpb24gtyBQcml2YWN5IFBvbGljeSC3IENvZGUgb2YgQ29uZHVjdCC3IFRlcm1zIG9mIFVzZSC3IERpc2NsYWltZXIgtyBDQy1CWS1TQSC3IEdQTApZb3VyIGJyb3dzZXIgdGltZXpvbmUgc2V0dGluZyBkaWZmZXJzIGZyb20gdGhlIHRpbWV6b25lIHNldHRpbmcgaW4geW91ciBwcm9maWxlLCBjbGljayB0byByZWNvbmNpbGUu