Andrew and some community members have report that our Debian stretch base images are failing to boot.
The initial tests suggest this is related to some virtual hardware change within libvirt/qemu/nova due to the recent upgrade to Openstack Ocata T237749: Upgrade wmcs OpenStack version to Ocata.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Partially revert "nova.conf ocata: remove [spice] config section" | operations/puppet | production | +11 -0 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Andrew | T240851 CloudVPS: stretch base images fails to boot | |||
| Resolved | • JHedden | T240871 CloudVPS: nova messing with instance disks |
first issue, I just created a test VM and apparently it lacks the console device which we can use in other VMs:
<serial type='pty'> <source path='/dev/pts/2'/> <target port='1'/> <alias name='serial1'/> </serial>
This is a diff of a good VM and the bad one I just created:
--- good.xml 2019-12-16 12:55:38.091199644 +0000 +++ bad.xml 2019-12-16 12:55:44.615753066 +0000 @@ -1,27 +1,30 @@ -<domain type='kvm' id='2'> - <name>i-00000167</name> - <uuid>5f162b81-465f-49f5-baf6-eb90c3ebe020</uuid> +<domain type='kvm' id='437'> + <name>i-0001009e</name> + <uuid>9511e800-b998-43f8-9058-d968622fa2b5</uuid> <metadata> <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0"> - <nova:package version="14.0.0"/> - <nova:name>login-test</nova:name> - <nova:creationTime>2019-10-23 09:08:14</nova:creationTime> - <nova:flavor name="m1.small"> - <nova:memory>2048</nova:memory> - <nova:disk>20</nova:disk> + <nova:package version="15.1.5"/> + <nova:name>stretch-boot-arturo-01</nova:name> + <nova:creationTime>2019-12-16 12:09:30</nova:creationTime> + <nova:flavor name="m1.medium"> + <nova:memory>4096</nova:memory> + <nova:disk>40</nova:disk> <nova:swap>0</nova:swap> <nova:ephemeral>0</nova:ephemeral> - <nova:vcpus>1</nova:vcpus> + <nova:vcpus>2</nova:vcpus> </nova:flavor> - <nova:owner/> - <nova:root type="image" uuid="6e6d743a-64f0-4cd3-b1b9-327bbf57e03b"/> + <nova:owner> + <nova:user uuid="aborrero">Arturo Borrero Gonzalez</nova:user> + <nova:project uuid="testlabs">testlabs</nova:project> + </nova:owner> + <nova:root type="image" uuid="6255ff81-db78-4521-b770-076fe16d365f"/> </nova:instance> </metadata> - <memory unit='KiB'>2097152</memory> - <currentMemory unit='KiB'>2097152</currentMemory> - <vcpu placement='static'>1</vcpu> + <memory unit='KiB'>4194304</memory> + <currentMemory unit='KiB'>4194304</currentMemory> + <vcpu placement='static'>2</vcpu> <cputune> - <shares>1024</shares> + <shares>2048</shares> </cputune> <resource> <partition>/machine</partition> @@ -30,9 +33,9 @@ <system> <entry name='manufacturer'>OpenStack Foundation</entry> <entry name='product'>OpenStack Nova</entry> - <entry name='version'>14.0.0</entry> + <entry name='version'>15.1.5</entry> <entry name='serial'>84a378a9-6288-40e5-a745-d4b5a5b13b87</entry> - <entry name='uuid'>5f162b81-465f-49f5-baf6-eb90c3ebe020</entry> + <entry name='uuid'>9511e800-b998-43f8-9058-d968622fa2b5</entry> <entry name='family'>Virtual Machine</entry> </system> </sysinfo> @@ -48,7 +51,7 @@ <cpu mode='custom' match='exact'> <model fallback='forbid'>Skylake-Client-IBRS</model> <vendor>Intel</vendor> - <topology sockets='1' cores='1' threads='1'/> + <topology sockets='2' cores='1' threads='1'/> <feature policy='require' name='ds'/> <feature policy='require' name='acpi'/> <feature policy='require' name='ss'/> @@ -91,15 +94,15 @@ <emulator>/usr/bin/kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> - <source file='/var/lib/nova/instances/5f162b81-465f-49f5-baf6-eb90c3ebe020/disk'/> + <source file='/var/lib/nova/instances/9511e800-b998-43f8-9058-d968622fa2b5/disk'/> <backingStore type='file' index='1'> <format type='raw'/> - <source file='/var/lib/nova/instances/_base/7dc4654a43592cac49efeea9a5451ac3669d1c6a'/> + <source file='/var/lib/nova/instances/_base/bf8ac2e9cf1b4bab5d8c07b61ef68995f9ddeb6c'/> <backingStore/> </backingStore> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> - <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> + <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </disk> <controller type='usb' index='0' model='piix3-uhci'> <alias name='usb'/> @@ -108,57 +111,36 @@ <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> - <controller type='virtio-serial' index='0'> - <alias name='virtio-serial0'/> - <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> - </controller> <interface type='bridge'> - <mac address='fa:16:3e:49:5b:4a'/> + <mac address='fa:16:3e:23:ec:9c'/> <source bridge='brq7425e328-56'/> - <target dev='tap85a6d509-1f'/> + <target dev='tap2e5097e6-45'/> <model type='virtio'/> <alias name='net0'/> - <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> + <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface> - <serial type='file'> - <source path='/var/lib/nova/instances/5f162b81-465f-49f5-baf6-eb90c3ebe020/console.log'/> + <serial type='pty'> + <source path='/dev/pts/6'/> + <log file='/var/lib/nova/instances/9511e800-b998-43f8-9058-d968622fa2b5/console.log' append='off'/> <target port='0'/> <alias name='serial0'/> </serial> - <serial type='pty'> - <source path='/dev/pts/2'/> - <target port='1'/> - <alias name='serial1'/> - </serial> - <console type='file'> - <source path='/var/lib/nova/instances/5f162b81-465f-49f5-baf6-eb90c3ebe020/console.log'/> + <console type='pty' tty='/dev/pts/6'> + <source path='/dev/pts/6'/> + <log file='/var/lib/nova/instances/9511e800-b998-43f8-9058-d968622fa2b5/console.log' append='off'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> - <channel type='pty'> - <source path='/dev/pts/3'/> - <target type='virtio' name='com.redhat.spice.0' state='disconnected'/> - <alias name='channel0'/> - <address type='virtio-serial' controller='0' bus='0' port='1'/> - </channel> <input type='mouse' bus='ps2'> <alias name='input0'/> </input> <input type='keyboard' bus='ps2'> <alias name='input1'/> </input> - <graphics type='spice' port='5901' autoport='yes' listen='0.0.0.0' keymap='en-us'> - <listen type='address' address='0.0.0.0'/> - </graphics> - <video> - <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/> - <alias name='video0'/> - <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> - </video> <memballoon model='virtio'> <stats period='10'/> <alias name='balloon0'/> - <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> + <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </memballoon> </devices> <seclabel type='none' model='none'/>
for the record, the console thing is tracked in T240660: openstack nova: virsh consoles broken in Ocata
At first sight, this would look like an issue somewhere in the virtualization stack. Interestingly, a buster VM can boot with mostly the same libvirt XML definition:
--- buster.xml 2019-12-16 14:28:17.792328295 +0100 +++ stretch.xml 2019-12-16 14:28:40.832248284 +0100 @@ -1,11 +1,11 @@ -<domain type='kvm' id='1058'> - <name>i-000100a6</name> - <uuid>8e4ce11a-6bc5-48a3-accc-b3e7a51b7b8d</uuid> +<domain type='kvm' id='437'> + <name>i-0001009e</name> + <uuid>9511e800-b998-43f8-9058-d968622fa2b5</uuid> <metadata> <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0"> <nova:package version="15.1.5"/> - <nova:name>buster-boot-arturo-01</nova:name> - <nova:creationTime>2019-12-16 13:00:27</nova:creationTime> + <nova:name>stretch-boot-arturo-01</nova:name> + <nova:creationTime>2019-12-16 12:09:30</nova:creationTime> <nova:flavor name="m1.medium"> <nova:memory>4096</nova:memory> <nova:disk>40</nova:disk> @@ -17,7 +17,7 @@ <nova:user uuid="aborrero">Arturo Borrero Gonzalez</nova:user> <nova:project uuid="testlabs">testlabs</nova:project> </nova:owner> - <nova:root type="image" uuid="fc6fb78b-4515-4dcc-8254-591b9fe01762"/> + <nova:root type="image" uuid="6255ff81-db78-4521-b770-076fe16d365f"/> </nova:instance> </metadata> <memory unit='KiB'>4194304</memory> @@ -34,8 +34,8 @@ <entry name='manufacturer'>OpenStack Foundation</entry> <entry name='product'>OpenStack Nova</entry> <entry name='version'>15.1.5</entry> - <entry name='serial'>fa753a55-9f95-4992-a3de-001a0d3fca33</entry> - <entry name='uuid'>8e4ce11a-6bc5-48a3-accc-b3e7a51b7b8d</entry> + <entry name='serial'>84a378a9-6288-40e5-a745-d4b5a5b13b87</entry> + <entry name='uuid'>9511e800-b998-43f8-9058-d968622fa2b5</entry> <entry name='family'>Virtual Machine</entry> </system> </sysinfo> @@ -94,10 +94,10 @@ <emulator>/usr/bin/kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> - <source file='/var/lib/nova/instances/8e4ce11a-6bc5-48a3-accc-b3e7a51b7b8d/disk'/> + <source file='/var/lib/nova/instances/9511e800-b998-43f8-9058-d968622fa2b5/disk'/> <backingStore type='file' index='1'> <format type='raw'/> - <source file='/var/lib/nova/instances/_base/abd2fbb3fa7d5c62250f3d90c17369a7c097e00e'/> + <source file='/var/lib/nova/instances/_base/bf8ac2e9cf1b4bab5d8c07b61ef68995f9ddeb6c'/> <backingStore/> </backingStore> <target dev='vda' bus='virtio'/> @@ -112,22 +112,22 @@ <alias name='pci.0'/> </controller> <interface type='bridge'> - <mac address='fa:16:3e:05:9b:9f'/> + <mac address='fa:16:3e:23:ec:9c'/> <source bridge='brq7425e328-56'/> - <target dev='tapb4ad7c9b-40'/> + <target dev='tap2e5097e6-45'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface> <serial type='pty'> - <source path='/dev/pts/35'/> - <log file='/var/lib/nova/instances/8e4ce11a-6bc5-48a3-accc-b3e7a51b7b8d/console.log' append='off'/> + <source path='/dev/pts/6'/> + <log file='/var/lib/nova/instances/9511e800-b998-43f8-9058-d968622fa2b5/console.log' append='off'/> <target port='0'/> <alias name='serial0'/> </serial> - <console type='pty' tty='/dev/pts/35'> - <source path='/dev/pts/35'/> - <log file='/var/lib/nova/instances/8e4ce11a-6bc5-48a3-accc-b3e7a51b7b8d/console.log' append='off'/> + <console type='pty' tty='/dev/pts/6'> + <source path='/dev/pts/6'/> + <log file='/var/lib/nova/instances/9511e800-b998-43f8-9058-d968622fa2b5/console.log' append='off'/> <target type='serial' port='0'/> <alias name='serial0'/> </console>
The serial console in Buster is still missing though.
Mentioned in SAL (#wikimedia-cloud) [2019-12-16T13:59:05Z] <arturo> powering down puppet-stretch-test VM to test stuff related to T240851
More differences. This is how qemu is used in a GOOD instance:
qemu-system-x86_64 -enable-kvm -name guest=i-00000167,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-2-i-00000167/master-key.aes -machine pc-i440fx-2.8,accel=kvm,usb=off,dump-guest-core=off -cpu Skylake-Client-IBRS,+ds,+acpi,+ss,+ht,+tm,+pbe,+dtes64,+monitor,+ds_cpl,+vmx,+smx,+est,+tm2,+xtpr,+pdcm,+dca,+osxsave,+tsc_adjust,+avx512f,+avx512dq,+clflushopt,+avx512cd,+avx512bw,+avx512vl,+pku,+ospke,+md-clear,+ssbd,+pdpe1gb -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 5f162b81-465f-49f5-baf6-eb90c3ebe020 -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=14.0.0,serial=84a378a9-6288-40e5-a745-d4b5a5b13b87,uuid=5f162b81-465f-49f5-baf6-eb90c3ebe020,family=Virtual Machine -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-2-i-00000167/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive file=/var/lib/nova/instances/5f162b81-465f-49f5-baf6-eb90c3ebe020/disk,format=qcow2,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=30 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:49:5b:4a,bus=pci.0,addr=0x3 -add-fd set=2,fd=32 -chardev file,id=charserial0,path=/dev/fdset/2,append=on -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -chardev pty,id=charchannel0 -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -spice port=5901,addr=0.0.0.0,disable-ticketing,seamless-migration=on -k en-us -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on
This is how it is called in a BAD instance:
qemu-system-x86_64 -enable-kvm -name guest=i-0001009e,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-437-i-0001009e/master-key.aes -machine pc-i440fx-2.8,accel=kvm,usb=off,dump-guest-core=off -cpu Skylake-Client-IBRS,+ds,+acpi,+ss,+ht,+tm,+pbe,+dtes64,+monitor,+ds_cpl,+vmx,+smx,+est,+tm2,+xtpr,+pdcm,+dca,+osxsave,+tsc_adjust,+avx512f,+avx512dq,+clflushopt,+avx512cd,+avx512bw,+avx512vl,+pku,+ospke,+md-clear,+ssbd,+pdpe1gb -m 4096 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 9511e800-b998-43f8-9058-d968622fa2b5 -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=15.1.5,serial=84a378a9-6288-40e5-a745-d4b5a5b13b87,uuid=9511e800-b998-43f8-9058-d968622fa2b5,family=Virtual Machine -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-437-i-0001009e/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/instances/9511e800-b998-43f8-9058-d968622fa2b5/disk,format=qcow2,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=64,id=hostnet0,vhost=on,vhostfd=66 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:23:ec:9c,bus=pci.0,addr=0x2 -add-fd set=2,fd=68 -chardev pty,id=charserial0,logfile=/dev/fdset/2,logappend=on -device isa-serial,chardev=charserial0,id=serial0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on
This is directly related to the XML diff above.
We may want to investigate the libvirt service errors we have:
aborrero@cloudvirt1025:~$ sudo journalctl -u libvirtd.service Dec 03 00:39:13 cloudvirt1025 libvirtd[92841]: 2019-12-03 00:39:13.890+0000: 92846: error : qemuDomainAgentAvailable:5499 : argument unsupported: QEMU guest agent is not configured Dec 03 01:24:27 cloudvirt1025 libvirtd[92841]: 2019-12-03 01:24:27.038+0000: 92846: error : qemuDomainAgentAvailable:5499 : argument unsupported: QEMU guest agent is not configured Dec 04 03:22:00 cloudvirt1025 libvirtd[92841]: 2019-12-04 03:22:00.177+0000: 92841: error : qemuMonitorIO:710 : internal error: End of file from qemu monitor Dec 04 03:25:04 cloudvirt1025 libvirtd[92841]: 2019-12-04 03:25:04.206+0000: 92841: error : qemuMonitorIO:710 : internal error: End of file from qemu monitor Dec 07 01:13:02 cloudvirt1025 libvirtd[92841]: 2019-12-07 01:13:02.221+0000: 92845: error : qemuDomainAgentAvailable:5499 : argument unsupported: QEMU guest agent is not configured Dec 08 21:39:58 cloudvirt1025 libvirtd[92841]: 2019-12-08 21:39:58.732+0000: 92845: error : qemuDomainAgentAvailable:5499 : argument unsupported: QEMU guest agent is not configured Dec 10 19:51:50 cloudvirt1025 libvirtd[92841]: 2019-12-10 19:51:50.115+0000: 92842: error : qemuDomainAgentAvailable:5499 : argument unsupported: QEMU guest agent is not configured Dec 10 20:09:15 cloudvirt1025 libvirtd[92841]: 2019-12-10 20:09:15.132+0000: 92846: error : qemuDomainAgentAvailable:5499 : argument unsupported: QEMU guest agent is not configured Dec 11 01:20:23 cloudvirt1025 libvirtd[92841]: 2019-12-11 01:20:23.566+0000: 92845: error : qemuDomainAgentAvailable:5499 : argument unsupported: QEMU guest agent is not configured Dec 11 20:46:55 cloudvirt1025 libvirtd[92841]: 2019-12-11 20:46:55.845+0000: 92841: error : qemuMonitorIO:710 : internal error: End of file from qemu monitor Dec 12 06:14:25 cloudvirt1025 libvirtd[92841]: 2019-12-12 06:14:25.280+0000: 92841: error : virNetSocketReadWire:1801 : End of file while reading data: Input/output error Dec 12 07:19:10 cloudvirt1025 libvirtd[92841]: 2019-12-12 07:19:10.526+0000: 92841: error : virNetSocketReadWire:1801 : End of file while reading data: Input/output error Dec 12 07:23:30 cloudvirt1025 libvirtd[92841]: 2019-12-12 07:23:30.279+0000: 92841: error : virNetSocketReadWire:1801 : End of file while reading data: Input/output error Dec 12 07:25:54 cloudvirt1025 libvirtd[92841]: 2019-12-12 07:25:54.263+0000: 92841: error : virNetSocketReadWire:1801 : End of file while reading data: Input/output error Dec 12 09:43:34 cloudvirt1025 libvirtd[92841]: 2019-12-12 09:43:34.711+0000: 92841: error : virNetSocketReadWire:1801 : End of file while reading data: Input/output error Dec 12 09:43:42 cloudvirt1025 libvirtd[92841]: 2019-12-12 09:43:42.029+0000: 92841: error : qemuMonitorIO:710 : internal error: End of file from qemu monitor Dec 12 21:14:09 cloudvirt1025 systemd[1]: Stopping Virtualization daemon... Dec 12 21:14:09 cloudvirt1025 systemd[1]: Stopped Virtualization daemon. Dec 12 21:14:09 cloudvirt1025 systemd[1]: Starting Virtualization daemon... Dec 12 21:14:09 cloudvirt1025 systemd[1]: Started Virtualization daemon. Dec 15 02:43:38 cloudvirt1025 libvirtd[160067]: 2019-12-15 02:43:38.260+0000: 160067: info : libvirt version: 3.0.0, package: 4+deb9u4 (Guido Günther <agx@sigxcpu.org> Wed, 12 Jun 2019 10:13:38 +0200) Dec 15 02:43:38 cloudvirt1025 libvirtd[160067]: 2019-12-15 02:43:38.260+0000: 160067: info : hostname: cloudvirt1025 Dec 15 02:43:38 cloudvirt1025 libvirtd[160067]: 2019-12-15 02:43:38.260+0000: 160067: error : virNetSocketReadWire:1801 : End of file while reading data: Input/output error Dec 16 12:49:44 cloudvirt1025 libvirtd[160067]: 2019-12-16 12:49:44.373+0000: 160090: error : qemuDomainOpenConsole:16083 : internal error: cannot find character device serial1 Dec 16 16:16:12 cloudvirt1025 libvirtd[160067]: 2019-12-16 16:16:12.714+0000: 160067: error : qemuMonitorIO:710 : internal error: End of file from qemu monitor [..]
This is the commit that broke console output on the stretch hosts. https://gerrit.wikimedia.org/r/c/operations/puppet/+/554151
I've confirmed that when using this nova config snippet, the stretch hosts can boot and write to the console.log
[spice] enabled = true agent_enabled = false
<serial type='pty'> <log file='/var/lib/nova/instances/800e1e5b-ecd5-4fb8-94d3-5dfd1267f51b/console.log' append='off'/> <target port='0'/> </serial> <console type='pty'> <log file='/var/lib/nova/instances/800e1e5b-ecd5-4fb8-94d3-5dfd1267f51b/console.log' append='off'/> <target type='serial' port='0'/> </console> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel>
When spice is disabled the serial0 device doesn't get setup properly to match our VM configuration. I'm looking into what makes buster work with spice disabled, hoping we can use that instead.
We could also work around this with another hack, disabling spice and adding in just the ttyS1 serial interface to the nova libvirt guest config process.
<serial type='pty'> <target port='1'/> </serial>
I'm wondering if we should review our current requirements for these hacks, and ensure they're still needed.
I believe that the serial1 console is also broken with Buster. According to the paste at https://phabricator.wikimedia.org/T240660#5738941 the nova people really want us to use nova-defined consoles, so I'm going to pursue that a bit. I'm guessing that including that in the nova config will work as well as including Spice.
possibly
[serial_console]
enabled=True
That's meant to work with a whole elaborate proxying system but maybe it'll add the device we need on its own.
Change 558284 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Revert "nova.conf ocata: remove [spice] config section"
Change 558284 merged by Andrew Bogott:
[operations/puppet@production] Partially revert "nova.conf ocata: remove [spice] config section"