See also T324774: Openstack hacks

Started evaluating kolla-ansible, found a blocker in nova-compute not starting due to it unable to auth to libvirtd, and opened an upstream bug https://bugs.launchpad.net/kolla-ansible/+bug/2004579

In T326758#8582716, @aborrero wrote:

Started evaluating kolla-ansible, found a blocker in nova-compute not starting due to it unable to auth to libvirtd, and opened an upstream bug https://bugs.launchpad.net/kolla-ansible/+bug/2004579

I can't reproduce this problem on a VM in my laptop. There is some kind of bad interaction for running kolla-ansible inside Cloud VPS :-(

• zed @ Cloud VPS: libvirt <-> nova-compute auth failure
• yoga @ Cloud VPS: libvirt <-> nova-compute auth failure
• zed @ my laptop: all fine regarding libvirt & nova-compute

Have you checked if the image you use for the VM is the same? Maybe the one on Cloud is missing something/user setup/etc.

Incoming news, TripleO is being discontinued upstream.

In T326758#8590478, @aborrero wrote:

In T326758#8582716, @aborrero wrote:

Started evaluating kolla-ansible, found a blocker in nova-compute not starting due to it unable to auth to libvirtd, and opened an upstream bug https://bugs.launchpad.net/kolla-ansible/+bug/2004579

I can't reproduce this problem on a VM in my laptop. There is some kind of bad interaction for running kolla-ansible inside Cloud VPS :-(

• zed @ Cloud VPS: libvirt <-> nova-compute auth failure
• yoga @ Cloud VPS: libvirt <-> nova-compute auth failure
• zed @ my laptop: all fine regarding libvirt & nova-compute

I found the problem. We're affected by https://bugs.launchpad.net/kolla-ansible/+bug/1989791 The bug I reported is a duplicate, but I was unaware about the hostname being the root problem.

The workaround is simple, in /etc/hosts:

--- hosts_old	2023-02-10 12:22:40.241881291 +0000
+++ hosts_new	2023-02-10 12:22:05.537866714 +0000
@@ -5,7 +5,7 @@
 # b.) change or remove the value of 'manage_etc_hosts' in
 #     /etc/cloud/cloud.cfg or cloud-config from user-data
 #
-172.16.3.208 kolla-test.testlabs.eqiad1.wikimedia.cloud kolla-test
+172.16.3.208 kolla-test kolla-test.testlabs.eqiad1.wikimedia.cloud
 127.0.0.1 localhost
 
 # The following lines are desirable for IPv6 capable hosts

Before the change:

root@kolla-test:~# hostname -f
kolla-test.testlabs.eqiad1.wikimedia.cloud

After the change:

root@kolla-test:~# hostname -f
kolla-test

With the change libvirt and nova-compute daemons can communicate.

This /etc/host file is generated by us via cloud-init most likely.

Change 888222 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloud: openstack: add kolla-ansible evaluation recipe

https://gerrit.wikimedia.org/r/888222

Change 888222 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloud: openstack: add kolla-ansible evaluation recipe

https://gerrit.wikimedia.org/r/888222

Steps are:

• try from a VM with the puppet role::wmcs::openstack::kolla_ansible_evaluation
• then run wmcs-kolla-ansible-evaluation.sh

Wait and enjoy.

Patching kolla containers: https://docs.openstack.org/kolla/latest/admin/image-building.html

Bootstrapped: https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/2023_Openstack_deployment_workflow

Note, by default openstack-ansible all-in-one setup renames the VM hostname, introducing a severe drift wrt. other Cloud VPS context (like puppet, etc), therefore making it difficult to operate inside Cloud VPS for evaluation & testing purposes. Will investigate next if this renaming can be disabled.

In T326758#8644613, @aborrero wrote:

Note, by default openstack-ansible all-in-one setup renames the VM hostname, introducing a severe drift wrt. other Cloud VPS context (like puppet, etc), therefore making it difficult to operate inside Cloud VPS for evaluation & testing purposes. Will investigate next if this renaming can be disabled.

I know now why this happens.

• as part of the AIO quistack guide, we are expected to run the scripts/bootstrap-aio.sh script
• internally, it calls ansible-playbook to run the bootstrap-aio.yml playbook
• this playbook eventually calls the bootstrap-host role
• which has a task prepare_hostname.yml to rename the hostname

The task is guarded by a prepare-hostname tag, which in theory makes it possible to skip using ansible-playbook --skip-tags prepare-hostname, however:

• the wrapper script bootstrap-aio.sh doesn't seem to have an option to pass the --skip-tags argument to the inner ansible-playbook call. We would need to patch that ourselves in the script (no big deal, but means a patch in this early stage...)
• there is a comment in the code hinting that the target hostname aio1 is expected per the inventory.
• I understand the use case for the hard hostname, since they are using this AIO mode to test openstack itself, and they need strict control of all the environment.

Therefore I don't see an easy way to play with openstack-ansible in AIO mode within Cloud VPS VMs.
I'll run another test, not using the AIO mode and will see if things behave differently.

In T326758#8676365, @aborrero wrote:

Therefore I don't see an easy way to play with openstack-ansible in AIO mode within Cloud VPS VMs.

To be clear, the AIO mode works just great. The thing with the hostname rename is that it effectively removes the VM from puppet control. Which makes me nervous.

Another option is to make sure we're using an aio1 hostname from the beginning. Will try that!

Change 895789 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: create openstack-ansible evaluation role

https://gerrit.wikimedia.org/r/895789

In T326758#8676367, @aborrero wrote:

Another option is to make sure we're using an aio1 hostname from the beginning. Will try that!

Still unable to operate. After a while, openstack-ansible tries to ssh using the aio1 name rather than localhost, which clashes with the SSH configuration we have for the host via puppet:

TASK [Ensure python is installed] ********************************************************************************************************************************************************************************************************************************************************************************************
fatal: [aio1]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Warning: Permanently added '172.29.236.100' (ECDSA) to the list of known hosts.\r\nroot@172.29.236.100: Permission denied (publickey).", "unreachable": true}

Also, the hostname rename also involves changing the domain (from whatever.eqiad1.wikimedia.cloud to aio1.openstack.local, meaning the puppet agent certificate gets invalidated.

I'm starting to have doubts about if it is worth keep trying to run openstack-ansible inside Cloud VPS.

Change 895789 abandoned by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: create openstack-ansible evaluation role

Reason:

don't merging this as the script doesn't work as expected.

https://gerrit.wikimedia.org/r/895789

We should evaluate as well https://docs.airshipit.org/

In T326758#8741415, @aborrero wrote:

We should evaluate as well https://docs.airshipit.org/

Regarding airship: the community has been very quiet for a little while now. There are mainly maintenance activities going on for the components that are running in production at a few companies, carried out by only a handful of people.
Probably not the level of activity/adoption/support we're looking for.

We could evaluate as well https://www.starlingx.io/

After the research was completed, my recommendation was to try going with kolla-ansible.

However, I recently attended Kubecon EU and now I'm feeling like re-evaluating my conclusions to suggest openstack-helm instead.
I think that having an undercloud kubernetes deployment as bedrock would be very beneficial, even if we have to pay the price of the extra abstraction layer (k8s):

• I believe openstack-helm deploys everything inside a k8s namespace. We could simply have a IaaS deploy per kubernetes namespace, one of them being the Cloud VPS service, while opening the door for others in other namespaces.
• we already have puppet code to install a kubernetes cluster to baremetal (kubeadm, which is what toolforge uses).
• we could leverage lessons learned from toolforge lima-kilo and develop a local development cloud from day 0 based on kubernetes. Imagine running Cloud VPS on your laptop for testing / devel purposes.
• the cloud-native architecture, and in that regard it should be better suited for the next 10 years of industry evolution