Page MenuHomePhabricator

Cloud VPS upgrade to Openstack Mitaka release
Closed, ResolvedPublic

Description

As part of annual planning we have been lining up work that needs to be done and it seems clear that if we try to do migration away from Trusty and Neutron in serial rather than in parallel we will be in trouble timeline wise.

@bd808, @Andrew and I have all talked about it and it seems like the following is the best option to attempt to pursue:

  • Setup labtestn (to be renamed) using Debian Jessie which we can pull Mitaka from backports
  • Upgrade labtest (to be renamed ;) to Mitaka using the trusty cloud repo (this is so we can mix and match trusty and debian components)
  • Upgrade main to Mitaka

This means Mitaka is our "frozen" target for Neutron rather than Liberty but it seems like a necessity, and it's my understanding that Mitaka is the last release of OpenStack where nova-network is tested/supported/available.

(Note we could I think in theory grab packages from http://snapshot.debian.org/package/keystone/ (etc) and attempt to stay on Liberty with debian but it seems far more sane to follow backports last standing in Jessie, especially since @Andrew doesn't think the upgrade will be too bad and the only big open question is some designate changes at the moment)


https://www.stratoscale.com/blog/openstack/openstack-mitaka-whats-new-part-1/

https://docs.openstack.org/releasenotes/nova/mitaka.html

Latest API microversion supported for Mitaka is v2.25
Nova now requires a second database (called ‘API DB’).
A new nova-manage script allows you to perform all online DB migrations once you upgrade your cloud
EC2 API support is fully removed.

https://docs.openstack.org/releasenotes/keystone/mitaka.html
:* keystone-manage bootstrap
https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/openstack/templates/bootstrap/keystone/keystone_seed.sh.erb

https://docs.openstack.org/releasenotes/designate/mitaka.html
(Deprecation Notes)
Create / Update / Delete API methods on /v2/pools/ Using these messages will cause unforseen issues with pools. We highly recommend that the policy file be updated to restrict create / update / delete from all users
default-soa-refresh configuration option

https://docs.openstack.org/releasenotes/glance/mitaka.html
(Upgrade Notes)
Metadata definitions previously associated with OS::Nova::Instance have been changed to be associated with OS::Nova::Server in order to align with Heat and Searchlight. You may either upgrade them using glance-manage db load_metadefs [path] [merge] [prefer_new] or glance-manage db upgrade 44.
The task API is being deprecated and it has been made admin only. If deployers of Glance would like to have this API as a public one, it is necessary to change the policy.json file and remove role:admin from every task related field.


https://docs.openstack.org/releasenotes/horizon/mitaka.html
:* multi-region (vxlan and floating ips)
:* rebuild labtestweb2001?

They are creation a separate nova_api DB that makes this transition more complicated in this release.

https://bugs.launchpad.net/tripleo/+bug/1648504
https://review.openstack.org/#/c/408669/


Procedure for in place upgrade of existing ubuntu (labtest => main):

  • Add new mitaka cloud repo? -- beware mitaka and neutron config things --
  • Upgrade keystone?
  • Upgrade glance
  • Upgrade nova ( new nova_api db?) (nova_network)
  • Upgrade designate (fix scripts? downtime?)

Event Timeline

chasemp created this task.
chasemp renamed this task from Upgrade to Openstack Mitaka release to Cloud VPS upgrade to Openstack Mitaka release.Mar 14 2018, 3:25 PM

Borrowed from @aborrero notes as an agenda from todays meeting:

labtestn == jessie <--- new mitaka deployment directly with jessie
labtest == ubuntu <--- moving from liberty to mitaka
main == ubuntu <--- eventually moving from liberty to mitaka

  • Mitaka in jessie is in jessie-backports, Mitaka in ubuntu is in cloud repo
  • Plan is to upgrade/deploy all clusters with mitaka. Then, rebuild components as jessie.
  • Puppet code is shared between all mitaka deployments (chase refactoring)
  • keystone: issues with LDAP backend
  • neutron: no switch between overlays
  • desinate: suffers in upgradres

Change 419737 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: trial of mixed mitaka/liberty nova compute

https://gerrit.wikimedia.org/r/419737

Change 419737 merged by Rush:
[operations/puppet@production] openstack: trial of mixed mitaka/liberty nova compute

https://gerrit.wikimedia.org/r/419737

Change 419737 merged by Rush:
[operations/puppet@production] openstack: trial of mixed mitaka/liberty nova compute

https://gerrit.wikimedia.org/r/419737

Well, this infact does not work. While APIs are in theory N-/+1 the message versioning using the queue is not, and the model is only support for nova-compute N-1 and nova-conductor N and explicitly not reverse.

labvirt1021 is currently jessie and mitaka with everything seemingly setup except nova-compute throws and error:

IncompatibleObjectVersion: Version 1.3 of MigrationList is not supported

Tracking back this seems to be a variant of known errors when attempting to use a nova-compute that is newer than nova-conductor schema.

https://bugs.launchpad.net/nova/+bug/1483321/comments/7

http://lists.openstack.org/pipermail/openstack-dev/2015-August/072233.html
https://ask.cloudbase.it/question/1110/not-able-to-start-nova-compute-service-on-hyper-v-host/
https://ask.openstack.org/en/question/90315/not-able-to-start-nova-compute-service-on-hyper-v-host/
http://lists.openstack.org/pipermail/openstack-dev/2015-August/072237.html
https://bugs.launchpad.net/nova/+bug/1483321

2018-03-16 15:10:54.264 72304 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 461, in _send
2018-03-16 15:10:54.264 72304 ERROR oslo_service.service     raise result
2018-03-16 15:10:54.264 72304 ERROR oslo_service.service RemoteError: Remote error: IncompatibleObjectVersion Version 1.3 of MigrationList is not supported
2018-03-16 15:10:54.264 72304 ERROR oslo_service.service [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch\n    result = func(ctxt, **new_args)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 937, in object_class_action_versions\n    context, objname, objmethod, object_versions, args, kwargs)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 468, in object_class_action_versions\n    objname, object_versions[objname])\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 355, in obj_class_from_name\n    supported=latest_ver)\n', u'IncompatibleObjectVersion: Version 1.3 of MigrationList is not supported\n'].

Change 420056 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: labvirt102[12] as Ubuntu and Liberty

https://gerrit.wikimedia.org/r/420056

Change 420056 merged by Rush:
[operations/puppet@production] openstack: labvirt102[12] as Ubuntu and Liberty

https://gerrit.wikimedia.org/r/420056

Script wmf-auto-reimage was launched by rush on neodymium.eqiad.wmnet for hosts:

labvirt1021.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201803161609_rush_2520_labvirt1021_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['labvirt1021.eqiad.wmnet']

Of which those FAILED:

['labvirt1021.eqiad.wmnet']

Change 420084 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: labtestn pass in dhcp_domain for dhcp-agent

https://gerrit.wikimedia.org/r/420084

Change 420085 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: neutron-server ferm rule for default port

https://gerrit.wikimedia.org/r/420085

Change 420084 merged by Rush:
[operations/puppet@production] openstack: labtestn pass in dhcp_domain for dhcp-agent

https://gerrit.wikimedia.org/r/420084

Change 420085 merged by Rush:
[operations/puppet@production] openstack: neutron-server ferm rule for default port

https://gerrit.wikimedia.org/r/420085

Change 422352 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] keystone-paste.ini: Remove deprecated extension filters

https://gerrit.wikimedia.org/r/422352

Change 422433 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova.conf: remove memcached setting

https://gerrit.wikimedia.org/r/422433

Change 422432 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova.conf: use entry point name for scheduler_driver

https://gerrit.wikimedia.org/r/422432

Change 422352 merged by Andrew Bogott:
[operations/puppet@production] keystone-paste.ini: Remove deprecated extension filters

https://gerrit.wikimedia.org/r/422352

Change 422433 merged by Andrew Bogott:
[operations/puppet@production] nova.conf: remove memcached setting

https://gerrit.wikimedia.org/r/422433

Change 422432 merged by Andrew Bogott:
[operations/puppet@production] nova.conf: use entry point name for scheduler_driver

https://gerrit.wikimedia.org/r/422432

Change 424030 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] designate: s/domain/zone/ throughout

https://gerrit.wikimedia.org/r/424030

Change 424030 merged by Andrew Bogott:
[operations/puppet@production] designate: s/domain/zone/ throughout

https://gerrit.wikimedia.org/r/424030

Change 424031 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] designate: add a pool config for labtest

https://gerrit.wikimedia.org/r/424031

Change 424031 merged by Andrew Bogott:
[operations/puppet@production] designate: add a pool config for labtest

https://gerrit.wikimedia.org/r/424031

Change 426100 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] designate: add a pool config for the main deploy

https://gerrit.wikimedia.org/r/426100

Change 426100 merged by Andrew Bogott:
[operations/puppet@production] designate: add a pool config for the main deploy

https://gerrit.wikimedia.org/r/426100

chasemp claimed this task.