Page MenuHomePhabricator
Create Task
Maniphest T212302

CloudVPS: upgrade: jessie -> stretch & mitaka -> newton
Closed, ResolvedPublic
Actions

Description

We need to upgrade our Cloud VPS infra:

  • from Debian Jessie to Debian Stretch
  • from Mitaka to Newton

Since there is no way to install Newton in Jessie, we need to try to install Mitaka in Stretch and then upgrade Mitaka -> Newton.

All Mitaka packages are in the jessie-backports Debian repository, and we may try hacking doing the trick of using that repo in Stretch.
My proposal is to try something like this:

  1. Image a server with Debian Stretch
  2. Enable the jessie-backports repo
  3. Install Mitaka packages from that repo
  4. Upgrade Mitaka (jessie-backports) to Newton (stretch)
  5. Cleanup any remaining jessie-backports package and use only what's provided in stretch

PS: There is a small summary of the versioning matrix in T169099#4676842

Details

SubjectRepoBranchLines +/-
openstack: clientpackages: vms: refresh comments and messagesoperations/puppetproduction+33 -5
openstack: drop jessie codeoperations/puppetproduction+0 -177
openstack: Update jessie openstack clients in eqiad1operations/puppetproduction+4 -4
openstack: pin mitaka version on jessie openstack clientsoperations/puppetproduction+4 -0
Openstack: move eqiad1 glance/keystone/nova/neutron to Newtonoperations/puppetproduction+7 -7
Horizon: put in maintenance mode for the mitaka->newton upgradeoperations/puppetproduction+1 -1
nova: add a few more Newton filesoperations/puppetproduction+79 -0
Openstack: add some missing files for Newtonoperations/puppetproduction+3 K -0
Designate: upgrade eqiad1 to Newtonoperations/puppetproduction+4 -4
Designate: move to OpenStack version 'newton'operations/puppetproduction+4 -0
Horizon: put into maintenance mode during designate upgradeoperations/puppetproduction+1 -1
Designate: add Newton config files and resourcesoperations/puppetproduction+1 K -0
Openstack Neutron: added config files and templates for version Newtonoperations/puppetproduction+5 K -0
keystone: update policy.json for Newtonoperations/puppetproduction+2 -2
keystone: forward mitaka config to newtonoperations/puppetproduction+3 K -0
glance: add Newton config filesoperations/puppetproduction+451 -0
openstack: nova: mitaka: stretch: install python-dogpile.core from jessieoperations/puppetproduction+19 -1
cloudvirt1018: when we reimage, do so as Stretchoperations/puppetproduction+0 -2
openstack: remove redundant sqlite3 declaration in cloudvirt hostsoperations/puppetproduction+8 -5
openstack: enable net nodes in the mitaka/stretch combinationoperations/puppetproduction+149 -72
openstack: linuxbridge_agent: typo in libosinfo package nameoperations/puppetproduction+1 -1
openstack: nova: mitaka/stretch: require packages before installing novaoperations/puppetproduction+6 -1
hiera: cloudvirt1030: override interface names for bridge mappingoperations/puppetproduction+2 -0
openstack: nova: introduce config files for newtonoperations/puppetproduction+419 -0
openstack: nova: compute: the libvirt service in stretch depends on other pkgsoperations/puppetproduction+1 -1
openstack: introduce templates for newtonoperations/puppetproduction+485 -0
cloudvps: swith cloudvirt1030 to openstack newtonoperations/puppetproduction+14 -2
openstack: add more stretch/mitaka supportoperations/puppetproduction+9 -6
openstack: introduce basic support for running Mitaka on Stretch on virt nodesoperations/puppetproduction+14 -3
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedAndrewT237749 Upgrade wmcs OpenStack version to Ocata
 ResolvedAndrewT210715 cloudvps: PDNS 3.x vs 4.x
 OpenNoneT132225 Add SSHFP dns records to bastions
ResolvedMoritzMuehlenhoffT224549 Track remaining jessie systems in production
 OpenNoneT224708 Drop most of mwopenstackclients.DnsManager in favour of designateclient
 ResolvedaborreroT212302 CloudVPS: upgrade: jessie -> stretch & mitaka -> newton
 ResolvedaborreroT214299 cloudvps: neutron: upgrade jessie -> stretch
 ResolvedaborreroT214167 labtestneutron2001: reimage to stretch & rename to cloudnet2001-dev
 ResolvedPapaulT214181 codfw: rename/relabel labtestneutron2001 to cloudnet2001-dev
 Resolved• JHeddenT214297 cloudvps: neutron: investigate keepalived warning
 ResolvedaborreroT214303 labtestneutron2002: reimage to stretch & rename to cloudnet2002-dev
 ResolvedaborreroT214322 cloudnet2002-dev: ACPI error
 ResolvedPapaulT214370 labtestneutron2002: refresh/rename to cloudnet2002-dev
 ResolvedaborreroT215407 cloudvps: cloudcontrol support for mitaka/stretch
 ResolvedaborreroT215605 cloudvps: missing packages in stretch for cloudcontrol servers
 ResolvedaborreroT223832 puppet vs. stretch vs. keystone
 ResolvedaborreroT216497 CloudVPS: workaround archival of jessie-backports repo
 ResolvedAndrewT221769 Upgrade cloudservices1003/1004 to stretch/mitaka
 ResolvedaborreroT224354 backport pdns-server version 3.x to Stretch
 ResolvedaborreroT224877 prometheus-pdns-exporter: add stretch support
 ResolvedAndrewT221770 Upgrade cloucontrol1003/1004 to stretch/mitaka
 ResolvedaborreroT224345 stretch build of prometheus-openstack-exporter incompatible with our mitaka apt repo
 ResolvedaborreroT224424 cloudservices1003: gateway timeout error
 ResolvedAndrewT233258 designate: switch from designate-pool-manager to designate-producer/designate-worker
 Resolved• MarosteguiT233978 Drop 'designate_pool_manager' database from m5 and remove associated grants
 ResolvedaborreroT233665 Forward our neutron-l3-agent routing hacks to Openstack Newton
 ResolvedAndrewT234834 Various user visible errors in Cloud VPS projects following OpenStack upgrade on 2019-10-07
 DuplicateNoneT234830 CloudVPS: m5-master databases for openstack may require re-encoding
 ResolvedAndrewT234876 nova-conductor running out of mysql connections
 ResolvedAndrewT234836 CloudVPS: update DNS record for eqiad1 egress (routing_source_ip) & ingress
Invalid• JHeddenT235070 Resolve keystone schema errors

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
aborrero added a comment.Jan 3 2019, 1:56 PM

It seems cloudvirt1030.eqiad.wmnet is happy now with our puppet code for mitaka/stretch. Will try now with cloudvirt1029

ops-monitoring-bot added a comment.Jan 3 2019, 4:01 PM

Script wmf-auto-reimage was launched by aborrero on cumin1001.eqiad.wmnet for hosts:

['cloudvirt1013.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201901031601_aborrero_125781.log.

• GTirloni awarded a token.Jan 3 2019, 9:51 PM
aborrero added a comment.Jan 4 2019, 2:04 PM

I will be using the openstack CloudVPS project to try more stuff related to this, specifically the stretch/mitaka combo for cloudnet servers (and then cloudcontrol servers)

Stashbot added a comment.Jan 4 2019, 2:05 PM

Mentioned in SAL (#wikimedia-cloud) [2019-01-04T14:05:36Z] <arturo> T212302 creating openstack-puppetmaster-01 and cloudvps-upgrade-test VM instances

aborrero added a comment.EditedJan 4 2019, 5:37 PM

I was able to begin testing installation of cloudnet nodes in a VM following these steps:

  • puppetmaster: openstack-puppetmaster-01.openstack.eqiad.wmflabs
  • vm: cloudvps-upgrade-test.openstack.eqiad.wmflabs (stretch)
  1. in the puppetmaster, apply this patch:
diff --git a/modules/role/manifests/labs/instance.pp b/modules/role/manifests/labs/instance.pp
index 91833b8c32..320492e25b 100644
--- a/modules/role/manifests/labs/instance.pp
+++ b/modules/role/manifests/labs/instance.pp
@@ -4,7 +4,7 @@ class role::labs::instance {
     include ::profile::base::labs
     include sudo
     include ::base::instance_upstarts
-    include ::profile::openstack::main::observerenv
+    #include ::profile::openstack::main::observerenv
     include ::profile::openstack::main::cumin::target
 
     sudo::group { 'ops':
  1. in horizon, apply this basic hiera config to the vm:

(this hiera config won't be useful for running neutron, but it is for checking package installation, which is what I'm looking for)

profile::openstack::base::neutron::db_user: x
profile::openstack::base::neutron::physical_interface_mappings: {}
profile::openstack::base::neutron::rabbit_user: x
profile::openstack::eqiad1::keystone_host: x.example.com
profile::openstack::eqiad1::ldap_user_pass: x
profile::openstack::eqiad1::neutron::agent_down_time: 2
profile::openstack::eqiad1::neutron::db_host: x.example.com
profile::openstack::eqiad1::neutron::db_pass: x
profile::openstack::eqiad1::neutron::dmz_cidr:
- 0.0.0.0
profile::openstack::eqiad1::neutron::l3_agent_bridge_mappings:
  br: x
profile::openstack::eqiad1::neutron::l3_agent_bridges:
  br:
    addif: eth1.0
profile::openstack::eqiad1::neutron::log_agent_heartbeats: x
profile::openstack::eqiad1::neutron::metadata_proxy_shared_secret: x
profile::openstack::eqiad1::neutron::network_compat_interface: eth1.0
profile::openstack::eqiad1::neutron::network_compat_interface_vlan: 0
profile::openstack::eqiad1::neutron::network_flat_interface: eth1.1
profile::openstack::eqiad1::neutron::network_flat_interface_external: eth1.2
profile::openstack::eqiad1::neutron::network_flat_interface_vlan: 1
profile::openstack::eqiad1::neutron::network_flat_interface_vlan_external: 2
profile::openstack::eqiad1::neutron::network_public_ip: 0.0.0.0
profile::openstack::eqiad1::neutron::rabbit_pass: x
profile::openstack::eqiad1::neutron::report_interval: x
profile::openstack::eqiad1::neutron::tld: x.x
profile::openstack::eqiad1::nova::dhcp_domain: x
profile::openstack::eqiad1::nova_controller: x.example.com
profile::openstack::eqiad1::observer_password: x
profile::openstack::eqiad1::region: test-r
profile::openstack::eqiad1::version: mitaka
puppetmaster: openstack-puppetmaster-01.openstack.eqiad.wmflabs
  1. in the vm, create a dummy eth1 interface: sudo ip link add eth1 type dummy
  2. in horizon, apply this role to the vm: role::wmcs::openstack::eqiad1::net
aborrero added a subscriber: bd808.Jan 8 2019, 7:06 PM

For the record, @bd808 helped me install neutron-common in a CloudVPS VM instance by deleting the neutron user from the cloud LDAP: P7967

gerritbot added a comment.Jan 10 2019, 1:24 PM

Change 483408 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: enable net nodes in the mitaka/stretch combination

https://gerrit.wikimedia.org/r/483408

gerritbot added a comment.Jan 10 2019, 1:29 PM

Change 483408 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: enable net nodes in the mitaka/stretch combination

https://gerrit.wikimedia.org/r/483408

aborrero added a comment.EditedJan 10 2019, 1:36 PM
In T212302#4869456, @gerritbot wrote:

Change 483408 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: enable net nodes in the mitaka/stretch combination

https://gerrit.wikimedia.org/r/483408

After this patch, we should be able to rebuild cloudnet1003 and cloudnet1004 as mitaka/stretch. @Andrew let me know if this is OK:

  • I would select the inactive node in the HA pair
  • rebuild it in stretch
  • see if mitaka/stretch can work with mitaka/jessie, i.e, they can be a HA pair again, working as expected
  • if all is fine, switch the HA active node and do the same with the remaining mitaka/jessie node

This process can cause downtime:

  • while rebuilding the inactive node, we won't have HA support.
  • while switching the active node from mitaka/jessie to mitaka/stretch, that's the critical moment of truth to see if the mitaka/stretch combo is in well shape for actual workload

Also, please note that I have no idea yet how we will do mitaka/stretch -> newton/stretch. That would likely mean another reimage of the servers.

Stashbot added a comment.Jan 10 2019, 1:51 PM

Mentioned in SAL (#wikimedia-operations) [2019-01-10T13:51:18Z] <arturo> T212302 icinga downtime for 2h cloudvirt[1013,1024,1026-1030].eqiad.wmnet bc wrong puppet code

gerritbot added a comment.Jan 10 2019, 2:04 PM

Change 483416 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: remove redundant sqlite3 declaration in cloudvirt hosts

https://gerrit.wikimedia.org/r/483416

gerritbot added a comment.Jan 10 2019, 2:07 PM

Change 483416 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: remove redundant sqlite3 declaration in cloudvirt hosts

https://gerrit.wikimedia.org/r/483416

Andrew added a comment.Jan 10 2019, 2:19 PM

After this patch, we should be able to rebuild cloudnet1003 and cloudnet1004 as mitaka/stretch. @Andrew let me know if this is OK

That all sounds good to me. I'm not sure I understand how the HA aspects of neutron are implemented here, but as long as switching from the passive to the active server is simple then it's a good plan. I would notify users ahead of time though.

Also, regarding mitaka -> newton, I wouldn't expect that to require a rebuild (at least, version upgrade haven't in the past) but we can cross that when we get to it :)

gerritbot added a comment.Jan 18 2019, 1:49 PM

Change 485185 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] labtestneutron2001: reimage to stretch and rename to cloudnet2001-dev

https://gerrit.wikimedia.org/r/485185

aborrero closed subtask T214167: labtestneutron2001: reimage to stretch & rename to cloudnet2001-dev as Resolved.Jan 21 2019, 9:54 AM
aborrero removed a subtask: T214167: labtestneutron2001: reimage to stretch & rename to cloudnet2001-dev.Jan 21 2019, 1:07 PM
aborrero removed a subtask: T214297: cloudvps: neutron: investigate keepalived warning.
aborrero closed subtask T214299: cloudvps: neutron: upgrade jessie -> stretch as Resolved.Jan 24 2019, 11:08 AM
aborrero moved this task from Doing to Soon! on the cloud-services-team (Kanban) board.Jan 28 2019, 11:21 AM
gerritbot added a comment.Feb 14 2019, 3:26 PM

Change 490619 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloudvirt1018: when we reimage, do so as Stretch

https://gerrit.wikimedia.org/r/490619

gerritbot added a comment.Feb 14 2019, 3:27 PM

Change 490619 merged by Andrew Bogott:
[operations/puppet@production] cloudvirt1018: when we reimage, do so as Stretch

https://gerrit.wikimedia.org/r/490619

aborrero closed subtask T216497: CloudVPS: workaround archival of jessie-backports repo as Resolved.Mar 1 2019, 6:41 PM
aborrero reopened subtask T216497: CloudVPS: workaround archival of jessie-backports repo as Open.Mar 12 2019, 1:17 PM
aborrero closed subtask T216497: CloudVPS: workaround archival of jessie-backports repo as Resolved.Mar 15 2019, 10:16 AM
• Bstorm reopened subtask T216497: CloudVPS: workaround archival of jessie-backports repo as Open.Mar 15 2019, 9:24 PM
• Bstorm closed subtask T216497: CloudVPS: workaround archival of jessie-backports repo as Resolved.
bd808 mentioned this in T218423: Add python 3 packages to openstack::clientpackages::common.Mar 19 2019, 11:49 PM
• GTirloni added a comment.Mar 21 2019, 4:43 PM

Just for reference, here's the currentl list of OpenStack releases.

Screenshot from 2019-03-21 13-31-58.png (356×856 px, 65 KB)

If we started right now, it seems that even Queens would be risky release since Extended Maintenance does not seem to ensure security fixes (only that community members decided to keep it in a best effort basis). With our current capacity, it seems that Rocky or even Stein would yield more future-proof results.

However, such a big jump means we would have to implement a new region and migrate VMs over (like we did for Neutron). That makes having shared storage critical (for faster migrations).

The alternative is to go Mitaka -> Newton -> Ocata -> Pike -> Queen -> Rocky -> Stein. Since we have had historical trouble keeping up with OpenStack releases, it seems improbable we will be able to handle 5-6 major upgrades in a short time to reach a release that is in Maintained status.

And there's the question of aligning with Debian releases. Debian Buster seems to come with OpenStack Rocky.

• GTirloni mentioned this in T145920: (eventually) Upgrade Labs to OpenStack Newton.Mar 21 2019, 4:48 PM
• GTirloni merged a task: T145920: (eventually) Upgrade Labs to OpenStack Newton.
• GTirloni added subscribers: • AlexMonk-WMF, tom29739.
• Bstorm added a comment.EditedMar 21 2019, 4:51 PM

If we started right now, it seems that even Queens would be risky release since Extended Maintenance does not seem to ensure security fixes (only that community members decided to keep it in a best effort basis). With our current capacity, it seems that Rocky or even Stein would yield more future-proof results.

One reason we had trouble (besides the fact that we use OS-based Debian packages, which is an obvious problem for this) is Neutron. We couldn't get out of our old version as long as we were on novanetwork. We might be able to move faster than in the past once novanetwork dies?

And there's the question of aligning with Debian releases. Debian Buster seems to come with OpenStack Rocky.

I don't personally see any value in aligning with Debian releases for a core product because it isn't Debian's core product. I've been thinking this is a cool thing, however, that also means we cannot easily upgrade while following the Debian release cycle. So maybe we'll *have* to stop following Debian packages eventually.

• GTirloni added a project: Cloud-VPS.Mar 21 2019, 4:56 PM
• GTirloni unsubscribed.Mar 21 2019, 9:06 PM
aborrero added a comment.Mar 22 2019, 9:45 AM

We have all this information at hand here: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Openstack_source

Not sure why you bring this up now though.

aborrero closed subtask T215407: cloudvps: cloudcontrol support for mitaka/stretch as Resolved.Mar 29 2019, 12:40 PM
Krenair added a parent task: T132225: Add SSHFP dns records to bastions .Apr 14 2019, 3:53 AM
Andrew closed subtask T221770: Upgrade cloucontrol1003/1004 to stretch/mitaka as Resolved.May 30 2019, 2:29 PM
Andrew closed subtask T221769: Upgrade cloudservices1003/1004 to stretch/mitaka as Resolved.Jun 3 2019, 5:10 PM
bd808 merged a task: T224561: Migrate remaining cloudvirt hosts to Stretch/Mitaka.Aug 5 2019, 11:38 PM
bd808 added a parent task: T224549: Track remaining jessie systems in production.
bd808 added a subscriber: MoritzMuehlenhoff.
aborrero mentioned this in T149589: Puppet tab in Horizon unusably slow.Aug 7 2019, 4:40 PM
gerritbot added a comment.Sep 2 2019, 3:15 PM

Change 533923 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] glance: add Newton config files

https://gerrit.wikimedia.org/r/533923

gerritbot added a comment.Sep 2 2019, 3:15 PM

Change 533924 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] keystone: forward mitaka config to newton

https://gerrit.wikimedia.org/r/533924

gerritbot added a comment.Sep 2 2019, 3:15 PM

Change 533925 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] keystone: update policy.json for Newton

https://gerrit.wikimedia.org/r/533925

gerritbot added a comment.Sep 2 2019, 3:15 PM

Change 533926 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Designate: add Newton config files and resources

https://gerrit.wikimedia.org/r/533926

gerritbot added a comment.Sep 2 2019, 3:15 PM

Change 533927 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack Neutron: added config files and templates for version Newton

https://gerrit.wikimedia.org/r/533927

gerritbot added a comment.Sep 4 2019, 8:56 PM

Change 533923 merged by Andrew Bogott:
[operations/puppet@production] glance: add Newton config files

https://gerrit.wikimedia.org/r/533923

gerritbot added a comment.Sep 4 2019, 9:10 PM

Change 533924 merged by Andrew Bogott:
[operations/puppet@production] keystone: forward mitaka config to newton

https://gerrit.wikimedia.org/r/533924

gerritbot added a comment.Sep 4 2019, 9:10 PM

Change 533925 merged by Andrew Bogott:
[operations/puppet@production] keystone: update policy.json for Newton

https://gerrit.wikimedia.org/r/533925

gerritbot added a comment.Sep 8 2019, 3:26 AM

Change 533927 merged by Andrew Bogott:
[operations/puppet@production] Openstack Neutron: added config files and templates for version Newton

https://gerrit.wikimedia.org/r/533927

gerritbot added a comment.Sep 8 2019, 3:27 AM

Change 533926 merged by Andrew Bogott:
[operations/puppet@production] Designate: add Newton config files and resources

https://gerrit.wikimedia.org/r/533926

Andrew moved this task from Soon! to Doing on the cloud-services-team (Kanban) board.Sep 11 2019, 3:28 PM
Andrew added a parent task: T210715: cloudvps: PDNS 3.x vs 4.x.
gerritbot added a comment.Sep 19 2019, 1:54 PM

Change 538027 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: put into maintenance mode during designate upgrade

https://gerrit.wikimedia.org/r/538027

gerritbot added a comment.Sep 19 2019, 1:54 PM

Change 538028 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Designate: move to OpenStack version 'newton'

https://gerrit.wikimedia.org/r/538028

gerritbot added a comment.Sep 19 2019, 1:56 PM

Change 538027 merged by Andrew Bogott:
[operations/puppet@production] Horizon: put into maintenance mode during designate upgrade

https://gerrit.wikimedia.org/r/538027

gerritbot added a comment.Sep 19 2019, 1:57 PM

Change 538028 merged by Andrew Bogott:
[operations/puppet@production] Designate: move to OpenStack version 'newton'

https://gerrit.wikimedia.org/r/538028

gerritbot added a comment.Sep 19 2019, 2:01 PM

Change 538031 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Designate: upgrade eqiad1 to Newton

https://gerrit.wikimedia.org/r/538031

gerritbot added a comment.Sep 19 2019, 2:02 PM

Change 538031 merged by Andrew Bogott:
[operations/puppet@production] Designate: upgrade eqiad1 to Newton

https://gerrit.wikimedia.org/r/538031

Andrew added a comment.Sep 19 2019, 2:22 PM

Cloudservices1003 and 1004 are now running Designate version Newton. There are a few more steps that we should take before we're ready for Ocata there, though -- we need to move to the worker/producer model and also (probably) to pdns4.

gerritbot added a comment.Sep 19 2019, 6:49 PM

Change 538085 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] designate: switch to the worker/producer model

https://gerrit.wikimedia.org/r/538085

gerritbot added a comment.Sep 22 2019, 6:44 PM

Change 538430 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack: add some missing files for Newton

https://gerrit.wikimedia.org/r/538430

gerritbot added a comment.Sep 22 2019, 6:46 PM

Change 538430 merged by Andrew Bogott:
[operations/puppet@production] Openstack: add some missing files for Newton

https://gerrit.wikimedia.org/r/538430

gerritbot added a comment.Sep 22 2019, 6:54 PM

Change 538431 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova: add a few more Newton files

https://gerrit.wikimedia.org/r/538431

gerritbot added a comment.Sep 22 2019, 7:06 PM

Change 538431 merged by Andrew Bogott:
[operations/puppet@production] nova: add a few more Newton files

https://gerrit.wikimedia.org/r/538431

gerritbot added a comment.Sep 25 2019, 8:41 AM

Change 539065 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: drop jessie code

https://gerrit.wikimedia.org/r/539065

aborrero removed aborrero as the assignee of this task.Sep 25 2019, 3:22 PM
aborrero added a subscriber: • JHedden.

Un-claiming this tasks myself. @JHedden and @Andrew are actively working on this, probably more than me at this point.

aborrero closed subtask T233665: Forward our neutron-l3-agent routing hacks to Openstack Newton as Resolved.Oct 2 2019, 4:03 PM
gerritbot added a comment.Oct 3 2019, 6:03 PM

Change 540643 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack: move eqiad1 glance/keystone/nova/neutron to Newton

https://gerrit.wikimedia.org/r/540643

Krenair added a parent task: T224708: Drop most of mwopenstackclients.DnsManager in favour of designateclient.Oct 5 2019, 7:17 PM
gerritbot added a comment.Oct 7 2019, 2:22 AM

Change 541133 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: put in maintenance mode for the mitaka->newton upgrade

https://gerrit.wikimedia.org/r/541133

gerritbot added a comment.Oct 7 2019, 2:01 PM

Change 541133 merged by Andrew Bogott:
[operations/puppet@production] Horizon: put in maintenance mode for the mitaka->newton upgrade

https://gerrit.wikimedia.org/r/541133

gerritbot added a comment.Oct 7 2019, 2:05 PM

Change 540643 merged by Andrew Bogott:
[operations/puppet@production] Openstack: move eqiad1 glance/keystone/nova/neutron to Newton

https://gerrit.wikimedia.org/r/540643

Stashbot added a comment.Oct 7 2019, 2:07 PM

Mentioned in SAL (#wikimedia-cloud) [2019-10-07T14:07:09Z] <arturo> horizon is disabled for maintenance (T212302)

Stashbot added a comment.Oct 7 2019, 2:25 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-07T14:25:16Z] <arturo> upgrading openstack in CloudVPS. Some IRC bots and related stuff may be unavailable (T212302)

Jdforrester-WMF mentioned this in T234834: Various user visible errors in Cloud VPS projects following OpenStack upgrade on 2019-10-07.Oct 7 2019, 4:26 PM
bd808 added a subtask: T234834: Various user visible errors in Cloud VPS projects following OpenStack upgrade on 2019-10-07.Oct 7 2019, 4:34 PM
gerritbot added a comment.Oct 7 2019, 4:58 PM

Change 541310 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] openstack: pin mitaka version on jessie openstack clients

https://gerrit.wikimedia.org/r/541310

gerritbot added a comment.Oct 7 2019, 6:15 PM

Change 541310 merged by Jhedden:
[operations/puppet@production] openstack: pin mitaka version on jessie openstack clients

https://gerrit.wikimedia.org/r/541310

gerritbot added a comment.Oct 7 2019, 6:27 PM

Change 541342 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] openstack: Update jessie openstack clients in eqiad1

https://gerrit.wikimedia.org/r/541342

gerritbot added a comment.Oct 7 2019, 6:30 PM

Change 541342 merged by Jhedden:
[operations/puppet@production] openstack: Update jessie openstack clients in eqiad1

https://gerrit.wikimedia.org/r/541342

aborrero removed a subtask: T234830: CloudVPS: m5-master databases for openstack may require re-encoding.Oct 8 2019, 9:31 AM
Andrew closed subtask T233258: designate: switch from designate-pool-manager to designate-producer/designate-worker as Resolved.Oct 10 2019, 1:47 PM
gerritbot added a comment.Oct 17 2019, 9:07 AM

Change 539065 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: drop jessie code

https://gerrit.wikimedia.org/r/539065

Stashbot added a comment.Oct 17 2019, 2:41 PM

Mentioned in SAL (#wikimedia-cloud) [2019-10-17T14:41:28Z] <jeh> deleting failed stresstest VMs that have multiple designate records stresstest1024-16-[16,17,64] left over from newton upgrade T212302

Andrew closed subtask T234834: Various user visible errors in Cloud VPS projects following OpenStack upgrade on 2019-10-07 as Resolved.Oct 18 2019, 3:28 PM
aborrero closed this task as Resolved.Oct 23 2019, 3:45 PM
aborrero claimed this task.

Closing task now, this work has been done already.

gerritbot added a comment.Nov 13 2019, 10:59 AM

Change 550659 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: clientpackages: vms: refresh comments and messages

https://gerrit.wikimedia.org/r/550659

gerritbot added a comment.Nov 13 2019, 11:08 AM

Change 550659 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: clientpackages: vms: refresh comments and messages

https://gerrit.wikimedia.org/r/550659

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL