Page MenuHomePhabricator
Create Task
Maniphest T214299

cloudvps: neutron: upgrade jessie -> stretch
Closed, ResolvedPublic
Actions

Description

Tracking task to upgrade cloudnet servers from jessie -> stretch (mitaka).

Steps are:

  • reimage standby server as stretch
  • check new interface names and update hiera accordingly
  • setup neutron l3 agent
  • failover standby (stretch as active)
  • repeat with the remaining jessie box

Details

SubjectRepoBranchLines +/-
cloudnet1004: reimage to Debian Stretchoperations/puppetproduction+4 -14
cloudvps: neutron: l3_agent: merge similar code for eqiad1/labtestnoperations/puppetproduction+55 -56
cloudvps: neutron: introduce base interface hiera keyoperations/puppetproduction+11 -6
cloudnet1003: hiera: refresh interface namesoperations/puppetproduction+6 -6
cloudnet1003: reimage to stretchoperations/puppetproduction+10 -2
labtestn: neutron: refresh hiera settings for interface namesoperations/puppetproduction+6 -19
cloudnet2001-dev: hiera cleanup for stretch/mitakaoperations/puppetproduction+13 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

aborrero triaged this task as Medium priority.Jan 21 2019, 1:06 PM
aborrero created this task.

I'm conducting the steps in labtestn first for the sake of testing.

aborrero added a subtask: T214167: labtestneutron2001: reimage to stretch & rename to cloudnet2001-dev.Jan 21 2019, 1:07 PM
aborrero added a subtask: T214297: cloudvps: neutron: investigate keepalived warning.
aborrero moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.
aborrero added a comment.Jan 21 2019, 1:10 PM

For the record, I needed these neutron commands when enrolling the new l3 agent in labtestn:

root@labtestcontrol2003:~# neutron l3-agent-list-hosting-router cloudinstances2b-gw
+--------------------------------------+--------------------+----------------+-------+----------+
| id                                   | host               | admin_state_up | alive | ha_state |
+--------------------------------------+--------------------+----------------+-------+----------+
| 569a3a0b-824f-4189-b6c7-b067d14a0fa3 | labtestneutron2001 | True           | xxx   | standby  |
| ac4771f4-bf50-4607-9b2b-45a3fb0d574b | labtestneutron2002 | True           | :-)   | active   |
+--------------------------------------+--------------------+----------------+-------+----------+

root@labtestcontrol2003:~# neutron agent-delete 569a3a0b-824f-4189-b6c7-b067d14a0fa3
Deleted agent: 569a3a0b-824f-4189-b6c7-b067d14a0fa3

root@labtestcontrol2003:~# neutron l3-agent-list-hosting-router cloudinstances2b-gw
+--------------------------------------+--------------------+----------------+-------+----------+
| id                                   | host               | admin_state_up | alive | ha_state |
+--------------------------------------+--------------------+----------------+-------+----------+
| ac4771f4-bf50-4607-9b2b-45a3fb0d574b | labtestneutron2002 | True           | :-)   | active   |
+--------------------------------------+--------------------+----------------+-------+----------+

root@labtestcontrol2003:~# neutron agent-list | grep cloudnet
| 312ede1a-e0ea-4556-88f9-3a7ce45a02f8 | Metadata agent     | cloudnet2001-dev   |                   | :-)   | True           | neutron-metadata-agent    |
| 498c6ec0-1e65-4514-9633-be34e5f3a486 | Linux bridge agent | cloudnet2001-dev   |                   | :-)   | True           | neutron-linuxbridge-agent |
| 61d2165e-0269-4964-96df-24bc61b2569d | L3 agent           | cloudnet2001-dev   | nova              | :-)   | True           | neutron-l3-agent          |
| 8bfffc22-509d-4ca0-930f-17e3f78da1d9 | DHCP agent         | cloudnet2001-dev   | nova              | :-)   | True           | neutron-dhcp-agent        |

root@labtestcontrol2003:~# neutron l3-agent-router-add 61d2165e-0269-4964-96df-24bc61b2569d cloudinstances2b-gw
Added router cloudinstances2b-gw to L3 agent

root@labtestcontrol2003:~# neutron l3-agent-list-hosting-router cloudinstances2b-gw
+--------------------------------------+--------------------+----------------+-------+----------+
| id                                   | host               | admin_state_up | alive | ha_state |
+--------------------------------------+--------------------+----------------+-------+----------+
| 61d2165e-0269-4964-96df-24bc61b2569d | cloudnet2001-dev   | True           | :-)   | standby  |
| ac4771f4-bf50-4607-9b2b-45a3fb0d574b | labtestneutron2002 | True           | :-)   | active   |
+--------------------------------------+--------------------+----------------+-------+----------+
aborrero added a comment.Jan 21 2019, 1:36 PM

I was unable to get the new cloudnet2001-dev working with neutron. I just saw that interface names are different and we need some hiera overrides.

gerritbot added a comment.Jan 21 2019, 1:48 PM

Change 485657 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudnet2001-dev: hiera cleanup for stretch/mitaka

https://gerrit.wikimedia.org/r/485657

gerritbot added a project: Patch-For-Review.Jan 21 2019, 1:48 PM

Change 485657 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudnet2001-dev: hiera cleanup for stretch/mitaka

https://gerrit.wikimedia.org/r/485657

aborrero updated the task description. (Show Details)Jan 21 2019, 1:52 PM
aborrero closed subtask T214303: labtestneutron2002: reimage to stretch & rename to cloudnet2002-dev as Resolved.Jan 22 2019, 12:37 PM
gerritbot added a comment.Jan 22 2019, 12:40 PM

Change 485800 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] labtestn: neutron: refresh hiera settings for interface names

https://gerrit.wikimedia.org/r/485800

gerritbot added a comment.Jan 22 2019, 12:41 PM

Change 485800 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] labtestn: neutron: refresh hiera settings for interface names

https://gerrit.wikimedia.org/r/485800

aborrero updated the task description. (Show Details)Jan 22 2019, 1:15 PM
gerritbot added a comment.Jan 22 2019, 6:31 PM

Change 485869 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudnet1003: reimage to stretch

https://gerrit.wikimedia.org/r/485869

gerritbot added a comment.Jan 22 2019, 6:32 PM

Change 485869 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudnet1003: reimage to stretch

https://gerrit.wikimedia.org/r/485869

ops-monitoring-bot subscribed.Jan 22 2019, 6:36 PM

Script wmf-auto-reimage was launched by aborrero on cumin1001.eqiad.wmnet for hosts:

cloudnet1003.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201901221836_aborrero_53986_cloudnet1003_eqiad_wmnet.log.

Stashbot added a comment.Jan 22 2019, 6:36 PM

Mentioned in SAL (#wikimedia-operations) [2019-01-22T18:36:56Z] <arturo> T214299 reimaging cloudnet1003 as debian stretch

Stashbot added a comment.Jan 22 2019, 6:37 PM

Mentioned in SAL (#wikimedia-cloud) [2019-01-22T18:37:08Z] <arturo> T214299 reimaging cloudnet1003 as debian stretch

Stashbot added a comment.Jan 22 2019, 6:40 PM

Mentioned in SAL (#wikimedia-cloud) [2019-01-22T18:40:31Z] <arturo> T214299 manually delete from neutron agents from cloudnet1003 (must be added again after reimage, with new uuids)

ops-monitoring-bot added a comment.Jan 22 2019, 7:17 PM

Completed auto-reimage of hosts:

['cloudnet1003.eqiad.wmnet']

and were ALL successful.

Stashbot added a comment.Jan 22 2019, 7:19 PM

Mentioned in SAL (#wikimedia-cloud) [2019-01-22T19:19:47Z] <arturo> T214299 stretch cloudnet1003 is apparently all set

gerritbot added a comment.Jan 22 2019, 7:27 PM

Change 485878 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudnet1003: hiera: refresh interface names

https://gerrit.wikimedia.org/r/485878

gerritbot added a comment.Jan 22 2019, 7:28 PM

Change 485878 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudnet1003: hiera: refresh interface names

https://gerrit.wikimedia.org/r/485878

Stashbot added a comment.Jan 22 2019, 7:30 PM

Mentioned in SAL (#wikimedia-operations) [2019-01-22T19:30:41Z] <arturo> T214299 additional reboot for cloudnet1003

Dzahn subscribed.Jan 23 2019, 12:04 AM

cloudnet1003 - CRITICAL - degraded: The system is operational but one or more units failed.

https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=cloudnet1003&service=Check+systemd+state

gerritbot added a comment.Jan 23 2019, 10:32 AM

Change 486047 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudvps: neutron: introduce base interface hiera key

https://gerrit.wikimedia.org/r/486047

gerritbot added a comment.Jan 23 2019, 10:58 AM

Change 486047 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudvps: neutron: introduce base interface hiera key

https://gerrit.wikimedia.org/r/486047

Stashbot added a comment.Jan 23 2019, 11:04 AM

Mentioned in SAL (#wikimedia-operations) [2019-01-23T11:04:18Z] <arturo> T214299 reboot cloudnet2001-dev, cloudnet2002-dev and cloudnet1003 for new interface names

aborrero added a comment.Jan 23 2019, 11:09 AM
In T214299#4900925, @Dzahn wrote:

cloudnet1003 - CRITICAL - degraded: The system is operational but one or more units failed.

https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=cloudnet1003&service=Check+systemd+state

Thanks @Dzahn for the heads-up :-) really appreciated (I didn't noticed myself)

gerritbot added a comment.Jan 23 2019, 11:25 AM

Change 486060 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudvps: neutron: l3_agent: merge similar code for eqiad1/labtestn

https://gerrit.wikimedia.org/r/486060

gerritbot added a comment.Jan 23 2019, 11:37 AM

Change 486060 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudvps: neutron: l3_agent: merge similar code for eqiad1/labtestn

https://gerrit.wikimedia.org/r/486060

Stashbot added a comment.Jan 24 2019, 9:51 AM

Mentioned in SAL (#wikimedia-cloud) [2019-01-24T09:51:48Z] <arturo> T214299 failover cloudnet1004 to cloudnet1003

gerritbot added a comment.Jan 24 2019, 9:58 AM

Change 486236 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudnet1004: reimage to Debian Stretch

https://gerrit.wikimedia.org/r/486236

gerritbot added a comment.Jan 24 2019, 10:00 AM

Change 486236 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudnet1004: reimage to Debian Stretch

https://gerrit.wikimedia.org/r/486236

ops-monitoring-bot added a comment.Jan 24 2019, 10:03 AM

Script wmf-auto-reimage was launched by aborrero on cumin1001.eqiad.wmnet for hosts:

cloudnet1004.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201901241003_aborrero_47033_cloudnet1004_eqiad_wmnet.log.

Stashbot added a comment.Jan 24 2019, 10:03 AM

Mentioned in SAL (#wikimedia-operations) [2019-01-24T10:03:46Z] <arturo> T214299 reimage cloudnet1004 to debian stretch

Stashbot added a comment.Jan 24 2019, 10:03 AM

Mentioned in SAL (#wikimedia-cloud) [2019-01-24T10:03:55Z] <arturo> T214299 reimage cloudnet1004 to debian stretch

ops-monitoring-bot added a comment.Jan 24 2019, 10:44 AM

Completed auto-reimage of hosts:

['cloudnet1004.eqiad.wmnet']

and were ALL successful.

Stashbot added a comment.Jan 24 2019, 10:59 AM

Mentioned in SAL (#wikimedia-operations) [2019-01-24T10:59:07Z] <arturo> T214299 additional reboot for cloudnet1004

Stashbot added a comment.Jan 24 2019, 11:07 AM

Mentioned in SAL (#wikimedia-cloud) [2019-01-24T11:07:09Z] <arturo> T214299 failover cloudnet1003 to cloudnet1004

aborrero closed this task as Resolved.Jan 24 2019, 11:08 AM

All seems fine. All cloudnet servers are running mitaka/stretch.

• JHedden closed subtask T214297: cloudvps: neutron: investigate keepalived warning as Resolved.Nov 1 2019, 3:36 PM
Maintenance_bot removed a project: Patch-For-Review.Nov 1 2019, 4:11 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL