Status	Assigned	Task
Open	None	T138150 Purge stale data from LDAP
Resolved	Andrew	T148781 Clean up ldap host entries and references
Resolved	Krenair	T108625 Remove reliance on ldap $::projectid from shinkengen
Resolved	Andrew	T42022 Add icinga checks for all nova, glance, and keystone related services
Invalid	None	T90784 Monitor nova-scheduler log for lost contact with compute nodes
Resolved	Andrew	T104575 Don't rely on wikitech API for production services
Resolved	Andrew	T104588 Give 'novaobserver' keystone account rights to read everything, everywhere, write or change nothing
Resolved	Andrew	T104586 upgrade to Openstack Kilo
Resolved	Andrew	T104587 Upgrade Labs to Openstack Juno
Resolved	yuvipanda	T105720 Labs team reliability goal for Q1 2015/16
Resolved	Andrew	T106144 Make an evacuation plan for labs instances
Declined	None	T106146 Get instance block-migration working reliably; script and document
Resolved	Andrew	T90821 Upgrade labs cluster to Trusty
Resolved	Andrew	T90823 Upgrade labs network node to trusty
Resolved	Andrew	T99701 Setup/Install/Deploy labnet1002
Resolved	RobH	T98740 labnet1002
Resolved	• Cmjohnson	T99702 relocate WMF3153 (now labnet1002) from row C to B
Resolved	• Cmjohnson	T102131 check labnet1002 for pci slot space
Resolved	Andrew	T109329 Try to fail over to labnet1002
Resolved	Andrew	T109653 move nova api to labnet1002

Event Timeline

RobH created this task.May 19 2015, 7:13 PM

RobH claimed this task.

RobH raised the priority of this task from to Medium.

RobH updated the task description. (Show Details)

RobH added a project: acl*sre-team.

RobH added subscribers: RobH, Andrew.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 19 2015, 7:13 PM

RobH added a subtask: T98740: labnet1002.May 19 2015, 7:13 PM

• Cmjohnson closed subtask T99702: relocate WMF3153 (now labnet1002) from row C to B as Resolved.May 19 2015, 8:00 PM

Very happy to see this chugging along!

This will be a Trusty box. Note also that the network setup for labnet1001 is weird, and labnet1002 needs the same setup. I believe that eth1,2,3 are bonded and eth0 has a separate connection; hopefully you can just look at 1001 and copy the setup.

The 10G upgrade was somehow neglected in the moving and allocation of the server (my bad.)

I've created https://rt.wikimedia.org/Ticket/Display.html?id=9407 to track the quote/order of said NIC.

RobH updated the task description. (Show Details)Jun 10 2015, 8:51 PM

RobH set Security to None.

RobH updated the task description. (Show Details)

Andrew added a parent task: T90823: Upgrade labs network node to trusty.Jun 17 2015, 1:33 AM

• Cmjohnson closed subtask T102131: check labnet1002 for pci slot space as Resolved.Jun 17 2015, 11:36 AM

I added the 10G Nic card, and updated dhcpd file with the correct MAC. I also disabled pxe for the 1Gb Eth.

I do want to check with someone about the network aspect. labnet1001 is currently using both xe-2/1/0 and xe-2/1/2. I believe we may have to make a direct connection to cr1 and cr2 to a different uplink module.

I do have an additional uplink module and if have space in B3.

Andrew added a project: Labs-Sprint-104.Jun 29 2015, 5:46 PM

Restricted Application added a subscriber: Matanya. · View Herald TranscriptJun 29 2015, 5:46 PM

Andrew added a project: Labs-Sprint-105.Jul 6 2015, 5:41 PM

I disabled the internal nic and got the console unstuck. I assume some cables need to be switched around now, so that that the 10g interface can act as the new eth0 and eth1 during startup.

This is blocked pending a replacement 10g card.

Andrew moved this task from To Do to Code Review / Blocked on the Labs-Sprint-105 board.Jul 10 2015, 3:26 PM

Andrew added a parent task: T105720: Labs team reliability goal for Q1 2015/16.Jul 17 2015, 2:52 PM

Andrew removed a parent task: T105720: Labs team reliability goal for Q1 2015/16.Jul 17 2015, 2:59 PM

Ricordisamoa subscribed.Jul 17 2015, 3:15 PM

Andrew added a project: Labs-Sprint-107.Jul 27 2015, 5:33 PM

I'm not sure what the status is on this, other than the task was assigned to me.

Is the new NIC not working?

The box is up and running. Getting it to actually do useful Nova things is up to me now.

Now, how do we switch to actually using this box? One option is

https://phabricator.wikimedia.org/T107731

Another option is to try to do some clever in-place swapping where labnet1002 steps into the role of labnet1001 without the instances noticing... I've no idea how to make that happen though.

I've no idea how to make that happen though.

Well, that's not exactly true. It is /probably/ as simple as changing

templates/wmnet:eth4-1102.labnet1001 1H IN A 10.68.16.1
to
templates/wmnet:eth4-1102.labnet1002 1H IN A 10.68.16.1

And then restarting nova-network on labnet1002. Scary, though!

whoah, except why is that eth4-1102 when the actual interface is eth1.1102@eth1?

More info from code-diving: When associating a new instance with a network in flatdhcp mode, it just grabs the list of networks and associates the first one in the list.

Andrew moved this task from To Do to Code Review / Blocked on the Labs-Sprint-107 board.Aug 3 2015, 5:31 PM

Andrew moved this task from Code Review / Blocked to Doing on the Labs-Sprint-107 board.

Andrew added a project: Labs-Sprint-108.Aug 3 2015, 5:45 PM

Andrew moved this task from To Do to Code Review / Blocked on the Labs-Sprint-108 board.Aug 10 2015, 5:34 PM

Andrew added a project: Labs-Sprint-109.Aug 10 2015, 5:42 PM

Change 231406 had a related patch set uploaded (by Andrew Bogott):
Replace some hardcoded labnet1001 refs with hiera values.

https://gerrit.wikimedia.org/r/231406

gerritbot added a project: Patch-For-Review.Aug 13 2015, 8:06 PM

• chasemp subscribed.Aug 13 2015, 9:57 PM

Andrew moved this task from To Do to Code Review / Blocked on the Labs-Sprint-109 board.Aug 14 2015, 11:43 PM

Andrew added a project: labs-sprint-110.Aug 17 2015, 8:18 PM

Andrew moved this task from To do to Code Review/Blocked on the labs-sprint-110 board.Aug 17 2015, 8:28 PM