Fri, May 18
eth1 on both should be connected and configured to be in the cloud-instance-ports interface-range which makes them trunks that pass the instance network.
I tried to bring the eth1 interfaces up and no dice. My thought is they are not connected.
Thu, May 17
Wed, May 16
You know it! +1
heyo -- I talked to the team about this situation yesterday and the outcome is to mimic labstore1004/5 (except in the public VLAN):
Tue, May 15
We ran through our normal procedure to fail traffic from labnet1002 back to labnet1001 (post move this morning). Labnet1001 saw incoming traffic from external parties hit eth0 but could not route that to any instances. The bridge interface and addressing came up for br1102 and the gateway IP transferred but still no connectivity. I looked at eth1 there and the corresponding switch interface and did not see anything that immediately made me think this was a quick fix so we decided to move traffic back to labnet1002.
Mon, May 14
I am tossing your way so you can dig through this and review my thinking ...and code :)
I am reopening for a bit of investigation on existing functionality. The currently set routing_source_ip does not appear in the router namespace as a secondary address. There is concern that this may have undesired effects.
Planning to talk to the team about this during the normal meeting tomorrow.
Sat, May 12
Fri, May 11
Thu, May 10
@Cmjohnson did you get a chance to move labnet1002?
Seems like the total list for cloud-services-team to really worry about from the physical diagrams is:
Wed, May 2
This is still currently a SPOF and we are probably weeks out on the replacement systems (soon to be racked). Probably best to replace this ASAP if we can.
fyi on labtestneutron2001 atm
Tue, May 1
I have previously talked with @ayounsi about this and promised a task weeks ago :) I did assign this but only bc of that and I know @ayounsi is the human who can help the Cloud team sort this out. Thanks man!
Mon, Apr 30
Apr 27 2018
@Bstorm fyi I imagine this is headed your way once the initial hard blocker of T191845 is resolved
in favor of T193264
Seems to be working
Seems to be working
labtestcontrol2003.wikimedia.org Debian GNU/Linux 8 \n \l
Note T193196 is related for next phases here but this is racked/stack/imaged
Apr 26 2018
These are now debian jessie shoutout to @RobH for helping me work through some install issues :)
Apr 24 2018
I jumped in on mwoffliner4.mwoffliner.eqiad.wmflabs and hot patched it to:
Apr 21 2018
Apr 20 2018
Apr 12 2018
Apr 10 2018
Apr 9 2018
@aborrero can you reimage labtestservices2002.wikimedia.org as jessie as part of this? There is a lot of work to be done but I think post that this can be closed.
Apr 5 2018
Apr 2 2018
Sincerest thanks to you all :)
@RobH figured out what he believed is the eth0 issue described, unless a screenshot was captured I don't think there are logs but the message he pasted in irc from console was something very literal like "failed as eth0 is not connected". It was my understanding that this has been seen in the past on Trusty and was a somewhat forgotten but known old issue. Trying to image with Trusty reproduces. I thought @RobH was going to try to circle back on this and see if the issue can be overcome easily but I was away on vacation and then we have both been busy, I figured T187373 was the priority of the two pending cloud hardware isues from a dcops perspective so I wasn't too worried.
Mar 20 2018
@RobH just a ping that we are talking about this in our weekly, are you going to have time to check into where to go from here? easy money says maybe we should just connect up the 10G interfaces to image these for a short term thing.
Mar 16 2018
2018-03-16 15:10:54.264 72304 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 461, in _send 2018-03-16 15:10:54.264 72304 ERROR oslo_service.service raise result 2018-03-16 15:10:54.264 72304 ERROR oslo_service.service RemoteError: Remote error: IncompatibleObjectVersion Version 1.3 of MigrationList is not supported 2018-03-16 15:10:54.264 72304 ERROR oslo_service.service [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n executor_callback))\n', u' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\n', u' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 937, in object_class_action_versions\n context, objname, objmethod, object_versions, args, kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 468, in object_class_action_versions\n objname, object_versions[objname])\n', u' File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 355, in obj_class_from_name\n supported=latest_ver)\n', u'IncompatibleObjectVersion: Version 1.3 of MigrationList is not supported\n'].
@Andrew tried to merge the change to allow nova to be more gracious and it didn't work out.
closed in favor of T189871
Mar 15 2018