chasemp (Chase)Administrator
Lead Operations Engineer (Wikimedia Cloud Services)

Projects (31)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Sep 16 2014, 11:39 AM (191 w, 5 d)
Roles
Administrator
Availability
Available
IRC Nick
chasemp
LDAP User
Rush
MediaWiki User
CPettet (WMF)

Recent Activity

Fri, May 18

chasemp added a comment to T194964: Connect or troubleshoot eth1 on labvirt1019 and labvirt1020.

eth1 on both should be connected and configured to be in the cloud-instance-ports interface-range which makes them trunks that pass the instance network.

Fri, May 18, 6:06 PM · Cloud-Services, Operations, ops-eqiad
chasemp added a comment to T194964: Connect or troubleshoot eth1 on labvirt1019 and labvirt1020.

I tried to bring the eth1 interfaces up and no dice. My thought is they are not connected.

Fri, May 18, 6:02 PM · Cloud-Services, Operations, ops-eqiad

Thu, May 17

chasemp added a comment to T193496: Allocate public v4 IPs for Neutron setup in eqiad.

The /25 -> /24 renumbering seems fairly straightforward, but given a) IPv4's depletion (we effectively cannot get more IPv4 space from any of the RIRs), b) the Neutron redesign and c) Cloud Services' growth and needs like T122406's, I think it's worthwhile to look at it a bit more broadly in order to make sure we avoid e.g. depletion or fragmentation of our IP space. Perhaps for instance we need to be looking at a larger assignment :)

So first of:

  • My understanding from the description is that the intention is for this to be used for floating IPs. Do you foresee any other needs in terms of public IPv4 space besides those?
  • Relatedly, do we have any kind of historical growth figures and/or estimates for future growth, in the short-term or mid-term, besides those extra 20 IPs for Toolforge that you mentioned?
  • Is this going to be routed just to eqiad, or does the upcoming addition of the second zone in the next FY means that this is also going to be partially routed to codfw? I'm asking because routing a single /24 to two DCs is possible, but it would cause an impact to availability in case of a split brain between our data centers that may or may not be acceptable?
  • How do we intend to subnet this /24? Is this all going to be flat for floating IPs, or partitioned somehow per data center, row, other kind of availability zone, etc.?

    Finally, I think it would be useful to look at this holistically and figure out the addressing plan for {eqiad, codfw} × {IPv4, IPv6} × {public, private} for at least as far as we can reasonably foresee. Do we have a sense for that yet?
Thu, May 17, 4:05 PM · netops, Operations, Cloud-Services
chasemp updated the task description for T193496: Allocate public v4 IPs for Neutron setup in eqiad.
Thu, May 17, 3:40 PM · netops, Operations, Cloud-Services
chasemp added a comment to T193655: rack/setup/install labstore1008 & labstore1009.

@chasemp The only row I have available that I can put in adjacent racks is A5 and A6. will that work for you?

Thu, May 17, 3:10 PM · ops-eqiad, cloud-services-team, Cloud-VPS, Operations

Wed, May 16

chasemp updated subscribers of T194859: Toolforge maintain-kubeusers stauck in infinite sleeps of 10 seconds.

@Bstorm or @aborrero can you look at this? New tools will be in a bad way until this is fixed. Ldap issues...?

Wed, May 16, 9:21 PM · Toolforge
bd808 awarded T194853: Request creation of clouddb-services VPS project a Goat token.
Wed, May 16, 8:39 PM · Cloud-VPS (Project-requests)
chasemp closed T194853: Request creation of clouddb-services VPS project as Resolved.
Wed, May 16, 8:13 PM · Cloud-VPS (Project-requests)
chasemp added a comment to T194853: Request creation of clouddb-services VPS project.

You know it! +1

Wed, May 16, 8:13 PM · Cloud-VPS (Project-requests)
chasemp triaged T194853: Request creation of clouddb-services VPS project as Normal priority.
Wed, May 16, 8:13 PM · Cloud-VPS (Project-requests)
chasemp reassigned T193655: rack/setup/install labstore1008 & labstore1009 from chasemp to Cmjohnson.

heyo -- I talked to the team about this situation yesterday and the outcome is to mimic labstore1004/5 (except in the public VLAN):

Wed, May 16, 1:56 PM · ops-eqiad, cloud-services-team, Cloud-VPS, Operations

Tue, May 15

chasemp added a comment to T193579: Update and move labnet1001/1002.

cloud-instances1-b-eqiad was not trunked between asw2 and asw, it is now.

Tue, May 15, 8:28 PM · Patch-For-Review, Cloud-VPS, ops-eqiad, Operations
chasemp added a comment to T193579: Update and move labnet1001/1002.

We ran through our normal procedure to fail traffic from labnet1002 back to labnet1001 (post move this morning). Labnet1001 saw incoming traffic from external parties hit eth0 but could not route that to any instances. The bridge interface and addressing came up for br1102 and the gateway IP transferred but still no connectivity. I looked at eth1 there and the corresponding switch interface and did not see anything that immediately made me think this was a quick fix so we decided to move traffic back to labnet1002.

Tue, May 15, 5:28 PM · Patch-For-Review, Cloud-VPS, ops-eqiad, Operations
chasemp updated the task description for T193579: Update and move labnet1001/1002.
Tue, May 15, 5:21 PM · Patch-For-Review, Cloud-VPS, ops-eqiad, Operations
chasemp updated the task description for T193579: Update and move labnet1001/1002.
Tue, May 15, 4:29 PM · Patch-For-Review, Cloud-VPS, ops-eqiad, Operations
chasemp updated the task description for T193579: Update and move labnet1001/1002.
Tue, May 15, 2:10 PM · Patch-For-Review, Cloud-VPS, ops-eqiad, Operations
chasemp updated the task description for T193579: Update and move labnet1001/1002.
Tue, May 15, 1:37 PM · Patch-For-Review, Cloud-VPS, ops-eqiad, Operations

Mon, May 14

chasemp reassigned T168580: Neutron implementation of routing_source_ip definition from chasemp to aborrero.

I am tossing your way so you can dig through this and review my thinking ...and code :)

Mon, May 14, 8:19 PM · Patch-For-Review, Epic, Cloud-Services
chasemp reopened T168580: Neutron implementation of routing_source_ip definition as "Open".

I am reopening for a bit of investigation on existing functionality. The currently set routing_source_ip does not appear in the router namespace as a secondary address. There is concern that this may have undesired effects.

Mon, May 14, 8:18 PM · Patch-For-Review, Epic, Cloud-Services
chasemp reopened T168580: Neutron implementation of routing_source_ip definition, a subtask of T167293: Nova-network to Neutron migration, as Open.
Mon, May 14, 8:18 PM · Epic, Cloud-Services
chasemp added a comment to T193655: rack/setup/install labstore1008 & labstore1009.

Planning to talk to the team about this during the normal meeting tomorrow.

Mon, May 14, 7:15 PM · ops-eqiad, cloud-services-team, Cloud-VPS, Operations

Sat, May 12

chasemp closed T194554: Trash as Invalid.
Sat, May 12, 3:39 AM · Trash

Fri, May 11

chasemp added a comment to T187962: Rack/cable/configure asw2-c-eqiad switch stack.

I'm flying on the 29th. If Chase wants to manage these things without me that's fine with me though :)

Fri, May 11, 6:27 PM · Patch-For-Review, Operations, ops-eqiad, netops

Thu, May 10

chasemp added a comment to T193579: Update and move labnet1001/1002.

@Cmjohnson did you get a chance to move labnet1002?

Thu, May 10, 5:56 PM · Patch-For-Review, Cloud-VPS, ops-eqiad, Operations
chasemp updated subscribers of T187962: Rack/cable/configure asw2-c-eqiad switch stack.

Seems like the total list for cloud-services-team to really worry about from the physical diagrams is:

Thu, May 10, 5:56 PM · Patch-For-Review, Operations, ops-eqiad, netops
chasemp added a comment to T193496: Allocate public v4 IPs for Neutron setup in eqiad.

@chasemp Can you provide an ETA for returning the /25?

Thu, May 10, 5:44 PM · netops, Operations, Cloud-Services
chasemp updated the task description for T193496: Allocate public v4 IPs for Neutron setup in eqiad.
Thu, May 10, 5:44 PM · netops, Operations, Cloud-Services

Wed, May 2

chasemp added a project to T193651: labstore1003 SMART failure: Cloud-VPS.
Wed, May 2, 5:49 PM · Cloud-VPS, ops-eqiad, cloud-services-team, Operations
chasemp assigned T193651: labstore1003 SMART failure to Cmjohnson.

This is still currently a SPOF and we are probably weeks out on the replacement systems (soon to be racked). Probably best to replace this ASAP if we can.

Wed, May 2, 5:49 PM · Cloud-VPS, ops-eqiad, cloud-services-team, Operations
chasemp added a comment to T188392: package prometheus-rabbitmq-exporter for Debian jessie.

fyi on labtestneutron2001 atm

Wed, May 2, 5:44 PM · Patch-For-Review, Cloud-VPS, Operations
chasemp added a parent task for T193657: integrate nova.conf missing settings into neutron setup: T167293: Nova-network to Neutron migration.
Wed, May 2, 5:36 PM · Patch-For-Review, Cloud-VPS
chasemp added a subtask for T167293: Nova-network to Neutron migration: T193657: integrate nova.conf missing settings into neutron setup.
Wed, May 2, 5:36 PM · Epic, Cloud-Services
chasemp triaged T193657: integrate nova.conf missing settings into neutron setup as Normal priority.
Wed, May 2, 5:36 PM · Patch-For-Review, Cloud-VPS
chasemp created T193657: integrate nova.conf missing settings into neutron setup.
Wed, May 2, 5:35 PM · Patch-For-Review, Cloud-VPS
chasemp added a comment to T193579: Update and move labnet1001/1002.

@Andrew @chasemp One other thing we should do here is move labnet1002 to the new switch.

Can we do this on May 15? 1500UTC/1000 EST

Wed, May 2, 2:46 PM · Patch-For-Review, Cloud-VPS, ops-eqiad, Operations
chasemp added a parent task for T193196: labnet1003 and labnet1004 moving and enabling 10G NICs: Unknown Object (Task).
Wed, May 2, 2:01 PM · Cloud-VPS, Operations, ops-eqiad

Tue, May 1

chasemp added a comment to T193272: Prometheus vs. CPU usage vs. hyperthreading.

https://www.percona.com/blog/2015/01/15/hyper-threading-double-cpu-throughput/

Tue, May 1, 3:57 PM · Operations, cloud-services-team, monitoring
chasemp added a comment to T122406: Consider renumbering Labs to separate address spaces.

T193496: Allocate public v4 IPs for Neutron setup in eqiad is related

Tue, May 1, 2:56 PM · Cloud-Services, netops, Operations
chasemp added a comment to T193496: Allocate public v4 IPs for Neutron setup in eqiad.

I have previously talked with @ayounsi about this and promised a task weeks ago :) I did assign this but only bc of that and I know @ayounsi is the human who can help the Cloud team sort this out. Thanks man!

Tue, May 1, 2:55 PM · netops, Operations, Cloud-Services
chasemp triaged T193496: Allocate public v4 IPs for Neutron setup in eqiad as Normal priority.
Tue, May 1, 2:53 PM · netops, Operations, Cloud-Services
chasemp created P7059 floating ips.
Tue, May 1, 2:45 PM

Mon, Apr 30

chasemp updated subscribers of T193196: labnet1003 and labnet1004 moving and enabling 10G NICs.

@chasemp Confirmed both are 10G w/2 nics, labnet1004 can go to B2...I do not currently have any labnet server in that rack. labnet1001 is in B3 and labnet1002 is B4. FYI all of these servers will be moving to new switches soon. labnet1003 will need to move to B2/B4 or B7

I am thinking this
labnet1001 moves from B3 to B2
labnet1002 stays in B4
labnet1003 stay in B7
labnet1004 moves to B2

Let me know if that works for you and let's schedule the move sooner rather than later. Thanks

Mon, Apr 30, 2:41 PM · Cloud-VPS, Operations, ops-eqiad
chasemp added a comment to T167559: Create a detailed migration plan for implementing Neutron as our OpenStack SDN layer.

groupings at the moment

main (nova-network)

labcontrol1001.wikimedia.org
 labcontrol1002.wikimedia.org
 labmon1001.eqiad.wmnet
 labmon1002.eqiad.wmnet
 labnet1001.eqiad.wmnet
 labnet1002.eqiad.wmnet
 labnodepool1001.eqiad.wmnet
 labnodepool1002.eqiad.wmnet
 labservices1001.wikimedia.org
 labservices1002.wikimedia.org
 labvirt1001.eqiad.wmnet
 labvirt1002.eqiad.wmnet
 labvirt1003.eqiad.wmnet
 labvirt1004.eqiad.wmnet
 labvirt1005.eqiad.wmnet
 labvirt1006.eqiad.wmnet
 labvirt1007.eqiad.wmnet
 labvirt1008.eqiad.wmnet
 labvirt1009.eqiad.wmnet
 labvirt1010.eqiad.wmnet
 labvirt1011.eqiad.wmnet
 labvirt1012.eqiad.wmnet
 labvirt1013.eqiad.wmnet
 labvirt1014.eqiad.wmnet
 labvirt1015.eqiad.wmnet
 labvirt1016.eqiad.wmnet
 labvirt1017.eqiad.wmnet
 labvirt1018.eqiad.wmnet
 labvirt1019.eqiad.wmnet
 labvirt1020.eqiad.wmnet
 labvirt1021.eqiad.wmnet
 labvirt1022.eqiad.wmnet
 labweb1001.wikimedia.org
 labweb1002.wikimedia.org

main (neutron)

labcontrol1003.wikimedia.org
 labcontrol1004.wikimedia.org
 labnet1003.wikimedia.org
 labnet1004.wikimedia.org

labtest

labtestcontrol2001.wikimedia.org
 labtestnet2001.codfw.wmnet
 labtestnet2002.codfw.wmnet
 labtestpuppetmaster2001.wikimedia.org
 labtestservices2001.wikimedia.org
 labtestvirt2001.codfw.wmnet
 labtestvirt2002.codfw.wmnet
 labtestweb2001.wikimedia.org

labtestn

labtestcontrol2003.wikimedia.org
 labtestneutron2001.codfw.wmnet
 labtestneutron2002.codfw.wmnet
 labtestservices2002.wikimedia.org
 labtestservices2003.wikimedia.org
 labtestvirt2003.codfw.wmnet
 labtestmetal2001.codfw.wmnet (as virt)
Mon, Apr 30, 1:56 PM · Patch-For-Review, Goal, cloud-services-team (FY2017-18)

Fri, Apr 27

chasemp updated subscribers of T193264: Replace toolsdb and co with instances on labvirt1019 and labvirt1020.

@Bstorm fyi I imagine this is headed your way once the initial hard blocker of T191845 is resolved

Fri, Apr 27, 6:54 PM · Patch-For-Review, Epic, Cloud-VPS, cloud-services-team
chasemp closed T172538: rack/setup/install labvirt10(19|20).eqiad.wmnet as Resolved.

in favor of T193264

Fri, Apr 27, 6:53 PM · cloud-services-team (Kanban), Operations, Cloud-Services
chasemp merged task T187373: Rebuild raids on labvirt1019 and 1020 into Unknown Object (Task).
Fri, Apr 27, 6:52 PM · cloud-services-team (Kanban), Operations, Cloud-Services
chasemp added a subtask for T193264: Replace toolsdb and co with instances on labvirt1019 and labvirt1020: Unknown Object (Task).
Fri, Apr 27, 6:51 PM · Patch-For-Review, Epic, Cloud-VPS, cloud-services-team
chasemp triaged T193264: Replace toolsdb and co with instances on labvirt1019 and labvirt1020 as Normal priority.
Fri, Apr 27, 6:50 PM · Patch-For-Review, Epic, Cloud-VPS, cloud-services-team
chasemp created T193264: Replace toolsdb and co with instances on labvirt1019 and labvirt1020.
Fri, Apr 27, 6:50 PM · Patch-For-Review, Epic, Cloud-VPS, cloud-services-team
chasemp added a comment to T167559: Create a detailed migration plan for implementing Neutron as our OpenStack SDN layer.

In progress https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Neutron_Notes/phases

Fri, Apr 27, 6:44 PM · Patch-For-Review, Goal, cloud-services-team (FY2017-18)
chasemp closed T167357: Determine need and replacement for dmz_cidr configuration in nova-network as Resolved.

Seems to be working

Fri, Apr 27, 6:30 PM · Patch-For-Review, Cloud-Services
chasemp closed T167357: Determine need and replacement for dmz_cidr configuration in nova-network, a subtask of T167293: Nova-network to Neutron migration, as Resolved.
Fri, Apr 27, 6:30 PM · Epic, Cloud-Services
chasemp closed T168580: Neutron implementation of routing_source_ip definition as Resolved.

Seems to be working

Fri, Apr 27, 6:30 PM · Patch-For-Review, Epic, Cloud-Services
chasemp closed T168580: Neutron implementation of routing_source_ip definition, a subtask of T167293: Nova-network to Neutron migration, as Resolved.
Fri, Apr 27, 6:30 PM · Epic, Cloud-Services
chasemp assigned T188392: package prometheus-rabbitmq-exporter for Debian jessie to aborrero.
Fri, Apr 27, 6:30 PM · Patch-For-Review, Cloud-VPS, Operations
chasemp closed T187954: Cloud VPS upgrade to Openstack Mitaka release as Resolved.
Fri, Apr 27, 6:26 PM · cloud-services-team (FY2017-18), Patch-For-Review, Cloud-VPS
chasemp closed T187954: Cloud VPS upgrade to Openstack Mitaka release, a subtask of T167293: Nova-network to Neutron migration, as Resolved.
Fri, Apr 27, 6:25 PM · Epic, Cloud-Services
chasemp closed T188266: labtestn to Mitaka on Jessie as Resolved.
labtestcontrol2003.wikimedia.org
Debian GNU/Linux 8 \n \l
Fri, Apr 27, 6:25 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS
chasemp closed T188266: labtestn to Mitaka on Jessie, a subtask of T187954: Cloud VPS upgrade to Openstack Mitaka release, as Resolved.
Fri, Apr 27, 6:25 PM · cloud-services-team (FY2017-18), Patch-For-Review, Cloud-VPS
chasemp closed Restricted Task, a subtask of T132324: Tracking and Reducing cron-spam to root@ , as Resolved.
Fri, Apr 27, 1:51 PM · Patch-For-Review, Operations
chasemp closed T165779: rack/setup/install labnet100[34] as Resolved.

Note T193196 is related for next phases here but this is racked/stack/imaged

Fri, Apr 27, 1:50 PM · cloud-services-team (Kanban), Patch-For-Review, Cloud-Services, Operations

Thu, Apr 26

chasemp placed T158883: Issues in enabling NFS for new projects (was 'adding project to nfs-mounts.yaml does not create directories') up for grabs.
Thu, Apr 26, 7:46 PM · Data-Services, cloud-services-team (Kanban), Patch-For-Review
chasemp placed T164123: tools-k8s-master-01 has two floating IPs up for grabs.
Thu, Apr 26, 7:46 PM · cloud-services-team (Kanban), Operations, Cloud-Services
chasemp closed T165781: rack/setup/install labcontrol100[34] as Resolved.

These are now debian jessie shoutout to @RobH for helping me work through some install issues :)

Thu, Apr 26, 7:45 PM · cloud-services-team (Kanban), Cloud-Services, Operations
chasemp closed T178405: create a wmcs alerting group in icinga and review alerting as Resolved.
Thu, Apr 26, 7:35 PM · cloud-services-team (Kanban), Patch-For-Review
chasemp removed a parent task for T177850: Page if the grid engine master is unreachable: T178405: create a wmcs alerting group in icinga and review alerting.
Thu, Apr 26, 7:35 PM · Patch-For-Review, monitoring, Toolforge, cloud-services-team (Kanban)
chasemp removed a subtask for T178405: create a wmcs alerting group in icinga and review alerting: T177850: Page if the grid engine master is unreachable.
Thu, Apr 26, 7:35 PM · cloud-services-team (Kanban), Patch-For-Review
chasemp renamed T193196: labnet1003 and labnet1004 moving and enabling 10G NICs from labnet1003 and labnet1004 extra NIC connections to labnet1003 and labnet1004 moving and enabling 10G NICs.
Thu, Apr 26, 7:07 PM · Cloud-VPS, Operations, ops-eqiad
chasemp updated the task description for T193196: labnet1003 and labnet1004 moving and enabling 10G NICs.
Thu, Apr 26, 6:58 PM · Cloud-VPS, Operations, ops-eqiad
chasemp triaged T193196: labnet1003 and labnet1004 moving and enabling 10G NICs as Normal priority.
Thu, Apr 26, 6:56 PM · Cloud-VPS, Operations, ops-eqiad
chasemp created T193196: labnet1003 and labnet1004 moving and enabling 10G NICs.
Thu, Apr 26, 6:56 PM · Cloud-VPS, Operations, ops-eqiad

Tue, Apr 24

chasemp triaged T192892: Request creation of wikibase-registry VPS project as Normal priority.
Tue, Apr 24, 4:04 PM · wikibase-registry, Cloud-VPS (Project-requests)
chasemp added a comment to T192713: labstore1003 load spikes.

I jumped in on /usr/local/sbin/tc-setup and hot patched it to:

Tue, Apr 24, 3:26 PM · Cloud-VPS

Sat, Apr 21

chasemp triaged T192713: labstore1003 load spikes as Normal priority.
Sat, Apr 21, 12:48 PM · Cloud-VPS
chasemp created T192713: labstore1003 load spikes.
Sat, Apr 21, 12:48 PM · Cloud-VPS

Apr 20 2018

chasemp added a comment to T192162: Set up proper apt pinning for Mitaka on Jessie, upgrade Jessie hosts to Mitaka.

Now I see that we have possibly-conflicting solutions for this. Most Jessie hosts aren't getting Arturo's new pinning rules, rather they are getting Chase's openstack::backports class.

openstack::backports only mentions three files, so I'm inclined to discard it in favor of Arturo's openstack::jessie_mitaka_pinning.  Would like input from @chasemp about this first in case somehow those three files are all we need.
Apr 20 2018, 11:14 PM · Patch-For-Review, Cloud-Services

Apr 12 2018

chasemp created P6987 (An Untitled Masterwork).
Apr 12 2018, 6:43 PM

Apr 10 2018

chasemp added a comment to T190895: Request increased quota for toolsbeta.

+1

Apr 10 2018, 3:35 PM · Cloud-VPS (Quota-requests)

Apr 9 2018

chasemp triaged T191791: Make same glance images available to multiple regions as Normal priority.
Apr 9 2018, 1:06 PM · cloud-services-team (Kanban), Cloud-Services
chasemp triaged T191790: Figure out how to migrate an instance from a nova-network region to a neutron region as Normal priority.
Apr 9 2018, 1:04 PM · cloud-services-team (Kanban), Cloud-Services
chasemp assigned T181523: labtest puppetmaster is not working for clients to aborrero.
Apr 9 2018, 12:56 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS, Epic
chasemp reassigned T188266: labtestn to Mitaka on Jessie from chasemp to aborrero.

@aborrero can you reimage labtestservices2002.wikimedia.org as jessie as part of this? There is a lot of work to be done but I think post that this can be closed.

Apr 9 2018, 12:54 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS

Apr 5 2018

chasemp added a comment to T187373: Rebuild raids on labvirt1019 and 1020.

I think we probably need a new controller.

Apr 5 2018, 7:34 PM · cloud-services-team (Kanban), Operations, Cloud-Services

Apr 2 2018

chasemp changed the status of T191155: Request creation of wikimisc VPS project from Open to Stalled.

@KATMAKROFAN: Where is the WikiMiscellanea project *creation proposal*? See my question in T191155#4095794. I do not see why to already host content if there has been no decision about the future of such a project?

Apr 2 2018, 1:26 PM · Cloud-VPS (Project-requests)
chasemp added a comment to T191149: labsdb1009 crashed.

Sincerest thanks to you all :)

Apr 2 2018, 12:37 PM · Patch-For-Review, Data-Services, DBA
chasemp added a comment to T183937: rack/setup/install labvirt102[12].

@RobH figured out what he believed is the eth0 issue described, unless a screenshot was captured I don't think there are logs but the message he pasted in irc from console was something very literal like "failed as eth0 is not connected". It was my understanding that this has been seen in the past on Trusty and was a somewhat forgotten but known old issue. Trying to image with Trusty reproduces. I thought @RobH was going to try to circle back on this and see if the issue can be overcome easily but I was away on vacation and then we have both been busy, I figured T187373 was the priority of the two pending cloud hardware isues from a dcops perspective so I wasn't too worried.

Apr 2 2018, 12:35 PM · cloud-services-team (Kanban), Operations

Mar 20 2018

chasemp reassigned T190182: implement renewed *.tools.wmflabs.org cert/key pair from chasemp to aborrero.

@madhuvishy graciously said she would walk @aborrero through the process from last year. Not sure if documented anywhere?

Mar 20 2018, 5:26 PM · cloud-services-team (Kanban), Patch-For-Review, Toolforge, Operations
chasemp added a comment to T183937: rack/setup/install labvirt102[12].

@RobH just a ping that we are talking about this in our weekly, are you going to have time to check into where to go from here? easy money says maybe we should just connect up the 10G interfaces to image these for a short term thing.

Mar 20 2018, 3:28 PM · cloud-services-team (Kanban), Operations

Mar 16 2018

chasemp created P6855 (An Untitled Masterwork).
Mar 16 2018, 7:17 PM
chasemp added a comment to T187954: Cloud VPS upgrade to Openstack Mitaka release.
2018-03-16 15:10:54.264 72304 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 461, in _send
2018-03-16 15:10:54.264 72304 ERROR oslo_service.service     raise result
2018-03-16 15:10:54.264 72304 ERROR oslo_service.service RemoteError: Remote error: IncompatibleObjectVersion Version 1.3 of MigrationList is not supported
2018-03-16 15:10:54.264 72304 ERROR oslo_service.service [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch\n    result = func(ctxt, **new_args)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 937, in object_class_action_versions\n    context, objname, objmethod, object_versions, args, kwargs)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 468, in object_class_action_versions\n    objname, object_versions[objname])\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 355, in obj_class_from_name\n    supported=latest_ver)\n', u'IncompatibleObjectVersion: Version 1.3 of MigrationList is not supported\n'].
Mar 16 2018, 3:41 PM · cloud-services-team (FY2017-18), Patch-For-Review, Cloud-VPS
chasemp added a comment to T187954: Cloud VPS upgrade to Openstack Mitaka release.

Change 419737 merged by Rush:
[operations/puppet@production] openstack: trial of mixed mitaka/liberty nova compute

https://gerrit.wikimedia.org/r/419737

Mar 16 2018, 3:40 PM · cloud-services-team (FY2017-18), Patch-For-Review, Cloud-VPS
chasemp reopened T188589: db1009 overloaded by idle connections to the nova database as "Open".

@Andrew tried to merge the change to allow nova to be more gracious and it didn't work out.

Mar 16 2018, 2:25 PM · Patch-For-Review, Operations, Cloud-Services, DBA
chasemp created P6853 (An Untitled Masterwork).
Mar 16 2018, 2:09 PM
chasemp triaged T189871: labmon1002 as cold standby for labmon1001 as Normal priority.
Mar 16 2018, 1:01 PM · cloud-services-team (Kanban), Patch-For-Review, Cloud-VPS
chasemp closed T165784: rack/setup/install labmon1002 as Resolved.

closed in favor of T189871

Mar 16 2018, 1:01 PM · cloud-services-team (Kanban), Cloud-Services, Operations
chasemp created T189871: labmon1002 as cold standby for labmon1001.
Mar 16 2018, 1:01 PM · cloud-services-team (Kanban), Patch-For-Review, Cloud-VPS
chasemp added a comment to T183585: Rack/cable/configure asw2-b-eqiad switch stack.

https://phabricator.wikimedia.org/T183937#4056221

Mar 16 2018, 12:48 PM · cloud-services-team, Cloud-VPS, ops-eqiad, Operations
chasemp updated subscribers of T183937: rack/setup/install labvirt102[12].

Ok, escalating this to @chasemp for completion. The systems are installed and calling into puppet. Their 1G ports are showing as eth2/3, with eth2 being the primary used interface. eth0/1 are the 10G ports. Irc discussion noted that will cause some refactoring of some puppet code to accommodate the change.

I resolved https://phabricator.wikimedia.org/T188297#4053178 (hope that's cool)

I don't believe eth3 is connected for labvirt1021:

lldpcli show neighbors only shows eth2 connected and icinga is reporting eth3 reporting no carrier. FWIW eth3 here should be connected and configured as a trunk switch (match eth1 for existing labvirts).

Mar 16 2018, 12:48 PM · cloud-services-team (Kanban), Operations

Mar 15 2018

chasemp added a comment to T189165: Request creation of WCDO VPS project.

+1'ing

Mar 15 2018, 3:31 PM · Cloud-VPS (Project-requests)
chasemp reassigned T183937: rack/setup/install labvirt102[12] from chasemp to RobH.

Ok, escalating this to @chasemp for completion. The systems are installed and calling into puppet. Their 1G ports are showing as eth2/3, with eth2 being the primary used interface. eth0/1 are the 10G ports. Irc discussion noted that will cause some refactoring of some puppet code to accommodate the change.

Mar 15 2018, 2:21 PM · cloud-services-team (Kanban), Operations