chasemp (Chase)Administrator
Lead Operations Engineer (Wikimedia Cloud Services)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Monday

  • Clear sailing ahead.

User Details

User Since
Sep 16 2014, 11:39 AM (144 w, 4 d)
Roles
Administrator
Availability
Available
IRC Nick
chasemp
LDAP User
Rush
MediaWiki User
CPettet (WMF)

Recent Activity

Wed, Jun 21

chasemp created T168580: Neutron implementation of routing_source_ip definition.
Wed, Jun 21, 9:31 PM · Labs-Infrastructure, Epic, Labs

Tue, Jun 20

chasemp added a comment to T168344: labmon1001 disk filling up.

I would think a year is a first good step.

Tue, Jun 20, 7:45 PM · Patch-For-Review, cloud-services-team (Kanban), Labs
chasemp added a comment to T165531: rack/setup/install labvirt101[5-8].

@Cmjohnson @RobH thanks guys, post install assign to me and I'll take care of it.

Tue, Jun 20, 5:04 PM · cloud-services-team (Kanban), Patch-For-Review, ops-eqiad, Labs-Infrastructure, Labs, Operations

Thu, Jun 15

chasemp updated subscribers of T167985: Horizon bug: hidden web proxy after deleting instance.

I think possibly @Andrew just fixed this but maybe it wouldn't have effected historical artifacts.

Thu, Jun 15, 6:00 PM · Horizon, cloud-services-team (Kanban), User-bd808, Labs
chasemp reassigned T167984: rack/setup/install labstore100[67].wikimedia.org from chasemp to RobH.

Note from irc: these are closer in function to the old dataset boxes rather than existing labstores. They need to go in a public VLAN being externally accessible. :)

Thu, Jun 15, 5:59 PM · Labs-Infrastructure, ops-eqiad, Labs, Operations

Wed, Jun 14

chasemp added a project to T167920: Impending load test: Traffic.
Wed, Jun 14, 8:30 PM · Traffic, Wikimedia-General-or-Unknown, Operations
chasemp updated the task description for T154664: codfw: (2) hardware access request for labtest.
Wed, Jun 14, 8:22 PM · hardware-requests, Operations
chasemp edited P5581 (An Untitled Masterwork).
Wed, Jun 14, 7:13 PM
chasemp created P5581 (An Untitled Masterwork).
Wed, Jun 14, 7:13 PM

Tue, Jun 13

chasemp added a comment to T83732: virbr0 interface present in some virt hosts.

Post hoc note. I noticed that /etc/libvirt/qemu/networks/autostart/default.xml is ensured absent in our nova compute role. This is a file that libvirt seems to generate and the contents of it stock are:

Tue, Jun 13, 10:20 PM · Operations, Labs
chasemp added a comment to T167086: Consider moving PAWS to its own k8s cluster, rather than using Tools' k8s cluster.

I was thinking a separate nginx ingress in total, and ignoring tools-proxy-xx here for sure. This is a consideration of Tools the environment being bigger than one implementation inside of it, etc. If you don't want to go with Puppet then it's a non-issue. We aren't ready to be that cool atm. I was thinking more detached than it came across, but if you go far enough adrift what's the point of course :) It's your call. Maybe at some point it will be a PITA either way chosen and changing it up will make sense. I don't feel too strongly about it as it's outlined.

Tue, Jun 13, 8:05 PM · Patch-For-Review, Tool-Labs, Tools-Kubernetes, PAWS, Labs
chasemp added a comment to T167086: Consider moving PAWS to its own k8s cluster, rather than using Tools' k8s cluster.

Small thought to which I'm not terribly attached but it is worth mentioning:

Tue, Jun 13, 1:39 PM · Patch-For-Review, Tool-Labs, Tools-Kubernetes, PAWS, Labs
chasemp added a comment to T155041: Prepare and check storage layer for wikimania2018wiki.

DBA gents please let us know when it's ready for views and meta_p fixup and we'll get on it! Thanks

Tue, Jun 13, 1:36 PM · Labs, DBA

Fri, Jun 9

chasemp updated the task description for T167518: update queue_server.pp for user/pass setup or one-time setup instructions.
Fri, Jun 9, 3:56 PM · Labs
chasemp created T167518: update queue_server.pp for user/pass setup or one-time setup instructions.
Fri, Jun 9, 3:51 PM · Labs

Thu, Jun 8

chasemp created T167467: Create nova service account for openstack.
Thu, Jun 8, 8:22 PM · Labs
chasemp added a watcher for cloud-services-team: chasemp.
Thu, Jun 8, 4:06 PM
chasemp lowered the priority of T167114: Open view for term_full_entity_id in wb_terms table in labs from High to Normal.
Thu, Jun 8, 2:25 PM · User-Ladsgroup, Patch-For-Review, Labs, Wikidata-Sprint, Wikidata
chasemp changed the status of T167114: Open view for term_full_entity_id in wb_terms table in labs from Open to Stalled.

Change 357369 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[operations/software/labsdb-auditor@master] Whitelist term_full_entity_id in wb_terms table

https://gerrit.wikimedia.org/r/357369

Thu, Jun 8, 2:24 PM · User-Ladsgroup, Patch-For-Review, Labs, Wikidata-Sprint, Wikidata
chasemp changed the status of T167114: Open view for term_full_entity_id in wb_terms table in labs, a subtask of T159851: Add a column for full entity ID to wb_terms table, from Open to Stalled.
Thu, Jun 8, 2:24 PM · Wikidata-Sprint, Wikidata
chasemp added a parent task for T167412: host-vmem.erb is doing operations that make no sense: T162955: rebuild tools-grid-master as a large instance.
Thu, Jun 8, 2:14 PM · Patch-For-Review, Operations, Labs
chasemp added a subtask for T162955: rebuild tools-grid-master as a large instance: T167412: host-vmem.erb is doing operations that make no sense.
Thu, Jun 8, 2:14 PM · cloud-services-team (Kanban), Patch-For-Review, Operations, Labs
chasemp renamed T167412: host-vmem.erb is doing operations that make no sense from Tools puppet failing: Detail: undefined method `>>' for "24443.99":String to host-vmem.erb is doing operations that make no sense.
Thu, Jun 8, 2:14 PM · Patch-For-Review, Operations, Labs
chasemp updated subscribers of T167412: host-vmem.erb is doing operations that make no sense.

Quoting @faidon from irc:

Thu, Jun 8, 2:14 PM · Patch-For-Review, Operations, Labs
chasemp added a comment to T167412: host-vmem.erb is doing operations that make no sense.

turns out 37b83e8b2c04a58f555ee5627a415561ab792d26 unintentionally resulted in this

Thu, Jun 8, 2:13 PM · Patch-For-Review, Operations, Labs
chasemp triaged T167412: host-vmem.erb is doing operations that make no sense as Normal priority.
Thu, Jun 8, 2:04 PM · Patch-For-Review, Operations, Labs
chasemp added a comment to T167412: host-vmem.erb is doing operations that make no sense.

This is probably from an operation against this fact:

Thu, Jun 8, 2:02 PM · Patch-For-Review, Operations, Labs
chasemp added a comment to T167412: host-vmem.erb is doing operations that make no sense.
Commit:  d3dc61097073773b308f2cc1bb9352c4aea61be8
Author:  Alexandros Kosiaris <akosiaris@wikimedia.org>
Date:    (5 hours ago) 2017-06-08 12:12:13 +0300
Subject: puppetmaster: Set stringify_facts = false
Thu, Jun 8, 2:01 PM · Patch-For-Review, Operations, Labs
chasemp created T167412: host-vmem.erb is doing operations that make no sense.
Thu, Jun 8, 1:57 PM · Patch-For-Review, Operations, Labs
chasemp updated subscribers of T115194: Some labs instances IP have multiple PTR entries in DNS.
Thu, Jun 8, 1:55 PM · Patch-For-Review, Wikimedia-Incident, Labs-Infrastructure, Operations, Labs
chasemp reopened T115194: Some labs instances IP have multiple PTR entries in DNS as "Open".
elukey@deployment-aqs03:~$ dig -x 10.68.17.125 +short
elukey
ci-jessie-wikimedia-505374.contintcloud.eqiad.wmflabs.
elukey
deployment-aqs03.deployment-prep.eqiad.wmflabs.
elukey
is it normal? :D
Thu, Jun 8, 1:55 PM · Patch-For-Review, Wikimedia-Incident, Labs-Infrastructure, Operations, Labs
chasemp closed T83732: virbr0 interface present in some virt hosts as Resolved.

It seems not, I'm going to close this but anyone who knows differently please reopen

Thu, Jun 8, 1:47 PM · Operations, Labs

Wed, Jun 7

chasemp added a parent task for T122406: Consider renumbering Labs to separate address spaces: T167293: Nova-network to Neutron migration.
Wed, Jun 7, 9:35 PM · Labs, netops, Operations
chasemp added a subtask for T167293: Nova-network to Neutron migration: T122406: Consider renumbering Labs to separate address spaces.
Wed, Jun 7, 9:35 PM · Labs-Infrastructure, Epic, Labs
chasemp created T167357: Determine need and replacement for dmz_cidr configuration in nova-network.
Wed, Jun 7, 9:32 PM · Labs
chasemp reopened T167357: Determine need and replacement for dmz_cidr configuration in nova-network, a subtask of T167293: Nova-network to Neutron migration, as Open.
Wed, Jun 7, 9:32 PM · Labs-Infrastructure, Epic, Labs
chasemp created T167356: Undo trunking to virts for a sane flat networking model.
Wed, Jun 7, 9:22 PM · Labs
chasemp reopened T167356: Undo trunking to virts for a sane flat networking model, a subtask of T167293: Nova-network to Neutron migration, as Open.
Wed, Jun 7, 9:22 PM · Labs-Infrastructure, Epic, Labs
chasemp renamed T166310: Grant root access for Bryan Davis on labstore* and admin for maintain scripts for labsdb* from Grant sudo access for Bryan Davis for labstore* and labsdb* to Grant root access for Bryan Davis on labstore* and admin for maintain scripts for labsdb*.
Wed, Jun 7, 4:58 PM · Patch-For-Review, Operations, Ops-Access-Requests
chasemp added a comment to T166310: Grant root access for Bryan Davis on labstore* and admin for maintain scripts for labsdb*.

If we want to holdoff on the labsdb root inclusion I am going to propose in the opsen meeting this task become:

  • root on labstore* things
  • wmcs-admin group creation with ability to run only maintain-views and maintain-meta_p on labsdb* hosts
Wed, Jun 7, 4:58 PM · Patch-For-Review, Operations, Ops-Access-Requests
chasemp awarded Blog Post: Improving time-to-logo performance with preload links a Like token.
Wed, Jun 7, 3:05 PM · Performance-Team
chasemp added a comment to T166310: Grant root access for Bryan Davis on labstore* and admin for maintain scripts for labsdb*.

If we want to holdoff on the labsdb root inclusion I am going to propose in the opsen meeting this task become:

Wed, Jun 7, 3:01 PM · Patch-For-Review, Operations, Ops-Access-Requests
chasemp closed T41787: Add more network nodes as Resolved.

We did add a second network node to our single openstack deployment with a custom failover procedure, but more verbose versions of this will be tracked elsewhere. Failover is currently documented at https://wikitech.wikimedia.org/wiki/Portal:Wikimedia_VPS/Admin/Troubleshooting#Fail-over

Wed, Jun 7, 2:55 PM · Labs, Labs-Infrastructure
chasemp created T167295: Discontinue use of admin_token for keystone.
Wed, Jun 7, 1:47 PM · Labs
chasemp reopened T167295: Discontinue use of admin_token for keystone, a subtask of T167293: Nova-network to Neutron migration, as Open.
Wed, Jun 7, 1:47 PM · Labs-Infrastructure, Epic, Labs
chasemp added a parent task for T167294: use a service role project for openstack components: T167293: Nova-network to Neutron migration.
Wed, Jun 7, 1:46 PM · Labs-Infrastructure, Labs
chasemp added a subtask for T167293: Nova-network to Neutron migration: T167294: use a service role project for openstack components.
Wed, Jun 7, 1:46 PM · Labs-Infrastructure, Epic, Labs
chasemp created T167294: use a service role project for openstack components.
Wed, Jun 7, 1:46 PM · Labs-Infrastructure, Labs
chasemp added a subtask for T167293: Nova-network to Neutron migration: Unknown Object (Task).
Wed, Jun 7, 1:44 PM · Labs-Infrastructure, Epic, Labs
chasemp added a parent task for T153099: Initial OpenStack Neutron PoC deployment in Labtest: T167293: Nova-network to Neutron migration.
Wed, Jun 7, 1:44 PM · cloud-services-team (Kanban), Labs, Operations
chasemp added a subtask for T167293: Nova-network to Neutron migration: T153099: Initial OpenStack Neutron PoC deployment in Labtest.
Wed, Jun 7, 1:44 PM · Labs-Infrastructure, Epic, Labs
chasemp created T167293: Nova-network to Neutron migration.
Wed, Jun 7, 1:43 PM · Labs-Infrastructure, Epic, Labs

Tue, Jun 6

chasemp added a comment to T167160: rack/setup/install labtestneutron2002.

@Papaul yes, thank you

Tue, Jun 6, 8:16 PM · Labs, Labs-Infrastructure, Operations
chasemp added a comment to T116607: How to handle mgmt lan for labs bare metal?.

This task was to make a plan for user mgmt access to bare metal as a service @Dzahn to help clarify, which we have no plans to do.

Tue, Jun 6, 12:30 AM · labs-sprint-118, Labs, Labs-Infrastructure, labs-sprint-117, Operations

Mon, Jun 5

chasemp moved T166985: Install DjVuLibre and XPDF packages for Kubernetes containers on Tool Labs from Inbox to To-Do on the cloud-services-team (Kanban) board.
Mon, Jun 5, 8:17 PM · cloud-services-team (Kanban), Tools-Kubernetes, Tool-Labs, Labs
chasemp added a project to T166985: Install DjVuLibre and XPDF packages for Kubernetes containers on Tool Labs: cloud-services-team (Kanban).
Mon, Jun 5, 8:17 PM · cloud-services-team (Kanban), Tools-Kubernetes, Tool-Labs, Labs
chasemp added a hashtag to cloud-services-team (Kanban): #cskanban.
Mon, Jun 5, 8:16 PM
chasemp added a comment to T166985: Install DjVuLibre and XPDF packages for Kubernetes containers on Tool Labs.

@Dzahn I believe this user is in the k8s environment which is handled separately from SGE

Mon, Jun 5, 8:15 PM · cloud-services-team (Kanban), Tools-Kubernetes, Tool-Labs, Labs
chasemp triaged T166985: Install DjVuLibre and XPDF packages for Kubernetes containers on Tool Labs as Normal priority.
Mon, Jun 5, 8:10 PM · cloud-services-team (Kanban), Tools-Kubernetes, Tool-Labs, Labs
chasemp closed T116607: How to handle mgmt lan for labs bare metal? as Declined.

For now this is totally off the books

Mon, Jun 5, 7:31 PM · labs-sprint-118, Labs, Labs-Infrastructure, labs-sprint-117, Operations
chasemp closed T116607: How to handle mgmt lan for labs bare metal?, a subtask of T114435: Labs test cluster in codfw, as Declined.
Mon, Jun 5, 7:31 PM · labs-sprint-118, Operations, labs-sprint-117, hardware-requests, Labs-Infrastructure, Labs
chasemp closed T116607: How to handle mgmt lan for labs bare metal?, a subtask of T117095: eqiad: 1 hardware access request for labs on real hardware (mwoffliner), as Declined.
Mon, Jun 5, 7:31 PM · Operations
chasemp closed T85609: Labs available in the new data centre (with Neutron/IPv6) as Declined.

With T85610 also being declined I'm going to say any work towards this end is a ways off and will be tracked in other tasks

Mon, Jun 5, 7:30 PM · Labs
chasemp closed T85609: Labs available in the new data centre (with Neutron/IPv6), a subtask of T85610: Distributing tools, deployment-prep to both data centers (availability/redundancy), as Declined.
Mon, Jun 5, 7:30 PM · Tool-Labs, Labs
chasemp closed T85609: Labs available in the new data centre (with Neutron/IPv6), a subtask of T85611: Neutron networking, with IPv6 at eqiad, as Declined.
Mon, Jun 5, 7:30 PM · Labs
chasemp awarded Blog Post: Status update (June 3rd, 2017) a Love token.
Mon, Jun 5, 1:22 PM · Documentation, periodic-update

Fri, Jun 2

chasemp updated subscribers of T166561: Rollout prometheus-node-exporter 0.14 in labs.

thanks! I think the version in aptly has been removed so we should be set for tools too, what's the best way I can run a command on all labs + tools ?

Fri, Jun 2, 1:47 PM · User-fgiunchedi, Tool-Labs, Labs, Labs-Infrastructure

Thu, Jun 1

chasemp added a comment to T166237: rack/setup/install labtestvirt2003.

@chasemp what partman recipe do you want to use for the server? We have :

Thu, Jun 1, 3:18 PM · Patch-For-Review, cloud-services-team, ops-codfw, Operations
chasemp added a comment to T165555: nova-fullstack is losing instances on creation.

Still no failures.

Thu, Jun 1, 12:45 PM · cloud-services-team (Kanban), Labs
chasemp added a comment to T166561: Rollout prometheus-node-exporter 0.14 in labs.

sounds good to me, let us know when you hit the Tools road block with aptly and one of us can untangle (i.e. this)

Thu, Jun 1, 12:43 PM · User-fgiunchedi, Tool-Labs, Labs, Labs-Infrastructure
chasemp triaged T166561: Rollout prometheus-node-exporter 0.14 in labs as Normal priority.
Thu, Jun 1, 12:43 PM · User-fgiunchedi, Tool-Labs, Labs, Labs-Infrastructure
chasemp updated subscribers of T166237: rack/setup/install labtestvirt2003.

It seems that there is confusion, due to the fact that racktables shows two labtestvirt2001 systems.

I went ahead and connected to the mgmt dns for the existing labtestvirt2002, and it points to this system: https://racktables.wikimedia.org/index.php?page=object&object_id=1317

WMF3810 is labtestvirt2002. (There will be a new task to correct that one shortly.)

The system being setup on this task T166237, should be renamed to labtestvirt2003, since the other system already exists, and is online as labtestvirt2002.

Thu, Jun 1, 12:40 PM · Patch-For-Review, cloud-services-team, ops-codfw, Operations
chasemp added a comment to T166237: rack/setup/install labtestvirt2003.

@chasemp the others lab servers in the DHCP file are pointing to the Trusty install do you want to install Trusty on labtestvirt2003 or Jessie ?

Thu, Jun 1, 12:30 PM · Patch-For-Review, cloud-services-team, ops-codfw, Operations
chasemp added a comment to T165531: rack/setup/install labvirt101[5-8].

@Cmjohnson @RobH the row b requirement for labvirts and labnets is unfortunately still real as of now. We are working on it though.

Thu, Jun 1, 12:23 PM · cloud-services-team (Kanban), Patch-For-Review, ops-eqiad, Labs-Infrastructure, Labs, Operations

Thu, May 25

chasemp triaged T166295: tools.wmflabs.org needs a documentation update as Normal priority.
Thu, May 25, 2:13 PM · Patch-For-Review, User-bd808, Documentation, Labs, Tool-Labs
chasemp added a comment to T166295: tools.wmflabs.org needs a documentation update.

Thanks for logging this task! We have had a bunch of different scattered instructions, where (what page(s)) were you working from?

Thu, May 25, 2:13 PM · Patch-For-Review, User-bd808, Documentation, Labs, Tool-Labs
chasemp changed the "Can Create Blogs" policy for application Phame from "All Users" to "Members of Project: acl*phabricator".
Thu, May 25, 2:09 PM
chasemp changed the "Can Create Blogs" policy for application Phame from "Members of Project: acl*phabricator" to "All Users".
Thu, May 25, 2:09 PM
chasemp changed the edit policy for Clouds & Unicorns.
Thu, May 25, 2:08 PM · cloud-services-team
chasemp changed the edit policy for Clouds & Unicorns.
Thu, May 25, 2:05 PM · cloud-services-team

May 24 2017

chasemp added a comment to T165875: Update maintain-kubeusers to allow tool's to write to $HOME/.kube.

Change 354839 merged by Rush:
[operations/puppet@production] tools: have maintain-kubeusers chown $HOME/.kube

https://gerrit.wikimedia.org/r/354839

May 24 2017, 2:31 PM · cloud-services-team (Kanban), User-bd808, Patch-For-Review, Tool-Labs, Labs
chasemp renamed Clouds & Unicorns blog from Labs and Tools News to Wikimedia Cloud Services Related News.
May 24 2017, 2:17 PM · cloud-services-team

May 21 2017

chasemp triaged T165939: Wikitech rights for Quiddity as Normal priority.
May 21 2017, 8:33 AM · User-bd808, cloud-services-team, wikitech.wikimedia.org, Labs
chasemp awarded T165939: Wikitech rights for Quiddity a Like token.
May 21 2017, 8:33 AM · User-bd808, cloud-services-team, wikitech.wikimedia.org, Labs

May 17 2017

chasemp added a comment to T165555: nova-fullstack is losing instances on creation.

Small note just for posterity as I think there is no relation (per volans):

May 17 2017, 8:03 AM · cloud-services-team (Kanban), Labs
chasemp assigned T165555: nova-fullstack is losing instances on creation to Andrew.

I am suspecting this is some odd DNS issue that happens in a race (that's mostly been your speciality :D) I'm tossing your way to grab attention but will look into this too. I'll try to sync up later today.

May 17 2017, 7:52 AM · cloud-services-team (Kanban), Labs
chasemp added a comment to T165555: nova-fullstack is losing instances on creation.

I am dropping 2 of the 4 holdovers to give us some headroom:

May 17 2017, 7:50 AM · cloud-services-team (Kanban), Labs
chasemp updated subscribers of T165555: nova-fullstack is losing instances on creation.
May 17 2017, 7:49 AM · cloud-services-team (Kanban), Labs
chasemp triaged T165555: nova-fullstack is losing instances on creation as High priority.
May 17 2017, 7:48 AM · cloud-services-team (Kanban), Labs
chasemp created T165555: nova-fullstack is losing instances on creation.
May 17 2017, 7:48 AM · cloud-services-team (Kanban), Labs
chasemp updated the task description for T161753: eqiad: (1) hardware access request for labnodepool1002.
May 17 2017, 7:37 AM · hardware-requests, Labs, Operations

May 13 2017

chasemp claimed T153099: Initial OpenStack Neutron PoC deployment in Labtest.
May 13 2017, 2:37 AM · cloud-services-team (Kanban), Labs, Operations
chasemp created T165211: Disable keystone admin_token usage.
May 13 2017, 2:37 AM · Patch-For-Review, Labs, Operations

May 9 2017

chasemp renamed T164515: codfw: (1) labtest puppetmaster from codfw: (1) labs puppetmaster to codfw: (1) labtest puppetmaster.
May 9 2017, 8:19 PM · hardware-requests, Operations

May 8 2017

chasemp added a comment to T164675: labservices1002 slow puppet runs and IO issues.

I'm not sure what the story is other than during debugging it disappeared for now. Pattern as I understand it is: puppet slow seemingly from degraded IO sunday, failed away from the host as primary and rebooted, symptoms still persisted, I did the above poking looking for some IO issue indicator and went back to demonstrate and the issue would no longer reproduce.

May 8 2017, 5:48 PM · Labs, Labs-Infrastructure
chasemp triaged T164675: labservices1002 slow puppet runs and IO issues as Normal priority.
May 8 2017, 4:30 PM · Labs, Labs-Infrastructure
chasemp changed the status of T164675: labservices1002 slow puppet runs and IO issues from Open to Stalled.

Well. Now the issue has gone dormant or fsck addressed it.

May 8 2017, 4:29 PM · Labs, Labs-Infrastructure
chasemp added a comment to T164675: labservices1002 slow puppet runs and IO issues.

I updated /etc/default/rcS:

May 8 2017, 4:09 PM · Labs, Labs-Infrastructure
chasemp added a comment to T164675: labservices1002 slow puppet runs and IO issues.

I tried dropping into LifeCycle controller to run diagnostics via F-10

May 8 2017, 3:15 PM · Labs, Labs-Infrastructure
chasemp renamed T164675: labservices1002 slow puppet runs and IO issues from labservices1002 slow puppet runs to labservices1002 slow puppet runs and IO issues.
May 8 2017, 3:08 PM · Labs, Labs-Infrastructure