chasemp (Chase)Administrator
Lead Operations Engineer (Wikimedia Cloud Services)

Projects (34)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Sep 16 2014, 11:39 AM (204 w, 23 h)
Roles
Administrator
Availability
Available
IRC Nick
chasemp
LDAP User
Rush
MediaWiki User
CPettet (WMF) [ Global Accounts ]

Recent Activity

Yesterday

chasemp closed T201925: Server error when viewing web proxies as Resolved.

seems related to T201924: Can't create new proxies so I'll close but reopen for the cloud folks if really not resolved still

Tue, Aug 14, 3:15 PM · Cloud-Services
chasemp added a comment to T194150: Decide future of *.security-tools vms in labs.

novaproxy domains that should go along with these

Tue, Aug 14, 1:02 PM · Cloud-VPS, Security-Team

Mon, Aug 13

chasemp triaged T201861: Puppetize drupal7 on a Cloud host as Normal priority.
Mon, Aug 13, 6:55 PM · Security-Team
chasemp triaged T201860: deploy drupal as a GRC CMS for risk and compliance management as Normal priority.
Mon, Aug 13, 6:53 PM · Operations, Security-Team
chasemp added a comment to T194150: Decide future of *.security-tools vms in labs.
| f4d46a5c-f8cd-405d-9cbf-82710a9564e2 | two-factor | SHUTOFF | public=10.68.21.34                  |
| a82089e3-b0c7-4f01-a2d3-4d15dc113ade | jobs       | SHUTOFF | public=10.68.17.248                 |
| 49fb74ed-734a-4332-9fb3-6d2c4903e98e | openvas    | SHUTOFF | public=10.68.16.85                  |
| b0859108-799d-4dbc-a4cc-d2179686bcf4 | keystone2  | SHUTOFF | public=10.68.21.241                 |
| fbed895f-ccfe-4403-ba13-5a12183daab9 | xsstest    | SHUTOFF | public=10.68.17.178, 208.80.155.231 |
| 1f5637e7-ecc8-482c-8b3b-6ded44878c06 | torproxy   | SHUTOFF | public=10.68.18.15                  |
Mon, Aug 13, 5:51 PM · Cloud-VPS, Security-Team
chasemp claimed T194150: Decide future of *.security-tools vms in labs.
Mon, Aug 13, 5:50 PM · Cloud-VPS, Security-Team
chasemp added a comment to T194150: Decide future of *.security-tools vms in labs.

Since no one seems to know what most of these are for I removed one in https://phabricator.wikimedia.org/T194150#4499280 and am shutting down the others to flush anyone who cares out of the woodwork :)

Mon, Aug 13, 5:49 PM · Cloud-VPS, Security-Team
chasemp triaged T194150: Decide future of *.security-tools vms in labs as Normal priority.
Mon, Aug 13, 5:48 PM · Cloud-VPS, Security-Team
chasemp added a comment to T194150: Decide future of *.security-tools vms in labs.

Considering https://phabricator.wikimedia.org/T161107#3123690 I am going to purge scanner00.security-tools.eqiad.wmflabs

Mon, Aug 13, 5:45 PM · Cloud-VPS, Security-Team
chasemp updated the task description for T194150: Decide future of *.security-tools vms in labs.
Mon, Aug 13, 5:44 PM · Cloud-VPS, Security-Team

Wed, Jul 25

chasemp added a comment to T198420: Improve unmount/relink setup for dumps (labstore1006/1007) failovers.

Small note we had major troubles today w/ 1007 going away and though instances by all accounts should not have cared they did and load was out of control. It sure seemed like instances were trying to mount 1007 but from what directive I don't understand. /etc/fstab had the entry removed and puppet as disabled. Rebooting a tools-worker and cheating to make it appear no NFS mounts were at that IP seeemd to settle things but I don't feel like I know much about what is going on

Wed, Jul 25, 7:04 PM · Patch-For-Review, cloud-services-team (Kanban)
chasemp added a comment to T196542: Private space for tracking internal security team activities.

A space for this type of work does exist but was more narrowly scoped at the time. All the stakeholders there have since left. Instead of creating a new space I did some shuffling to reuse S11 seen in T150046. S11 is owned by https://phabricator.wikimedia.org/project/profile/2356/ for editing the space itself, and visible to those folks and acl*operations_team. I also disabled the old herald rule H205.

@chasemp:
@bbogaert who has left is still set as assignee of two open WMF-SIEM tasks in S11. I cannot access S11.
As I don't know which exact "shuffling" took place, who can reset the assignee of these two tasks (because cookie-licking)? Looking at acl*security_team renamed from #acl*wmf_siem_policy_admins I guess it's Reedy, Bawolff, JBennett and you who could fix that?

Wed, Jul 25, 3:06 PM · Phabricator

Sun, Jul 22

chasemp updated subscribers of T195121: Contribution from the IGN to Structured Data on Commons.
Sun, Jul 22, 9:49 AM · Cloud-Services, GLAM, Wikidata, Structured-Data-Commons, cloud-services-team

Sat, Jul 21

chasemp awarded T200125: Nomination of Addshore for Toolforge admin role a Love token.
Sat, Jul 21, 11:35 AM · User-Addshore, cloud-services-team (Kanban), Toolforge
chasemp added a comment to T200124: Nomination of Legoktm for Toolforge admin role.

+1

Sat, Jul 21, 11:23 AM · cloud-services-team (Kanban), Toolforge

Thu, Jul 19

chasemp added a comment to T199987: Enable ores_classification tables on labsDB.

After a bit of poking and feeling like things /should/ be working if the right scripts were run I issued:

Thu, Jul 19, 11:21 AM · Data-Services
chasemp added a comment to T199532: Move 6 instances for general-k8s project each to a different physical host.

I did a survey of instance count and load on existing hypervisors in the scheduler, and then took stock of how the existing spread was here for these nodes and it already looked fairly good. In the end I created the 6th node on a distinct hypervisor and the current situation is:

Thu, Jul 19, 3:22 AM · cloud-services-team (Kanban), Cloud-VPS
chasemp closed T90491: Create policy projects and convert people projects to open as Resolved.
Thu, Jul 19, 2:47 AM · Project-Admins, Phabricator
chasemp closed T90491: Create policy projects and convert people projects to open, a subtask of T88762: Enable select Fundraising people to modify policy for tasks, as Resolved.
Thu, Jul 19, 2:47 AM · Security, WMF-NDA, Phabricator
chasemp closed T90491: Create policy projects and convert people projects to open, a subtask of T89053: unable to subscribe to operations tag after migration and merge from ops-core and ops-request, as Resolved.
Thu, Jul 19, 2:47 AM · Operations, Phabricator

Wed, Jul 18

chasemp added a comment to T199532: Move 6 instances for general-k8s project each to a different physical host.

Oh I see they are created :). Could we start from scratch/are they configured?

Wed, Jul 18, 12:07 PM · cloud-services-team (Kanban), Cloud-VPS
chasemp added a comment to T199532: Move 6 instances for general-k8s project each to a different physical host.

It is indeed easier to distribute them at the time of creation but yeah that's a root function. I am not sure if we have 6 virts that have headroom. What size instances and would 3 virts with 2 each work?

Wed, Jul 18, 12:05 PM · cloud-services-team (Kanban), Cloud-VPS
chasemp added a comment to T199688: Decide if H141 ("Add security tag when Security flag is set to Software security bug") is needed or not.

We are going to review the Security workflow soon (august or sept?) so I'll keep this in mind

Wed, Jul 18, 10:28 AM · Phabricator, Security

Jul 12 2018

chasemp added a comment to T193496: Allocate public v4 IPs for Neutron setup in eqiad.
+    /* Cloud public prefix via labnet100[45] */
 +    route 185.15.56.0/25 next-hop 10.64.22.4;
Jul 12 2018, 6:26 PM · Cloud-Services, netops, Operations
chasemp added a comment to T193496: Allocate public v4 IPs for Neutron setup in eqiad.
+    /* Cloud public prefix via labnet100[45] */
+    route 185.15.56.0/25 next-hop 10.64.22.4;
Jul 12 2018, 6:24 PM · Cloud-Services, netops, Operations

Jul 11 2018

chasemp updated the task description for T199374: Delegate public IP range for Eqiad1-r OpenStack deployment to designate (neutron).
Jul 11 2018, 9:07 PM · Patch-For-Review, Cloud-Services
chasemp updated the task description for T199374: Delegate public IP range for Eqiad1-r OpenStack deployment to designate (neutron).
Jul 11 2018, 9:07 PM · Patch-For-Review, Cloud-Services
chasemp triaged T199374: Delegate public IP range for Eqiad1-r OpenStack deployment to designate (neutron) as Normal priority.
Jul 11 2018, 9:05 PM · Patch-For-Review, Cloud-Services
chasemp updated subscribers of T193496: Allocate public v4 IPs for Neutron setup in eqiad.

https://apps.db.ripe.net/db-web-ui/#/lookup?source=ripe&key=185.15.56.0%2F24AS14907&type=route created.

185.15.56.0/25   - 1:1 replacement of 208.80.155.128/25
185.15.56.128/26 - reserved for groth beyond first /25
185.15.56.192/27 - reserved for future use
185.15.56.224/27 - reserved for labtest
Jul 11 2018, 9:02 PM · Cloud-Services, netops, Operations
chasemp added a comment to T198974: Rate-limit is too harsh and affects human users.

I have been hit too :)

Jul 11 2018, 6:07 PM · Patch-For-Review, Phabricator
chasemp added a comment to T198138: Disable agent forwarding to important hosts.

Is there a technical reason to only do it selectively for a few WMCS hosts and not in general? Or in other words, is there any known component (like something in Toolforge or so) which is known to depend on agent forwarding being enabled?

Jul 11 2018, 11:39 AM · Operations, Security

Jul 10 2018

chasemp closed T199123: labtestn (and possibly eqiad1) VM subnets missing from network constants as Resolved.
Jul 10 2018, 9:04 PM · Patch-For-Review, Cloud-Services
chasemp added a comment to T199123: labtestn (and possibly eqiad1) VM subnets missing from network constants.

https://gerrit.wikimedia.org/r/c/operations/puppet/+/444611

Jul 10 2018, 9:04 PM · Patch-For-Review, Cloud-Services
chasemp reassigned T199125: rack/setup/install cloudvirt102[34] from chasemp to Andrew.
Jul 10 2018, 9:03 PM · cloud-services-team (Kanban), ops-eqiad, Cloud-VPS, Operations
chasemp added a comment to T198420: Improve unmount/relink setup for dumps (labstore1006/1007) failovers.

Change 444914 merged by Rush:
[operations/puppet@production] dumps: add labstore1006 back for dumps serving to cloud

https://gerrit.wikimedia.org/r/444914

Jul 10 2018, 4:52 PM · Patch-For-Review, cloud-services-team (Kanban)
chasemp triaged T199248: Smart alert on labstore1006 as Normal priority.
Jul 10 2018, 4:51 PM · cloud-services-team (Kanban)

Jul 9 2018

chasemp updated subscribers of T199125: rack/setup/install cloudvirt102[34].

wait, can we make these cloudvirt1023 and cloudvirt1024? I think @aborrero is getting into the naming adjustments now.

Jul 9 2018, 4:45 PM · cloud-services-team (Kanban), ops-eqiad, Cloud-VPS, Operations
chasemp added a comment to T199125: rack/setup/install cloudvirt102[34].

@chasemp: Can you confirm the racking proposal above and also comment if these are trusty or another OS? Once we have that, please assign to @Cmjohnson

Jul 9 2018, 4:18 PM · cloud-services-team (Kanban), ops-eqiad, Cloud-VPS, Operations
chasemp added a watcher for Security: chasemp.
Jul 9 2018, 3:04 PM
chasemp closed T197169: 10G ports seem not to work on new HP hardware as Resolved.

So for at least labvirt1019 it was indeed about PXE not working (the card worked under Linux) and that was due to a BIOS misconfiguration (the "network boot" option for the card set to disabled). T194964#4283034 has more details and troubleshooting steps.

I haven't verified whether that's the case for labnet1003/1004 or even labvirt1020. I guess we should do that first, before resolving this task?

Jul 9 2018, 1:12 PM · Cloud-Services, Operations
chasemp closed T197169: 10G ports seem not to work on new HP hardware, a subtask of T167293: Nova-network to Neutron migration, as Resolved.
Jul 9 2018, 1:12 PM · Epic, Cloud-Services

Jul 8 2018

chasemp created P7334 (An Untitled Masterwork).
Jul 8 2018, 7:11 PM

Jul 5 2018

chasemp assigned T198863: Reallocate labvirt from main to eqiad1 to aborrero.

Want to reimage as cloudvirt? Would need subtasks to handle the long list of things to update :)

Jul 5 2018, 6:24 PM · Cloud-Services
chasemp updated subscribers of T198863: Reallocate labvirt from main to eqiad1.

@Andrew which labvirt should we put in eqiad1 deployment?

Jul 5 2018, 2:25 PM · Cloud-Services
chasemp triaged T198863: Reallocate labvirt from main to eqiad1 as Normal priority.
Jul 5 2018, 2:24 PM · Cloud-Services

Jul 2 2018

chasemp awarded Blog Post: 2017 Toolforge survey results a Yellow Medal token.
Jul 2 2018, 9:29 PM · Toolforge
chasemp added a member for Security-Core: chasemp.
Jul 2 2018, 9:25 PM
chasemp awarded T190015: Create separate user group for editing sitewide CSS/JavaScript that does not include administrators by default a Like token.
Jul 2 2018, 9:25 PM · MW-1.32-release-notes (WMF-deploy-2018-07-24 (1.32.0-wmf.14)), User-Tgr, Trust-and-Safety, Wikimedia-General-or-Unknown, Patch-For-Review, Security, JavaScript, Security-Core
chasemp assigned T194012: labsdb1004 and labsdb1005 some hard disks not healthy to Cmjohnson.
Jul 2 2018, 9:23 PM · DC-Ops, Operations, ops-eqiad, Cloud-Services

Jun 30 2018

chasemp added a comment to T198420: Improve unmount/relink setup for dumps (labstore1006/1007) failovers.

This also needs to remove from fstab. I really suspect we should have nfs mounts managed outside of the normal puppet mount provider and not have them automounted on boot but instead through some explicit puppet action as otherwise a missing NFS server is really tricky to recover from.

Jun 30 2018, 4:49 PM · Patch-For-Review, cloud-services-team (Kanban)

Jun 28 2018

chasemp updated subscribers of T196651: rack upgraded storage capacity in labstore100[67].eqiad.wmnet.

labstore1007 has been restored to service and NFS clients and web users are pointed at it (https://gerrit.wikimedia.org/r/c/operations/puppet/+/442913)

Jun 28 2018, 8:00 PM · Patch-For-Review, Datasets-General-or-Unknown, ops-eqiad, Cloud-VPS, Operations
chasemp updated subscribers of T196651: rack upgraded storage capacity in labstore100[67].eqiad.wmnet.

We ran into trouble here:

Jun 28 2018, 6:25 PM · Patch-For-Review, Datasets-General-or-Unknown, ops-eqiad, Cloud-VPS, Operations
chasemp added a comment to T198407: Degraded RAID on labstore1007.

possibly related to T196651: rack upgraded storage capacity in labstore100[67].eqiad.wmnet?

Jun 28 2018, 3:18 PM · ops-eqiad, Operations
chasemp updated subscribers of T198407: Degraded RAID on labstore1007.

@Volans can you help make sense of this?

Jun 28 2018, 3:16 PM · ops-eqiad, Operations
chasemp added a comment to T198407: Degraded RAID on labstore1007.

and labstore1006 as well? [from irc]

Jun 28 2018, 3:16 PM · ops-eqiad, Operations
chasemp triaged T198407: Degraded RAID on labstore1007 as High priority.
Jun 28 2018, 3:15 PM · ops-eqiad, Operations
chasemp assigned T198407: Degraded RAID on labstore1007 to Cmjohnson.

I don't quite understand this. Is this trying to say 6 failed drives?

Jun 28 2018, 3:14 PM · ops-eqiad, Operations

Jun 27 2018

chasemp closed T193196: labnet1003 and labnet1004 moving and enabling 10G NICs as Resolved.

dug up an old task that said rootdelay is the way to address this in jessie, with permanent fixes having landed in stretch. So far that seems to have corrected the issue. Both labnet1003 and 1004 are now running jessie using 10G nics. Need to get a bit further to confirm the instance facing interface works but I'm considering this closed until I know better ;)

Jun 27 2018, 8:36 PM · Patch-For-Review, Cloud-VPS, ops-eqiad, Operations
chasemp added a comment to T193196: labnet1003 and labnet1004 moving and enabling 10G NICs.

We moved past the DHCP/NIC issue and now are failing with

Jun 27 2018, 4:54 PM · Patch-For-Review, Cloud-VPS, ops-eqiad, Operations
chasemp added a comment to T198267: Request increased quota for reading-web-staging Cloud VPS project.

+1!!!!

Jun 27 2018, 2:59 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
chasemp added a comment to T193196: labnet1003 and labnet1004 moving and enabling 10G NICs.

trying https://gerrit.wikimedia.org/r/c/operations/puppet/+/442295 to see if there is any change

Jun 27 2018, 1:15 PM · Patch-For-Review, Cloud-VPS, ops-eqiad, Operations
chasemp added a comment to T193196: labnet1003 and labnet1004 moving and enabling 10G NICs.
Attempting Boot From NIC
Jun 27 2018, 1:00 PM · Patch-For-Review, Cloud-VPS, ops-eqiad, Operations

Jun 26 2018

chasemp added a comment to P7296 (An Untitled Masterwork).

openstack role add --project admin --user novaadmin admin

Jun 26 2018, 1:14 PM
chasemp created P7296 (An Untitled Masterwork).
Jun 26 2018, 1:13 PM

Jun 25 2018

chasemp added a comment to T196418: Google Maps keyless usage sunset on 2018-06-11.

I don't think there is anything for us to do other than notify cloud-announce, is there?

Jun 25 2018, 6:28 PM · CommRel-Specialists-Support (Jul-Sep-2018), Toolforge
chasemp added a comment to T198074: Fixing response time and enabling larger models in Wembedder.

Please use the template from https://phabricator.wikimedia.org/project/view/2880/

Jun 25 2018, 4:20 PM · Tools
chasemp added a comment to T197550: Remove approval requirement for new accounts, or patch everything in Phabricator to allow unapproved users to be treated as logged out for permissions purposes .

Change 441806 had a related patch set uploaded (by Aklapper; owner: Aklapper):
[operations/puppet@production] Phab: Explain to not-yet-approved users how they can access tasks

https://gerrit.wikimedia.org/r/441806

Jun 25 2018, 4:10 PM · Phabricator

Jun 20 2018

chasemp added a watcher for acl*stewards: chasemp.
Jun 20 2018, 9:44 AM
chasemp added a member for acl*operations-team: Vgutierrez.
Jun 20 2018, 7:42 AM

Jun 14 2018

chasemp added a comment to T187962: Rack/cable/configure asw2-c-eqiad switch stack.

We met today to sync up on moving the remaining lab* servers. Hopefully these days/times al work for @Cmjohnson (I added him to the calendar invites to confirm)

Jun 14 2018, 2:53 PM · Patch-For-Review, Operations, ops-eqiad, netops
chasemp added a comment to T188012: PAWS userhomes permission error.
timeout 180s bash -x /data/project/paws/paws-userhomes-hack.bash
+ true
+ find /data/project/paws/userhomes/ -maxdepth 1 -user root
+ xargs -L1 chown -v tools.paws:tools.paws
changed ownership of ‘/data/project/paws/userhomes/51990302’ from root:root to tools.paws:tools.paws
changed ownership of ‘/data/project/paws/userhomes/8631757’ from root:root to tools.paws:tools.paws
changed ownership of ‘/data/project/paws/userhomes/54479680’ from root:root to tools.paws:tools.paws
changed ownership of ‘/data/project/paws/userhomes/1281’ from root:root to tools.paws:tools.paws
changed ownership of ‘/data/project/paws/userhomes/54533986’ from root:root to tools.paws:tools.paws
changed ownership of ‘/data/project/paws/userhomes/54469129’ from root:root to tools.paws:tools.paws
changed ownership of ‘/data/project/paws/userhomes/54485747’ from root:root to tools.paws:tools.paws
changed ownership of ‘/data/project/paws/userhomes/54475927’ from root:root to tools.paws:tools.paws
changed ownership of ‘/data/project/paws/userhomes/52360419’ from root:root to tools.paws:tools.paws
changed ownership of ‘/data/project/paws/userhomes/53338127’ from root:root to tools.paws:tools.paws
changed ownership of ‘/data/project/paws/userhomes/46482467’ from root:root to tools.paws:tools.paws
changed ownership of ‘/data/project/paws/userhomes/54491825’ from root:root to tools.paws:tools.paws
changed ownership of ‘/data/project/paws/userhomes/5424947’ from root:root to tools.paws:tools.paws
changed ownership of ‘/data/project/paws/userhomes/54476398’ from root:root to tools.paws:tools.paws
changed ownership of ‘/data/project/paws/userhomes/54485761’ from root:root to tools.paws:tools.paws
+ sleep 1
+ true
+ find /data/project/paws/userhomes/ -maxdepth 1 -user root
+ xargs -L1 chown -v tools.paws:tools.paws
chown: missing operand after ‘tools.paws:tools.paws’
Try 'chown --help' for more information.
Jun 14 2018, 2:18 PM · PAWS (JupyterHub 0.9)
chasemp added a comment to T188012: PAWS userhomes permission error.

@Chicocvenancio

Jun 14 2018, 2:15 PM · PAWS (JupyterHub 0.9)

Jun 13 2018

chasemp updated subscribers of T197169: 10G ports seem not to work on new HP hardware.

ping @aborrero who indicated he had seem a similar issue in the past

Jun 13 2018, 6:47 PM · Cloud-Services, Operations
chasemp added a comment to T197169: 10G ports seem not to work on new HP hardware.

@Cmjohnson could you describe a bit what you've tried to get the 10G ports to work?

Jun 13 2018, 6:37 PM · Cloud-Services, Operations
chasemp triaged T197169: 10G ports seem not to work on new HP hardware as Normal priority.
Jun 13 2018, 6:36 PM · Cloud-Services, Operations
chasemp added a comment to T191791: Make same glance images available to multiple regions.

Nope, it's entirely possible that multi-region glance is just that simple because glance isn't too picky. I think though that what you want is glance running on labtestcontrol2003 (debian) to use the databases from labtestcontrol2001's glance. small change here and if this works I imagine that's no big deal.

Jun 13 2018, 3:16 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-Services

Jun 12 2018

chasemp changed the status of T196545: Create new labs project for development & testing of WMDE technical wishlist features/extensions from Open to Stalled.

We decided to mark this as stalled for our next meeting as we need to get input from team members who were not present. Nothing bad here just a very busy few weeks.

Jun 12 2018, 8:40 PM · Cloud-VPS (Project-requests)
chasemp closed T196999: Request creation of sciencesource VPS project as Resolved.

Created sciencesource project. Let me know if you have issues!

Jun 12 2018, 8:39 PM · Cloud-VPS (Project-requests)
chasemp triaged T196999: Request creation of sciencesource VPS project as Normal priority.
Jun 12 2018, 8:36 PM · Cloud-VPS (Project-requests)
chasemp closed T196796: Request creation of openocr VPS project as Resolved.

created opencr project with you as a member. Let me know if you have any issues.

Jun 12 2018, 8:35 PM · Cloud-VPS (Project-requests)
chasemp closed T196094: Request creation of general-k8s VPS project as Resolved.

So I understand the plan would be to create a single project encompassing T196095 T195214 and this request.

Would it be possible to get:

  • 8 instances
  • 16 CPUs (doubled from the regular quota)
  • 32 GB memory (doubled from the regular quota)
  • 1 floating IP
Jun 12 2018, 8:31 PM · Cloud-VPS (Project-requests)
chasemp added a comment to T193655: rack/setup/install labstore1008 & labstore1009.

I think this is ready for OS install and such? I spoke with @Bstorm who is going to take this on and may need some advice.

Jun 12 2018, 3:55 PM · cloud-services-team (Kanban), Patch-For-Review, ops-eqiad, Cloud-VPS, Operations
chasemp reassigned T193655: rack/setup/install labstore1008 & labstore1009 from Cmjohnson to Bstorm.
Jun 12 2018, 3:48 PM · cloud-services-team (Kanban), Patch-For-Review, ops-eqiad, Cloud-VPS, Operations
chasemp triaged T196973: toolforge: k8s alert about nodes not healthy as Normal priority.
Jun 12 2018, 3:43 PM · cloud-services-team, Toolforge
chasemp added a comment to T196796: Request creation of openocr VPS project.

approved

Jun 12 2018, 3:43 PM · Cloud-VPS (Project-requests)
chasemp added a comment to T196999: Request creation of sciencesource VPS project.

approved

Jun 12 2018, 3:43 PM · Cloud-VPS (Project-requests)
chasemp added a comment to T196094: Request creation of general-k8s VPS project.

approved as 'general-k8s'

Jun 12 2018, 3:43 PM · Cloud-VPS (Project-requests)
chasemp added a comment to T196973: toolforge: k8s alert about nodes not healthy.

By chance was there an update to some global ferm variable?

Jun 12 2018, 2:51 PM · cloud-services-team, Toolforge
chasemp added a comment to T196973: toolforge: k8s alert about nodes not healthy.

I believe tools-worker-1029.tools.eqiad.wmflabs needs to be deleted, but I'm unsure why these three are cordon'd. relic of forgotten maint?

Jun 12 2018, 2:50 PM · cloud-services-team, Toolforge

Jun 11 2018

chasemp removed a member for acl*security_team: ggellerman.
Jun 11 2018, 2:56 PM
chasemp closed T196542: Private space for tracking internal security team activities as Resolved.

A space for this type of work does exist but was more narrowly scoped at the time. All the stakeholders there have since left. Instead of creating a new space I did some shuffling to reuse S11 seen in T150046. S11 is owned by https://phabricator.wikimedia.org/project/profile/2356/ for editing the space itself, and visible to those folks and acl*operations_team. I also disabled the old herald rule H205.

Jun 11 2018, 2:55 PM · Phabricator
chasemp added members for acl*security_team: JBennett, Bawolff, Reedy.
Jun 11 2018, 2:50 PM
chasemp added a member for acl*security_team: chasemp.
Jun 11 2018, 2:48 PM
chasemp triaged T196542: Private space for tracking internal security team activities as Normal priority.
Jun 11 2018, 2:47 PM · Phabricator
chasemp changed the edit policy for acl*security_team.
Jun 11 2018, 2:43 PM
chasemp renamed acl*security_team from acl*wmf_siem_policy_admins to acl*security_team.
Jun 11 2018, 2:40 PM
chasemp closed T196704: labstore1003 SMART failure (again) as Resolved.
root@labstore1003:~# /usr/local/lib/nagios/plugins/check_raid megacli
OK: optimal, 5 logical, 34 physical
OK
Jun 11 2018, 1:59 PM · Operations, cloud-services-team, ops-eqiad, DC-Ops
Gerrit Code Review <gerrit@wikimedia.org> committed rODITe748358ad8b1: Update patch set 8 (authored by chasemp).
Update patch set 8
Jun 11 2018, 11:35 AM
Gerrit Code Review <gerrit@wikimedia.org> committed rODIT7a4efc34f8ab: Update patch set 3 (authored by chasemp).
Update patch set 3
Jun 11 2018, 11:35 AM

Jun 8 2018

chasemp added a comment to T183920: 2018-01-02: labstore Tools and Misc share very full.

I totally forgot to tell @Bstorm we have announced the invasive parts of this in the past, old example: https://lists.wikimedia.org/pipermail/labs-l/2016-May/004493.html

Jun 8 2018, 6:03 PM · cloud-services-team (Kanban), Operations, Cloud-VPS