Page MenuHomePhabricator

Andrew (Andrew Bogott)
User

Projects (10)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Nov 2 2014, 11:35 PM (229 w, 10 h)
Availability
Available
IRC Nick
andrewbogott
LDAP User
Unknown
MediaWiki User
Andrewbogott [ Global Accounts ]

Recent Activity

Fri, Mar 22

Andrew added a comment to T218448: Volunteer NDA for Alex Monk.

fyi @jcrespo, access levels (in particular 'cloud-wide root') are defined in the policy document here: https://wikitech.wikimedia.org/wiki/Help:Access_policies

Fri, Mar 22, 8:17 PM · Operations, WMF-NDA-Requests
Andrew reopened T46722: Add two read-only LDAP servers in eqiad as "Open".

this might need a bit of indexing work -- I see this in syslog:

Fri, Mar 22, 3:55 AM · Patch-For-Review, cloud-services-team (Kanban), LDAP, Cloud-VPS
Andrew reopened T46722: Add two read-only LDAP servers in eqiad, a subtask of T46720: Only list LDAP servers location in the same datacenter in the nslcd configuration, as Open.
Fri, Mar 22, 3:55 AM · LDAP, cloud-services-team (Kanban), Cloud-VPS
Andrew added a comment to T218133: Put our ldap servers behind LVS.

There are now two read-only replicas in eqiad behind the endpoint ldap-ro.eqiad.wikimedia.org

Fri, Mar 22, 3:51 AM · Patch-For-Review, LDAP
Andrew closed T46722: Add two read-only LDAP servers in eqiad as Resolved.

ldap-eqiad-replica01.wikimedia.org and ldap-eqiad-replica02.wikimedia.org are online now and seem to be working

Fri, Mar 22, 3:47 AM · Patch-For-Review, cloud-services-team (Kanban), LDAP, Cloud-VPS
Andrew closed T46722: Add two read-only LDAP servers in eqiad, a subtask of T46720: Only list LDAP servers location in the same datacenter in the nslcd configuration, as Resolved.
Fri, Mar 22, 3:47 AM · LDAP, cloud-services-team (Kanban), Cloud-VPS
Andrew closed T46722: Add two read-only LDAP servers in eqiad, a subtask of T218133: Put our ldap servers behind LVS, as Resolved.
Fri, Mar 22, 3:47 AM · Patch-For-Review, LDAP

Thu, Mar 21

Andrew added a comment to T218934: Horizon login failures.

This is probably a side-effect of issues with Striker; they most likely share the same consumer token. Horizon should really have its own.

Thu, Mar 21, 8:14 PM · cloud-services-team (Kanban), Horizon, wikitech.wikimedia.org
Andrew added a comment to T218934: Horizon login failures.

On the keystone server (cloudcontrol2003):

Thu, Mar 21, 7:30 PM · cloud-services-team (Kanban), Horizon, wikitech.wikimedia.org
Andrew created T218934: Horizon login failures.
Thu, Mar 21, 7:29 PM · cloud-services-team (Kanban), Horizon, wikitech.wikimedia.org
Mill <mill@mail.com> committed rCUMINf842163b82c9: vxaaaaaaaaaaaa (authored by Andrew).
vxaaaaaaaaaaaa
Thu, Mar 21, 12:37 AM

Tue, Mar 19

Andrew added a comment to T218119: Disable jessie VM creation in VPS.

https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/VM_images#Private_Images

Tue, Mar 19, 2:47 PM · Cloud-VPS, cloud-services-team (Kanban)

Mon, Mar 18

Andrew added a comment to T218609: Figure out future for newly created deployment-prep jessie instances.

Jessie creation is now disabled in most projects (including deployment-prep). I'd prefer to leave it that way in order to provide some mild resistance to new Jessie VMs showing up in the cloud.

Mon, Mar 18, 10:30 PM · Beta-Cluster-Infrastructure
Andrew closed T218119: Disable jessie VM creation in VPS as Resolved.

This image is now private to the 'admin' project, with shared access extended to the 'integration' and 'testlabs' projects.

Mon, Mar 18, 10:29 PM · Cloud-VPS, cloud-services-team (Kanban)
Andrew added a comment to T218119: Disable jessie VM creation in VPS.

The Integration project will need to keep creating Jessie VMs for a bit.

Mon, Mar 18, 3:34 PM · Cloud-VPS, cloud-services-team (Kanban)
Andrew added a comment to T218119: Disable jessie VM creation in VPS.

Announced here:

Mon, Mar 18, 2:34 PM · Cloud-VPS, cloud-services-team (Kanban)
Andrew closed T218398: Update openldap profile to use LE as Resolved.

This is working with acme now.

Mon, Mar 18, 1:57 PM · Patch-For-Review, Traffic, Operations, cloud-services-team (Kanban), LDAP, Cloud-VPS
Andrew closed T218398: Update openldap profile to use LE, a subtask of T46722: Add two read-only LDAP servers in eqiad, as Resolved.
Mon, Mar 18, 1:57 PM · Patch-For-Review, cloud-services-team (Kanban), LDAP, Cloud-VPS
Andrew added a comment to T218022: decommission: labtestservices2001.wikimedia.org.

I don't really know what that database is about. But perhaps we want to do it at the same time as T218569: Openstack codfw DBs: move to m5-master.eqiad.wmnet. Would you mind updating that tickets so we have all the DB-reallocating info in a single place?

Mon, Mar 18, 1:54 PM · Operations, decommission, ops-codfw, cloud-services-team (Kanban)

Sun, Mar 17

Andrew added a comment to T218514: puppet breakage on Jessie tools nodes (and probably on Jessie VMs everywhere).

Sometime when it's not the weekend let's audit all instances for stuck dpkg processes. This might be happening all over the place.

Sun, Mar 17, 5:46 PM · cloud-services-team (Kanban)
Andrew triaged T218514: puppet breakage on Jessie tools nodes (and probably on Jessie VMs everywhere) as High priority.
Sun, Mar 17, 3:33 PM · cloud-services-team (Kanban)
Andrew created T218514: puppet breakage on Jessie tools nodes (and probably on Jessie VMs everywhere).
Sun, Mar 17, 3:33 PM · cloud-services-team (Kanban)

Fri, Mar 15

Andrew added a comment to T218022: decommission: labtestservices2001.wikimedia.org.

right now labtestservices2001 is the only host for the labtest ldap db. So we should move that someplace before we decom, unless we want to start with a fresh db entirely.

Fri, Mar 15, 3:55 PM · Operations, decommission, ops-codfw, cloud-services-team (Kanban)
Andrew created T218398: Update openldap profile to use LE.
Fri, Mar 15, 2:12 PM · Patch-For-Review, Traffic, Operations, cloud-services-team (Kanban), LDAP, Cloud-VPS

Wed, Mar 13

Andrew closed T218224: Ganeti request: two new VMs in eqiad for ldap as Resolved.

These are now created and puppetized:

Wed, Mar 13, 8:02 PM · Patch-For-Review, cloud-services-team (Kanban), LDAP, Cloud-VPS
Andrew closed T218224: Ganeti request: two new VMs in eqiad for ldap, a subtask of T46722: Add two read-only LDAP servers in eqiad, as Resolved.
Wed, Mar 13, 8:02 PM · Patch-For-Review, cloud-services-team (Kanban), LDAP, Cloud-VPS
Andrew added a comment to T215012: cloudvirt1015: apparent hardware errors in CPU/Memory.

@aborrero the CPU is here...let me know when it's safe for me to change.

Wed, Mar 13, 6:36 PM · Patch-For-Review, Operations, ops-eqiad, DC-Ops, cloud-services-team (Kanban)
Andrew added a comment to T218224: Ganeti request: two new VMs in eqiad for ldap.

Oh, I was confused -- these do need to be on public IPs.

Wed, Mar 13, 6:20 PM · Patch-For-Review, cloud-services-team (Kanban), LDAP, Cloud-VPS
Andrew added a comment to T218224: Ganeti request: two new VMs in eqiad for ldap.

Ldap isn't really a cloud-specific service. So I propose ldap-replica01 and ldap-replica02. My instinct is to put them on private IPs (because LVM will provide public endpoint anyway) but I'm open to suggestion.

Wed, Mar 13, 5:36 PM · Patch-For-Review, cloud-services-team (Kanban), LDAP, Cloud-VPS
Andrew updated the task description for T218224: Ganeti request: two new VMs in eqiad for ldap.
Wed, Mar 13, 3:52 PM · Patch-For-Review, cloud-services-team (Kanban), LDAP, Cloud-VPS
Andrew assigned T218224: Ganeti request: two new VMs in eqiad for ldap to akosiaris.

Alex, I'm hoping that you have time to do some/all of this. If not, then please answer these questions and refer back to me so I can muddle through:

Wed, Mar 13, 3:52 PM · Patch-For-Review, cloud-services-team (Kanban), LDAP, Cloud-VPS
Andrew created T218224: Ganeti request: two new VMs in eqiad for ldap.
Wed, Mar 13, 3:51 PM · Patch-For-Review, cloud-services-team (Kanban), LDAP, Cloud-VPS
Andrew added a comment to T46722: Add two read-only LDAP servers in eqiad.

That sounds like a plan! I've re-titled this task to fit that plan. We'll need two more Ganeti VMs. The current ldap servers have 8 cores and 4Gb of RAM. More RAM would be great, and fewer CPUs would probably be tolerable.

Wed, Mar 13, 3:43 PM · Patch-For-Review, cloud-services-team (Kanban), LDAP, Cloud-VPS
Andrew added a comment to T218133: Put our ldap servers behind LVS.

Current plan is to add two new read-only hosts (on internal IPs) and put LVS in front of them, then use that endpoint exclusively for cloud VMs access.

Wed, Mar 13, 3:41 PM · Patch-For-Review, LDAP
Andrew renamed T46722: Add two read-only LDAP servers in eqiad from Have two LDAP servers per datacenter to Add two read-only LDAP servers in eqiad.
Wed, Mar 13, 3:39 PM · Patch-For-Review, cloud-services-team (Kanban), LDAP, Cloud-VPS
Andrew added a comment to T216218: Cloud VPS outage on cloudvirt1024 and cloudvirt1018 due to storage failure.

We haven't bothered to rebuild this host since it's pending some datacenter work. The alert is silenced and described I believe?

Wed, Mar 13, 3:32 PM · Cloud-VPS, cloud-services-team (Kanban)

Tue, Mar 12

Andrew added a comment to T46722: Add two read-only LDAP servers in eqiad.

I'd like to set aside the issue of ldap-on-cloud for now and just get a couple more servers up on Ganeti. Which I don't immediately know how to do but I bet @GTirloni knows how

Tue, Mar 12, 10:37 PM · Patch-For-Review, cloud-services-team (Kanban), LDAP, Cloud-VPS
Andrew added a subtask for T217280: LDAP server running out of memory frequently and disrupting Cloud VPS clients: T218133: Put our ldap servers behind LVS.
Tue, Mar 12, 5:38 PM · cloud-services-team (Kanban), Patch-For-Review, Operations, Cloud-VPS, LDAP, Toolforge
Andrew added a parent task for T218133: Put our ldap servers behind LVS: T217280: LDAP server running out of memory frequently and disrupting Cloud VPS clients.
Tue, Mar 12, 5:37 PM · Patch-For-Review, LDAP
Andrew added a parent task for T46722: Add two read-only LDAP servers in eqiad: T218133: Put our ldap servers behind LVS.
Tue, Mar 12, 5:37 PM · Patch-For-Review, cloud-services-team (Kanban), LDAP, Cloud-VPS
Andrew added a subtask for T218133: Put our ldap servers behind LVS: T46722: Add two read-only LDAP servers in eqiad.
Tue, Mar 12, 5:37 PM · Patch-For-Review, LDAP
Andrew updated the task description for T218133: Put our ldap servers behind LVS.
Tue, Mar 12, 5:36 PM · Patch-For-Review, LDAP
Andrew created T218133: Put our ldap servers behind LVS.
Tue, Mar 12, 5:36 PM · Patch-For-Review, LDAP
Andrew created T218119: Disable jessie VM creation in VPS.
Tue, Mar 12, 4:07 PM · Cloud-VPS, cloud-services-team (Kanban)
Andrew added a comment to T218029: CloudVPS: evaluate convenience of having codfw openstack DBs in proper DB hosts.

We would also want labtestwiki here (the MW database for labtestwikitech)

Tue, Mar 12, 3:02 PM · cloud-services-team (Kanban)
Andrew awarded T218064: dynamicproxy data contains domains ending in '.' which can not be deleted via Horizon or cli tools a Barnstar token.
Tue, Mar 12, 2:59 PM · cloud-services-team (Kanban), Horizon
Andrew awarded T204506: cloudvps: maps project trusty deprecation a Yellow Medal token.
Tue, Mar 12, 2:58 PM · cloud-services-team (Kanban), User-TheDJ, Cloud-VPS (Ubuntu Trusty Deprecation), Maps

Sat, Mar 9

Andrew added a comment to T217838: Toolforge Stretch - Increased LDAP utilization.
root@serpens:/etc/ldap# grep deref slapd.conf
moduleload  deref
overlay deref
Sat, Mar 9, 9:44 PM · cloud-services-team (Kanban), LDAP, Toolforge
Andrew triaged T217943: Stretch grid problem: cannot increase memory for Java as Normal priority.
Sat, Mar 9, 9:33 PM · cloud-services-team, Toolforge
Andrew added a comment to T217954: Recover ign.ign2commons.eqiad.wmflabs.

I'm sorry @abian, that VM does not appear to have survived the hardware failure. Its disk image is entirely missing so there's nothing to salvage.

Sat, Mar 9, 9:28 PM · cloud-services-team, Cloud-Services

Fri, Mar 8

Andrew added a comment to T217838: Toolforge Stretch - Increased LDAP utilization.

Here's the kind of thing nslcd is doing on stretch:

Fri, Mar 8, 11:38 PM · cloud-services-team (Kanban), LDAP, Toolforge
Andrew added a comment to T217838: Toolforge Stretch - Increased LDAP utilization.

It seems moderately possible that we need to install the 'deref' overlay on openldap to catch up with the new expectations of nslcd. That's only barely supported, but one brave soul seems to have made it work: https://www.openldap.org/lists/openldap-technical/201401/msg00025.html

Fri, Mar 8, 11:32 PM · cloud-services-team (Kanban), LDAP, Toolforge
Andrew added a comment to T217838: Toolforge Stretch - Increased LDAP utilization.
Fri, Mar 8, 9:45 PM · cloud-services-team (Kanban), LDAP, Toolforge
Andrew added a comment to T217838: Toolforge Stretch - Increased LDAP utilization.

I've been watching ldap traffic (with tcpdump) while starting and stopping jobs on particular grid nodes. The Stretch grid produces about 50-60 times as many ldap requests when a job starts as the old grid.

Fri, Mar 8, 8:38 PM · cloud-services-team (Kanban), LDAP, Toolforge
Andrew added a comment to T216239: CloudVPS: drain and rebuild labvirt1009 as cloudvirt1009.

Unless is somehow disabled in BIOS, I only see this:

10Gb nics are disabled in the bios for every one of our cloudvirts that aren't already running 10Gb. Details on https://phabricator.wikimedia.org/T216195

Fri, Mar 8, 2:11 PM · Patch-For-Review, cloud-services-team (Kanban)

Thu, Mar 7

Andrew added a comment to T217838: Toolforge Stretch - Increased LDAP utilization.

I'm comparing the ldap/pam behavior of test stretch, jessie and trusty VMs. So far I don't see any significant difference.

Thu, Mar 7, 10:18 PM · cloud-services-team (Kanban), LDAP, Toolforge

Wed, Mar 6

Andrew closed T217530: Remove 2FA on wikitech.wikimedia.org for user Avicennasis as Resolved.

@Avicennasis, I have disabled 2fa on your wikitech account. Sorry for the slow response time!

Wed, Mar 6, 9:10 PM · wikitech.wikimedia.org
Andrew added a comment to T213546: Prepare puppet infrastructure for Debian buster.

If you would like to test, we now have a VPS base image for buster in wmcs. You can use it in the 'testlabs' project, or I can add it to a project of your choice.

Wed, Mar 6, 6:09 PM · Patch-For-Review, Packaging, Puppet, Operations
Andrew closed T216781: Create Buster image as Resolved.

I've merged buster patches upstream, and built a +wmf release with those patches. It's in repropro, named 'python-bootstrap-vz'. I presume that eventually we can go back to using the upstream package if it's rebuilt for future buster releases.

Wed, Mar 6, 6:07 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS

Tue, Mar 5

Andrew added a comment to T216781: Create Buster image.

I have a pending pull request for bootstrap-vz which should allow us to build without local hacks: https://github.com/andsens/bootstrap-vz/pull/496

Tue, Mar 5, 9:57 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS
Andrew moved T215012: cloudvirt1015: apparent hardware errors in CPU/Memory from Epics to Blocked on the cloud-services-team (Kanban) board.
Tue, Mar 5, 5:53 PM · Patch-For-Review, Operations, ops-eqiad, DC-Ops, cloud-services-team (Kanban)
Andrew moved T215012: cloudvirt1015: apparent hardware errors in CPU/Memory from Important to Epics on the cloud-services-team (Kanban) board.
Tue, Mar 5, 5:53 PM · Patch-For-Review, Operations, ops-eqiad, DC-Ops, cloud-services-team (Kanban)
Andrew moved T208843: WMCS - Remove unused legacy code from Important to Epics on the cloud-services-team (Kanban) board.
Tue, Mar 5, 5:52 PM · Patch-For-Review, cloud-services-team (Kanban)

Mon, Mar 4

Andrew closed T215847: eqiad1-r instances not currently tracked by shinken as Resolved.
Mon, Mar 4, 4:54 PM · Patch-For-Review, Shinken, Toolforge
Andrew added a comment to T216769: wmcs: audit old hardware.

That spreadsheet looks quite up-to-date to me; I'm not sure there's more to be done here.

Mon, Mar 4, 3:20 PM · cloud-services-team (Kanban)
Andrew reassigned T214106: Creating projects in openstack hangs on CLI from Andrew to Bstorm.
Mon, Mar 4, 3:00 PM · Patch-For-Review, Cloud-VPS, cloud-services-team (Kanban)

Fri, Mar 1

Andrew added a comment to T216041: Order spare cloudvirt SSDs for eqiad.

I'm less sure that we need drives on hand now. We seem to be able to get replacements more-or-less overnight, and adding spare drives to the RAIDs will reduce the urgency of replacement.

Fri, Mar 1, 5:19 PM · Wikimedia-Incident, cloud-services-team (Kanban)
Andrew closed T216768: cloud vps: document capacity planning strategies, a subtask of T216218: Cloud VPS outage on cloudvirt1024 and cloudvirt1018 due to storage failure, as Resolved.
Fri, Mar 1, 5:18 PM · Cloud-VPS, cloud-services-team (Kanban)
Andrew closed T216768: cloud vps: document capacity planning strategies as Resolved.
Fri, Mar 1, 5:18 PM · cloud-services-team (Kanban)
Andrew added a comment to T216768: cloud vps: document capacity planning strategies.

First pass at this is https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Virt_capacity

Fri, Mar 1, 4:58 PM · cloud-services-team (Kanban)
Andrew awarded T214160: globaleducation outreachdashboard instance has been found to be running an unsupported operating system a Barnstar token.
Fri, Mar 1, 12:39 AM · Education-Program-Dashboard, Cloud-VPS (Ubuntu Trusty Deprecation)
Andrew closed T217216: Ebernhardson accepted at wikitech, but rejected from logging into horizon as Resolved.

To resolve this, I tried to abolish the mixed-case username 'EBernhardson':

Fri, Mar 1, 12:30 AM · cloud-services-team (Kanban), Horizon
Andrew renamed T217216: Ebernhardson accepted at wikitech, but rejected from logging into horizon from User accepted at wikitech, but rejected from logging into horizon to Ebernhardson accepted at wikitech, but rejected from logging into horizon.
Fri, Mar 1, 12:25 AM · cloud-services-team (Kanban), Horizon

Thu, Feb 28

Andrew added a comment to T217216: Ebernhardson accepted at wikitech, but rejected from logging into horizon.

This turned out to be a case of T165795

Thu, Feb 28, 11:55 PM · cloud-services-team (Kanban), Horizon
Andrew renamed T216218: Cloud VPS outage on cloudvirt1024 and cloudvirt1018 due to storage failure from Cloud VPS outage on cloudvirt1025 and cloudvirt1018 due to storage failure to Cloud VPS outage on cloudvirt1024 and cloudvirt1018 due to storage failure.
Thu, Feb 28, 5:40 PM · Cloud-VPS, cloud-services-team (Kanban)
Andrew added a comment to T216218: Cloud VPS outage on cloudvirt1024 and cloudvirt1018 due to storage failure.

I am 90% sure this was cloudvirt1024 and not 1025, so changing title...

Thu, Feb 28, 5:39 PM · Cloud-VPS, cloud-services-team (Kanban)
Andrew added a project to T216724: relocate cloudvirt1024 from b8-eqiad:u24 to b2-eqiad:u17: DC-Ops.
Thu, Feb 28, 5:19 PM · DC-Ops, Operations, ops-eqiad, cloud-services-team (Kanban)
Andrew added a project to T217347: Move cloudvirt1018 to a 10G rack, connect 10G nics: DC-Ops.
  • Move host to a rack with 10G -- B2, B4 or B7 I believe
  • Enable the 10G nic in the bios
  • Move/install cables
  • Update switch config
  • Re-image
Thu, Feb 28, 5:19 PM · DC-Ops, ops-eqiad, cloud-services-team (Kanban), Operations
Andrew added projects to T217140: cloudvirt1015: update raid config and move to 10Gb: DC-Ops, ops-eqiad.
  • Move host to a rack with 10G -- B2, B4 or B7 I believe
  • Enable the 10G nic in the bios
  • Move/install cables
  • Update switch config
  • Re-image
Thu, Feb 28, 5:19 PM · ops-eqiad, DC-Ops, Operations
Andrew added a project to T217346: Move cloudvirt1012 to a 10G rack and connect 10g nics: DC-Ops.
  • Move host to a rack with 10G -- B2, B4 or B7 I believe
  • Enable the 10G nic in the bios
  • Move/install cables
  • Update switch config
  • Re-image
Thu, Feb 28, 5:17 PM · DC-Ops, Operations, ops-eqiad, cloud-services-team (Kanban)
Andrew added projects to T216661: cloudVPS: drain and rebuild labvirt1008 as cloudvirt1008: DC-Ops, ops-eqiad.

the next step is moving this to a new rack for 10G connections (either 2, 4 or 7 in row B) so I'm tagging dc-ops. You can hand it back to me for the rename/rebuild once it's in place.

Thu, Feb 28, 5:16 PM · Operations, ops-eqiad, DC-Ops, cloud-services-team (Kanban)
Andrew updated the task description for T216195: Move cloudvirt hosts to 10Gb ethernet.
Thu, Feb 28, 5:12 PM · cloud-services-team (Kanban)
Andrew updated the task description for T216195: Move cloudvirt hosts to 10Gb ethernet.
Thu, Feb 28, 5:12 PM · cloud-services-team (Kanban)
Andrew created T217347: Move cloudvirt1018 to a 10G rack, connect 10G nics.
Thu, Feb 28, 5:11 PM · DC-Ops, ops-eqiad, cloud-services-team (Kanban), Operations
Andrew updated the task description for T216195: Move cloudvirt hosts to 10Gb ethernet.
Thu, Feb 28, 5:09 PM · cloud-services-team (Kanban)
Andrew created T217346: Move cloudvirt1012 to a 10G rack and connect 10g nics.
Thu, Feb 28, 5:09 PM · DC-Ops, Operations, ops-eqiad, cloud-services-team (Kanban)
Andrew updated the task description for T216190: Rebuild labvirt1012 as cloudvirt1012.
Thu, Feb 28, 5:08 PM · cloud-services-team (Kanban), Patch-For-Review
Andrew updated the task description for T216195: Move cloudvirt hosts to 10Gb ethernet.
Thu, Feb 28, 5:07 PM · cloud-services-team (Kanban)
Andrew updated the task description for T216195: Move cloudvirt hosts to 10Gb ethernet.
Thu, Feb 28, 5:06 PM · cloud-services-team (Kanban)
Andrew updated the task description for T216195: Move cloudvirt hosts to 10Gb ethernet.
Thu, Feb 28, 5:06 PM · cloud-services-team (Kanban)

Wed, Feb 27

Andrew added a comment to T217216: Ebernhardson accepted at wikitech, but rejected from logging into horizon.

I'll pay more attention to this later, but quick drive-by thoughts:

Wed, Feb 27, 2:47 PM · cloud-services-team (Kanban), Horizon
Dalba awarded T197930: Build or backport .deb for kubernetes python client for Jessie and Trusty a Pterodactyl token.
Wed, Feb 27, 2:21 PM · cloud-services-team (Kanban), Toolforge, Kubernetes

Tue, Feb 26

GTirloni awarded T215778: Publish new Debian 9.7 images a Yellow Medal token.
Tue, Feb 26, 7:02 PM · Patch-For-Review, Cloud-VPS, cloud-services-team (Kanban)
Andrew closed T215778: Publish new Debian 9.7 images as Resolved.

Done

Tue, Feb 26, 6:58 PM · Patch-For-Review, Cloud-VPS, cloud-services-team (Kanban)
Andrew reassigned T191362: decom promethium/WMF3571 from Andrew to ayounsi.
Tue, Feb 26, 6:30 PM · decommission, Operations, DC-Ops, ops-eqiad
Andrew added a comment to T191362: decom promethium/WMF3571.

@ayounsi I assume you're talking about this?

Tue, Feb 26, 6:30 PM · decommission, Operations, DC-Ops, ops-eqiad
Andrew closed T216960: Request creation of structurednavigation VPS project as Resolved.

@SamanthaNguyen I've created this project and added you as a projectadmin. You can assign other members as needed.

Tue, Feb 26, 5:14 PM · cloud-services-team (Kanban), Cloud-VPS (Project-requests)
Andrew added a comment to T212657: Close qna VPS project once Meza functionality has been ported to its own project.

Hello! The title of this looks like an action item ('delete qna project') but I'm not clear if that's right -- are you ready for me to delete the project or are there other things that need to happen first?

Tue, Feb 26, 4:58 PM · cloud-services-team (Kanban), Cloud-VPS (Project-requests)
Andrew closed T216767: nova: Provide a simple way to disable all VM creation as Resolved.

https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Disable_VM_scheduling

Tue, Feb 26, 3:54 PM · cloud-services-team (Kanban)
Andrew closed T216767: nova: Provide a simple way to disable all VM creation, a subtask of T216218: Cloud VPS outage on cloudvirt1024 and cloudvirt1018 due to storage failure, as Resolved.
Tue, Feb 26, 3:54 PM · Cloud-VPS, cloud-services-team (Kanban)