Page MenuHomePhabricator

Andrew (Andrew Bogott)
User

Projects (12)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Saturday

  • Clear sailing ahead.

User Details

User Since
Nov 2 2014, 11:35 PM (339 w, 3 d)
Availability
Available
IRC Nick
andrewbogott
LDAP User
Unknown
MediaWiki User
Andrewbogott [ Global Accounts ]

Recent Activity

Today

Andrew added a comment to T278390: Toolforge root for Majavah.
  • Gerrit group "toollabs-trusted": This allows a user to +2 changes in repositories exclusive to Toolforge.
Thu, May 6, 2:48 AM · cloud-services-team (Kanban), Toolforge
Andrew added a comment to T278390: Toolforge root for Majavah.
  • 'admin' tool maintainer: This allows a user to log into infrastructure instances and perform tasks as the admin tool.
Thu, May 6, 2:39 AM · cloud-services-team (Kanban), Toolforge
Andrew added a comment to T278390: Toolforge root for Majavah.

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T19:27:03Z] <andrewbogott> adding taavi as a sudo root to project toolforge for T278390

Thu, May 6, 2:37 AM · cloud-services-team (Kanban), Toolforge

Yesterday

Andrew added a comment to T278390: Toolforge root for Majavah.
[x] Project administrator: This allows a user to add and delete other users from the Toolforge project.
Wed, May 5, 7:24 PM · cloud-services-team (Kanban), Toolforge
Andrew added a comment to T278390: Toolforge root for Majavah.
  • Project administrator: This allows a user to add and delete other users from the Toolforge project.
  • sudo policy "roots": This allows a user to use sudo to become root on Toolforge instances.
  • 'admin' tool maintainer: This allows a user to log into infrastructure instances and perform tasks as the admin tool.
  • Gerrit group "toollabs-trusted": This allows a user to +2 changes in repositories exclusive to Toolforge.
Wed, May 5, 7:20 PM · cloud-services-team (Kanban), Toolforge
nskaggs awarded T278390: Toolforge root for Majavah a Party Time token.
Wed, May 5, 7:14 PM · cloud-services-team (Kanban), Toolforge
Andrew added a comment to T281045: labstore1007 crashed after storage controller errors--replace disk?.

Just to clarify: we (the wmcs team) are in agreement that we should spend the money and buy a new drive. No objections if y'all want to mess with firmware upgrades first.

Wed, May 5, 3:52 PM · SRE, ops-eqiad, cloud-services-team (Hardware)
Andrew added a comment to T281907: inadequate ptr record cleanup for 185.15.56.18/185.15.56.19.

OK, this is confusing!

Wed, May 5, 2:29 PM · Patch-For-Review, cloud-services-team (Kanban)
Andrew added a comment to T281514: Cinder API no longer provides used quotas to novaobserver/openstack-browser.

Thanks @Majavah !

Wed, May 5, 1:57 PM · User-Majavah, Cloud-VPS, cloud-services-team (Kanban)
Andrew updated the task description for T281399: cloudvirt1040 primary NIC disconnected.
Wed, May 5, 12:08 AM · SRE, ops-eqiad, cloud-services-team (Kanban)

Tue, May 4

Andrew added a comment to T281399: cloudvirt1040 primary NIC disconnected.

this server isn't in use currently, you're welcome to reboot it or shut it down as part of troubleshooting.

Tue, May 4, 11:16 PM · SRE, ops-eqiad, cloud-services-team (Kanban)
Andrew reassigned T281907: inadequate ptr record cleanup for 185.15.56.18/185.15.56.19 from Andrew to aborrero.

@arturo, please lmk if there's an issue here, otherwise feel free to close this.

Tue, May 4, 8:29 PM · Patch-For-Review, cloud-services-team (Kanban)
Andrew added a comment to T281907: inadequate ptr record cleanup for 185.15.56.18/185.15.56.19.

I actually see all of those set as forward records in Horizon, so I'm not sure there's an actual bug here.

Tue, May 4, 8:24 PM · Patch-For-Review, cloud-services-team (Kanban)
Andrew created T281907: inadequate ptr record cleanup for 185.15.56.18/185.15.56.19.
Tue, May 4, 5:38 PM · Patch-For-Review, cloud-services-team (Kanban)

Mon, May 3

Andrew added a comment to T281655: Investigate, adjust default access policies for Trove and trove-dashboard.

Upstream trove-dashboard bug: https://bugs.launchpad.net/trove-dashboard/+bug/1926929

Mon, May 3, 2:10 AM · Patch-For-Review, cloud-services-team (Kanban), Data-Services, Cloud-VPS, User-Marostegui

Sun, May 2

Andrew created T281655: Investigate, adjust default access policies for Trove and trove-dashboard.
Sun, May 2, 10:13 PM · Patch-For-Review, cloud-services-team (Kanban), Data-Services, Cloud-VPS, User-Marostegui
Andrew created T281654: Trove-dashboard: web app crashes when a db instance is deleted..
Sun, May 2, 10:11 PM · cloud-services-team (Kanban), Data-Services, Cloud-VPS, User-Marostegui
Andrew added a comment to T212595: [Feature request] Database as a Service (Trove) for Cloud VPS projects.

From irc:

Sun, May 2, 10:10 PM · Patch-For-Review, cloud-services-team (Kanban), Data-Services, Cloud-VPS, User-Marostegui
Andrew closed T281384: wmcs.openstack.eqiad: nova.exception.TooOldComputeService after upgrade to victoria as Resolved.
Sun, May 2, 10:09 PM · Patch-For-Review, cloud-services-team (Kanban)
Andrew reassigned T281514: Cinder API no longer provides used quotas to novaobserver/openstack-browser from Andrew to Majavah.
Sun, May 2, 10:08 PM · User-Majavah, Cloud-VPS, cloud-services-team (Kanban)

Thu, Apr 29

Andrew added a comment to T281514: Cinder API no longer provides used quotas to novaobserver/openstack-browser.

I can't entirely tell when this changed, but there's a 'usage' flag you can pass in to the query that gets usage:

Thu, Apr 29, 8:05 PM · User-Majavah, Cloud-VPS, cloud-services-team (Kanban)
Andrew claimed T281514: Cinder API no longer provides used quotas to novaobserver/openstack-browser.
Thu, Apr 29, 6:26 PM · User-Majavah, Cloud-VPS, cloud-services-team (Kanban)
Andrew added a comment to R2073:eff36cadc01b: Load network quota information from Neutron.

I'm excited to see this! It's been a long (multi-year) road.

Thu, Apr 29, 6:10 PM

Wed, Apr 28

Andrew added a comment to T276961: Support Openstack Swift APIs via the radosgw.

I recommend we look at erasure coding the pools for Swift, since I don't know if I've said it directly related to this yet https://ceph.io/planet/erasure-coding-in-ceph/

I think that's pretty simple -- radosgw creates the pools but you can convert them to erasure after the fact. Do you have an opinion about what ratio we should use?

Wed, Apr 28, 10:58 PM · cloud-services-team (Kanban), Data-Services, Cloud-VPS, User-Marostegui
Andrew added a comment to T281384: wmcs.openstack.eqiad: nova.exception.TooOldComputeService after upgrade to victoria.

The second warning was due to cloudvirt1038 being offline, having most recently run an old version, and nova obsessing over it. Unfortuantely, deleting the service didn't cause any of the services to refresh so I spent a ton of time digging into this before realizing that I just needed to restart all the conductor services.

Wed, Apr 28, 10:02 PM · Patch-For-Review, cloud-services-team (Kanban)
Andrew closed T261137: upgrade cloud-vps openstack to Openstack version 'Victoria' as Resolved.
  • update IRC topic
  • downtime everything in icinga through 14:00CDT
Wed, Apr 28, 9:23 PM · Patch-For-Review, cloud-services-team (Kanban)
Andrew closed T261137: upgrade cloud-vps openstack to Openstack version 'Victoria', a subtask of T211777: Can't get quota information from Neutron API, as Resolved.
Wed, Apr 28, 9:23 PM · cloud-services-team (Kanban), Upstream, Cloud-VPS
Andrew added a comment to T281384: wmcs.openstack.eqiad: nova.exception.TooOldComputeService after upgrade to victoria.

These two warnings seem to be unrelated.

Wed, Apr 28, 8:07 PM · Patch-For-Review, cloud-services-team (Kanban)
Andrew updated the task description for T281384: wmcs.openstack.eqiad: nova.exception.TooOldComputeService after upgrade to victoria.
Wed, Apr 28, 8:05 PM · Patch-For-Review, cloud-services-team (Kanban)
Andrew added a comment to T281399: cloudvirt1040 primary NIC disconnected.

...and it's offline again

Wed, Apr 28, 7:56 PM · SRE, ops-eqiad, cloud-services-team (Kanban)
Andrew added a comment to T281399: cloudvirt1040 primary NIC disconnected.

this recovered after a reboot. We'll see if it holds...

Wed, Apr 28, 7:40 PM · SRE, ops-eqiad, cloud-services-team (Kanban)
Andrew updated the task description for T281399: cloudvirt1040 primary NIC disconnected.
Wed, Apr 28, 6:34 PM · SRE, ops-eqiad, cloud-services-team (Kanban)
Andrew created T281399: cloudvirt1040 primary NIC disconnected.
Wed, Apr 28, 6:32 PM · SRE, ops-eqiad, cloud-services-team (Kanban)
Andrew added a comment to T275081: (Need By: TBD) rack/setup/install cloudvirt104[0-6].eqiad.wmnet.

These servers are now installed and running; 1040, 1041 and 1042 are now active in the 'ceph' host aggregate, the others are held back as spares for now.

Wed, Apr 28, 6:12 PM · Patch-For-Review, SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
Andrew claimed T281384: wmcs.openstack.eqiad: nova.exception.TooOldComputeService after upgrade to victoria.

This might be addressed by 'nova-manage cell_v2 discover_hosts' which I just ran

Wed, Apr 28, 4:30 PM · Patch-For-Review, cloud-services-team (Kanban)
Andrew added a comment to T276961: Support Openstack Swift APIs via the radosgw.

We discussed this in our weekly meeting today. Issues to sort out before radosgw goes in eqiad1 are:

Wed, Apr 28, 4:20 PM · cloud-services-team (Kanban), Data-Services, Cloud-VPS, User-Marostegui
Andrew renamed T276961: Support Openstack Swift APIs via the radosgw from Swift before Trove? to Support Openstack Swift APIs via the radosgw.
Wed, Apr 28, 4:19 PM · cloud-services-team (Kanban), Data-Services, Cloud-VPS, User-Marostegui
Andrew added a comment to T212595: [Feature request] Database as a Service (Trove) for Cloud VPS projects.

Next topic: Where does Trove get its docker images? We should get them from key.io or self-host.

Wed, Apr 28, 3:57 PM · Patch-For-Review, cloud-services-team (Kanban), Data-Services, Cloud-VPS, User-Marostegui
Andrew moved T212595: [Feature request] Database as a Service (Trove) for Cloud VPS projects from Inbox to Needs discussion on the cloud-services-team (Kanban) board.
Wed, Apr 28, 3:34 PM · Patch-For-Review, cloud-services-team (Kanban), Data-Services, Cloud-VPS, User-Marostegui
Andrew closed T280514: novafullstack: investigate leftover VMs as Resolved.

We've made it through a couple of days without a leak, so I think at least the main cause of these is resolved by the above patch. We'll see if it holds!

Wed, Apr 28, 1:24 PM · Patch-For-Review, cloud-services-team (Kanban)

Tue, Apr 27

Andrew added a comment to T280792: Investigate Openstack Magnum.

The packages appear to be available. Next step would be to try it out.

Tue, Apr 27, 9:40 PM · cloud-services-team (Kanban)
Andrew added a comment to T280792: Investigate Openstack Magnum.

Magnum is an API and a conductor that interacts with Heat which is, itself, another API and another conductor. At first glance I don't think these will be a lot harder to set up than Trove, but I need to confirm package availability.

Tue, Apr 27, 9:37 PM · cloud-services-team (Kanban)
Andrew created T281276: Upgrade cloud-vps openstack hosts to Debian 'Bullseye'.
Tue, Apr 27, 2:46 PM · Cloud-VPS, cloud-services-team (Kanban)
Andrew created T281275: upgrade cloud-vps openstack to Openstack version 'Wallaby'.
Tue, Apr 27, 2:45 PM · Cloud-VPS, cloud-services-team (Kanban)
Andrew added a comment to T261137: upgrade cloud-vps openstack to Openstack version 'Victoria'.

https://etherpad.wikimedia.org/p/eqiad1_victoria_upgrade

Tue, Apr 27, 1:56 PM · Patch-For-Review, cloud-services-team (Kanban)
Andrew updated the task description for T261137: upgrade cloud-vps openstack to Openstack version 'Victoria'.
Tue, Apr 27, 1:55 PM · Patch-For-Review, cloud-services-team (Kanban)

Mon, Apr 26

Andrew added a comment to T280801: Cloud VPS pre-release Debian Bullseye images.

Moritz thinks that this will be fixed in the daily builds at the end of this week.

Mon, Apr 26, 8:58 PM · cloud-services-team (Kanban), Cloud-VPS
Andrew added a comment to T280801: Cloud VPS pre-release Debian Bullseye images.

^ is resolved, now I'm blocked by the fact that puppet won't run on Bullseye at all.

Mon, Apr 26, 8:54 PM · cloud-services-team (Kanban), Cloud-VPS
Andrew added a comment to T276922: cloudvirt1038: PCIe error.

Thanks for the update! I agree that we should leave this in Dell's hands as much as possible. I'm not impatient to have it back online so do whatever you need to to get them to take responsibility.

Mon, Apr 26, 4:20 PM · SRE, ops-eqiad, DC-Ops, cloud-services-team (Hardware)
Legoktm awarded T278390: Toolforge root for Majavah a Like token.
Mon, Apr 26, 3:49 PM · cloud-services-team (Kanban), Toolforge

Sun, Apr 25

Andrew committed rLPRId0e954dfe101: dummy secrets for radosgw (authored by Andrew).
dummy secrets for radosgw
Sun, Apr 25, 2:46 PM

Sat, Apr 24

Andrew added a comment to T280801: Cloud VPS pre-release Debian Bullseye images.

Whoever/whatever populates https://cdimage.debian.org/cdimage/openstack/ is not building Bullseye images; the 'testing' directory only contains stale Buster builds.

Sat, Apr 24, 10:16 PM · cloud-services-team (Kanban), Cloud-VPS
Andrew closed T226647: nova-fullstack crashed from a keystone timeout as Resolved.

I don't think this is happening anymore; closing in favor of T280514

Sat, Apr 24, 8:19 PM · cloud-services-team (Kanban)
Andrew added a comment to T276961: Support Openstack Swift APIs via the radosgw.

I don't think we need to actually run swift in wmcs; radosgw supports swift APIs.

Sat, Apr 24, 2:04 PM · cloud-services-team (Kanban), Data-Services, Cloud-VPS, User-Marostegui

Fri, Apr 23

Andrew added a comment to T263680: Learn about Barbican.

I've worked more with the horizon UI but still don't have it working. It appears to be largely abandoned so to get things in shape I would probably need to adopt the project, at least temporarily.

Fri, Apr 23, 5:50 PM · Patch-For-Review, cloud-services-team (Kanban)
Andrew added a comment to T278390: Toolforge root for Majavah.

@nskaggs, any update on the legal/NDA situation here?

Fri, Apr 23, 5:48 PM · cloud-services-team (Kanban), Toolforge
Andrew updated subscribers of T212595: [Feature request] Database as a Service (Trove) for Cloud VPS projects.

Trove is now up and running in codfw1dev. Going to pause development for now until I get some user feedback from @Bstorm

Fri, Apr 23, 5:47 PM · Patch-For-Review, cloud-services-team (Kanban), Data-Services, Cloud-VPS, User-Marostegui
Andrew closed T280835: Wrong version of Trove (13) in Victoria osbpo, a subtask of T212595: [Feature request] Database as a Service (Trove) for Cloud VPS projects, as Resolved.
Fri, Apr 23, 5:46 PM · Patch-For-Review, cloud-services-team (Kanban), Data-Services, Cloud-VPS, User-Marostegui
Andrew closed T280835: Wrong version of Trove (13) in Victoria osbpo as Resolved.

All set now thanks to Zigo and @Bstorm

Fri, Apr 23, 5:46 PM · cloud-services-team (Kanban), Data-Services, Cloud-VPS, User-Marostegui
Andrew updated the task description for T261137: upgrade cloud-vps openstack to Openstack version 'Victoria'.
Fri, Apr 23, 2:43 PM · Patch-For-Review, cloud-services-team (Kanban)
Andrew committed rLPRI6c7119236e74: Dummy passwords for Trove service user (authored by Andrew).
Dummy passwords for Trove service user
Fri, Apr 23, 1:38 PM

Wed, Apr 21

Andrew added a comment to T280835: Wrong version of Trove (13) in Victoria osbpo.

After a quick chat with Zio, version 14 is now available in the debian repo. I don't immediately know how to pull those into our repo.

Wed, Apr 21, 7:11 PM · cloud-services-team (Kanban), Data-Services, Cloud-VPS, User-Marostegui
Andrew created T280835: Wrong version of Trove (13) in Victoria osbpo.
Wed, Apr 21, 6:48 PM · cloud-services-team (Kanban), Data-Services, Cloud-VPS, User-Marostegui
Andrew added a comment to T274945: (Need By: TBD) rack/setup/install cloudcephosd10[16-20].eqiad.wmnet.

The secondary IPs are set by puppet and are on a subnet that's separate from anything managed by netbox so I think we're all good... I'll see about getting a node online.

Wed, Apr 21, 4:05 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
Andrew created T280792: Investigate Openstack Magnum.
Wed, Apr 21, 2:18 PM · cloud-services-team (Kanban)
Andrew added a comment to T274945: (Need By: TBD) rack/setup/install cloudcephosd10[16-20].eqiad.wmnet.

The secondary IPs are set by puppet and are on a subnet that's separate from anything managed by netbox so I think we're all good... I'll see about getting a node online.

Wed, Apr 21, 1:36 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops

Tue, Apr 20

Andrew added a comment to T280310: communication on port 5672 (amqp) between guest instances and cloudcontrols.

it works! thanks @aborrero

Tue, Apr 20, 8:05 PM · Patch-For-Review, cloud-services-team (Kanban), Data-Services, Cloud-VPS, User-Marostegui
Andrew added a comment to T274945: (Need By: TBD) rack/setup/install cloudcephosd10[16-20].eqiad.wmnet.

I have updated the networking request yet again -- with luck I got it right this time. For reference it's probably worth comparing with existing cloudcephosd1001-1015 on netbox; these are intended to be part of the same cluster.

Tue, Apr 20, 7:43 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
Andrew updated the task description for T274945: (Need By: TBD) rack/setup/install cloudcephosd10[16-20].eqiad.wmnet.
Tue, Apr 20, 7:42 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
Andrew reopened T274945: (Need By: TBD) rack/setup/install cloudcephosd10[16-20].eqiad.wmnet as "Open".

These need to be re-imaged with internal IPs and names in .eqiad.wmnet. Sorry for the confusion!

Tue, Apr 20, 4:45 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
Andrew updated the task description for T274945: (Need By: TBD) rack/setup/install cloudcephosd10[16-20].eqiad.wmnet.
Tue, Apr 20, 4:44 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
Andrew committed rOHPU77a6b2eac1e8: firewall: cloud-in4: new TCP port in cloudcontrol servers for rabbitmq (authored by aborrero).
firewall: cloud-in4: new TCP port in cloudcontrol servers for rabbitmq
Tue, Apr 20, 4:16 PM
Andrew added a comment to T274945: (Need By: TBD) rack/setup/install cloudcephosd10[16-20].eqiad.wmnet.

I think that our very first initial ceph cluster was in .wikimedia.org and that might have accidentally carried over.

Tue, Apr 20, 3:42 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
Andrew added a comment to T274945: (Need By: TBD) rack/setup/install cloudcephosd10[16-20].eqiad.wmnet.

They should definitely be .eqiad.wmnet, these are meant to resemble existing osd nodes. There might be some broken copypasta in the task description; I think that our very first initial ceph cluster was in .wikimedia.org and that might have accidentally carried over.

Tue, Apr 20, 3:40 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
Andrew added a comment to T280641: ceph: Upgrade to latest Nautilus/Octopus to fix CVE-2021-20288.

Currently we can't build fresh compute nodes because of mixed dependencies:

Tue, Apr 20, 3:34 PM · Patch-For-Review, cloud-services-team (Kanban)
Andrew closed T252831: cloudvirt ceph nodes can't launch new VMs as Resolved.

everything that was going to move is moved

Tue, Apr 20, 3:12 PM · cloud-services-team (Kanban)

Fri, Apr 16

Andrew reassigned T280310: communication on port 5672 (amqp) between guest instances and cloudcontrols from Andrew to aborrero.

Probably the easiest place to test this is is on the VM abogott-victoria-packages.andrewtestproject.codfw1dev.wikimedia.cloud.

Fri, Apr 16, 3:20 AM · Patch-For-Review, cloud-services-team (Kanban), Data-Services, Cloud-VPS, User-Marostegui
Andrew created T280310: communication on port 5672 (amqp) between guest instances and cloudcontrols.
Fri, Apr 16, 3:19 AM · Patch-For-Review, cloud-services-team (Kanban), Data-Services, Cloud-VPS, User-Marostegui

Wed, Apr 14

Andrew assigned T280103: Request increased quota for library-upgrader Cloud VPS project to aborrero.

+1 from me

Wed, Apr 14, 3:32 PM · LibUp, Cloud-VPS (Quota-requests)
Andrew assigned T279956: Request increased quota for globaleducation Cloud VPS project (multi-node) to aborrero.

+1 from me

Wed, Apr 14, 3:32 PM · Cloud-VPS (Quota-requests)
Andrew assigned T279246: Request creation of commons-corruption-checker VPS project (2) to aborrero.
Wed, Apr 14, 3:31 PM · cloud-services-team (Kanban), Cloud-VPS (Project-requests)
Andrew moved T279870: Request creation of masz VPS project from Inbox to Feedback needed on the Cloud-VPS (Project-requests) board.
Wed, Apr 14, 2:53 PM · Anti-Harassment, Cloud-VPS (Project-requests)
Andrew added a comment to T280046: horizon: creating a server group complains about insufficient permissions.

This is a result of hitting the server group quota for the project -- the default quota is 10 groups. Horizon doesn't seem to be aware of the quotas so it passes the request on to the nova api which fails due to the quota limit.

Wed, Apr 14, 3:15 AM · cloud-services-team (Kanban), Horizon

Tue, Apr 13

Andrew added a comment to T276018: Investigate new roles and policies in openstack train.

I just overheard people in the Horizon irc channel talking about future support for scoped roles, so we probably don't want to adopt them just yet.

Tue, Apr 13, 7:11 PM · cloud-services-team (Kanban), Cloud-VPS
Andrew added a comment to T278723: ORES deployment - Spring 2021.

this may or may not be the issue but I'd encourage you to change your hosts list to use .wikimedia.cloud names rather than .wmflabs; any new or rebuilt hosts won't exist under .wmflabs

Tue, Apr 13, 6:04 PM · artificial-intelligence, drafttopic-modeling, articlequality-modeling, ORES, Machine-Learning-Team (Active Tasks)
Andrew closed T266777: integration instances suffer from high IO latency due to Ceph as Resolved.
Tue, Apr 13, 4:36 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS, Release-Engineering-Team (CI & Testing services), Continuous-Integration-Infrastructure
Andrew closed T266777: integration instances suffer from high IO latency due to Ceph, a subtask of T265615: Terminating MySQL takes several minutes in (Wikibase?) CI jobs, as Resolved.
Tue, Apr 13, 4:35 PM · Patch-For-Review, Release-Engineering-Team (CI & Testing services), Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), Continuous-Integration-Infrastructure, Wikidata, Quibble
Andrew moved T267683: Move PAWS to it's own database (and away from ToolsDB) from Inbox to Blocked on the cloud-services-team (Kanban) board.
Tue, Apr 13, 4:34 PM · cloud-services-team (Kanban), PAWS
Andrew moved T269211: Convert labsdb1012 from multi-source to multi-instance from Inbox to Watching on the cloud-services-team (Kanban) board.
Tue, Apr 13, 4:34 PM · Analytics-Kanban, cloud-services-team (Kanban), Data-Services, DBA, Patch-For-Review, Analytics-Clusters
Andrew closed T278845: cinder: Upgrade to same in codfw than eqiad as Invalid.

Since we've done U and V upgrades on the codfw1dev boxes this shouldn't matter anymore

Tue, Apr 13, 4:30 PM · cloud-services-team (Kanban)
Andrew moved T279106: Establish replacement for PodPresets in Toolforge Kubernetes from Inbox to Soon! on the cloud-services-team (Kanban) board.
Tue, Apr 13, 4:26 PM · Toolforge, cloud-services-team (Kanban)
Andrew lowered the priority of T279687: [ceph,puppet] review the classes and remove any non-used ones from High to Medium.
Tue, Apr 13, 4:25 PM · cloud-services-team (Kanban)
Andrew moved T279687: [ceph,puppet] review the classes and remove any non-used ones from Inbox to Doing on the cloud-services-team (Kanban) board.
Tue, Apr 13, 4:24 PM · cloud-services-team (Kanban)
Andrew triaged T277078: Support Cinder for CI docker workers as Medium priority.
Tue, Apr 13, 4:22 PM · Release-Engineering-Team (Seen), Continuous-Integration-Infrastructure, Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS
Andrew closed T277319: Glance-image backup failure on cloudcontrol1005 as Resolved.

This is covered by other competing tickets :)

Tue, Apr 13, 4:21 PM · cloud-services-team (Kanban)
Andrew triaged T277542: toolschecker: naming refresh as Low priority.
Tue, Apr 13, 4:21 PM · cloud-services-team (Kanban)
Andrew triaged T277778: Toolforge: consider decoupling user & accounts from CloudVPS accounts as Medium priority.
Tue, Apr 13, 4:20 PM · Toolforge, cloud-services-team (Kanban)
Andrew triaged T277549: neutron: investigate using IRC conntrack helpers to improve IRC bots connectiviy as Low priority.
Tue, Apr 13, 4:20 PM · cloud-services-team (Kanban)
Andrew moved T277778: Toolforge: consider decoupling user & accounts from CloudVPS accounts from Inbox to Graveyard on the cloud-services-team (Kanban) board.
Tue, Apr 13, 4:20 PM · Toolforge, cloud-services-team (Kanban)
Andrew triaged T278017: import posts from 'Clouds & Unicorns' into Wikimedia Tech Blog as Medium priority.
Tue, Apr 13, 4:19 PM · cloud-services-team (Kanban), Technical blog