Page MenuHomePhabricator

Request increased quota for xtools Cloud VPS project
Closed, ResolvedPublic

Description

Project Name: xtools
Type of quota increase requested: CPU/RAM
Reason:

We are in the process of upgrading XTools to run on PHP 7.2 and Symfony 4.1. The only sane way to do this without downtime is to set up a new instance running on the latest and greatest, make sure everything is working properly, then edit the web proxies to move traffic to the new server. Unfortunately we don't have enough quota to do this. So, I'm asking to temporarily give the xtools project enough quota to perform the migration, then we can delete the old instances and the quota can be reduced back to the norm.

There are two production instances: xtools-prod01 (m1.large), and xtools-prod02 (m1.medium). xtools-dev03 (m1.small) is the new test instance. I'll be deleting xtools-dev02 (m1.small) soon, meaning to perform the migration, we will need our quota to be increased by:

  • 5 VCPUs
  • 10GB RAM

If we could get up to a week's time to do this, that'd be awesome. That'd allow us to diagnose and fix bugs, and be able to switch traffic back and forth as needed to avoid downtime. We can report back here once we're satisfied everything is stable.

Thank you!

Details

Related Gerrit Patches:
operations/puppet : productionHorizon: move xtools to eqiad1-r

Event Timeline

MusikAnimal renamed this task from Request increased quota for <Replace Me> Cloud VPS project to Request increased quota for xtools Cloud VPS project.Sep 24 2018, 1:17 AM
Andrew added a subscriber: Andrew.Sep 25 2018, 4:05 PM

Hello! You might not be following this news, but we're in the process of gradually moving VMs from one cloud 'region' to another one. If you are about to tear down your existing VMs and build new ones, that's great news for me because you can save me having to move things in the future :)

So, here's what I propose: I can enable instance creation in the new region (and disable it in the old region) and duplicate your existing quota in the new region. Then you can create your new VMs in the new region (it's marked as 'eqiad1-r' in the Horizon user interface) and then delete things in the old region as appropriate.

Does that make sense/sound OK?

-Andrew

Andrew claimed this task.Sep 25 2018, 4:05 PM

Hello! You might not be following this news, but we're in the process of gradually moving VMs from one cloud 'region' to another one. If you are about to tear down your existing VMs and build new ones, that's great news for me because you can save me having to move things in the future :)
So, here's what I propose: I can enable instance creation in the new region (and disable it in the old region) and duplicate your existing quota in the new region. Then you can create your new VMs in the new region (it's marked as 'eqiad1-r' in the Horizon user interface) and then delete things in the old region as appropriate.
Does that make sense/sound OK?
-Andrew

Sounds fine and dandy! I think :) One strong desire is the ability to move traffic back and forth (between the old VMs and the new), in case something goes wrong. Will we be able to do that?

Change 462841 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move xtools to eqiad1-r

https://gerrit.wikimedia.org/r/462841

Change 462841 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move xtools to eqiad1-r

https://gerrit.wikimedia.org/r/462841

Andrew closed this task as Resolved.Sep 26 2018, 2:58 AM

OK, you should be all set -- now in Horizon you'll be permitted to create new instances in the eqiad1-r region but not in the eqiad region. All quotas and security groups should be the same in each region.

VMs can definitely communicate between the two regions, but keep in mind that the new VMs will be in a new IP range, 172.16.0.0/21. So, in order to allow traffic from the new VMs to the old ones you'll probably need to adjust security groups to include that IP range.

Izno moved this task from Blocked (Non-XTools) to Complete on the XTools board.Sep 30 2018, 5:35 PM

@Andrew Thanks!

Sorry if this is a stupid question: how do I SSH into the new instances? For the equiad region I have the following in my SSH config:

Host *.eqiad.wmflabs
  ProxyCommand ssh -a -W %h:%p musikanimal@primary.bastion.wmflabs.org

and I connect with ssh xtools-prod01.xtools.eqiad.wmflabs.

I added the same config for eqiad1-r, but:

> ssh xtools-prod01.xtools.eqiad1-r.wmflabs
channel 0: open failed: administratively prohibited: open failed
stdio forwarding failed
ssh_exchange_identification: Connection closed by remote host

The security groups on the new instances are the same as the old. It should accompany the new IPs already. Maybe we're doing this wrong or going against best practice; you can see for yourself at https://horizon.wikimedia.org/project/security_groups/d3c946b9-719c-4b8e-9ee8-d9c667cd4b16/ But anyway I'm guessing the issue with SSH'ing is not because of the security groups.

Andrew added a comment.Oct 1 2018, 6:45 PM
ssh xtools-prod01.xtools.eqiad1-r.wmflabs

Even though the VMs are hosted on a new set of hardware, the DNS is the same for both reasons. So you just want to

ssh xtools-prod01.xtools.eqiad.wmflabs

Even though the VMs are hosted on a new set of hardware, the DNS is the same for both reasons.

Gotcha. What's interesting is that it let me create new VMs with the same names. Luckily the our proxy settings (e.g. for https://xtools.wmflabs.org) didn't get confused and go to the new xtools-prod01, which was not set up yet!

I've recreated the new VMs using new names, and indeed I can SSH in now :)

Just a final note: I've got a lot of things on my plate so it may not be until next week before we are able to delete the old VMs.

bd808 added a subscriber: bd808.Oct 1 2018, 7:59 PM

What's interesting is that it let me create new VMs with the same names.

This is kind of a bug. The two regions have separate "nova" components which is the part of OpenStack that deals with allocating and managing VMs. XTools is the first project that we have allowed to be active in both regions at once. Nobody thought to warn you that you really should make sure yourself that the VM names are unique across both regions. Good to hear that at least thus far having a naming collision is not causing you any problems.

Krenair added a comment.EditedOct 1 2018, 8:00 PM

What's interesting is that it let me create new VMs with the same names.

please don't. Your project is in a special state of having changed active regions while not having migrated all existing instances. Bad things may happen if you try to make VMs in the two regions have the same name. OpenStack thinks they only need to be unique within the region but Wikimedia's setup assumes they are unique across the two (with both OpenStack regions' instances getting DNS records under eqiad.wmflabs being the example that springs to mind).

No problem! I guess it's a miracle something bad didn't happen. I've deleted all the new VMs that had preexisting names, so hopefully we are safe now. And thank you immensely for letting our project be in this special state! :)

I guess it's a miracle something bad didn't happen. I've deleted all the new VMs that had preexisting names, so hopefully we are safe now.

At first I was horrified at the truly terrible, evil bugs that you may have unleashed from pandora's box.

I have a feeling you likely just ended up with some exceptions in some of the openstack service logs and a broken instance though. Hopefully that's all it was.

MusikAnimal reopened this task as Open.EditedSep 12 2019, 9:27 PM
MusikAnimal moved this task from Approved to Inbox on the Cloud-VPS (Quota-requests) board.

@Andrew I'm going to reuse this task. Per the Debian Jessie deprecation, we need to create new instances running on Buster. Same situation as before, except this time it is in the same region so we actually will need increased quota. We'll be using the same version of PHP, etc., so there's no big risks here like there was last time. It should take me no more than an hour or so to do the full transition. I can comment here after I delete the old instances for you to remove the increased quota, if you'd like.

Thanks!

Approved, I'll help with this shortly.

@MusikAnimal, I'm temporarily doubled the ram and CPU quotas in this project. Once you've created the new VMs and deleted the old ones let me know and I'll revert the change.

(And, if you can, skip to Buster! It'll save you trouble later.)

@Andrew All set! Everything is now running on Buster, and I've deleted the old VMs. We do not need the additional quota anymore. Thanks for the help!

Andrew closed this task as Resolved.Sep 24 2019, 5:45 PM

@MusikAnimal nice work! I've revered the quota boost.