Maniphest T205232

Request increased quota for xtools Cloud VPS project
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	MusikAnimal
	Sep 24 2018, 1:08 AM

Description

Project Name: xtools
Type of quota increase requested: CPU/RAM
Reason:

We are in the process of upgrading XTools to run on PHP 7.2 and Symfony 4.1. The only sane way to do this without downtime is to set up a new instance running on the latest and greatest, make sure everything is working properly, then edit the web proxies to move traffic to the new server. Unfortunately we don't have enough quota to do this. So, I'm asking to temporarily give the xtools project enough quota to perform the migration, then we can delete the old instances and the quota can be reduced back to the norm.

There are two production instances: xtools-prod01 (m1.large), and xtools-prod02 (m1.medium). xtools-dev03 (m1.small) is the new test instance. I'll be deleting xtools-dev02 (m1.small) soon, meaning to perform the migration, we will need our quota to be increased by:

5 VCPUs
10GB RAM

If we could get up to a week's time to do this, that'd be awesome. That'd allow us to diagnose and fix bugs, and be able to switch traffic back and forth as needed to avoid downtime. We can report back here once we're satisfied everything is stable.

Thank you!

Details

	Subject	Repo	Branch	Lines +/-
	Horizon: move xtools to eqiad1-r	operations/puppet	production	+1 -1

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Invalid		None	T197804 Puppet: forbid new Python2 code
Open		None	T218426 Upgrade various Cloud VPS Python 2 scripts to Python 3
Resolved	BUG REPORT	• Bstorm	T218423 Add python 3 packages to openstack::clientpackages::common
Resolved		• MoritzMuehlenhoff	T232677 Remove support for Debian Jessie in Cloud Services
Resolved		MusikAnimal	T232787 XTools: Move Debian Jessie instances to Buster
Resolved		Andrew	T205232 Request increased quota for xtools Cloud VPS project

Event Timeline

MusikAnimal created this task.Sep 24 2018, 1:08 AM

MusikAnimal renamed this task from Request increased quota for <Replace Me> Cloud VPS project to Request increased quota for xtools Cloud VPS project.Sep 24 2018, 1:17 AM

Matthewrbowker subscribed.Sep 24 2018, 1:45 AM

Hello! You might not be following this news, but we're in the process of gradually moving VMs from one cloud 'region' to another one. If you are about to tear down your existing VMs and build new ones, that's great news for me because you can save me having to move things in the future :)

So, here's what I propose: I can enable instance creation in the new region (and disable it in the old region) and duplicate your existing quota in the new region. Then you can create your new VMs in the new region (it's marked as 'eqiad1-r' in the Horizon user interface) and then delete things in the old region as appropriate.

Does that make sense/sound OK?

-Andrew

Andrew claimed this task.Sep 25 2018, 4:05 PM

Matthewrbowker moved this task from Backlog to Blocked (Non-XTools) on the XTools board.Sep 25 2018, 4:20 PM

In T205232#4615666, @Andrew wrote:

Hello! You might not be following this news, but we're in the process of gradually moving VMs from one cloud 'region' to another one. If you are about to tear down your existing VMs and build new ones, that's great news for me because you can save me having to move things in the future :)

So, here's what I propose: I can enable instance creation in the new region (and disable it in the old region) and duplicate your existing quota in the new region. Then you can create your new VMs in the new region (it's marked as 'eqiad1-r' in the Horizon user interface) and then delete things in the old region as appropriate.

Does that make sense/sound OK?

-Andrew

Sounds fine and dandy! I think :) One strong desire is the ability to move traffic back and forth (between the old VMs and the new), in case something goes wrong. Will we be able to do that?

Krenair subscribed.Sep 25 2018, 9:54 PM

Change 462841 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: move xtools to eqiad1-r

https://gerrit.wikimedia.org/r/462841

Change 462841 merged by Andrew Bogott:
[operations/puppet@production] Horizon: move xtools to eqiad1-r

https://gerrit.wikimedia.org/r/462841

OK, you should be all set -- now in Horizon you'll be permitted to create new instances in the eqiad1-r region but not in the eqiad region. All quotas and security groups should be the same in each region.

VMs can definitely communicate between the two regions, but keep in mind that the new VMs will be in a new IP range, 172.16.0.0/21. So, in order to allow traffic from the new VMs to the old ones you'll probably need to adjust security groups to include that IP range.

Izno moved this task from Blocked (Non-XTools) to Complete on the XTools board.Sep 30 2018, 5:35 PM

@Andrew Thanks!

Sorry if this is a stupid question: how do I SSH into the new instances? For the equiad region I have the following in my SSH config:

Host *.eqiad.wmflabs
  ProxyCommand ssh -a -W %h:%p musikanimal@primary.bastion.wmflabs.org

and I connect with ssh xtools-prod01.xtools.eqiad.wmflabs.

I added the same config for eqiad1-r, but:

> ssh xtools-prod01.xtools.eqiad1-r.wmflabs
channel 0: open failed: administratively prohibited: open failed
stdio forwarding failed
ssh_exchange_identification: Connection closed by remote host

The security groups on the new instances are the same as the old. It should accompany the new IPs already. Maybe we're doing this wrong or going against best practice; you can see for yourself at https://horizon.wikimedia.org/project/security_groups/d3c946b9-719c-4b8e-9ee8-d9c667cd4b16/ But anyway I'm guessing the issue with SSH'ing is not because of the security groups.

ssh xtools-prod01.xtools.eqiad1-r.wmflabs

Even though the VMs are hosted on a new set of hardware, the DNS is the same for both reasons. So you just want to

ssh xtools-prod01.xtools.eqiad.wmflabs

Even though the VMs are hosted on a new set of hardware, the DNS is the same for both reasons.

Gotcha. What's interesting is that it let me create new VMs with the same names. Luckily the our proxy settings (e.g. for https://xtools.wmflabs.org) didn't get confused and go to the new xtools-prod01, which was not set up yet!

I've recreated the new VMs using new names, and indeed I can SSH in now :)

Just a final note: I've got a lot of things on my plate so it may not be until next week before we are able to delete the old VMs.

In T205232#4631437, @MusikAnimal wrote:

What's interesting is that it let me create new VMs with the same names.

This is kind of a bug. The two regions have separate "nova" components which is the part of OpenStack that deals with allocating and managing VMs. XTools is the first project that we have allowed to be active in both regions at once. Nobody thought to warn you that you really should make sure yourself that the VM names are unique across both regions. Good to hear that at least thus far having a naming collision is not causing you any problems.

In T205232#4631437, @MusikAnimal wrote:

What's interesting is that it let me create new VMs with the same names.

please don't. Your project is in a special state of having changed active regions while not having migrated all existing instances. Bad things may happen if you try to make VMs in the two regions have the same name. OpenStack thinks they only need to be unique within the region but Wikimedia's setup assumes they are unique across the two (with both OpenStack regions' instances getting DNS records under eqiad.wmflabs being the example that springs to mind).

Krenair awarded a token.Oct 1 2018, 8:03 PM

No problem! I guess it's a miracle something bad didn't happen. I've deleted all the new VMs that had preexisting names, so hopefully we are safe now. And thank you immensely for letting our project be in this special state! :)

In T205232#4631526, @MusikAnimal wrote:

I guess it's a miracle something bad didn't happen. I've deleted all the new VMs that had preexisting names, so hopefully we are safe now.

At first I was horrified at the truly terrible, evil bugs that you may have unleashed from pandora's box.

I have a feeling you likely just ended up with some exceptions in some of the openstack service logs and a broken instance though. Hopefully that's all it was.

bd808 moved this task from Inbox to Approved on the Cloud-VPS (Quota-requests) board.Feb 5 2019, 4:49 PM

@Andrew I'm going to reuse this task. Per the Debian Jessie deprecation, we need to create new instances running on Buster. Same situation as before, except this time it is in the same region so we actually will need increased quota. We'll be using the same version of PHP, etc., so there's no big risks here like there was last time. It should take me no more than an hour or so to do the full transition. I can comment here after I delete the old instances for you to remove the increased quota, if you'd like.

Thanks!

MusikAnimal added a parent task: T232787: XTools: Move Debian Jessie instances to Buster.Sep 12 2019, 9:28 PM

Maintenance_bot removed a project: Patch-For-Review.Sep 12 2019, 10:10 PM

Approved, I'll help with this shortly.

@MusikAnimal, I'm temporarily doubled the ram and CPU quotas in this project. Once you've created the new VMs and deleted the old ones let me know and I'll revert the change.

(And, if you can, skip to Buster! It'll save you trouble later.)

aborrero moved this task from Inbox to Approved on the Cloud-VPS (Quota-requests) board.Sep 24 2019, 11:13 AM

@Andrew All set! Everything is now running on Buster, and I've deleted the old VMs. We do not need the additional quota anymore. Thanks for the help!

@MusikAnimal nice work! I've revered the quota boost.

Re-using the same task again, same situation, same request :) We need to upgrade to Debian Bullseye and will need the quota for the xtools project to be temporarily increased by 5 VCPUs and 10GB RAM.

I will note that this time we're deploying major changes to the codebase, so I will likely want additional time to ensure everything is running smoothly before deleting the old VMs, perhaps 2 weeks or so. I assume this isn't a problem.

Another quick question: Will the old wmflabs.org domain continue to automatically redirect to wmcloud.org? Because I see I can't create new web proxies with wmflabs.org. I was thinking just to be safe, I could keep the xtools.wmflabs.org proxy and change it to point to the new VM, and add the additional <ServerName> direction to the Apache config. But, if we are certain wmflabs.org will more or less always automatically redirect to wmcloud.org, then I won't need to do that. It's just that xtools.wmflabs.org is linked to from a bajillion places so I want to make sure that continues to work :)

Thanks as always!

MusikAnimal moved this task from Complete to General / other on the XTools board.Mar 31 2023, 6:43 PM

MusikAnimal mentioned this in T333442: Puppetize all our WMCS VMs.Mar 31 2023, 9:11 PM

+1 sgtm

Old .wmflabs.org domains will continue to redirect but please move your users to the new domains when possible. Eventually we will stop noticing/maintaining the old domain.

Mentioned in SAL (#wikimedia-cloud) [2023-04-03T19:45:31Z] <taavi> bump quotas by 5 VCPUs and 10GB RAM T205232

In T205232#8746668, @MusikAnimal wrote:

Re-using the same task again, same situation, same request :) We need to upgrade to Debian Bullseye and will need the quota for the xtools project to be temporarily increased by 5 VCPUs and 10GB RAM.

Please create a new task next time, makes it easier to track it on our end. Anyways, done, please let us know when it can be reverted.

Another quick question: Will the old wmflabs.org domain continue to automatically redirect to wmcloud.org? Because I see I can't create new web proxies with wmflabs.org. I was thinking just to be safe, I could keep the xtools.wmflabs.org proxy and change it to point to the new VM, and add the additional <ServerName> direction to the Apache config. But, if we are certain wmflabs.org will more or less always automatically redirect to wmcloud.org, then I won't need to do that. It's just that xtools.wmflabs.org is linked to from a bajillion places so I want to make sure that continues to work :)

What Andrew said so yes, if there is no proxy for a wmflabs.org subdomain defined it will redirect to the wmcloud.org proxy with the same name at least for the next while. We would like to eventually drop wmflabs.org but that's not going to happen overnight.

MusikAnimal moved this task from General / other to Complete on the XTools board.Apr 4 2023, 10:16 PM

@taavi The old VMs have now been deleted, so the additional quota given can be reverted. Thanks! (Not sure if a simple comment is all you wanted or a new task… but acknowledging that requesting additional quota moving forward will be with a new task!)

Reverted, thanks!

Request increased quota for xtools Cloud VPS projectClosed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

Request increased quota for xtools Cloud VPS project
Closed, ResolvedPublic
Actions

Related Objects
Search...