Page MenuHomePhabricator

Request increased quota for mwoffliner Cloud VPS project
Closed, ResolvedPublic

Description

Project Name: mwoffliner
Type of quota increase requested: <8 cores/18GB/1/1/200GB of storage>
Reason:

The WP1 tools have 4 parts:

This tools are fundamental for the ENWP community, work together, need relatively robust HW resources to run properly. The openZIM/Kiwix relies also a lot on this tools and maintain them... and is currently actively working on them to improve their overall reliability.

As we want to have them all running together, have remarked that in many cases the Toolforge system seemed to suffer from instability (at least by running our - quite old - solutions), we require to be able to run them all on an adequate box, beside the 5 we already have to scrape Wikimedia projects. The fact that we have to leave the old Ubuntu trusty toolforge immedialy (deadline is next Monday) make us think that this is an opportunity to do that move.

Event Timeline

Kelson created this task.Mar 23 2019, 6:57 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 23 2019, 6:57 PM

FYI the schedule is that these requests get reviewed on Tuesdays, so it might be too late for a toolforge trusty move without downtime.

@Krenair Yes, we fully understand that and this is our fault. We will have to deal with a few days downtime.

Kelson added a subscriber: bd808.Mar 24 2019, 7:51 AM
bd808 added a comment.Mar 24 2019, 9:46 PM

Type of quota increase requested: <8 cores/18GB/1/1/200GB of storage>

These specs do not match any standard instance type. The closest would be an m1.xlarge instance which needs 8 cores and 16GB RAM and can allocate up to 160GB of disk.

Your stated goal is to replace the processes which have been running as the enwp10 tool with a self-managed instance. This is your prerogative, but I am a bit taken aback by the resources that you believe are needed to replace that tool. The tool seems to be running 2 continuous grid jobs and one lighttpd webservice. Are these jobs really so resource intensive as to justify the permanent consumption of the equivalent of two Toolforge grid engine exec hosts?

I would think that a single m1.medium instance (4 cores, 8GB ram, 80GB storage) would provide enough ram and cpu to meet or exceed the needs of the current job load.

Kelson added a comment.EditedMar 25 2019, 9:23 AM

@bd808 I think a m1.xlarge might be enough for now, but not so sure how long it will last. Your analysis is good but you are missing an information. The jobs which need the most memory are not running on the grid. The datamining jobs were running on the pre-existing mwoffliner VPS boxes, but as we use these boxes more intensively than before, I want (or have) to stop to do so.

bd808 claimed this task.Mar 26 2019, 4:50 PM
bd808 closed this task as Resolved.Mar 26 2019, 8:33 PM

I have added enough quota in the eqiad1-r region to allow you to provision an additional m1.xlarge instance.

$ openstack quota show mwoffliner
+----------------------+------------+
| Field                | Value      |
+----------------------+------------+
| cores                | 36         |
| fixed-ips            | 200        |
| floating-ips         | 3          |
| injected-file-size   | 10240      |
| injected-files       | 5          |
| injected-path-size   | 255        |
| instances            | 8          |
| key-pairs            | 100        |
| network              | 10         |
| port                 | 50         |
| project              | mwoffliner |
| properties           | 128        |
| ram                  | 73728      |
| rbac_policy          | 10         |
| router               | 10         |
| secgroup-rules       | 100        |
| secgroups            | 40         |
| server_group_members | 10         |
| server_groups        | 10         |
| subnet               | 10         |
| subnetpool           | -1         |
+----------------------+------------+
$ openstack quota set --cores $(( 36 + 8 )) --ram $(( 73728 + 16384 )) mwoffliner
$ openstack quota show mwoffliner
+----------------------+------------+
| Field                | Value      |
+----------------------+------------+
| cores                | 44         |
| fixed-ips            | 200        |
| floating-ips         | 3          |
| injected-file-size   | 10240      |
| injected-files       | 5          |
| injected-path-size   | 255        |
| instances            | 8          |
| key-pairs            | 100        |
| network              | 10         |
| port                 | 50         |
| project              | mwoffliner |
| properties           | 128        |
| ram                  | 90112      |
| rbac_policy          | 10         |
| router               | 10         |
| secgroup-rules       | 100        |
| secgroups            | 40         |
| server_group_members | 10         |
| server_groups        | 10         |
| subnet               | 10         |
| subnetpool           | -1         |
+----------------------+------------+

@bd808 Thx, I have successfully created the new WP1 instance.