# Custom Flavour for Wikidumpparse Cloud VPS projectClosed, ResolvedPublicActions

Assigned To
 notconfusing
Authored By
 notconfusing Nov 18 2020, 11:04 PM2020-11-18 23:04:38 (UTC+0)
Referenced Files
None

# Description

Project Name: Wikidumpparse
Type of quota increase requested: Custom Flavour g2.cores8.ram16.disk1500
Reason: The main point of this project is to pre-compute wikidata-dump level statistics about all humans, and store them in a database. Allow me to explain the calculation that leads to the 1.5TB request. Detailed calculation this google doc.

• The version 1, proof of concept , of this project "Denelezh" ran for 2 years and generated 300GB of data. See T263703#6622726
• Assuming we want this to run for 3 years or 150 weeks. (Project follows the weekly wikidata dump cycle).
• Empirically, from the proof of concept, the "input" data can be modelled by y=(4.5+0.005x)GB where x:=weeks since launch
• Empirically, from the proof of concept, the "output" data can be modelled by y=(4.5+0.013x)GB where x:=weeks since launch
• Given the strategy of keeping just the last 1-year of input data after 3 years, we have integral y=5.5+0.005x from 0 to 50 weeks (wolfram alpha link)
• Given keeping all of the output data for 3 yeas, we have integral y=4.5+0.013x from 0 to 150 weeks

Totals:

 total (GB) description calculation 256 last 1 year of input data integral y=5.5+0.005x from 0 to 50 weeks 821 3 years of output data integral y=4.5+0.013x from 0 to 150 weeks 300 2 years of 'backfill data' from denelezh.org (WMFR) empirical 75 5 years of 'backfill data' from whgi.wmflabs.org empirical 1452 grand total for 3 years of running humaniki project sum of above

Thank you in advance. I am happy to answer any questions you have.
-Max

### Event Timeline

Restricted Application added a subscriber: Aklapper. Nov 18 2020, 11:04 PM

How much data do you have today? There are 2 reasons that I ask:

1. There is a very low probability of single cloud vps instance surviving for a 3 year life cycle. It is very likely that an instance you make today will need to be replaced in ~2 years max for operating system deprecation.
2. I would hope that within the next 12 months we will be able to offer "attachable block storage" using OpenStack Cinder. That feature will functionally be like getting a second hard disk for an instance with the added bonus of that disk being separate from the operating system disk. This will allow the attached storage to be unmounted from instance A and mounted to instance B which will make OS upgrades much, much simpler for large storage needs like yours.

Hi @bd808 thanks for the response.

It is very likely that an instance you make today will need to be replaced in ~2 years max for operating system deprecation.

Fair point, I updated my calculations for a 2 years run way.

I would hope that within the next 12 months we will be able to offer "attachable block storage" using OpenStack Cinder.

That'd be excellent, and perfect for our use case.

∴ Given those two facts. Let's plan on our instance just lasting two years. Updated calculations would indicate that we'll need 1.12TB for 2 years. At that point we would be happy to migrate to attached storage. Does that sound like a reasonable path forwards?

Andrew subscribed.

This is approved, someone will create the new flavor soon

New flavor created and assigned to the project, @notconfusing can you verify and close the task if ok?
Or bounce back to me if there's any issues ;)

```# wmcs-openstack flavor show g2.cores8.ram16.disk1120
+----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                      | Value                                                                                                                                                        |
+----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-FLV-DISABLED:disabled   | False                                                                                                                                                        |
| OS-FLV-EXT-DATA:ephemeral  | 0                                                                                                                                                            |
| access_project_ids         | wikidumpparse                                                                                                                                                |
| disk                       | 1120                                                                                                                                                         |
| id                         | 46ce2174-f133-4561-8404-83707ee11367                                                                                                                         |
| name                       | g2.cores8.ram16.disk1120                                                                                                                                     |
| os-flavor-access:is_public | False                                                                                                                                                        |
| properties                 | aggregate_instance_extra_specs:ceph=''true'', quota:disk_read_iops_sec=''5000'', quota:disk_total_bytes_sec=''200000000'', quota:disk_write_iops_sec=''500'' |
| ram                        | 16384                                                                                                                                                        |
| rxtx_factor                | 1.0                                                                                                                                                          |
| swap                       |                                                                                                                                                              |
| vcpus                      | 8                                                                                                                                                            |
+----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+```

Mentioned in SAL (#wikimedia-cloud) [2020-11-26T15:53:31Z] <dcaro> Created private flavor g2.cores8.ram16.disk1120 for wikidumpparse (T268190)

Mentioned in SAL (#wikimedia-cloud) [2020-11-26T15:53:48Z] <dcaro> Created private flavor g2.cores8.ram16.disk1120 (T268190)