Project Name: wikidata-realtime-dumps
Wikitech Usernames of requestors: Bekh6ex, Addshore
Purpose: Creating and maintaining Wikidata JSON dumps in real time for people to use.
Brief description: Needed software is git, rust toolchain, some utilities.
How soon you are hoping this can be fulfilled: this week. It is currently running on Google Cloud.
Hello! Can you give us a bit of info about what resources you expect to use? Ram, Cores, Disk space? Also, are you hoping to offload storage onto NFS or NFS/scratch? (If the latter, you may be disappointed by performance)
Currently working on VPS with following utilization: 12 cores (avg cpu utilization 70%), avg mem usage is 2,5-3,5 GB, Storage: read avg: 16 MB/s, write avg: 41 MB/s, 130GB SSD drive.
So, resources needed are somewhat like:
Ram: around 5 GB to be sure.
Cores: at least 10 cores.
Storage: 130GB should be enough. Regarding speed, it seems that drive should have at least 30+MB/s stable write throughput. (currently working perfectly on 60MB/s drive, 15MB/s was WAY too little).
I'm working on algorithm optimizations, so I hope that requirements will be reduced in the future.
I can not find a Wikimedia developer account for @Bekh6ex. I will proceed with the creation, make @Addshore the project admin, and hope that he can get things sorted out.
The metrics given in T244277#5862957 are above the default project quota (8 cores, 16GB RAM). The WMCS team would like this project to try working within the default quota before being granted additional cores. A Cloud-VPS (Quota-requests) ticket can be created to request additional quota in the future once the default limits can be shown definitively to be insufficient.
$ ssh cloudcontrol1003.wikimedia.org $ sudo wmcs-openstack user list | grep -Ei '(Bekh6ex|Addshore)' | addshore | Addshore | $ sudo wmcs-openstack --os-region-name eqiad1-r project create --enable --description "Creating and maintaining Wikidata JSON dumps in real time for people to use" wikidata-realtime-dumps +-------------+-----------------------------------------------------------------------------+ | Field | Value | +-------------+-----------------------------------------------------------------------------+ | description | Creating and maintaining Wikidata JSON dumps in real time for people to use | | domain_id | default | | enabled | True | | id | wikidata-realtime-dumps | | is_domain | False | | name | wikidata-realtime-dumps | | parent_id | default | +-------------+-----------------------------------------------------------------------------+ $ sudo wmcs-openstack role add --user addshore --project wikidata-realtime-dumps projectadmin $ sudo wmcs-openstack role add --user addshore --project wikidata-realtime-dumps user $ sudo wmcs-openstack role assignment list --project wikidata-realtime-dumps --names +--------------+----------------------+-------+---------------------------------+--------+-----------+ | Role | User | Group | Project | Domain | Inherited | +--------------+----------------------+-------+---------------------------------+--------+-----------+ | projectadmin | Addshore@Default | | wikidata-realtime-dumps@Default | | False | | user | Addshore@Default | | wikidata-realtime-dumps@Default | | False | | projectadmin | Novaadmin@Default | | wikidata-realtime-dumps@Default | | False | | user | Novaadmin@Default | | wikidata-realtime-dumps@Default | | False | | observer | Novaobserver@Default | | wikidata-realtime-dumps@Default | | False | +--------------+----------------------+-------+---------------------------------+--------+-----------+