Page MenuHomePhabricator

Request creation of MissSearch VPS project
Closed, ResolvedPublic

Description

*Project Name:* MissSearch

*Developer account usernames of requestors:* Elizaka181, Galrach, Ilanen1, Naorleizer, Rutsavi09

*Purpose:* A web-based tool that helps Wikipedia contributors identify articles with high potential for creation, expansion or improvement.

*Brief description:* "MissSearch" is a web-based support tool for Wikipedia contributors worldwide, currently under active development. It analyzes Wikipedia content and highlights articles with high potential for creation, improvement or expansion. The tool is currently hosted on Toolforge: https://miss-search.toolforge.org/
The source code is publicly available on GitLab: https://gitlab.wikimedia.org/toolforge-repos/miss-search
We are a group of students from the Technion – Israel Institute of Technology, working directly with the CEO of Wikimedia Israel, in coordination with the needs of the Wikipedia community.
The system is developed in Python and requires increased storage capacity due to the size of the generated data files, local SQL databases, indexes, and metadata used in article analysis.
We are requesting an increased disk quota to support our growing data needs and to ensure consistent availability and responsiveness for Wikipedia contributors. Currently, we are experiencing storage limitations that prevent us from continuing development, as our solution requires more storage than the standard Toolforge quota allows. Furthermore, as the project evolves, we anticipate a significant increase in data storage requirements and may require even greater capacity in the near future.

*How soon you are hoping this can be fulfilled:* As soon as possible, preferably within the coming weeks.

Details

Other Assignee
Naorleizer

Event Timeline

Hi @Galr600!
Can you elaborate on the storage needs of your tool? If you can give some numbers and estimations of usage it will help us understand the project needs.
If storage is all you need, maybe using https://wikitech.wikimedia.org/wiki/Help:Object_storage_user_guide#Toolforge meets your needs? (you'll need an openstack project also, so this task is ok, just we would provision it differently)

Hi @dcaro,
Thank you for your response!

Our current estimated storage requirement for MissSearch is approximately ~250GB. This is the minimum needed at this stage to continue active development, as the current quota limits are preventing further progress.

The storage is primarily used for locally generated data files, intermediate processing outputs, SQL databases, and indexes related to Wikipedia content analysis. As the project evolves and we expand the tool’s capabilities, we anticipate this requirement could grow up to ~500GB at most, depending on the scale and the number of Wikipedia language versions we support.

We reviewed the Object storage guide, and while it could be part of a long-term solution, our current architecture is optimized for local storage due to performance and response-time considerations. This is particularly important as our tool functions like a search engine intended to serve Wikipedia contributors worldwide.

Please let us know the next steps, and whether we should proceed with an OpenStack-based setup or if there’s an alternative that better suits our needs.

Thanks again for your help!

Yep, currently there's no easy on demand storage on toolforge side besides NFS (it's on our minds, so keep an eye!).

Fyi. there's an elasticsearch instance for toolforge users to use https://wikitech.wikimedia.org/wiki/Help:Toolforge/Elasticsearch , not sure if that helps (now or in the future).

+1 approved

dcaro triaged this task as Medium priority.May 4 2025, 4:24 AM
dcaro moved this task from Inbox to Approved on the Cloud-VPS (Project-requests) board.

Mentioned in SAL (#wikimedia-cloud-feed) [2025-05-06T18:59:43Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project misssearch in eqiad1 (T392903)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-05-06T19:00:54Z] <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.quota_increase (T392903)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-05-06T19:01:02Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T392903)

Andrew claimed this task.
Andrew subscribed.

You should be all set. Re-open this ticket if you run into trouble!