A single report that requires 100GB of RAM to generate seems like a opportunity for optimization.

The default quota for a project is 16GB of RAM, so splitting out into your own project is not going to get you closer to running a 100GB report automatically. You will need more than 6.25 times the default project quota and a custom flavor (description of an instance) to attempt managing a 100GB heap with your java process.

At first glance, you appear to be using bufferedreads and streams, but I agree that there must be a optimization or two that will enable these statistics to run in a smaller environment. I'm not a java expert, but it would be interesting to understand what is being held entirely in memory that needs 100 GB. I would encourage you to profile the application and make sure it's acting as you expect. Alternatively, batching might be a short-term solution to limit the size of work being processed at one time and then later combining the results.

Would you like a project with the default quota?

I would encourage you to profile the application and make sure it's acting as you expect

I have done that.

it would be interesting to understand what is being held entirely in memory that needs 100 GB

The global user matrix. There are about 69 million users, and each of them can have an account on up to 941 wikis. The number of elements in that matrix is itself ~6.5 billion.
Do I need to work with this full matrix? In some cases yes, such as when finding correlation.

Regarding optimisation: one way is to shift the bottleneck (from RAM to disk), but I think that would drastically increase the time to the extent that it wouldn't be feasible. As it currently stands my code takes two days to completely execute (when run on a university server) due to the output table being ~120 GB.

Would you like a project with the default quota?

What exactly would that be? https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS_Instances suggests that the maximum RAM I can have that way is 36 GB.

The default quota is 16G of RAM. Higher project quotas are possible with justification. You're correct that we don't have 100GB+ RAM instances. That's a large amount of resources for a single vm / container. However, it sounds like the toolforge environment is working for you, apart from the memory limitations. If that's correct, a cloud VPS project default quota won't improve things, and will also require you to maintain the environment.

Are there other collaborators or maintainers for this work? Can you explain more about the need for these statistics and their use cases?

However, it sounds like the toolforge environment is working for you, apart from the memory limitations.

That's true. If grid could support 100GB projects that would be enough.

Are there other collaborators or maintainers for this work?

No

Can you explain more about the need for these statistics and their use cases?

• To allow other users to easily download a table that contains all users' edit counts (kind of a dump). I release such data at https://github.com/Leader-board/Wikimedia-statistics/releases
• For me to perform various statistical tests and experiments on such data (mostly hobbyist, though I do report unusual examples, such as https://phabricator.wikimedia.org/T305329)

Is there an update to this request?

This was discussed some. I believe the main view was that the request that can be filled is mostly the same as what would be offered in toolforge. Which is mostly that we can't offer an instance with 100GB+ of RAM in cloud VPS or elsewhere. Thus, in this case, if this won't work in toolforge, we probably can't support it in cloud VPS either. @nskaggs could you comment further?

@Leaderboard I'm sorry, I was ill for a few days and was unable to reply to you. We cannot allocate that much memory to a single instance on Toolforge or within Cloud VPS. Given your answers to the questions I asked, it does sound like Toolforge is the correct place for this tool. We can adjust the tool quota to give you more memory than the default, however, I think you've already discovered the working limit of 16G within Toolforge.

So for this specific project request to create a new project, I would say no. A cloud VPS project also won't allow for a single VM to contain the amount of memory you are requesting.

I do wish you the best on your project. Thank you for your work within the movement! If there is something further we can help with, please let us know.

Hi @Leaderboard the flavor with 36 GB was actually created due to an arithmetic error and is in the slow process of being removed. Would you be alright with a 32GB instance? If so I believe there would be no issue in creating a project with a 32 GB quota. Would that be alright?

+1 from me

@rook That's fine.

All set!

openstack project create --description 'statanalyser' statanalyser --domain default
openstack role add --project statanalyser --user leaderboard projectadmin
openstack role add --project statanalyser --user leaderboard user
openstack quota set --ram 32768 statanalyser

Can do!

root@cloudcontrol1005:~# openstack quota set --gigabytes 200 statanalyser