Page MenuHomePhabricator

Request increased quota for cyberbot Cloud VPS project
Closed, ResolvedPublic

Description

Project Name: cyberbot
Type of quota increase requested: An instance large enough to contain the s51059__cyberbot DB currently on tools.db with room for growth
Reason: As InternetArchiveBot expands on to other wikis, it's memory core steadily grows and demands more resources to operate on the dozen+ wikis it currently runs on, learning new URL data. It's beginning to crowd out other tools using the shared database on Toolforge. As per the recommendation of @bd808 I am filing to add a new instance to the Cyberbot project large enough to sufficiently process queries, and allow for growth as IABot grows.

Also referencing T178294

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 16 2017, 7:57 PM
chasemp claimed this task.Oct 17 2017, 4:56 PM
chasemp triaged this task as Medium priority.
chasemp added a subscriber: chasemp.

+1

chasemp closed this task as Resolved.Oct 18 2017, 1:40 PM

I added one large instance worth of quota to the existing resources:

NameRAMDiskVCPU
m1.large8192804

Mentioned in SAL (#wikimedia-operations) [2017-10-18T14:03:36Z] <chasemp> bump up quota on cyberpower cloud project per T178332

I have now spawned a large instance named "cyberbot-db-01". Thank you for the increase.

Cyberpower678 reopened this task as Open.EditedOct 19 2017, 4:52 PM

Unfortunately with the DB at 45GB, I need the bigdisk flavor.

The /srv mount only has barely 55GB since the root is 20GB.

Andrew added a subscriber: Andrew.Oct 19 2017, 4:57 PM

@Cyberpower678 Can you provide any kind of long-term estimate about how big you expect this database to grow?

The DB is shared across every wiki IABot runs on. As it expands to more wikis the DB will grow a little more. The DB also contains log entries for https://tools.wmflabs.org/iabot, and a bot queue. So I was thinking 120GB, would be sufficient for a while.

chasemp added a comment.EditedOct 24 2017, 1:37 PM

My current understanding:

  • We bumped up for a large instance which can be formatted with a srv of 55GB
  • The current DB is 45GB which means not a ton of headroom there
  • @Cyberpower678 wants us to add quota that would take the large added here and make an xlarge possible (so the delta between a large and an xlarge)
  • The current new large gets deleted
  • A new xlarge is created

Is this where things stand?

My current understanding:

  • We bumped up for a large instance which can be formatted with a srv of 55GB
  • The current DB is 45GB which means not a ton of headroom there
  • @Cyberpower678 wants us to add quota that would take the large added here and make an xlarge possible (so the delta between a large and an xlarge)
  • The current new large gets deleted
  • A new xlarge is created

Is this where things stand?

Actually it is my understanding that there exists a large instance flavor with a bigdisk option. So the instance CPU and RAM are the same, but the SRV space is much larger. I don't want to needlessly waste CPU and RAM resources when the large flavor is sufficient. But apparently that flavor needs to be added to the cyberbot project.

chasemp reassigned this task from chasemp to Andrew.Oct 24 2017, 4:12 PM

Here is what we approved in our meeting:

  • Access to the 'bigdisk' flavor within openstack with 300GB disk
21e9047d-a60f-499d-b7f5-51f83ddf3611bigdisk2457630004False
  • No other quota bump from existing as this is a replacement of the current new-large instance that originally came from this task

@Andrew is going to facilitate

(please only create the single large bigdisk instance)

Here is what we approved in our meeting:

  • Access to the 'bigdisk' flavor within openstack with 300GB disk
21e9047d-a60f-499d-b7f5-51f83ddf3611bigdisk2457630004False
  • No other quota bump from existing as this is a replacement of the current new-large instance that originally came from this task

@Andrew is going to facilitate

(please only create the single large bigdisk instance)

Excellent. :-) Thank you. I appreciate the patience and access to 300GB. :-)

I've added the 'bigdisk' flavor to this project, so now when you create a new VM you should see that as an option. Let me know if you run into any trouble.

Cyberpower678 closed this task as Resolved.Oct 26 2017, 12:06 AM