DB hardware for Reading Lists
Open, NormalPublic

Description

Per the discussion in T164990, the x1 database cluster is close enough to full that a new service does not comfortably fit there. Reading Lists can temporarily live there with some restrictions but should eventually go to the to-be-established new cluster x2. That means new hardware needs to be procured - per T164990#3264314, probably 4 DB servers. This should be coordinated with the other would-be tenants of the new cluster.

(Not adding Operations hardware-requests yet as this is just a tracking task and not actionable right now; will probably happen in next annual budget planning.)

Tgr created this task.Oct 5 2017, 8:51 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Tgr updated the task description. (Show Details)Sat, Jun 30, 11:41 AM
Tgr updated the task description. (Show Details)Sat, Jun 30, 11:50 AM
Jhernandez triaged this task as High priority.Thu, Jul 5, 3:09 PM
Jhernandez added a subscriber: Jhernandez.

I'll be checking in to see how we can make this actionable.

jcrespo added a comment.EditedThu, Jul 5, 3:59 PM

new cluster x2

When budgeting this year's resources, SRE decision (not only mine, Mark signed it off) was to budget to increase x1 servers rather than setting up a separate, dedicated cluster. That is coming for codfw on Q1 and for eqiad in Q3.

There was already a small increase of resources in late Q4.

@Tgr Can you specify the needs of the service to @jcrespo to see if they can be met with the planned increase?

@jcrespo where can we see the planned increase in detail? Thanks for chiming in!

Tgr added a comment.Tue, Jul 10, 2:52 PM

I don't really have anything beyond what was projected in T164990. (The current load is around 1 read/sec an 0.1 write/sec, but the feature is not used on the web which would be the overwhelming majority of the load. OTOH there are some temporary limitations in place: no more than 10 lists or 100 list entries read in a single request - those would presumably be lifted if the DB permits.)

Jhernandez lowered the priority of this task from High to Normal.Tue, Jul 10, 4:55 PM

Lowering to normal since it is not clear when the web will implement this and apps traffic don't seem to pose a problem right now.