Page MenuHomePhabricator

Setup es4 and es5 replica sets for new read-write external store service
Open, NormalPublic

Description

es2 and es3 replica sets are at 70% capacity as of June 2019 (note those servers have so much capacity that means they still have 3.3TB of free capacity each). Predicted usage is to reach 80% around December 2019 - January 2020. Before they get to around 90% capacity, they will be converted into read-only hosts (in addition to es1 ones) and write service will be moved to a new es4/es5 service.

This ticket is a meta one to coordinate all steps needed for the migration.

Event Timeline

jcrespo created this task.Jun 27 2019, 11:15 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 27 2019, 11:15 AM
jcrespo triaged this task as Normal priority.Jun 27 2019, 11:15 AM
jcrespo moved this task from Triage to Meta/Epic on the DBA board.

@Anomie You mentioned some potential ES maintenance in the past. Server change would be a great opportunity to transition to a different mw configuration once we are on the new servers. I ping you with months in advance FYI in case you want/need to prepare for such transition.

Should we define new HW specs for these hosts?
Currently es1015 specs:
128GB RAM
12x1.819 TB SATA HDD

Thanks for the ping, but I don't think we (meaning Core Platform Team) are working on the things that would touch on that maintenance at this time.

If I recall correctly, the maintenance was along the lines of running MediaWiki's recompression scripts and other similar scripts, which would first require making sure the scripts still work right.

If I recall correctly, the maintenance was along the lines of running MediaWiki's recompression scripts and other similar scripts, which would first require making sure the scripts still work right.

Sorry, I misunderstood you, I thought some architectural change/maintenance was upcoming related to how ES worked based on a previous conversation (that I probably misunderstood- e.g. related to MCR). Based on your comment, you can unsubscribe to avoid spam here as there is then no blocker for us to proceed with this then.

No problem, it doesn't hurt to check. At one point we were considering the possibility of having the scripts update existing ES entries in place, but in the end we decided not to do that (instead the scripts would insert new ES entries so old ones can remain read-only). If we get back to either the recompression or reserialization projects, we'll consult you on the details then.

I don't recall any other proposed changes to ES on the database level.

Thanks, that was probably it. Thanks for clarifying it!

Anomie removed a subscriber: Anomie.Jul 12 2019, 2:04 PM
Marostegui added a subtask: Unknown Object (Task).Mon, Oct 21, 6:49 AM
Marostegui added a subtask: Unknown Object (Task).