es2 and es3 replica sets are at 70% capacity as of June 2019 (note those servers have so much capacity that means they still have 3.3TB of free capacity each). Predicted usage is to reach 80% around December 2019 - January 2020. Before they get to around 90% capacity, they will be converted into read-only hosts (in addition to es1 ones) and write service will be moved to a new es4/es5 service.
This ticket is a meta one to coordinate all steps needed for the migration.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| mysql: update CORE_SECTIONS for external storage | operations/software/spicerack | master | +2 -2 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • Marostegui | T226704 Setup es4 and es5 replica sets for new read-write external store service | |||
| Unknown Object (Task) | |||||
| Unknown Object (Task) | |||||
| Resolved | • Marostegui | T243052 Productionize es1020-es1025, es2020-es2025 | |||
| Resolved | • Cmjohnson | T241359 (Needed by 31st January) eqiad: rack/setup/install es102[0-5].eqiad.wmnet | |||
| Resolved | Papaul | T241336 (Needed By 31st January) codfw: rack/setup/install es202[0-5].codfw.wmnet | |||
| Resolved | Papaul | T242481 d-i fails to install on servers with BRCM 2P 1G BT + 2P 10G SFP NDC | |||
| Resolved | • Marostegui | T243512 Clean up wikiadmin2 user from core hosts | |||
| Resolved | Papaul | T245714 es2022 power supply issues | |||
| Resolved | CDanis | T245806 Create es4 and es5 sections in dbctl | |||
| Resolved | Reedy | T245720 Create new blobs_clusterXX tables on new es4 and es5 sections | |||
| Resolved | • Marostegui | T246072 Enable es4 and es5 as writable new external store sections and set es2 and es3 as read only |
@Anomie You mentioned some potential ES maintenance in the past. Server change would be a great opportunity to transition to a different mw configuration once we are on the new servers. I ping you with months in advance FYI in case you want/need to prepare for such transition.
Should we define new HW specs for these hosts?
Currently es1015 specs:
128GB RAM
12x1.819 TB SATA HDD
Thanks for the ping, but I don't think we (meaning Platform Engineering) are working on the things that would touch on that maintenance at this time.
If I recall correctly, the maintenance was along the lines of running MediaWiki's recompression scripts and other similar scripts, which would first require making sure the scripts still work right.
If I recall correctly, the maintenance was along the lines of running MediaWiki's recompression scripts and other similar scripts, which would first require making sure the scripts still work right.
Sorry, I misunderstood you, I thought some architectural change/maintenance was upcoming related to how ES worked based on a previous conversation (that I probably misunderstood- e.g. related to MCR). Based on your comment, you can unsubscribe to avoid spam here as there is then no blocker for us to proceed with this then.
No problem, it doesn't hurt to check. At one point we were considering the possibility of having the scripts update existing ES entries in place, but in the end we decided not to do that (instead the scripts would insert new ES entries so old ones can remain read-only). If we get back to either the recompression or reserialization projects, we'll consult you on the details then.
I don't recall any other proposed changes to ES on the database level.
Change 576297 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/spicerack@master] mysql: update CORE_SECTIONS for external storage
es4 and es5 are now in production
es2 and es3 are now read only
Thanks @jcrespo for all the help here!
Change 576297 merged by jenkins-bot:
[operations/software/spicerack@master] mysql: update CORE_SECTIONS for external storage