move data to /srv for the cirrus / elasticsearch clusters
Open, HighPublic

Description

The standard at WMF is to store application data on /srv. This standardisation allows to limit the number of specific partman recipes. Elasticsearch is one of the few exception (for the usual "historical" reasons). The relforge cluster is already using /srv for data, so this change should be mostly painless.

Gehel created this task.Nov 22 2016, 3:14 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 22 2016, 3:14 PM
Gehel triaged this task as "High" priority.Nov 23 2016, 12:49 PM
Gehel claimed this task.
Gehel added a subscriber: RobH.Nov 23 2016, 2:56 PM

The oldest elasticsearch servers (elastic1017-1031) have smaller SSD, configured as RAID0. We do not have a standard partman configuration that matches this. Relforge servers already use the standard raid10-gpt-srv-lvm-ext4.cfg configuration.

Current partman configs (from modules/install_server/files/autoinstall/netboot.cfg):

elastic101[7-9]|elastic102[0-9]|elastic103[0-1]) echo partman/elasticsearch-raid0.cfg ;; \
elastic103[2-9]|elastic104[0-9]|elastic20[0-2][0-9]|elastic203[0-1]) echo partman/elasticsearch-raid1.cfg ;; \
relforge100[12]) echo partman/raid10-gpt-srv-lvm-ext4.cfg ;; \

New configuration should be:

elastic101[7-9]|elastic102[0-9]|elastic103[0-1]) echo partman/??? ;; \
elastic103[2-9]|elastic104[0-9]|elastic20[0-2][0-9]|elastic203[0-1]) echo partman/raid1-lvm-ext4-srv.cfg ;; \
relforge100[12]) echo partman/raid10-gpt-srv-lvm-ext4.cfg ;; \

For old elasticsearch servers, we could create a new configuration based on raid0-lvm-srv.cfg, but with ext4 instead of XFS for the /srv partition.

@RobH you probably know more about this than I do. Any suggestion?

Change 323157 had a related patch set uploaded (by Gehel):
elasticsearch - codfw servers move to jessie and data on /srv

https://gerrit.wikimedia.org/r/323157

Change 335218 had a related patch set uploaded (by Gehel):
elasticsearch - ensure data directory with puppet

https://gerrit.wikimedia.org/r/335218

Change 335218 merged by Gehel:
elasticsearch - ensure data directory with puppet

https://gerrit.wikimedia.org/r/335218

Mentioned in SAL (#wikimedia-releng) [2017-02-01T13:29:25Z] <gehel> starting deployment-elastic* migration to jessie and moving data partition to /srv (T151326 / T151328)

Change 323157 merged by Gehel:
elasticsearch - codfw servers move to jessie and data on /srv

https://gerrit.wikimedia.org/r/323157

Change 336409 had a related patch set uploaded (by Gehel):
elasticsaerch - change the default data directory to /srv/elasticsearch

https://gerrit.wikimedia.org/r/336409

Change 336410 had a related patch set uploaded (by Gehel):
elasticsearch - codfw cirrus cluster - move data to /srv/elasticsearch

https://gerrit.wikimedia.org/r/336410

Change 336409 merged by Gehel:
elasticsaerch - change the default data directory to /srv/elasticsearch

https://gerrit.wikimedia.org/r/336409

Change 336410 merged by Gehel:
elasticsearch - codfw cirrus cluster - move data to /srv/elasticsearch

https://gerrit.wikimedia.org/r/336410

Change 336414 had a related patch set uploaded (by Gehel):
elasticsearch - reimage elastic200[12] to jessie and move data to /srv

https://gerrit.wikimedia.org/r/336414

Change 336414 merged by Gehel:
elasticsearch - reimage elastic200[12] to jessie and move data to /srv

https://gerrit.wikimedia.org/r/336414

Change 336589 had a related patch set uploaded (by Gehel):
elasticsearch - reimage elastic200[12] to jessie and move data to /srv

https://gerrit.wikimedia.org/r/336589

Change 336589 merged by Gehel:
elasticsearch - reimage elastic200[12] to jessie and move data to /srv

https://gerrit.wikimedia.org/r/336589

Change 336595 had a related patch set uploaded (by Gehel):
elasticsearch - reimage elastic200[5678] to jessie and move data to /srv

https://gerrit.wikimedia.org/r/336595

Change 336595 merged by Gehel:
elasticsearch - reimage elastic200[5678] to jessie and move data to /srv

https://gerrit.wikimedia.org/r/336595

Change 336625 had a related patch set uploaded (by Gehel):
elasticsearch - reimage elastic20(09|10|11|12) to jessie and move data to /srv

https://gerrit.wikimedia.org/r/336625

Change 336625 merged by Gehel:
elasticsearch - reimage elastic20(09|10|11|12) to jessie and move data to /srv

https://gerrit.wikimedia.org/r/336625

Change 336679 had a related patch set uploaded (by Gehel):
elasticsearch - reimage elastic20(13|14|15|16) to jessie and move data to /srv

https://gerrit.wikimedia.org/r/336679

Change 336679 merged by Gehel:
elasticsearch - reimage elastic20(13|14|15|16) to jessie and move data to /srv

https://gerrit.wikimedia.org/r/336679

Change 336765 had a related patch set uploaded (by Gehel):
elasticsearch - reimage elastic20(17|18|19|20) to jessie and move data to /srv

https://gerrit.wikimedia.org/r/336765

Change 336765 merged by Gehel:
elasticsearch - reimage elastic20(17|18|19|20) to jessie and move data to /srv

https://gerrit.wikimedia.org/r/336765

Change 336795 had a related patch set uploaded (by Gehel):
elasticsearch - reimage elastic20(21|22|23|24) to jessie and move data to /srv

https://gerrit.wikimedia.org/r/336795

Change 336795 merged by Gehel:
elasticsearch - reimage elastic20(21|22|23|24) to jessie and move data to /srv

https://gerrit.wikimedia.org/r/336795

Change 336973 had a related patch set uploaded (by Gehel):
elasticsearch - reimage elastic20(25|26|27|28) to jessie and move data to /srv

https://gerrit.wikimedia.org/r/336973

Change 336973 merged by Gehel:
elasticsearch - reimage elastic20(25|26|27|28) to jessie and move data to /srv

https://gerrit.wikimedia.org/r/336973

Change 337060 had a related patch set uploaded (by Gehel):
elasticsearch - reimage elastic20(29|30|31|32) to jessie and move data to /srv

https://gerrit.wikimedia.org/r/337060

Change 337060 merged by Gehel:
elasticsearch - reimage elastic20(29|30|31|32) to jessie and move data to /srv

https://gerrit.wikimedia.org/r/337060

Change 337218 had a related patch set uploaded (by Gehel):
elasticsearch - reimage elastic20(33|34|35|36) to jessie and move data to /srv

https://gerrit.wikimedia.org/r/337218

Change 337218 merged by Gehel:
elasticsearch - reimage elastic20(33|34|35|36) to jessie and move data to /srv

https://gerrit.wikimedia.org/r/337218

Change 337378 had a related patch set uploaded (by Gehel):
elasticsearch - reimage to jessie and move data to /srv - preliminary work

https://gerrit.wikimedia.org/r/337378

Change 337378 merged by Gehel:
elasticsearch - reimage to jessie and move data to /srv - preliminary work

https://gerrit.wikimedia.org/r/337378

Change 338327 had a related patch set uploaded (by Gehel):
elasticsearch - reimage elastic10(17|18|19|20) to jessie and move data to /srv

https://gerrit.wikimedia.org/r/338327

Change 338327 merged by Gehel:
elasticsearch - reimage elastic10(17|18|19|20) to jessie and move data to /srv

https://gerrit.wikimedia.org/r/338327

Change 338392 had a related patch set uploaded (by Gehel):
elasticsearch - reimage elastic10(21|22|23|24) to jessie and move data to /srv

https://gerrit.wikimedia.org/r/338392

Change 338392 merged by Gehel:
elasticsearch - reimage elastic10(21|22|23|24) to jessie and move data to /srv

https://gerrit.wikimedia.org/r/338392

Change 338761 had a related patch set uploaded (by Gehel):
elasticsearch - reimage elastic10(25|28|29|30) to jessie and move data to /srv

https://gerrit.wikimedia.org/r/338761

Change 338761 merged by Gehel:
elasticsearch - reimage elastic10(25|28|29|30) to jessie and move data to /srv

https://gerrit.wikimedia.org/r/338761

Change 338768 had a related patch set uploaded (by Gehel):
elasticsearch - reimage elastic10(26|31|36|40) to jessie and move data to /srv

https://gerrit.wikimedia.org/r/338768

Change 338768 merged by Gehel:
elasticsearch - reimage elastic10(26|31|36|40) to jessie and move data to /srv

https://gerrit.wikimedia.org/r/338768

Change 338811 had a related patch set uploaded (by Gehel):
elasticsearch - reimage elastic10(27|32|37|41) to jessie and move data to /srv

https://gerrit.wikimedia.org/r/338811

Change 338811 merged by Gehel:
elasticsearch - reimage elastic10(27|32|37|41) to jessie and move data to /srv

https://gerrit.wikimedia.org/r/338811

Change 338977 had a related patch set uploaded (by Gehel):
elasticsearch - reimage elastic10(33|34|38|42) to jessie and move data to /srv

https://gerrit.wikimedia.org/r/338977

Change 338977 merged by Gehel:
elasticsearch - reimage elastic10(33|34|38|42) to jessie and move data to /srv

https://gerrit.wikimedia.org/r/338977

Change 339017 had a related patch set uploaded (by Gehel):
elasticsearch - reimage elastic10(35|39|43|44) to jessie and move data to /srv

https://gerrit.wikimedia.org/r/339017

Change 339017 merged by Gehel:
elasticsearch - reimage elastic10(35|39|43|44) to jessie and move data to /srv

https://gerrit.wikimedia.org/r/339017

Change 339130 had a related patch set uploaded (by Gehel):
elasticsearch - reimage elastic10(45|46|47) to jessie and move data to /srv

https://gerrit.wikimedia.org/r/339130

Change 339130 merged by Gehel:
elasticsearch - reimage elastic10(45|46|47) to jessie and move data to /srv

https://gerrit.wikimedia.org/r/339130