Page MenuHomePhabricator

migrate elasticsearch to stretch (from jessie)
Closed, ResolvedPublic

Description

At some point, we will need to migrate our elasticsearch servers from jessie to stretch. Since we have planned to renew some of our end of life servers next fiscal year, we should probably take the occasion to do this migration.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic2002.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808071203_gehel_11445.log.

Completed auto-reimage of hosts:

['elastic2002.codfw.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic2003.codfw.wmnet', 'elastic2004.codfw.wmnet', 'elastic2005.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808071236_gehel_18988.log.

Completed auto-reimage of hosts:

['elastic2004.codfw.wmnet', 'elastic2005.codfw.wmnet', 'elastic2003.codfw.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic2006.codfw.wmnet', 'elastic2025.codfw.wmnet', 'elastic2026.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808071619_gehel_8691.log.

Completed auto-reimage of hosts:

['elastic2026.codfw.wmnet', 'elastic2025.codfw.wmnet', 'elastic2006.codfw.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic2027.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808081420_gehel_30049.log.

Completed auto-reimage of hosts:

['elastic2027.codfw.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic2007.codfw.wmnet', 'elastic2008.codfw.wmnet', 'elastic2009.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808090713_gehel_31520.log.

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic2007.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808090718_gehel_32556.log.

Completed auto-reimage of hosts:

['elastic2007.codfw.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic2008.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808090802_gehel_11261.log.

Completed auto-reimage of hosts:

['elastic2008.codfw.wmnet']

Of which those FAILED:

['elastic2008.codfw.wmnet']

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic2008.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808090803_gehel_11473.log.

Completed auto-reimage of hosts:

['elastic2008.codfw.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic2010.codfw.wmnet', 'elastic2011.codfw.wmnet', 'elastic2012.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808090835_gehel_19144.log.

Completed auto-reimage of hosts:

['elastic2011.codfw.wmnet', 'elastic2012.codfw.wmnet', 'elastic2010.codfw.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic2028.codfw.wmnet', 'elastic2029.codfw.wmnet', 'elastic2030.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808091026_gehel_12332.log.

Completed auto-reimage of hosts:

['elastic2030.codfw.wmnet', 'elastic2028.codfw.wmnet', 'elastic2029.codfw.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic2013.codfw.wmnet', 'elastic2014.codfw.wmnet', 'elastic2015.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808091236_gehel_9923.log.

Completed auto-reimage of hosts:

['elastic2013.codfw.wmnet', 'elastic2014.codfw.wmnet', 'elastic2015.codfw.wmnet']

Of which those FAILED:

['elastic2013.codfw.wmnet', 'elastic2014.codfw.wmnet', 'elastic2015.codfw.wmnet']

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic2013.codfw.wmnet', 'elastic2014.codfw.wmnet', 'elastic2015.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808091341_gehel_25217.log.

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic2013.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808091342_gehel_25504.log.

Completed auto-reimage of hosts:

['elastic2013.codfw.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic2014.codfw.wmnet', 'elastic2015.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808091412_gehel_545.log.

Completed auto-reimage of hosts:

['elastic2015.codfw.wmnet']

Of which those FAILED:

['elastic2015.codfw.wmnet']

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic2015.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808091444_gehel_8582.log.

Completed auto-reimage of hosts:

['elastic2015.codfw.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic2016.codfw.wmnet', 'elastic2017.codfw.wmnet', 'elastic2018.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808091547_gehel_22262.log.

Completed auto-reimage of hosts:

['elastic2017.codfw.wmnet']

Of which those FAILED:

['elastic2017.codfw.wmnet']

Script wmf-auto-reimage was launched by gehel on sarin.codfw.wmnet for hosts:

['elastic2017.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808100916_gehel_17662.log.

Completed auto-reimage of hosts:

['elastic2017.codfw.wmnet']

Of which those FAILED:

['elastic2017.codfw.wmnet']

Script wmf-auto-reimage was launched by gehel on sarin.codfw.wmnet for hosts:

['elastic2017.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808100918_gehel_18038.log.

Completed auto-reimage of hosts:

['elastic2017.codfw.wmnet']

Of which those FAILED:

['elastic2017.codfw.wmnet']

Script wmf-auto-reimage was launched by gehel on sarin.codfw.wmnet for hosts:

['elastic2017.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808101207_gehel_25596.log.

Completed auto-reimage of hosts:

['elastic2017.codfw.wmnet']

Of which those FAILED:

['elastic2017.codfw.wmnet']

Script wmf-auto-reimage was launched by gehel on sarin.codfw.wmnet for hosts:

['elastic2031.codfw.wmnet', 'elastic2032.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808101208_gehel_25796.log.

Script wmf-auto-reimage was launched by gehel on sarin.codfw.wmnet for hosts:

['elastic2017.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808101232_gehel_32748.log.

Completed auto-reimage of hosts:

['elastic2031.codfw.wmnet', 'elastic2032.codfw.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by gehel on sarin.codfw.wmnet for hosts:

['elastic2033.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808101255_gehel_4262.log.

Completed auto-reimage of hosts:

['elastic2033.codfw.wmnet']

and were ALL successful.

Completed auto-reimage of hosts:

['elastic2017.codfw.wmnet']

Of which those FAILED:

['elastic2017.codfw.wmnet']

Script wmf-auto-reimage was launched by gehel on sarin.codfw.wmnet for hosts:

['elastic2019.codfw.wmnet', 'elastic2020.codfw.wmnet', 'elastic2021.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808101341_gehel_16071.log.

Completed auto-reimage of hosts:

['elastic2020.codfw.wmnet', 'elastic2019.codfw.wmnet', 'elastic2021.codfw.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by gehel on sarin.codfw.wmnet for hosts:

['elastic2022.codfw.wmnet', 'elastic2023.codfw.wmnet', 'elastic2024.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808101604_gehel_17745.log.

Completed auto-reimage of hosts:

['elastic2022.codfw.wmnet', 'elastic2023.codfw.wmnet', 'elastic2024.codfw.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by gehel on sarin.codfw.wmnet for hosts:

['elastic2034.codfw.wmnet', 'elastic2035.codfw.wmnet', 'elastic2036.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808101719_gehel_3490.log.

Completed auto-reimage of hosts:

['elastic2035.codfw.wmnet']

Of which those FAILED:

['elastic2035.codfw.wmnet']

Script wmf-auto-reimage was launched by gehel on sarin.codfw.wmnet for hosts:

['elastic2035.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808101806_gehel_15592.log.

Completed auto-reimage of hosts:

['elastic2035.codfw.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by gehel on sarin.codfw.wmnet for hosts:

['elastic2001.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808130826_gehel_24144.log.

Completed auto-reimage of hosts:

['elastic2001.codfw.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by gehel on sarin.codfw.wmnet for hosts:

['elastic2017.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808130858_gehel_31793.log.

Change 450064 merged by Gehel:
[operations/puppet@production] elasticsearch: migrate eqiad cluster to Stretch and RAID0

https://gerrit.wikimedia.org/r/450064

Script wmf-auto-reimage was launched by gehel on sarin.codfw.wmnet for hosts:

['elastic2017.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808130953_gehel_10741.log.

Completed auto-reimage of hosts:

['elastic2017.codfw.wmnet']

and were ALL successful.

Mentioned in SAL (#wikimedia-operations) [2018-08-13T12:56:27Z] <gehel> reimaging of elasticsearch / cirrus / codfw cluster (RAID0 / Stretch) completed - T193649 / T198391

Mentioned in SAL (#wikimedia-operations) [2018-08-13T12:56:45Z] <gehel> start reimaging of elasticsearch / cirrus / eqiad cluster (RAID0 / Stretch) - T193649 / T198391

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic1017.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808131257_gehel_8573.log.

Completed auto-reimage of hosts:

['elastic1017.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic1043.eqiad.wmnet', 'elastic1051.eqiad.wmnet', 'elastic1052.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808131324_gehel_15514.log.

Completed auto-reimage of hosts:

['elastic1043.eqiad.wmnet', 'elastic1052.eqiad.wmnet', 'elastic1051.eqiad.wmnet']

Of which those FAILED:

['elastic1043.eqiad.wmnet', 'elastic1052.eqiad.wmnet', 'elastic1051.eqiad.wmnet']

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic1043.eqiad.wmnet', 'elastic1051.eqiad.wmnet', 'elastic1052.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808131326_gehel_15877.log.

Completed auto-reimage of hosts:

['elastic1043.eqiad.wmnet', 'elastic1051.eqiad.wmnet', 'elastic1052.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic1047.eqiad.wmnet', 'elastic1049.eqiad.wmnet', 'elastic1050.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808131704_gehel_4889.log.

Completed auto-reimage of hosts:

['elastic1047.eqiad.wmnet', 'elastic1050.eqiad.wmnet', 'elastic1049.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic1049.eqiad.wmnet', 'elastic1046.eqiad.wmnet', 'elastic1039.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808140838_gehel_10377.log.

Completed auto-reimage of hosts:

['elastic1039.eqiad.wmnet', 'elastic1046.eqiad.wmnet', 'elastic1049.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic1044.eqiad.wmnet', 'elastic1045.eqiad.wmnet', 'elastic1048.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808140932_gehel_24356.log.

Completed auto-reimage of hosts:

['elastic1044.eqiad.wmnet', 'elastic1045.eqiad.wmnet', 'elastic1048.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic1040.eqiad.wmnet', 'elastic1041.eqiad.wmnet', 'elastic1042.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808141225_gehel_27997.log.

Completed auto-reimage of hosts:

['elastic1042.eqiad.wmnet', 'elastic1040.eqiad.wmnet', 'elastic1041.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic1036.eqiad.wmnet', 'elastic1037.eqiad.wmnet', 'elastic1038.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808141450_gehel_28916.log.

Completed auto-reimage of hosts:

['elastic1037.eqiad.wmnet', 'elastic1036.eqiad.wmnet', 'elastic1038.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic1033.eqiad.wmnet', 'elastic1034.eqiad.wmnet', 'elastic1035.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808141657_gehel_23163.log.

Completed auto-reimage of hosts:

['elastic1033.eqiad.wmnet']

Of which those FAILED:

['elastic1033.eqiad.wmnet']

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic1033.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808141734_gehel_1346.log.

Completed auto-reimage of hosts:

['elastic1033.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic1030.eqiad.wmnet', 'elastic1031.eqiad.wmnet', 'elastic1032.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808150938_gehel_6979.log.

Completed auto-reimage of hosts:

['elastic1031.eqiad.wmnet', 'elastic1030.eqiad.wmnet', 'elastic1032.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic1025.eqiad.wmnet', 'elastic1026.eqiad.wmnet', 'elastic1027.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808151413_gehel_325.log.

Completed auto-reimage of hosts:

['elastic1025.eqiad.wmnet', 'elastic1027.eqiad.wmnet', 'elastic1026.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic1028.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808151458_gehel_11705.log.

Completed auto-reimage of hosts:

['elastic1028.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic1018.eqiad.wmnet', 'elastic1019.eqiad.wmnet', 'elastic1020.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808151907_gehel_30992.log.

Completed auto-reimage of hosts:

['elastic1019.eqiad.wmnet', 'elastic1020.eqiad.wmnet', 'elastic1018.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic1023.eqiad.wmnet', 'elastic1024.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808160749_gehel_23929.log.

Completed auto-reimage of hosts:

['elastic1023.eqiad.wmnet']

Of which those FAILED:

['elastic1023.eqiad.wmnet']

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic1023.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808160813_gehel_30364.log.

Completed auto-reimage of hosts:

['elastic1023.eqiad.wmnet']

Of which those FAILED:

['elastic1023.eqiad.wmnet']

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic1022.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808160916_gehel_12564.log.

Completed auto-reimage of hosts:

['elastic1022.eqiad.wmnet']

and were ALL successful.

Mentioned in SAL (#wikimedia-operations) [2018-08-16T09:46:34Z] <gehel> all elasticsearch nodes reimaged (except elastic1029, waiting on memory issue) - T198391 / T193649 / T201991

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['elastic1029.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808161736_gehel_27521.log.

Completed auto-reimage of hosts:

['elastic1029.eqiad.wmnet']

and were ALL successful.

Mentioned in SAL (#wikimedia-operations) [2018-08-16T18:45:05Z] <gehel> reimage of elasticsearch eqiad completed - T198391 / T193649

debt closed this task as Resolved.Aug 24 2018, 3:57 PM