This task is blocked until the related rack/setup/deploy one is completed.
Nodes to refresh: analytics10[42-57]
This task is blocked until the related rack/setup/deploy one is completed.
Nodes to refresh: analytics10[42-57]
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Resolved | None | T244211 Analytics Hardware for Fiscal Year 2019/2020 | |||
Resolved | Ottomata | T243521 Hadoop Hardware Orders FY2019-2020 | |||
Resolved | elukey | T255140 Refresh 16 nodes in the Hadoop Analytics cluster |
Change 630991 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Update rack settings for new Analytics Hadoop nodes in hiera
Change 630991 merged by Elukey:
[operations/puppet@production] Update rack settings for new Analytics Hadoop nodes in hiera
Change 631391 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add hadoop worker node role to an-worker1103
Change 631391 merged by Elukey:
[operations/puppet@production] Add hadoop worker node role to an-worker1103
Change 631434 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Set an-worker110[45] as Hadoop workers
Change 631434 merged by Elukey:
[operations/puppet@production] Set an-worker110[45] as Hadoop workers
The plan is to add the 16 new nodes (expanding the cluster) progressively, and then remove the 16 old ones (shrinking the cluster) later on.
Change 631764 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Set an-worker110[6-9] as Hadoop workers
Change 631764 merged by Elukey:
[operations/puppet@production] Set an-worker110[6-9] as Hadoop workers
Change 632202 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Set an-worker111[02] as Hadoop workers
Change 632202 merged by Elukey:
[operations/puppet@production] Set an-worker111[02] as Hadoop workers
Change 632294 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Set an-worker111[5-7] as Hadoop workers
Change 632294 merged by Elukey:
[operations/puppet@production] Set an-worker111[5-7] as Hadoop workers
All nodes are now in Hadoop, just closed the rack/setup/deploy task. I am going to update the docs on adding worker nodes, they probably need a refresh.
Next step is to remove analytics1042->57 in small steps.
During the first puppet run, datanode and nodemanager fail for different reasons:
2020-10-06 09:17:19,422 FATAL org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Failed to initialize spark_shuffle java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found
2020-10-06 09:17:10,065 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain java.io.FileNotFoundException: /etc/hadoop/conf.analytics-hadoop/ssl/server.p12 (No such file or directory) at java.io.FileInputStream.open0(Native Method)
Seems to be two missing require/dependency in puppet :)
Change 632650 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Decommission analytics1042 from Analytics Hadoop
Change 632653 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::hadoop::master::standby: improve hiera lookups
Change 632650 merged by Elukey:
[operations/puppet@production] Decommission analytics1042 from Analytics Hadoop
Change 632653 merged by Elukey:
[operations/puppet@production] profile::hadoop::master::standby: improve hiera lookups
Change 633140 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Decommission analytics1044 from Hadoop
Change 633140 merged by Elukey:
[operations/puppet@production] Decommission analytics1044 from Hadoop
Change 633296 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Remove analytics1045 from the Hadoop cluster
Change 633296 merged by Elukey:
[operations/puppet@production] Remove analytics1045 from the Hadoop cluster
Change 633350 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Decommission analytics1046 from Hadoop
Change 633350 merged by Elukey:
[operations/puppet@production] Decommission analytics1046 from Hadoop
Change 633385 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Decom analytics1047 from the Hadoop cluster
Change 633385 merged by Elukey:
[operations/puppet@production] Decom analytics1047 from the Hadoop cluster
Change 633605 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Remove analytics1048 from the Hadoop cluster
Change 633605 merged by Elukey:
[operations/puppet@production] Remove analytics1048 from the Hadoop cluster
Change 633864 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Decommission analytics1049 from the Hadoop cluster
Change 633864 merged by Elukey:
[operations/puppet@production] Decommission analytics1049 from the Hadoop cluster
Change 634145 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Decommission analytics1050 from the Hadoop cluster
Change 634145 merged by Elukey:
[operations/puppet@production] Decommission analytics1050 from the Hadoop cluster
Change 634474 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Decommission analytics1051 from the Hadoop cluster
Change 634474 merged by Elukey:
[operations/puppet@production] Decommission analytics1051 from the Hadoop cluster
Change 634673 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Decommission analytics1053 from the Hadoop cluster
Change 634673 merged by Elukey:
[operations/puppet@production] Decommission analytics1053 from the Hadoop cluster
Change 634766 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Decom analytics1054 from Hadoop
Change 634766 merged by Elukey:
[operations/puppet@production] Decom analytics1054 from Hadoop
Change 634905 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Remove analytics1055 from the hadoop cluster
Change 634905 merged by Elukey:
[operations/puppet@production] Remove analytics1055 from the hadoop cluster
Change 635235 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Decommission analytics1056 from the Hadoop cluster
Change 635235 merged by Elukey:
[operations/puppet@production] Decommission analytics1056 from the Hadoop cluster
Change 635507 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Remove analytics1052 from Hadoop HDFS Journal nodes
Change 635507 merged by Elukey:
[operations/puppet@production] Remove analytics1052 from Hadoop HDFS Journal nodes
Change 635521 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Decommission analytics1052 from the Hadoop cluster
Change 635521 merged by Elukey:
[operations/puppet@production] Decommission analytics1052 from the Hadoop cluster
Change 635742 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Decom analytics1057 from the Hadoop cluster
Change 635742 merged by Elukey:
[operations/puppet@production] Decom analytics1057 from the Hadoop cluster
Change 635750 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] hadoop: final clean up after the decommission of old nodes
Change 635750 merged by Elukey:
[operations/puppet@production] hadoop: final clean up after the decommission of old nodes