We know that we need to upgrade our Hadoop clusters from version 2.10.2 to a more recent version, probably 3.3.6 or 3.4.0
In addition, we need to make sure that all of the services that are integrated with Hadoop either continue to function, or can be deprecated.
These services include:
- Hive metastore
- Hive query engine (potential to deprecate?)
- Spark
- Flink
- Sqoop (Does not support Hadoop 3, but can allegedly be made to work with wire compatibility to Hadoop 2)
- Gobblin
- Presto
- Druid
The purpose of this task is to track the initial creation of an upgrade plan, upon which we can then iterate.
We also need to make sure that we have sufficient safeguards in place in order to avoid data loss on HDFS.