In order to deprecate Spark 2 and fully move to Spark 3 all the current jobs need to be migrated. This task is to create a migration plan and provide a communication to teams who own Spark 2 jobs and will need to budget time to handle the migration.
Key Tasks:
- Create an inventory of all the remaining jobs still using spark 2, including owners and stakeholders
- Mark those which are DEs responsibility to move and migrate them
- Write a statement of intent that we are moving to spark 3 including
- Changes to workflows
- Changes to data
- Features and Benefits of the move
- End Date for support of Spark 2
- Finalizations
- Send out Final Communication to Analytics Group