Page MenuHomePhabricator

Migrate Streaming Updater to newer Scala version
Open, MediumPublic

Description

As a Streaming Updater maintainter I want to be able to compile and run streaming updater on newer versions of Scala than 2.11, so that I can update Apache Flink version past the 2.11 support drop date.

As we learned (which isn't really that surprising) Flink might drop Scala 2.11 support soon, which is the version we used to develop Streaming Updater - a dependency from the Hadoop version we use. Since we are moving away from Hadoop to Kubernetes, it will not be our limitation for long. Since we don't really have a version requirement other than Hadoop - we can probably move to Scala 2.13 - but this has to be tested, since Flink is compiled with 2.12 (it should be binary compatible, though). In any case, we should go any lower than 2.12.8, because of binary incompatibility introduced since previous version.

AC:

  • We can safely run Flink and our pipeline on at least Scala 2.12.8
  • CI/CD is configured accordingly