Page MenuHomePhabricator

Fix wikimedia-event-utilities Guava dependencies issues
Closed, ResolvedPublic

Description

In https://gerrit.wikimedia.org/r/c/wikimedia-event-utilities/+/857779, I made an attempt at downgrading the Guava dependency of wikimedia-event-utilities to avoid conflicts with different deployed versions. This turned out to be harder than expected, so we removed guava from refinery-source instead.

It seems that we still have a conflict with Spark 3 and Hadoop though, and I believe @Milimetric had a similar problem with Flink/Iceberg.

To avoid issues like this, ideally we could remove Guava from wikimedia-event-utilities too. We can get most of the way there by replacing all direct usages of guava, e.g. ImmutableList, Resource, etc. However, Guava is still a transitive dependencies of the JSONSchema validator library event-utilities uses, so we can't totally eliminate it. Instead, we could isolate the transitive Guava dependency by putting the JSONSchema validator code into a standalone module.

  • Remove all direct uses of Guava
  • Create new maven module for code that transitively uses Guava, with shaded version of the artifact jar.
NOTE: There might be other ways to solve this too, so please investigate if you can think of any (relocating guava?)

Event Timeline

Change 923624 had a related patch set uploaded (by DCausse; author: DCausse):

[wikimedia-event-utilities@master] Added a new shaded jar with relocated guava classes

https://gerrit.wikimedia.org/r/923624

Change 923646 had a related patch set uploaded (by DCausse; author: DCausse):

[analytics/refinery/source@master] [WIP] Use eventutilites shaded jar

https://gerrit.wikimedia.org/r/923646

JArguello-WMF moved this task from Backlog to Sprint 14 B on the Event-Platform board.
gmodena added a subscriber: gmodena.

Change 923624 merged by jenkins-bot:

[wikimedia-event-utilities@master] Added a new shaded jar with relocated guava classes

https://gerrit.wikimedia.org/r/923624

Change 923646 merged by jenkins-bot:

[analytics/refinery/source@master] Use eventutilites shaded jar

https://gerrit.wikimedia.org/r/923646

From my POV this is done and should be available in refinery v0.2.16 (v0.2.17 seems to be the one deployed). Unless we encounter weird classloading issues in production I think we can close this task?

Ottomata claimed this task.