Some testing already done by Joseph as part of the Iceberg investigation. Need to validate that Spark 3 doesn't cause any problems and change the puppet code that runs Refine to use Spark 3 and a new jar.
If there are any problems, update code to be compatible.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Snwachukwu | T335308 Migrate Refine to Spark 3 | |||
| Resolved | Ottomata | T337421 Fix wikimedia-event-utilities Guava dependencies issues | |||
| Resolved | None | T338343 Wikimedia-event-utilities jenkins build failure |
Currently getting the error below when running refine jobs with spark3: Yet to get a solution for this but will update once I get one.
23/05/18 16:25:09 ERROR Client: Application diagnostics message: User class threw exception: java.lang.NoSuchMethodError: com.google.common.collect.ImmutableList.toImmutableList()Ljava/util/stream/Collector;
at org.wikimedia.eventutilities.core.util.ResourceLoader.asURLs(ResourceLoader.java:253)
at org.wikimedia.analytics.refinery.job.refine.EventSparkSchemaLoader$.buildEventSchemaLoader(SchemaLoaders.scala:112)
at org.wikimedia.analytics.refinery.job.refine.EventSparkSchemaLoader$.apply(SchemaLoaders.scala:100)
at org.wikimedia.analytics.refinery.job.refine.Refine$.getRefineTargetsFromFS(Refine.scala:631)
at org.wikimedia.analytics.refinery.job.refine.Refine$.getRefineTargets(Refine.scala:574)
at org.wikimedia.analytics.refinery.job.refine.Refine$.apply(Refine.scala:371)
at org.wikimedia.analytics.refinery.job.refine.Refine$.main(Refine.scala:345)
at org.wikimedia.analytics.refinery.job.refine.Refine.main(Refine.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:732)
Exception in thread "main" org.apache.spark.SparkException: Application application_1678266962370_410117 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1242)
at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1634)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1039)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1048)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
23/05/18 16:25:09 INFO ShutdownHookManager: Shutdown hook calledChange 928525 had a related patch set uploaded (by Snwachukwu; author: Snwachukwu):
[operations/puppet@production] Use refinery v0.2.16 in refine jobs.
Change 928525 merged by Stevemunene:
[operations/puppet@production] Use refinery v0.2.16 in refine jobs.
Change 930669 had a related patch set uploaded (by Snwachukwu; author: Joal):
[operations/puppet@production] Move spark_jobs from spark2 to spark3
Change 930669 merged by Ottomata:
[operations/puppet@production] Move spark_jobs from spark2 to spark3
Change 930684 had a related patch set uploaded (by Ottomata; author: Ottomata):
[operations/puppet@production] Remove reference to absent ::druid_load classes
Change 930684 merged by Ottomata:
[operations/puppet@production] Remove reference to absent ::druid_load classes
Change 930706 had a related patch set uploaded (by Ottomata; author: Ottomata):
[operations/puppet@production] refine - Use trailing / for schema base uris
Change 930706 merged by Ottomata:
[operations/puppet@production] refine - Use trailing / for schema base uris
Change 930708 had a related patch set uploaded (by Ottomata; author: Ottomata):
[operations/puppet@production] refine_test - Use trailing / for schema base uris
So: https://gerrit.wikimedia.org/r/c/operations/puppet/+/928525/ updated the jar version used by Refine. This broke, because the new version needs spark3, and spark2-submit was still being used to launch this.
Sandra submitted https://gerrit.wikimedia.org/r/c/operations/puppet/+/930669/ to fix this. I merged. Had to fix a puppet thing, but ultimately spark3 was used to launch refine.
There seems to be a new change somewhere in the code that caused something with URI joining to break. https://gerrit.wikimedia.org/r/c/operations/puppet/+/930706 seems to fix the problem.
As far as I can tell, Refine jobs look like they are running now.
@JAllemandou @Milimetric @Snwachukwu @Stevemunene, in the future, we should do this for only analytics_test first, and try to make sure refine works there. Also, a roll out like this needs a little more supervision! I probably should have reverted the jar bump instead of merging Sandra's fix, but I think things are working now.
Anyway... VERY EXCITING about Refine and Spark 3!!! Really hoping we can do T255818#7992353 soon!
Also! The logging change we made for T309717: Event Utilities partially downloads schemas should now be live with the updated refinery jar, so maybe we'll get some more info about why Refine fails sometimes?!
Change 930713 had a related patch set uploaded (by Ottomata; author: Ottomata):
[operations/puppet@production] refine & spark_job - parameterize spark_submit executable path and use spark2 for RefineSanitize
Change 930708 merged by Ottomata:
[operations/puppet@production] refine_test - Use trailing / for schema base uris
Change 930713 merged by Ottomata:
[operations/puppet@production] refine - parameterize spark_submit and use spark2 for RefineSanitize
Ah, and looks like something is not quite right with RefineSanitize.
Reverted to spark2:
https://gerrit.wikimedia.org/r/c/operations/puppet/+/930713
Change 930714 had a related patch set uploaded (by Ottomata; author: Ottomata):
[operations/puppet@production] refine_sanitize - Fix typo in spark_submit path
Change 930714 merged by Ottomata:
[operations/puppet@production] refine_sanitize - Fix typo in spark_submit path
in the future, we should do this for only analytics_test first, and try to make sure refine works there. Also, a roll out like this needs a little more supervision!
+1 for a more thorough deploy process, test cluster first, then production, with multiple eyes.
Change 930765 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Revert the change in jar version for refine_sanitize
Change 930765 merged by Btullis:
[operations/puppet@production] Revert the change in jar version for refine_sanitize
RefineSanitize still was failing on Friday, so Ben and Joseph helped to successfully revert RefineSanitize to spark2 by reverting the refinery jar version from v0.2.16 back to v0.1.15. https://gerrit.wikimedia.org/r/930765 to allow it work during the weekend.
Today, I have pushed this patch to test that refine_sanitize job. If all is fine then I'll go ahead and deploy in production to allow refine_sanitize work with Spark3.
Change 931284 had a related patch set uploaded (by Snwachukwu; author: Snwachukwu):
[operations/puppet@production] Test Refine_sanitize migration to spark3.
Change 931284 merged by Btullis:
[operations/puppet@production] Test Refine_sanitize migration to spark3.
Change 931623 had a related patch set uploaded (by Snwachukwu; author: Snwachukwu):
[operations/puppet@production] Migrate refine sanitize to spark3.
Change 931623 merged by Ottomata:
[operations/puppet@production] Migrate refine_sanitize to spark3.