Page MenuHomePhabricator
Create Task
Maniphest T370199

gobblin-wmf: bump event-utilities dependency to unblock MW on K8s migration.
Closed, ResolvedPublic
Actions

Description

Our gobblin-wmf code uses EventStreamConfig to discover streams to ingest into HDFS. We need version 1.3.6 or later of event-utilities to avoid WikimediaDefaults from contacting the deprecated MW API.

  • gobblin-wmf update event-utilties dependency to 1.3.6
  • analytics/refinery updated and deployed with latest gobblin-wmf version
  • gobblin.pp puppet job(s) updated to use latest gobblin-wmf jar version
  • It looks like gobblin.pp in puppet is using an undefined hiera value to configure the jar to use. This should be fixed.
NOTE: The right thing to do is to remove the statically instantiated WikimediaDefaults in event-utilities altogether.

Details

SubjectRepoBranchLines +/-
refinery::job::gobblin - use gobbin-wmf 1.0.2operations/puppetproduction+10 -18
refinery::job::test::gobblin - use gobbin-wmf 1.0.2operations/puppetproduction+5 -6
Fix location of gobblin-wmf-core-jar-with-dependencies.jar symlinkanalytics/refinerymaster+1 -2
artifacts: add gobblin-wmf 1.0.2analytics/refinerymaster+5 -1
Bump wikimedia-event-utilities version to 1.3.6analytics/gobblin-wmfmain+55 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Ottomata created this task.Jul 16 2024, 7:13 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 16 2024, 7:13 PM
gerritbot added a comment.Jul 16 2024, 7:14 PM

Change #1054652 had a related patch set uploaded (by Ottomata; author: Ottomata):

[analytics/gobblin-wmf@main] Bump wikimedia-event-utilities version to 1.3.6

https://gerrit.wikimedia.org/r/1054652

gerritbot added a project: Patch-For-Review.Jul 16 2024, 7:14 PM
Ottomata added a project: Data-Engineering.Jul 16 2024, 7:16 PM
Ottomata updated the task description. (Show Details)
Ottomata added a subscriber: BTullis.
Clement_Goubert subscribed.Jul 17 2024, 8:40 AM
gerritbot added a comment.Jul 17 2024, 2:06 PM

Change #1054652 merged by jenkins-bot:

[analytics/gobblin-wmf@main] Bump wikimedia-event-utilities version to 1.3.6

https://gerrit.wikimedia.org/r/1054652

Maintenance_bot removed a project: Patch-For-Review.Jul 17 2024, 2:30 PM
gerritbot added a comment.Jul 17 2024, 3:16 PM

Change #1054898 had a related patch set uploaded (by Gmodena; author: Gmodena):

[analytics/refinery@master] artifacts: add gobblin-wmf 1.0.2

https://gerrit.wikimedia.org/r/1054898

gerritbot added a project: Patch-For-Review.Jul 17 2024, 3:16 PM
gerritbot added a comment.Jul 17 2024, 3:41 PM

Change #1054898 merged by Ottomata:

[analytics/refinery@master] artifacts: add gobblin-wmf 1.0.2

https://gerrit.wikimedia.org/r/1054898

gerritbot added a comment.Jul 17 2024, 3:49 PM

Change #1054904 had a related patch set uploaded (by Ottomata; author: Ottomata):

[analytics/refinery@master] Fix location of gobblin-wmf-core-jar-with-dependencies.jar symlink

https://gerrit.wikimedia.org/r/1054904

gerritbot added a comment.Jul 17 2024, 3:49 PM

Change #1054904 merged by Ottomata:

[analytics/refinery@master] Fix location of gobblin-wmf-core-jar-with-dependencies.jar symlink

https://gerrit.wikimedia.org/r/1054904

gerritbot added a comment.Jul 17 2024, 4:01 PM

Change #1054906 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] refinery::job::test::gobblin - use gobbin-wmf 1.0.2

https://gerrit.wikimedia.org/r/1054906

gerritbot added a comment.Jul 17 2024, 4:06 PM

Change #1054906 merged by Ottomata:

[operations/puppet@production] refinery::job::test::gobblin - use gobbin-wmf 1.0.2

https://gerrit.wikimedia.org/r/1054906

gerritbot added a comment.Jul 17 2024, 4:12 PM

Change #1054908 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] refinery::job::gobblin - use gobbin-wmf 1.0.2

https://gerrit.wikimedia.org/r/1054908

gerritbot added a comment.Jul 17 2024, 4:24 PM

Change #1054908 merged by Ottomata:

[operations/puppet@production] refinery::job::gobblin - use gobbin-wmf 1.0.2

https://gerrit.wikimedia.org/r/1054908

Maintenance_bot removed a project: Patch-For-Review.Jul 17 2024, 4:30 PM
Ottomata updated the task description. (Show Details)Jul 17 2024, 4:47 PM
Ottomata moved this task from Incoming (new tickets) to Q1 2024 July 1st - September 30th on the Data-Engineering board.
Ottomata edited projects, added Data-Engineering (Q1 2024 July 1st - September 30th); removed Data-Engineering.
Ottomata moved this task from Next Up to Done on the Data-Engineering (Q1 2024 July 1st - September 30th) board.
gmodena mentioned this in T370368: [NEEDS GROOMING] We should improve the code health of gobblin-wmf.Jul 18 2024, 11:13 AM
Ahoelzl closed this task as Resolved.Oct 9 2024, 2:00 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits