Page MenuHomePhabricator
Create Task
Maniphest T271232

Replace Camus by Gobblin
Closed, ResolvedPublic
Actions

Description

After having tested that Gobblin works with Kafka using TLS encryption (Yay!), we decided to move with this tool.
This task is the parent task for the various smaller bits we'll have to do.

Things to do:

We have ideas of improvements that Gobblin could help us with (webrequest stats for instance), but we're gonna keep the work to just replacing camus for now.
We'll create new tasks with improvements as our knowledge of Gobblin levels up.

Details

SubjectRepoBranchLines +/-
Refine - replace default formatters with gobblin conventionanalytics/refinery/sourcemaster+20 -21
Remove refinery-camus moduleanalytics/refinery/sourcemaster+5 -1 K
Remove references to camusanalytics/refinerymaster+1 -129
refinery/job - remove already absented jobsoperations/puppetproduction+0 -52
Remove no longer used camus module and references to camusoperations/puppetproduction+2 -385
Set gobblin job.lock.dir after allanalytics/refinerymaster+1 -4
Remove camus puppetizationoperations/puppetproduction+0 -161
Ensure remaining camus jobs are absentoperations/puppetproduction+3 -0
Add 5 minutes offset to gobblin webrequest timeroperations/puppetproduction+5 -3
Bump refine refinery-job version to 0.1.15operations/puppetproduction+4 -4
Refine - explicitly uncache DataFrame when doneanalytics/refinery/sourcemaster+7 -1
Tune Refine jobs in production hadoopoperations/puppetproduction+28 -12
Bump Refine spark_executor_memory to 8Goperations/puppetproduction+3 -1
Ensure camus eventlogging job is absentoperations/puppetproduction+1 -0
Finalize eventlogging_legacy gobblin job migrationoperations/puppetproduction+55 -46
Add gobbln job eventlogging_legacyoperations/puppetproduction+7 -106
Add gobblin job eventlogging_legacyanalytics/refinerymaster+22 -0
Refine event - fix input_path_regex_capture_groups paramoperations/puppetproduction+1 -1
Finalze gobblin event migrationoperations/puppetproduction+27 -23
Finalize gobblin migration for event_default jobanalytics/refinerymaster+1 -1
gobbin event_default - Fix typoanalytics/refinerymaster+1 -1
Set number of max mappers for gobblin event_default to 128analytics/refinerymaster+7 -3
Add event_default gobblin jobanalytics/refinerymaster+26 -0
Add gobblin job event_defaultoperations/puppetproduction+9 -1
Remove already absented camus jobsoperations/puppetproduction+2 -166
test/data_purge - fix checksum on refinery-drop-raw-eventoperations/puppetproduction+1 -1
Gobblinize test refine_event and drop_event jobsoperations/puppetproduction+32 -17
Finalize event_default_test gobblin migrationanalytics/refinerymaster+1 -3
Fix for refine eventlogging_legacy_testoperations/puppetproduction+7 -3
Fix checksum for refinery-drop-eventlogging-legacy-raw-partitionsoperations/puppetproduction+1 -1
Gobblinize eventlogging_legacy_test Refine and data purge jobsoperations/puppetproduction+34 -20
Revert gobblin time-pattern to use padded hoursanalytics/refinerymaster+12 -11
Put gobblin essential-queue property in common fileanalytics/refinerymaster+2 -5
Add gobblin job in hadoop test event_default_testoperations/puppetproduction+5 -1
Add gobblin event_default_test jobanalytics/refinerymaster+27 -0
Make gobblin jobs run in essential queueanalytics/refinerymaster+8 -2
gobblin eventlogging_legacy_test - use final import pathanalytics/refinerymaster+5 -3
Add gobblin job in hadoop test eventlogging_legacy_testoperations/puppetproduction+3 -1
Fix refinery-dump_status-webrequest-partition after gobblinanalytics/refinerymaster+14 -17
Add gobblin/jobs/eventlogging_legacy_test.pullanalytics/refinerymaster+22 -0
Run gobblin netflow at 5 and 35 after the houroperations/puppetproduction+16 -2
Use hive/gobblin date path format for netflow data purge joboperations/puppetproduction+19 -14
Gobblinize netflow Hadoop ingestionoperations/puppetproduction+15 -32
Make gobblin-netflow use production directoryanalytics/refinerymaster+1 -1
RefineTarget - support gzipped json as input data formatanalytics/refinery/sourcemaster+57 -14
Update webrequest raw data purge job with new partition path format for gobblinoperations/puppetproduction+3 -3
Ensure absent webrequest camus jobsoperations/puppetproduction+2 -17
Make gobblin-webrequest use production folderanalytics/refinerymaster+1 -1
Add permissions settings for gobblinanalytics/refinerymaster+5 -0
Declare analytics webrequest and netflow gobblin jobsoperations/puppetproduction+28 -0
Add webrequest and netflow gobblin jobsanalytics/refinerymaster+34 -0
Fix webrequest datasets_raw.xml for gobblinanalytics/refinerymaster+6 -6
Remove camus webrequest job in analytics test clusteroperations/puppetproduction+2 -0
Update webrequest for gobblinanalytics/refinerymaster+9 -13
Update webrequest_test gobblin job to prod folderanalytics/refinerymaster+1 -1
gobblin_job - set PYTHONPATH in environmentoperations/puppetproduction+6 -2
gobblin_job - fix typo in path to default jobconfig fileoperations/puppetproduction+1 -1
Add gobblin_job define and declare first gobblin job in hadoop test clusteroperations/puppetproduction+92 -0
Add webrequest_test gobblin jobanalytics/refinerymaster+17 -0
Add gobblin-wmf.jar symlink to versioned gobblin jaranalytics/refinerymaster+1 -0
Add bin/gobblin wrapper and initial gobblin/ common properties filesanalytics/refinerymaster+240 -0
Add gobblin-wmf to artifacts folderanalytics/refinerymaster+1 -0
Rename parent package to follow wmf conventionanalytics/gobblinwmf+44 -53
Support getting EventStreamConfig settings by JsonPointer pathwikimedia-event-utilitiesmaster+147 -45
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Jul 9 2021, 3:16 PM

Change 703857 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Remove already absented camus jobs

https://gerrit.wikimedia.org/r/703857

gerritbot added a comment.Jul 9 2021, 3:22 PM

Change 703857 merged by Ottomata:

[operations/puppet@production] Remove already absented camus jobs

https://gerrit.wikimedia.org/r/703857

gerritbot added a comment.Jul 9 2021, 5:32 PM

Change 703866 had a related patch set uploaded (by Ottomata; author: Ottomata):

[analytics/refinery@master] Add event_default gobblin job

https://gerrit.wikimedia.org/r/703866

gerritbot added a comment.Jul 9 2021, 5:36 PM

Change 703867 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Add gobblin job event_default

https://gerrit.wikimedia.org/r/703867

gerritbot added a comment.Jul 9 2021, 6:59 PM

Change 703869 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Finalze gobblin event migration

https://gerrit.wikimedia.org/r/703869

gerritbot added a comment.Jul 12 2021, 1:32 PM

Change 703866 merged by Ottomata:

[analytics/refinery@master] Add event_default gobblin job

https://gerrit.wikimedia.org/r/703866

Stashbot added a comment.Jul 12 2021, 1:32 PM

Mentioned in SAL (#wikimedia-operations) [2021-07-12T13:32:55Z] <otto@deploy1002> Started deploy [analytics/refinery@1cb9e12]: Add event_default gobblin job - T271232

Stashbot added a comment.Jul 12 2021, 1:36 PM

Mentioned in SAL (#wikimedia-operations) [2021-07-12T13:36:32Z] <otto@deploy1002> Finished deploy [analytics/refinery@1cb9e12]: Add event_default gobblin job - T271232 (duration: 03m 37s)

gerritbot added a comment.Jul 12 2021, 1:37 PM

Change 703867 merged by Ottomata:

[operations/puppet@production] Add gobblin job event_default

https://gerrit.wikimedia.org/r/703867

gerritbot added a comment.Jul 12 2021, 1:45 PM

Change 704117 had a related patch set uploaded (by Ottomata; author: Ottomata):

[analytics/refinery@master] Set number of max mappers for gobblin event_default to 128

https://gerrit.wikimedia.org/r/704117

gerritbot added a comment.Jul 12 2021, 1:49 PM

Change 704117 merged by Ottomata:

[analytics/refinery@master] Set number of max mappers for gobblin event_default to 128

https://gerrit.wikimedia.org/r/704117

Stashbot added a comment.Jul 12 2021, 1:49 PM

Mentioned in SAL (#wikimedia-operations) [2021-07-12T13:49:28Z] <otto@deploy1002> Started deploy [analytics/refinery@0149c81]: Set event_default gobblin job max mappers=128 - T271232

Stashbot added a comment.Jul 12 2021, 1:52 PM

Mentioned in SAL (#wikimedia-operations) [2021-07-12T13:52:44Z] <otto@deploy1002> Finished deploy [analytics/refinery@0149c81]: Set event_default gobblin job max mappers=128 - T271232 (duration: 03m 16s)

gerritbot added a comment.Jul 12 2021, 1:55 PM

Change 704118 had a related patch set uploaded (by Ottomata; author: Ottomata):

[analytics/refinery@master] gobbin event_default - Fix typo

https://gerrit.wikimedia.org/r/704118

gerritbot added a comment.Jul 12 2021, 1:55 PM

Change 704118 merged by Ottomata:

[analytics/refinery@master] gobbin event_default - Fix typo

https://gerrit.wikimedia.org/r/704118

Stashbot added a comment.Jul 12 2021, 1:56 PM

Mentioned in SAL (#wikimedia-operations) [2021-07-12T13:56:01Z] <otto@deploy1002> Started deploy [analytics/refinery@dd65f38]: event_default gobblin job - fix typo - T271232

Stashbot added a comment.Jul 12 2021, 1:59 PM

Mentioned in SAL (#wikimedia-operations) [2021-07-12T13:59:31Z] <otto@deploy1002> Finished deploy [analytics/refinery@dd65f38]: event_default gobblin job - fix typo - T271232 (duration: 03m 30s)

gerritbot added a comment.Jul 12 2021, 6:36 PM

Change 704152 had a related patch set uploaded (by Ottomata; author: Ottomata):

[analytics/refinery@master] Finalize gobblin migration for event_default job

https://gerrit.wikimedia.org/r/704152

gerritbot added a comment.Jul 12 2021, 6:37 PM

Change 704152 merged by Ottomata:

[analytics/refinery@master] Finalize gobblin migration for event_default job

https://gerrit.wikimedia.org/r/704152

Stashbot added a comment.Jul 12 2021, 6:37 PM

Mentioned in SAL (#wikimedia-operations) [2021-07-12T18:37:27Z] <otto@deploy1002> Started deploy [analytics/refinery@200b502]: Finalize event_default gobblin job - T271232

Stashbot added a comment.Jul 12 2021, 6:41 PM

Mentioned in SAL (#wikimedia-operations) [2021-07-12T18:41:06Z] <otto@deploy1002> Finished deploy [analytics/refinery@200b502]: Finalize event_default gobblin job - T271232 (duration: 03m 39s)

gerritbot added a comment.Jul 12 2021, 6:42 PM

Change 703869 merged by Ottomata:

[operations/puppet@production] Finalze gobblin event migration

https://gerrit.wikimedia.org/r/703869

gerritbot added a comment.Jul 12 2021, 6:49 PM

Change 704154 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Refine event - fix input_path_regex_capture_groups param

https://gerrit.wikimedia.org/r/704154

gerritbot added a comment.Jul 12 2021, 6:50 PM

Change 704154 merged by Ottomata:

[operations/puppet@production] Refine event - fix input_path_regex_capture_groups param

https://gerrit.wikimedia.org/r/704154

gerritbot added a comment.Jul 12 2021, 8:17 PM

Change 704157 had a related patch set uploaded (by Ottomata; author: Ottomata):

[analytics/refinery@master] Add gobblin job eventlogging_legacy

https://gerrit.wikimedia.org/r/704157

gerritbot added a comment.Jul 12 2021, 8:25 PM

Change 704159 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Add gobbln job eventlogging_legacy

https://gerrit.wikimedia.org/r/704159

gerritbot added a comment.Jul 12 2021, 8:34 PM

Change 704161 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Finalize eventlogging_legacy gobblin job migration

https://gerrit.wikimedia.org/r/704161

gerritbot added a comment.Jul 13 2021, 1:53 PM

Change 704157 merged by Ottomata:

[analytics/refinery@master] Add gobblin job eventlogging_legacy

https://gerrit.wikimedia.org/r/704157

Stashbot added a comment.Jul 13 2021, 1:53 PM

Mentioned in SAL (#wikimedia-operations) [2021-07-13T13:53:35Z] <otto@deploy1002> Started deploy [analytics/refinery@a3bc8bc]: Add eventlogging_legacy gobblin job - T271232

Stashbot added a comment.Jul 13 2021, 1:57 PM

Mentioned in SAL (#wikimedia-operations) [2021-07-13T13:57:04Z] <otto@deploy1002> Finished deploy [analytics/refinery@a3bc8bc]: Add eventlogging_legacy gobblin job - T271232 (duration: 03m 28s)

gerritbot added a comment.Jul 13 2021, 1:58 PM

Change 704159 merged by Ottomata:

[operations/puppet@production] Add gobbln job eventlogging_legacy

https://gerrit.wikimedia.org/r/704159

gerritbot added a comment.Jul 13 2021, 7:31 PM

Change 704161 merged by Ottomata:

[operations/puppet@production] Finalize eventlogging_legacy gobblin job migration

https://gerrit.wikimedia.org/r/704161

gerritbot added a comment.Jul 13 2021, 7:34 PM

Change 704412 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Ensure camus eventlogging job is absent

https://gerrit.wikimedia.org/r/704412

gerritbot added a comment.Jul 13 2021, 7:35 PM

Change 704412 merged by Ottomata:

[operations/puppet@production] Ensure camus eventlogging job is absent

https://gerrit.wikimedia.org/r/704412

gerritbot added a comment.Jul 14 2021, 1:43 PM

Change 704541 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Bump Refine spark_executor_memory to 8G

https://gerrit.wikimedia.org/r/704541

gerritbot added a comment.Jul 14 2021, 1:47 PM

Change 704541 merged by Ottomata:

[operations/puppet@production] Bump Refine spark_executor_memory to 8G

https://gerrit.wikimedia.org/r/704541

Ottomata added a comment.Jul 14 2021, 2:52 PM

@JAllemandou These are the OOMs in Refine we are getting, def looks gzip related:

Container: container_e18_1623774792907_150589_01_000010 on an-worker1132.eqiad.wmnet_8041_1626272654052
LogAggregationType: AGGREGATED
=======================================================================================================
LogType:stderr
LogLastModifiedTime:Wed Jul 14 14:24:14 +0000 2021
LogLength:30698
LogContents:

[...]

21/07/14 13:51:44 INFO ZlibFactory: Successfully loaded & initialized native-zlib library
21/07/14 13:52:40 INFO UnsafeExternalSorter: Thread 120 spilling sort data of 1472.0 MB to disk (0  time so far)
org.apache.spark.SparkException: Exception thrown in awaitResult:
	at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:226)
	at org.apache.spark.network.BlockTransferService.fetchBlockSync(BlockTransferService.scala:121)
	at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: Boxed Error
	at scala.concurrent.impl.Promise$.resolver(Promise.scala:59)
	at scala.concurrent.impl.Promise$.scala$concurrent$impl$Promise$$resolveTry(Promise.scala:51)
	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
	at scala.concurrent.Promise$class.complete(Promise.scala:55)
	at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:157)
	at scala.concurrent.Promise$class.failure(Promise.scala:104)
	at scala.concurrent.impl.Promise$DefaultPromise.failure(Promise.scala:157)
	at org.apache.spark.network.BlockTransferService$$anon$1.onBlockFetchSuccess(BlockTransferService.scala:110)
	at org.apache.spark.network.client.TransportResponseHandler.handle(TransportResponseHandler.java:171)
	at org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:120)
	at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
	at org.apache.spark.network.crypto.TransportCipher$DecryptionHandler.channelRead(TransportCipher.java:186)
	... 1 more
Caused by: java.lang.OutOfMemoryError: Java heap space
	at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
	at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
	at org.apache.spark.network.BlockTransferService$$anon$1.onBlockFetchSuccess(BlockTransferService.scala:111)
	at org.apache.spark.network.client.TransportResponseHandler.handle(TransportResponseHandler.java:171)
	at org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:120)
	at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
	at org.apache.spark.network.crypto.TransportCipher$DecryptionHandler.channelRead(TransportCipher.java:186)
21/07/14 13:53:57 INFO ShutdownHookManager: Shutdown hook called

End of LogType:stderr

This is even after bumping executor memory to 8G. The jobs can be re-run. Is it possible that when tasks are scheduled on the same executor, they don't free up memory between each other well? Or, maybe too many tasks in parallel in the same executor?

gerritbot added a comment.Jul 14 2021, 3:59 PM

Change 704563 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Tune Refine jobs in production hadoop

https://gerrit.wikimedia.org/r/704563

gerritbot added a comment.Jul 14 2021, 4:55 PM

Change 704563 merged by Ottomata:

[operations/puppet@production] Tune Refine jobs in production hadoop

https://gerrit.wikimedia.org/r/704563

gerritbot added a comment.Jul 14 2021, 5:15 PM

Change 704576 had a related patch set uploaded (by Ottomata; author: Ottomata):

[analytics/refinery/source@master] Refine - explicitly uncache DataFrame when done

https://gerrit.wikimedia.org/r/704576

gerritbot added a comment.Jul 15 2021, 1:29 PM

Change 704576 merged by Ottomata:

[analytics/refinery/source@master] Refine - explicitly uncache DataFrame when done

https://gerrit.wikimedia.org/r/704576

gerritbot added a comment.Jul 15 2021, 4:40 PM

Change 704842 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Bump refine refinery-job version to 0.1.15

https://gerrit.wikimedia.org/r/704842

Stashbot added a comment.Jul 15 2021, 4:44 PM

Mentioned in SAL (#wikimedia-analytics) [2021-07-15T16:44:50Z] <ottomata> deploying refinery and refinery-source 0.1.15 for refine job fixes - T271232

Ottomata updated the task description. (Show Details)Jul 15 2021, 4:46 PM
gerritbot added a comment.Jul 15 2021, 5:28 PM

Change 704842 merged by Ottomata:

[operations/puppet@production] Bump refine refinery-job version to 0.1.15

https://gerrit.wikimedia.org/r/704842

gerritbot added a comment.Jul 20 2021, 8:49 AM

Change 705621 had a related patch set uploaded (by Joal; author: Joal):

[operations/puppet@production] Add 5 minutes offset to gobblin webrequest timer

https://gerrit.wikimedia.org/r/705621

gerritbot added a comment.Jul 20 2021, 10:05 AM

Change 705621 merged by Elukey:

[operations/puppet@production] Add 5 minutes offset to gobblin webrequest timer

https://gerrit.wikimedia.org/r/705621

JAllemandou moved this task from In Progress to Ready to Deploy on the Analytics-Kanban board.Jul 20 2021, 4:10 PM
gerritbot added a comment.Jul 22 2021, 2:22 PM

Change 706492 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Ensure remaining camus jobs are absent

https://gerrit.wikimedia.org/r/706492

gerritbot added a comment.Jul 22 2021, 2:31 PM

Change 706492 merged by Ottomata:

[operations/puppet@production] Ensure remaining camus jobs are absent

https://gerrit.wikimedia.org/r/706492

gerritbot added a comment.Jul 22 2021, 3:42 PM

Change 706541 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Remove camus puppetization

https://gerrit.wikimedia.org/r/706541

gerritbot added a comment.Jul 22 2021, 3:49 PM

Change 706541 merged by Ottomata:

[operations/puppet@production] Remove camus puppetization

https://gerrit.wikimedia.org/r/706541

Stashbot added a comment.Jul 22 2021, 6:38 PM

Mentioned in SAL (#wikimedia-operations) [2021-07-22T18:38:11Z] <otto@deploy1002> Started deploy [analytics/refinery@1ef4fe1]: bin/gobbin wrapper now avoids launching if job is already running - T271232

Ottomata updated the task description. (Show Details)Jul 22 2021, 6:38 PM
Stashbot added a comment.Jul 22 2021, 6:41 PM

Mentioned in SAL (#wikimedia-operations) [2021-07-22T18:41:29Z] <otto@deploy1002> Finished deploy [analytics/refinery@1ef4fe1]: bin/gobbin wrapper now avoids launching if job is already running - T271232 (duration: 03m 18s)

gerritbot added a comment.Jul 22 2021, 6:55 PM

Change 706673 had a related patch set uploaded (by Ottomata; author: Ottomata):

[analytics/refinery@master] Set gobblin job.lock.dir after all

https://gerrit.wikimedia.org/r/706673

gerritbot added a comment.Jul 22 2021, 6:55 PM

Change 706673 merged by Ottomata:

[analytics/refinery@master] Set gobblin job.lock.dir after all

https://gerrit.wikimedia.org/r/706673

Stashbot added a comment.Jul 22 2021, 6:56 PM

Mentioned in SAL (#wikimedia-operations) [2021-07-22T18:56:06Z] <otto@deploy1002> Started deploy [analytics/refinery@3115f9e]: Set gobblin job.lock.dir after all - T271232

Stashbot added a comment.Jul 22 2021, 6:59 PM

Mentioned in SAL (#wikimedia-operations) [2021-07-22T18:59:28Z] <otto@deploy1002> Finished deploy [analytics/refinery@3115f9e]: Set gobblin job.lock.dir after all - T271232 (duration: 03m 22s)

Stashbot added a comment.Jul 23 2021, 1:31 PM

Mentioned in SAL (#wikimedia-operations) [2021-07-23T13:31:52Z] <otto@deploy1002> Started deploy [analytics/refinery@15521b3]: Add property disabling gobblin lock - T271232

Stashbot added a comment.Jul 23 2021, 1:35 PM

Mentioned in SAL (#wikimedia-operations) [2021-07-23T13:35:24Z] <otto@deploy1002> Finished deploy [analytics/refinery@15521b3]: Add property disabling gobblin lock - T271232 (duration: 03m 32s)

gerritbot added a comment.Jul 29 2021, 3:16 PM

Change 708782 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Remove no longer used camus module and references to camus

https://gerrit.wikimedia.org/r/708782

gerritbot added a comment.Jul 29 2021, 3:20 PM

Change 708782 merged by Ottomata:

[operations/puppet@production] Remove no longer used camus module and references to camus

https://gerrit.wikimedia.org/r/708782

gerritbot added a comment.Jul 29 2021, 3:32 PM

Change 708785 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] refinery/job - remove already absented jobs

https://gerrit.wikimedia.org/r/708785

gerritbot added a comment.Jul 29 2021, 3:35 PM

Change 708785 merged by Ottomata:

[operations/puppet@production] refinery/job - remove already absented jobs

https://gerrit.wikimedia.org/r/708785

Ottomata added a comment.Jul 29 2021, 3:35 PM

Some documentation:
https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin

Ottomata added a comment.EditedJul 29 2021, 3:36 PM

Removing raw data we have stopped importing:

sudo -u hdfs hdfs dfs -rm -R /wmf/data/raw/eventlogging_client_side
sudo -u hdfs hdfs dfs -rm -R /wmf/data/raw/mediawiki_job
sudo -u hdfs hdfs dfs -rm -R /wmf/data/raw/atskafka_test_webrequest_text
Ottomata added a comment.Jul 29 2021, 3:38 PM

Removing old camus work job state dirs

sudo -u hdfs hdfs dfs -rm -R /wmf/camus
gerritbot added a comment.Jul 29 2021, 3:54 PM

Change 708786 had a related patch set uploaded (by Ottomata; author: Ottomata):

[analytics/refinery/source@master] Remove refinery-camus module

https://gerrit.wikimedia.org/r/708786

gerritbot added a comment.Jul 29 2021, 3:56 PM

Change 708787 had a related patch set uploaded (by Ottomata; author: Ottomata):

[analytics/refinery/source@master] Refine - replace default formatters with gobblin convention

https://gerrit.wikimedia.org/r/708787

gerritbot added a comment.Jul 29 2021, 6:41 PM

Change 708816 had a related patch set uploaded (by Ottomata; author: Ottomata):

[analytics/refinery@master] Remove references to camus

https://gerrit.wikimedia.org/r/708816

Ottomata added a comment.Jul 29 2021, 6:58 PM

Alright, I've gone through wikitech and updated relevant references to Camus.

There are 3 outstanding patches to refinery-source and refinery about removing Camus. Once those are merged and deployed, we can call this task done!

odimitrijevic added subtasks: T280649: Add a spark job loading Cassandra 3, T287084: Purge gobblin files.Aug 2 2021, 9:56 PM
odimitrijevic added projects: Epic, Data-Engineering, Data-Engineering-Kanban.
odimitrijevic mentioned this in T287889: Finalize Gobblin Migration.Aug 2 2021, 10:00 PM
gerritbot added a comment.Aug 3 2021, 3:25 PM

Change 708816 merged by Milimetric:

[analytics/refinery@master] Remove references to camus

https://gerrit.wikimedia.org/r/708816

gerritbot added a comment.Aug 3 2021, 3:41 PM

Change 708786 merged by Milimetric:

[analytics/refinery/source@master] Remove refinery-camus module

https://gerrit.wikimedia.org/r/708786

Ottomata removed a subtask: T280649: Add a spark job loading Cassandra 3.Aug 3 2021, 3:44 PM
gerritbot added a comment.Aug 3 2021, 6:07 PM

Change 708787 merged by Ottomata:

[analytics/refinery/source@master] Refine - replace default formatters with gobblin convention

https://gerrit.wikimedia.org/r/708787

odimitrijevic moved this task from Next Up to Q2 Epics on the Data-Engineering-Kanban board.Aug 3 2021, 6:57 PM
odimitrijevic removed a subtask: T286503: Push Gobblin import metrics to Prometheus and add alerts on some critical imports.Aug 3 2021, 7:59 PM
odimitrijevic removed a subtask: T286559: When gobblin fails, we should know about it.
Ottomata moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Aug 4 2021, 4:07 PM
odimitrijevic removed projects: Data-Engineering-Kanban, Data-Engineering.Aug 5 2021, 3:19 PM
JAllemandou closed subtask T287685: Delete HDFS raw *_camus directories 60 days after July 12 (after 2021-09-10) as Resolved.Aug 20 2021, 8:11 AM
JAllemandou moved this task from Done to Parent Tasks on the Analytics-Kanban board.Sep 28 2021, 12:58 PM
JAllemandou merged a task: T287991: Gobblin Monitoring.Sep 28 2021, 1:14 PM
JAllemandou added a subtask: T286503: Push Gobblin import metrics to Prometheus and add alerts on some critical imports.
JAllemandou added a subscriber: odimitrijevic.
odimitrijevic added projects: Data-Engineering, Data-Engineering-Kanban.Oct 27 2021, 10:13 PM
odimitrijevic moved this task from Incoming (new tickets) to Apache Iceberg Migration on the Data-Engineering board.Oct 28 2021, 6:00 AM
JAllemandou closed subtask T287084: Purge gobblin files as Resolved.Nov 30 2021, 5:12 PM
odimitrijevic removed a project: Analytics.Jan 12 2022, 12:33 AM
JAllemandou moved this task from Q2 Epics to Done on the Data-Engineering-Kanban board.Mar 22 2022, 6:52 PM
JAllemandou closed this task as Resolved.Mar 30 2022, 4:05 PM
JArguello-WMF closed subtask T286503: Push Gobblin import metrics to Prometheus and add alerts on some critical imports as Resolved.Jul 5 2022, 3:42 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL