Page MenuHomePhabricator
Create Task
Maniphest T203804

Refactor Refine job scalaopt to use property files and CLI overrides
Closed, ResolvedPublic13 Estimated Story Points
Actions

Details

Show related patches Customize query in gerrit

Event Timeline

Ottomata created this task.Sep 7 2018, 3:53 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 7 2018, 3:53 PM
Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.Sep 7 2018, 4:02 PM
gerritbot subscribed.Sep 7 2018, 6:38 PM

Change 458862 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/refinery/source@master] Use ConfigHelper for RefineMonitor instead of scopt

https://gerrit.wikimedia.org/r/458862

gerritbot added a project: Patch-For-Review.Sep 7 2018, 6:38 PM
Ottomata moved this task from In Progress to In Code Review on the Analytics-Kanban board.Sep 11 2018, 3:10 PM
gerritbot added a comment.Sep 11 2018, 7:40 PM

Change 459845 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/refinery/source@master] Throw usable exception when missing required field in ConfigHelper configure

https://gerrit.wikimedia.org/r/459845

gerritbot added a comment.Sep 12 2018, 1:46 PM

Change 459845 merged by Ottomata:
[analytics/refinery/source@master] Throw usable exception when missing required field in ConfigHelper configure

https://gerrit.wikimedia.org/r/459845

gerritbot added a comment.Sep 12 2018, 1:47 PM

Change 458862 merged by Ottomata:
[analytics/refinery/source@master] Use ConfigHelper for RefineMonitor instead of scopt

https://gerrit.wikimedia.org/r/458862

gerritbot added a comment.Sep 12 2018, 2:28 PM

Change 460023 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use properties file for RefineMonitor

https://gerrit.wikimedia.org/r/460023

Stashbot subscribed.Sep 12 2018, 5:34 PM

Mentioned in SAL (#wikimedia-analytics) [2018-09-12T17:34:13Z] <ottomata> deploying new version of refinery-source, and then refinery for properties based RefineMonitor job - https://phabricator.wikimedia.org/T203804

Stashbot added a comment.Sep 12 2018, 5:50 PM

Mentioned in SAL (#wikimedia-operations) [2018-09-12T17:50:47Z] <otto@deploy1001> Started deploy [analytics/refinery@407da92]: Deploying refinery-source 0.0.74 jars for T203804

gerritbot added a comment.Sep 12 2018, 5:51 PM

Change 460023 merged by Ottomata:
[operations/puppet@production] Use properties file for RefineMonitor

https://gerrit.wikimedia.org/r/460023

Stashbot added a comment.Sep 12 2018, 5:52 PM

Mentioned in SAL (#wikimedia-operations) [2018-09-12T17:52:28Z] <otto@deploy1001> Finished deploy [analytics/refinery@407da92]: Deploying refinery-source 0.0.74 jars for T203804 (duration: 01m 41s)

Stashbot added a comment.Sep 12 2018, 5:53 PM

Mentioned in SAL (#wikimedia-operations) [2018-09-12T17:53:04Z] <otto@deploy1001> Started deploy [analytics/refinery@407da92]: Deploying refinery-source 0.0.74 jars for T203804

Stashbot added a comment.Sep 12 2018, 5:59 PM

Mentioned in SAL (#wikimedia-operations) [2018-09-12T17:59:15Z] <otto@deploy1001> Finished deploy [analytics/refinery@407da92]: Deploying refinery-source 0.0.74 jars for T203804 (duration: 06m 11s)

gerritbot added a comment.Sep 12 2018, 6:02 PM

Change 460070 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Update refinery-job jar used for refine jobs

https://gerrit.wikimedia.org/r/460070

gerritbot added a comment.Sep 12 2018, 6:03 PM

Change 460070 merged by Ottomata:
[operations/puppet@production] Update refinery-job jar used for refine jobs

https://gerrit.wikimedia.org/r/460070

gerritbot added a comment.Sep 12 2018, 6:15 PM

Change 460071 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Fix properties file for refine job

https://gerrit.wikimedia.org/r/460071

gerritbot added a comment.Sep 12 2018, 6:15 PM

Change 460071 merged by Ottomata:
[operations/puppet@production] Fix properties file for refine job

https://gerrit.wikimedia.org/r/460071

gerritbot added a comment.Sep 12 2018, 9:11 PM

Change 460108 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/refinery/source@master] Use ConfigHelper for Refine job

https://gerrit.wikimedia.org/r/460108

gerritbot added a comment.Sep 13 2018, 7:16 PM

Change 460108 merged by jenkins-bot:
[analytics/refinery/source@master] Use ConfigHelper for Refine and EventLoggingSanitization

https://gerrit.wikimedia.org/r/460108

gerritbot added a comment.Sep 13 2018, 7:29 PM

Change 460417 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Refactor refine_job to use new spark_job and ConfigHelper properties

https://gerrit.wikimedia.org/r/460417

gerritbot added a comment.Sep 13 2018, 7:44 PM

Change 460418 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/refinery/source@master] Update changelog for 0.0.75 ConfigHelper changes

https://gerrit.wikimedia.org/r/460418

gerritbot added a comment.Sep 13 2018, 7:58 PM

Change 460418 merged by Ottomata:
[analytics/refinery/source@master] Update changelog for 0.0.75 ConfigHelper changes

https://gerrit.wikimedia.org/r/460418

Ottomata triaged this task as Medium priority.Sep 24 2018, 3:04 PM
Ottomata set the point value for this task to 13.
Stashbot added a comment.Sep 25 2018, 5:36 PM

Mentioned in SAL (#wikimedia-analytics) [2018-09-25T17:36:17Z] <ottomata> stopping refine jobs and deploying refinery source 0.0.75 - T203804

Stashbot added a comment.Sep 25 2018, 5:39 PM

Mentioned in SAL (#wikimedia-operations) [2018-09-25T17:39:34Z] <otto@deploy1001> Started deploy [analytics/refinery@ce8f0b3]: Deploying refinery-source 0.0.75 for ConfigHelper Refine - T203804

Stashbot added a comment.Sep 25 2018, 5:49 PM

Mentioned in SAL (#wikimedia-operations) [2018-09-25T17:49:57Z] <otto@deploy1001> Finished deploy [analytics/refinery@ce8f0b3]: Deploying refinery-source 0.0.75 for ConfigHelper Refine - T203804 (duration: 10m 22s)

gerritbot added a comment.Sep 25 2018, 5:55 PM

Change 460417 merged by Ottomata:
[operations/puppet@production] Refactor refine_job to use new spark_job and ConfigHelper properties

https://gerrit.wikimedia.org/r/460417

gerritbot added a comment.Sep 25 2018, 5:57 PM

Change 462782 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Fix rebase conflict error in refine.pp

https://gerrit.wikimedia.org/r/462782

gerritbot added a comment.Sep 25 2018, 5:58 PM

Change 462782 merged by Ottomata:
[operations/puppet@production] Fix rebase conflict error in refine.pp

https://gerrit.wikimedia.org/r/462782

gerritbot added a comment.Sep 25 2018, 6:07 PM

Change 462786 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Sort refine job config properties keys

https://gerrit.wikimedia.org/r/462786

gerritbot added a comment.Sep 25 2018, 6:20 PM

Change 462786 merged by Ottomata:
[operations/puppet@production] Sort refine job config properties keys

https://gerrit.wikimedia.org/r/462786

gerritbot added a comment.Sep 25 2018, 6:37 PM

Change 462789 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use --files to upload job properties file for refine spark job

https://gerrit.wikimedia.org/r/462789

gerritbot added a comment.Sep 25 2018, 6:39 PM

Change 462789 merged by Ottomata:
[operations/puppet@production] Use --files to upload job properties file for refine spark job

https://gerrit.wikimedia.org/r/462789

Ottomata added a comment.Sep 25 2018, 7:00 PM

Woohoo, did it! Re-running refine jobs is now WAY easier. I updated the docs at https://wikitech.wikimedia.org/wiki/Analytics/Systems/Refine#Rerunning_jobs

Diff:
https://wikitech.wikimedia.org/w/index.php?title=Analytics%2FSystems%2FRefine&type=revision&diff=1804158&oldid=1801978

Ottomata moved this task from In Code Review to Done on the Analytics-Kanban board.Sep 25 2018, 7:00 PM
gerritbot added a comment.Sep 25 2018, 9:24 PM

Change 462816 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Fix input regex and capture group configs for Refine jobs

https://gerrit.wikimedia.org/r/462816

gerritbot added a comment.Sep 25 2018, 9:26 PM

Change 462816 merged by Ottomata:
[operations/puppet@production] Fix input regex and capture group configs for Refine jobs

https://gerrit.wikimedia.org/r/462816

gerritbot added a comment.Sep 25 2018, 9:52 PM

Change 462820 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/refinery/source@master] Remove potentially dangerous Refine Config defaults

https://gerrit.wikimedia.org/r/462820

gerritbot added a comment.Sep 26 2018, 2:06 PM

Change 462820 merged by Ottomata:
[analytics/refinery/source@master] Remove potentially dangerous Refine Config defaults

https://gerrit.wikimedia.org/r/462820

• Nuria closed this task as Resolved.Sep 26 2018, 7:12 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL