Page MenuHomePhabricator
Create Task
Maniphest T235200

Create HDFS /tmp/ cleaner
Closed, ResolvedPublic5 Estimated Story Points
Actions

Description

Hadoop doesn't do this wth!

We should make a hadoop cleaner job that uses the Hadoop Java API to look for files with old mtimes.

Details

SubjectRepoBranchLines +/-
HDFSCleaner - change command argsoperations/puppetproduction+1 -5
Use HDFS trash settings as default everywhereoperations/puppetproduction+9 -31
HDFSCleaner improvementsanalytics/refinery/sourcemaster+150 -27
Add bin/hdfs-cleaner wrapper scriptanalytics/refinerymaster+5 -0
HDFSCleaner - use Path toString in log messages instead of getNameanalytics/refinery/sourcemaster+8 -8
Include hdfs_cleaner an an-coord node to clean HDFS /tmp diroperations/puppetproduction+32 -1
Add HDFSCleaner to aid in cleaning HDFS tmp directoriesanalytics/refinery/sourcemaster+210 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT234954 500k files in hdfs /tmp
 ResolvedOttomataT235200 Create HDFS /tmp/ cleaner

Event Timeline

Ottomata triaged this task as High priority.Oct 10 2019, 5:10 PM
Ottomata created this task.
Ottomata moved this task from Incoming to Operational Excellence on the Analytics board.
gerritbot added a comment.Oct 17 2019, 4:45 PM

Change 543897 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/refinery/source@master] Add HDFSCleaner to aid in cleaning HDFS tmp directories

https://gerrit.wikimedia.org/r/543897

gerritbot added a project: Patch-For-Review.Oct 17 2019, 4:45 PM
Ottomata added a project: Analytics-Kanban.Oct 17 2019, 4:45 PM
Ottomata moved this task from Next Up to In Code Review on the Analytics-Kanban board.
Ottomata moved this task from In Code Review to In Progress on the Analytics-Kanban board.Oct 22 2019, 4:05 PM
Ottomata moved this task from In Progress to In Code Review on the Analytics-Kanban board.Oct 29 2019, 2:30 PM
gerritbot added a comment.Oct 29 2019, 8:03 PM

Change 543897 merged by Ottomata:
[analytics/refinery/source@master] Add HDFSCleaner to aid in cleaning HDFS tmp directories

https://gerrit.wikimedia.org/r/543897

Ottomata moved this task from In Code Review to Ready to Deploy on the Analytics-Kanban board.Oct 29 2019, 8:03 PM
Maintenance_bot removed a project: Patch-For-Review.Oct 29 2019, 8:11 PM
Ottomata set the point value for this task to 5.Oct 30 2019, 2:31 PM
gerritbot added a comment.Nov 4 2019, 7:38 PM

Change 548468 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Include hdfs_cleaner an an-coord node to clean HDFS /tmp dir

https://gerrit.wikimedia.org/r/548468

gerritbot added a project: Patch-For-Review.Nov 4 2019, 7:38 PM
gerritbot added a comment.Nov 5 2019, 4:29 PM

Change 548468 merged by Ottomata:
[operations/puppet@production] Include hdfs_cleaner an an-coord node to clean HDFS /tmp dir

https://gerrit.wikimedia.org/r/548468

Ottomata moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Nov 5 2019, 4:29 PM
gerritbot added a comment.Nov 5 2019, 4:36 PM

Change 548783 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/refinery/source@master] HDFSCleaner - use Path toString in log messages instead of getName

https://gerrit.wikimedia.org/r/548783

gerritbot added a comment.Nov 5 2019, 4:45 PM

Change 548783 merged by Ottomata:
[analytics/refinery/source@master] HDFSCleaner - use Path toString in log messages instead of getName

https://gerrit.wikimedia.org/r/548783

Maintenance_bot removed a project: Patch-For-Review.Nov 5 2019, 5:10 PM
Ottomata moved this task from Done to In Progress on the Analytics-Kanban board.Nov 5 2019, 5:45 PM

Moving back to in progress to use HDFS Trash

gerritbot added a comment.Nov 5 2019, 7:41 PM

Change 548850 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use HDFS trash settings as default everywhere

https://gerrit.wikimedia.org/r/548850

gerritbot added a project: Patch-For-Review.Nov 5 2019, 7:41 PM
gerritbot added a comment.Nov 5 2019, 10:01 PM

Change 548909 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/refinery/source@master] HDFSCleaner improvements

https://gerrit.wikimedia.org/r/548909

elukey added a comment.Nov 6 2019, 7:07 AM

On an-coord1001 I can see:

-- Logs begin at Tue 2019-11-05 13:34:31 UTC, end at Wed 2019-11-06 07:06:56 UTC. --
Nov 05 23:00:01 an-coord1001 systemd[1]: Started Run the HDFSCleaner job to keep HDFS /tmp dir clean of old files..
Nov 05 23:00:02 an-coord1001 java[195946]: Error: Could not find or load main class classpath)
Nov 05 23:00:02 an-coord1001 systemd[1]: hdfs-cleaner.service: Main process exited, code=exited, status=1/FAILURE
Nov 05 23:00:02 an-coord1001 systemd[1]: hdfs-cleaner.service: Unit entered failed state.
Nov 05 23:00:02 an-coord1001 systemd[1]: hdfs-cleaner.service: Failed with result 'exit-code'.

The unit failed, I have acked the alarm in Icinga :)

gerritbot added a comment.Nov 6 2019, 2:36 PM

Change 549092 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/refinery@master] Add bin/hdfs-cleaner wrapper script

https://gerrit.wikimedia.org/r/549092

gerritbot added a comment.Nov 6 2019, 3:46 PM

Change 549092 merged by Ottomata:
[analytics/refinery@master] Add bin/hdfs-cleaner wrapper script

https://gerrit.wikimedia.org/r/549092

Ottomata moved this task from In Progress to In Code Review on the Analytics-Kanban board.Nov 6 2019, 5:00 PM
gerritbot added a comment.Nov 6 2019, 6:01 PM

Change 548909 merged by Ottomata:
[analytics/refinery/source@master] HDFSCleaner improvements

https://gerrit.wikimedia.org/r/548909

Milimetric moved this task from In Code Review to Ready to Deploy on the Analytics-Kanban board.Nov 6 2019, 8:23 PM
gerritbot added a comment.Nov 6 2019, 9:36 PM

Change 548850 abandoned by Ottomata:
Use HDFS trash settings as default everywhere

Reason:
Not needed.

https://gerrit.wikimedia.org/r/548850

gerritbot added a comment.Nov 6 2019, 9:37 PM

Change 549207 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] HDFSCleaner - change command args

https://gerrit.wikimedia.org/r/549207

gerritbot added a comment.Nov 6 2019, 9:40 PM

Change 549207 merged by Ottomata:
[operations/puppet@production] HDFSCleaner - change command args

https://gerrit.wikimedia.org/r/549207

Maintenance_bot removed a project: Patch-For-Review.Nov 6 2019, 10:11 PM
Ottomata moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Nov 11 2019, 4:09 PM
• Nuria closed this task as Resolved.Nov 14 2019, 7:56 PM
Aklapper removed a project: Analytics.Jul 4 2020, 7:59 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL