Move HdfsArchiver job to hdfs-tools repo on github.
- Add dependency on refinery-tools
- Add (non provided) scala dependencies to hdfs-tools.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Snwachukwu | T310542 [Airflow] Refactor HDFSArchiveOperator to run in Skein | |||
| Resolved | Snwachukwu | T317054 Add HdfsArchiver job to hdfs-tools |
Event Timeline
Comment Actions
There were 2 fix made in this repo:
- There was a bug with the hdfs-tools HdfsRsync job. The condition to update the Lastmodification time of the Destination file was wrong.
- The HdfsRsync test were failing. To resolve this a thread delay was added to the job.