Change Details

Due to the delayed new mw history snapshot, the related dumps on labstore nodes are failing since May 1st due to: ``` May 05 05:00:01 labstore1006 kerberos-run-command[19045]: User dumpsgen executes as user dumpsgen the command ['/bin/bash', '-c', '/usr/local/bin/hdfs-rsync -r -t --delete --exclude "readme.html" --chmod=go-w hdfs:///wmf/data/archive/mediawiki/history/{$(/bin/date --date="$ May 05 05:00:07 labstore1006 kerberos-run-command[19045]: Error: Argument parsing error: May 05 05:00:07 labstore1006 kerberos-run-command[19045]: Error validating src list: May 05 05:00:07 labstore1006 kerberos-run-command[19045]: hdfs:///wmf/data/archive/mediawiki/history/2020-04 does not exist May 05 05:00:07 labstore1006 kerberos-run-command[19045]: Try --help for more information. ``` @JAllemandou is there a way to make hdfs-rsync a little bit more lenient?

Due to the delayed new mw history snapshot, the related dumps on labstore nodes are failing since May 1st due to: ``` May 05 05:00:01 labstore1006 kerberos-run-command[19045]: User dumpsgen executes as user dumpsgen the command ['/bin/bash', '-c', '/usr/local/bin/hdfs-rsync -r -t --delete --exclude "readme.html" --chmod=go-w hdfs:///wmf/data/archive/mediawiki/history/{$(/bin/date --date="$ May 05 05:00:07 labstore1006 kerberos-run-command[19045]: Error: Argument parsing error: May 05 05:00:07 labstore1006 kerberos-run-command[19045]: Error validating src list: May 05 05:00:07 labstore1006 kerberos-run-command[19045]: hdfs:///wmf/data/archive/mediawiki/history/2020-04 does not exist May 05 05:00:07 labstore1006 kerberos-run-command[19045]: Try --help for more information. ``` Desired solution: Check for folder existence in HDFS before launching hdfs-rsync (see comments below).

Due to the delayed new mw history snapshot, the related dumps on labstore nodes are failing since May 1st due to: ``` May 05 05:00:01 labstore1006 kerberos-run-command[19045]: User dumpsgen executes as user dumpsgen the command ['/bin/bash', '-c', '/usr/local/bin/hdfs-rsync -r -t --delete --exclude "readme.html" --chmod=go-w hdfs:///wmf/data/archive/mediawiki/history/{$(/bin/date --date="$ May 05 05:00:07 labstore1006 kerberos-run-command[19045]: Error: Argument parsing error: May 05 05:00:07 labstore1006 kerberos-run-command[19045]: Error validating src list: May 05 05:00:07 labstore1006 kerberos-run-command[19045]: hdfs:///wmf/data/archive/mediawiki/history/2020-04 does not exist May 05 05:00:07 labstore1006 kerberos-run-command[19045]: Try --help for more information. ``` @JAllemandou is there a way to makeDesired solution: Check for folder existence in HDFS before launching hdfs-rsync a little bit more lenient? (see comments below).