Page MenuHomePhabricator
Paste P43526

refinery deploy to HDFS step
ActivePublic

Authored by Stevemunene on Feb 1 2023, 11:14 AM.
Tags
None
Referenced Files
F36571934: refinery deploy to HDFS step
Feb 1 2023, 11:14 AM
Subscribers
None
2023-02-01T11:04:48+00:00 hdfs dfs -D fs.permissions.umask-mode=022 -mkdir -p hdfs:///wmf/refinery
2023-02-01T11:04:53+00:00
2023-02-01T11:04:54+00:00 * Copying local checkout to versioned directory in HDFS ...
2023-02-01T11:04:54+00:00 hdfs dfs -rm -r -f -skipTrash hdfs:///wmf/refinery/2023-02-01T11.04.48+00.00--scap_sync_2023-02-01_0001.tmp
2023-02-01T11:04:59+00:00 hdfs dfs -D fs.permissions.umask-mode=022 -mkdir hdfs:///wmf/refinery/2023-02-01T11.04.48+00.00--scap_sync_2023-02-01_0001.tmp
2023-02-01T11:05:06+00:00 hdfs dfs -D fs.permissions.umask-mode=022 -put -f airflow artifacts bin diagrams druid gobblin HACKING.md hive hql oozie packaged-environments python README.md setup.cfg spark static_data hdfs:///wmf/refinery/2023-02-01T11.04.48+00.00--scap_sync_2023-02-01_0001.tmp
23/02/01 11:07:25 WARN hdfs.DataStreamer: Caught exception
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1257)
at java.lang.Thread.join(Thread.java:1331)
at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980)
at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807)
2023-02-01T11:07:26+00:00 hdfs dfs -D fs.permissions.umask-mode=022 -put /dev/fd/63 hdfs:///wmf/refinery/2023-02-01T11.04.48+00.00--scap_sync_2023-02-01_0001.tmp/.deployment
2023-02-01T11:07:39+00:00 hdfs dfs -D fs.permissions.umask-mode=022 -mv hdfs:///wmf/refinery/2023-02-01T11.04.48+00.00--scap_sync_2023-02-01_0001.tmp hdfs:///wmf/refinery/2023-02-01T11.04.48+00.00--scap_sync_2023-02-01_0001
2023-02-01T11:07:45+00:00
2023-02-01T11:07:45+00:00 * Setting up 'current' version on cluster ...
2023-02-01T11:07:45+00:00 hdfs dfs -D fs.permissions.umask-mode=022 -cp hdfs:///wmf/refinery/2023-02-01T11.04.48+00.00--scap_sync_2023-02-01_0001 hdfs:///wmf/refinery/current.tmp
23/02/01 11:09:59 WARN hdfs.DataStreamer: Caught exception
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1257)
at java.lang.Thread.join(Thread.java:1331)
at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980)
at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807)
2023-02-01T11:10:26+00:00 hdfs dfs -D fs.permissions.umask-mode=022 -mv hdfs:///wmf/refinery/current hdfs:///wmf/refinery/current.swap
2023-02-01T11:10:32+00:00 hdfs dfs -D fs.permissions.umask-mode=022 -mv hdfs:///wmf/refinery/current.tmp hdfs:///wmf/refinery/current
2023-02-01T11:10:39+00:00 hdfs dfs -rm -r -f -skipTrash hdfs:///wmf/refinery/current.swap
Deleted hdfs:///wmf/refinery/current.swap
2023-02-01T11:10:45+00:00 pass (Used parameters: --verbose --no-dry-run )
stevemunene@an-launcher1002:/srv/deployment/analytics/refinery$