Page MenuHomePhabricator
Paste P48501

Sqooping test following schema update
ActivePublic
Actions

Authored by Antoine_Quhen on Wed, May 24, 1:52 PM.
# Squooping test after schema update
# Prepare some directories with loose permissions
hdfs dfs -mkdir /tmp/test-aqu-20230522
hdfs dfs -chmod -R 777 /tmp/test-aqu-20230522
hdfs dfs -ls /tmp/ | grep test-aqu-20230522
mkdir -p /tmp/test-aqu-20230522
cd /tmp/test-aqu-20230522
git clone https://github.com/wikimedia/analytics-refinery.git
chmod 777 -R /tmp/test-aqu-20230522
# Edit sqoop script
vim analytics-refinery/python/refinery/sqoop.py
# Launch Scooping process
sudo -u analytics \
PYTHONPATH=/tmp/test-aqu-20230522/analytics-refinery/python \
/usr/bin/python3 \
/tmp/test-aqu-20230522/analytics-refinery/bin/sqoop-mediawiki-tables \
--job-name aqu-sqoop-mediawiki-monthly-2023-04 \
--clouddb \
--output-dir /tmp/test-aqu-20230522 \
--wiki-file /mnt/hdfs/wmf/refinery/current/static_data/mediawiki/grouped_wikis/grouped_wikis_test.csv \
--tables externallinks \
--user s53272 \
--password-file /user/analytics/mysql-analytics-labsdb-client-pw.txt \
--partition-name snapshot \
--partition-value 2023-04 \
--mappers 64 \
--processors 10 \
--yarn-queue production \
--output-format avrodata \
--log-file /tmp/test-aqu-20230522/sqoop-mediawiki.log \
--local-tmp-path /tmp/test-aqu-20230522/sqoop-jars \
--hdfs-tmp-path /tmp/test-aqu-20230522
# Read a generated avro file
sudo -u analytics hdfs dfs -chmod +r /tmp/test-aqu-20230522/externallinks/snapshot=2023-04/wiki_db=dkwikimedia/part-m-00006.avro
sudo -u analytics hdfs dfs -chmod -R +x /tmp/test-aqu-20230522/externallinks
spark3-shell --packages org.apache.spark:spark-avro_2.12:3.1.2
val path = "file:///tmp/test-aqu-20230522/part-m-00006.avro"
val path = "hdfs:///tmp/test-aqu-20230522/externallinks/snapshot=2023-04/wiki_db=dkwikimedia/part-m-00006.avro"
val df = spark.read.format("avro").load(path)
df.show(5, false)
+-----+-------+-------------------------+---------------------------------------------------------------------------------------+
|el_id|el_from|el_to_domain_index |el_to_path |
+-----+-------+-------------------------+---------------------------------------------------------------------------------------+
|1222 |467 |http://dk.politiken. |/kultur/medier/ECE1992455/wikipedia-gaar-til-kamp-mod-redigeringskrig/ |
|1223 |214 |https://org.wikipedia.da.|/w/index.php?title=Wikipedia:Tr%C3%A6f&oldid=7165729 |
|1224 |182 |https://org.wikipedia.da.|/wiki/Wikipedia:GLAM/Edit-a-thon_1864 |
|1225 |214 |http://dk.politiken. |/kultur/medier/ECE1992455/wikipedia-gaar-til-kamp-mod-redigeringskrig/ |
|1226 |214 |https://org.wikipedia.da.|/w/index.php?title=Wikipedia:Landsbybr%C3%B8nden/Kommunevalgs-edit-a-thon&oldid=7295725|
+-----+-------+-------------------------+---------------------------------------------------------------------------------------+
# Cleanup
sudo -u hdfs kerberos-run-command hdfs hdfs dfs -rm -R /tmp/test-aqu-20230522
sudo -u analytics rm -Rf /tmp/test-aqu-20230522/*
rm -Rf /tmp/test-aqu-20230522
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL