Move wmf_dumps.wikitext_rc1 to the correct HDFS directory
Closed, ResolvedPublic3 Estimated Story Points
Actions

Assigned To

Authored By

	xcollazo
	Sep 18 2023, 2:48 PM

Description

Just realized that wmf_dumps.wikitext_rc1 lives in the wrong HDFS directory:

$ hdfs dfs -ls /user/hive/warehouse/wmf_dumps.db
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8
Found 1 items
drwxrwx---   - analytics analytics-privatedata-users          0 2023-08-24 16:28 /user/hive/warehouse/wmf_dumps.db/wikitext_raw_rc1

While most of the production tables typically live under /wmf/data. We had done it correctly for wmf_dumps.wikitext_rc0:

$ hdfs dfs -ls /wmf/data/wmf_dumps
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8
Found 1 items
drwxrwxr-x   - analytics analytics-privatedata-users          0 2023-07-11 19:50 /wmf/data/wmf_dumps/wikitext_raw_rc0

In this ticket we should:

~~Iceberg unfortunately persist fully qualified filenames in its metadata. Figure if the community has developed an utility to mv tables.~~
~~If so, then use the utility and mv the table to the proper place.~~
If not, then let's copy its content over? Or declare a new wmf_dumps.wikitext_rc2 release candidate in the proper place?

Related Objects
Search...

Status	Assigned	Task
Open	VirginiaPoundstone	T345988 [Epic] XML MediaWiki data dumps for right to fork
Resolved	Milimetric	T330296 Proof of concept for MediaWiki XML content dump via Event Platform, Iceberg and Spark
Resolved	xcollazo	T346646 Move wmf_dumps.wikitext_rc1 to the correct HDFS directory

Event Timeline

xcollazo created this task.Sep 18 2023, 2:48 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 18 2023, 2:48 PM

xcollazo moved this task from Incoming to To be discussed/To be estimated on the Data Products board.Sep 18 2023, 2:59 PM

Moving is fine, let's not make a new RC until we have a new schema

xcollazo set the point value for this task to 3.Sep 19 2023, 1:45 PM

VirginiaPoundstone added a project: Dumps 2.0.Sep 19 2023, 2:48 PM

VirginiaPoundstone edited projects, added Data Products (Sprint 01); removed Data Products.Sep 19 2023, 3:18 PM

CC @JAllemandou for awareness.

xcollazo moved this task from Sprint 01 to Sprint 02 on the Data Products board.Oct 3 2023, 1:40 PM

xcollazo edited projects, added Data Products (Sprint 02); removed Data Products (Sprint 01).

We need a new schema for T340863, so to solve this, I will make sure that that new table gets created in the proper LOCATION.

Will leave this ticket one open for now so that we don't forget to clean up by deleting the wmf_dumps.wikitext_rc1 table.

WDoranWMF moved this task from Sprint Backlog to Teleport to Sprint 03 on the Data Products (Sprint 02) board.Oct 24 2023, 12:56 PM

WDoranWMF edited projects, added Data Products (Data Products (Sprint 03)); removed Data Products (Sprint 02).

VirginiaPoundstone triaged this task as Low priority.Oct 30 2023, 7:41 PM

Ran the following in Hive as analytics user:

$ hostname -f
an-launcher1002.eqiad.wmnet
$ sudo -u analytics hive
hive (wmf_dumps)> DROP TABLE wmf_dumps.wikitext_raw_rc0;
OK
Time taken: 0.188 seconds
hive (wmf_dumps)> DROP TABLE wmf_dumps.wikitext_raw_rc1;
OK
Time taken: 0.129 seconds

Then to delete files for wikitext_raw_rc0:

$ sudo -u analytics hdfs dfs -ls /wmf/data/wmf_dumps
Found 2 items
drwxrwxr-x   - analytics analytics-privatedata-users          0 2023-07-11 19:50 /wmf/data/wmf_dumps/wikitext_raw_rc0
drwxrwxr-x   - analytics analytics-privatedata-users          0 2023-10-24 17:10 /wmf/data/wmf_dumps/wikitext_raw_rc2
$ sudo -u analytics hdfs dfs -rm -r -skipTrash /wmf/data/wmf_dumps/wikitext_raw_rc0
Deleted /wmf/data/wmf_dumps/wikitext_raw_rc0

And for for wikitext_raw_rc1:

$ sudo -u analytics hdfs dfs -ls /user/hive/warehouse/wmf_dumps.db
Found 1 items
drwxrwx---   - analytics analytics-privatedata-users          0 2023-08-24 16:28 /user/hive/warehouse/wmf_dumps.db/wikitext_raw_rc1
$ sudo -u analytics hdfs dfs -rm -r -skipTrash /user/hive/warehouse/wmf_dumps.db/wikitext_raw_rc1
Deleted /user/hive/warehouse/wmf_dumps.db/wikitext_raw_rc1