Support importing Pandas datasets into the Data Lake using Wmfdata
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	nshahquinn-wmf
	Jan 28 2021, 4:10 PM

Description

Currently, Wmfdata has a hive.load_csv function, which is most notably used to upload the canonical datasets into the Data Lake.

However, it has some limitations (T327983, T355847), and in any case Hive is deprecated and should be removed from Wmfdata in the future (T384541).

We can build a better replacement using Spark's saveAsTable function. Rather than focusing on delimited files, we should simply build a function that uploads a Pandas data frame. Users can use Pandas directly to read in local files in any of the many formats it supports.

Details

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	Support uploading a Pandas data frame to the Data Lake	repos/data-engineering/wmfdata-python!67	nshahquinn-wmf	work/spark_upload_squashed	main

Customize query in GitLab

Related Objects
Search...

Status	Assigned	Task
Resolved	nshahquinn-wmf	T384541 Deprecate Wmfdata's Hive module
Open	nshahquinn-wmf	T355847 Upload canonical datasets to the Data Lake using Wmfdata's spark.upload_pandas
Resolved	nshahquinn-wmf	T273196 Support importing Pandas datasets into the Data Lake using Wmfdata

Event Timeline

nshahquinn-wmf created this task.Jan 28 2021, 4:10 PM

nshahquinn-wmf triaged this task as Medium priority.

nshahquinn-wmf moved this task from Triage to Backlog on the Product-Analytics board.

nshahquinn-wmf renamed this task from Support importing a Pandas dataframe directly into HDFS to Support importing a Parquet file into HDFS using wmfdata-python.Feb 23 2021, 5:55 PM

nshahquinn-wmf updated the task description. (Show Details)

nshahquinn-wmf moved this task from Backlog to Upcoming Quarter on the Product-Analytics board.Dec 7 2021, 10:43 PM

nshahquinn-wmf edited projects, added Product-Analytics (Kanban); removed Product-Analytics.Dec 9 2021, 11:36 PM

nshahquinn-wmf claimed this task.Dec 10 2021, 8:41 PM

nshahquinn-wmf moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.

I've put up a draft pull request on GitHub. I still need to make some tweaks, so I haven't requested review yet.

nshahquinn-wmf moved this task from Doing to Next 2 weeks on the Product-Analytics (Kanban) board.Jan 11 2022, 9:14 PM

nshahquinn-wmf moved this task from Next 2 weeks to Needs Review on the Product-Analytics (Kanban) board.Jan 29 2022, 7:06 PM

Sorry, wrong task.

Adding Data-Engineering to all Wmfdata-Python tasks, as requested by Dan and Andrew.

• EChetty moved this task from Incoming (new tickets) to Event Platform Backlog on the Data-Engineering board.Mar 24 2022, 3:05 PM

nshahquinn-wmf edited projects, added Product-Analytics; removed Product-Analytics (Kanban).Mar 25 2022, 11:35 PM

I plan to finish the pull request in July, after I return from sabbatical. But if someone else wants to take over while I'm gone, that's fine with me!

• EChetty moved this task from Event Platform Backlog to WMF-Data on the Data-Engineering board.Mar 28 2022, 10:53 AM

kzimmerman moved this task from Upcoming Quarter to Current Quarter on the Product-Analytics board.Jul 5 2022, 8:24 PM

nshahquinn-wmf lowered the priority of this task from Medium to Low.Sep 6 2022, 5:18 PM

nshahquinn-wmf moved this task from Current Quarter to Upcoming Quarter on the Product-Analytics board.

nshahquinn-wmf moved this task from Upcoming Quarter to Backlog on the Product-Analytics board.Apr 12 2023, 12:14 AM

JArguello-WMF moved this task from WMF-Data to Data Products & Metrics on the Data-Engineering board.Jun 29 2023, 10:42 PM

The draft pull request is still there, but it seems unlikely that I'll be able to pick it back up in the near future.

• lbowmaker moved this task from Data Products & Metrics to Tag with Icebox on the Data-Engineering board.Nov 10 2023, 2:42 PM

nshahquinn-wmf mentioned this in T355847: Upload canonical datasets to the Data Lake using Wmfdata's spark.upload_pandas.Jan 25 2024, 2:25 AM

nshahquinn-wmf updated the task description. (Show Details)Oct 16 2024, 10:39 PM

nshahquinn-wmf moved this task from Backlog to Ready for development on the Wmfdata-Python board.Nov 8 2024, 2:54 AM

• VirginiaPoundstone edited projects, added Data-Engineering-Icebox-DEPRECATED; removed Data-Engineering.Jan 6 2025, 9:23 PM

Restricted Application edited projects, added Data-Engineering; removed Data-Engineering-Icebox-DEPRECATED. · View Herald TranscriptJan 6 2025, 9:23 PM

• VirginiaPoundstone edited projects, added Data-Engineering-Icebox-DEPRECATED; removed Data-Engineering.Jan 6 2025, 9:38 PM

Restricted Application edited projects, added Data-Engineering; removed Data-Engineering-Icebox-DEPRECATED. · View Herald TranscriptJan 6 2025, 9:38 PM

Ottomata edited projects, added Data-Engineering-Icebox-DEPRECATED; removed Data-Engineering.Jan 6 2025, 9:49 PM

Restricted Application edited projects, added Data-Engineering; removed Data-Engineering-Icebox-DEPRECATED. · View Herald TranscriptJan 6 2025, 9:49 PM

Ottomata added a project: Data-Engineering-Icebox.Jan 9 2025, 10:09 PM

@fkaelin has put up MR 66 for this. I need to review it.

nshahquinn-wmf moved this task from Ready for development to Code review on the Wmfdata-Python board.Jan 22 2025, 9:39 PM

nshahquinn-wmf moved this task from Incoming to Backlog on the Movement-Insights board.Feb 12 2025, 7:40 PM

nshahquinn-wmf renamed this task from Support importing a Parquet file into HDFS using wmfdata-python to Support importing Parquet, delimited, and Pandas datasets into the Data Lake using Wmfdata.Aug 13 2025, 5:47 PM

nshahquinn-wmf updated the task description. (Show Details)

Increasing the priority since this blocks deprecating the Hive module (T384541).

nshahquinn-wmf added a project: Data-Engineering-Icebox.Aug 13 2025, 5:49 PM

nshahquinn-wmf added a parent task: T384541: Deprecate Wmfdata's Hive module.

nshahquinn-wmf moved this task from Code review to In progress on the Wmfdata-Python board.Aug 14 2025, 12:26 AM

nshahquinn-wmf moved this task from Backlog to Doing on the Movement-Insights (FY25-26 H1) board.Aug 14 2025, 12:58 AM

nshahquinn-wmf closed https://gitlab.wikimedia.org/repos/data-engineering/wmfdata-python/-/merge_requests/25

Support loading a Parquet file into a Hive table

nshahquinn-wmf added a parent task: T355847: Upload canonical datasets to the Data Lake using Wmfdata's spark.upload_pandas.Aug 15 2025, 12:27 AM

I'm cutting out the part about loading Parquet and delimited files, because those functions ended up being nothing more than two line wrappers around the appropriate Pandas read function and our Pandas-upload function. No need to increase the API surface area for that.

nshahquinn-wmf opened https://gitlab.wikimedia.org/repos/data-engineering/wmfdata-python/-/merge_requests/67

Support uploading a Pandas data frame to the Data Lake

nshahquinn-wmf moved this task from Doing to Waiting on others on the Movement-Insights (FY25-26 H1) board.Aug 15 2025, 9:36 PM

nshahquinn-wmf merged https://gitlab.wikimedia.org/repos/data-engineering/wmfdata-python/-/merge_requests/67

Support uploading a Pandas data frame to the Data Lake

nshahquinn-wmf closed this task as Resolved.Aug 18 2025, 7:35 PM