Maniphest T347524

Add integration tests to the PySpark jobs
Closed, ResolvedPublic5 Estimated Story Points
Actions

Assigned To

Authored By

	xcollazo
	Sep 27 2023, 7:38 PM

Tags

Referenced Files

None

Subscribers

Description

We have been heads down developing the ingestion pipelines for the backfill , events, and visbility events.

Now that the code has started to stabilize, we should invest in proper testing.

In this task we should:

Figure out what is best these days to test PySpark jobs.
Implement tests to a significant coverage, let's target 80%, for the mediawiki-content-dump project.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Milimetric	T330296 Dumps 2.0 Phase I: Proof of concept for MediaWiki XML content dump via Event Platform, Iceberg and Spark
		Resolved		JEbe-WMF	T347524 Add integration tests to the PySpark jobs

Event Timeline

xcollazo created this task.Sep 27 2023, 7:38 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 27 2023, 7:38 PM

Perhaps @mfossati has PySpark testing framework recommendations?

Hi there, for unit tests I can certainly suggest chispa, which was initially proposed by @MunizaA .
For integration tests I can't tell.

VirginiaPoundstone moved this task from Incoming to To be discussed/To be estimated on the Experimentation Lab board.Sep 28 2023, 9:29 PM

xcollazo updated the task description. (Show Details)Oct 2 2023, 1:58 PM

xcollazo moved this task from To be discussed/To be estimated to Sprint 02 on the Experimentation Lab board.Oct 3 2023, 1:43 PM

xcollazo edited projects, added Experimentation Lab (Sprint 02); removed Experimentation Lab.

xcollazo added a project: Dumps 2.0.Oct 3 2023, 1:49 PM

JEbe-WMF claimed this task.Oct 4 2023, 4:41 PM

Step 1: Figure out test are useful for pyspark jobs
Step 2: Decide on acceptable parameters for test
Step 3: Create/script tests on local
Step 4: Move test to prod- gitlab

• WDoranWMF moved this task from Sprint Backlog to In Process on the Experimentation Lab (Sprint 02) board.Oct 10 2023, 3:56 PM

JEbe-WMF set the point value for this task to 5.Oct 12 2023, 12:25 PM

• WDoranWMF moved this task from In Process to Teleport to Sprint 03 on the Experimentation Lab (Sprint 02) board.Oct 24 2023, 12:23 PM

• WDoranWMF edited projects, added Experimentation Lab (Data Products (Sprint 03)); removed Experimentation Lab (Sprint 02).

• WDoranWMF moved this task from Sprint Backlog to In Process on the Experimentation Lab (Data Products (Sprint 03)) board.

VirginiaPoundstone triaged this task as Low priority.Oct 30 2023, 7:44 PM

JEbe-WMF moved this task from In Process to In code review / Tech Input on the Experimentation Lab (Data Products (Sprint 03)) board.Nov 2 2023, 8:30 AM

cjming moved this task from In code review / Tech Input to In Process on the Experimentation Lab (Data Products (Sprint 03)) board.Nov 6 2023, 5:16 PM

JEbe-WMF moved this task from In Process to Ready for Code Review on the Experimentation Lab (Data Products (Sprint 03)) board.Nov 9 2023, 12:56 PM

JEbe-WMF moved this task from Ready for Code Review to In code review / Tech Input on the Experimentation Lab (Data Products (Sprint 03)) board.Nov 15 2023, 1:02 PM

• WDoranWMF moved this task from In code review / Tech Input to Wormhole to Sprint 04 on the Experimentation Lab (Data Products (Sprint 03)) board.Nov 16 2023, 12:09 PM

• WDoranWMF edited projects, added Experimentation Lab (Data Products Sprint 04); removed Experimentation Lab (Data Products (Sprint 03)).

• WDoranWMF moved this task from Sprint Backlog to Code Review / Tech Input on the Experimentation Lab (Data Products Sprint 04) board.

xcollazo updated https://gitlab.wikimedia.org/repos/data-engineering/dumps/mediawiki-content-dump/-/merge_requests/17

T347524 add integration tests to the pyspark jobs

JEbe-WMF moved this task from Code Review / Tech Input to Done on the Experimentation Lab (Data Products Sprint 04) board.Nov 20 2023, 9:15 AM

JEbe-WMF closed this task as Resolved.Nov 30 2023, 8:22 AM