Improve mediawiki-event-enrichment test suite
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	gmodena
	Jan 26 2023, 12:21 PM

Description

Goal/Background

We should improve testing for mediawiki-event-enrichment. A good baseline for comparison is the Scala version of this application, that has a more comprehensive unit and integration test suite.

Key Tasks/Dependencies

Scala test suite https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-stream-enrichment/-/tree/main/enrichment/src/test/scala
Related (but not required) https://phabricator.wikimedia.org/T326565

Acceptance Criteria

Unit tests for the enrichment function logic are provided
Integration tests (e2e) for the streaming enrichment application are provided. Tests do not depend on remote hosts. HTTPs responses should be mocked.
The repo is restructured with a more pythonic layout

Details

	Title	Reference	Author	Source Branch	Dest Branch
	Add test coverage	repos/data-engineering/mediawiki-event-enrichment!57	tchin	add-test-coverage	main
	Mock requests response for tests	repos/data-engineering/mediawiki-event-enrichment!54	tchin	mock-http	main

Customize query in GitLab

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		tchin	T328013 Improve mediawiki-event-enrichment test suite
		Resolved		tchin	T337400 Get coverage artifacts from Kokkuri

Event Timeline

gmodena created this task.Jan 26 2023, 12:21 PM

Restricted Application added a project: Data-Engineering. · View Herald TranscriptJan 26 2023, 12:21 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

gmodena updated the task description. (Show Details)Jan 26 2023, 12:23 PM

• EChetty edited projects, added Data-Engineering-Planning; removed Data-Engineering.Jan 31 2023, 4:09 PM

• EChetty moved this task from Backlog to Event Platform on the Data-Engineering-Planning board.Jan 31 2023, 4:11 PM

gmodena mentioned this in T326565: Tests for mediawiki-stream-enrichment-python flink job via eventutilities-python.Feb 2 2023, 11:58 AM

JArguello-WMF renamed this task from [NEEDS GROOMING] Improve mediawiki-event-enrichment test suite to Improve mediawiki-event-enrichment test suite.Mar 13 2023, 3:46 PM

JArguello-WMF edited projects, added Event-Platform (Sprint 10); removed Event-Platform.

JArguello-WMF edited projects, added Event-Platform (Sprint 11); removed Event-Platform (Sprint 10).Mar 31 2023, 2:38 PM

JArguello-WMF edited projects, added Event-Platform; removed Event-Platform (Sprint 11).Apr 24 2023, 3:07 PM

Oof, was looking at how to potentially mock the http session and response object, but turns out mocks don't work when pickled/multiprocessed. I guess the only option is to spin up a web server during testing and hit that instead

tchin claimed this task.May 8 2023, 12:59 PM

tchin edited projects, added Event-Platform (Sprint 12); removed Event-Platform.

tchin moved this task from Next Up to In progress on the Event-Platform (Sprint 12) board.

tchin opened https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/54

Draft: Mock requests response for tests

CodeReviewBot added a project: Patch-For-Review.May 11 2023, 1:11 PM

JArguello-WMF edited projects, added Event-Platform (Sprint 14 A); removed Event-Platform (Sprint 12).May 12 2023, 3:52 PM

JArguello-WMF moved this task from Next Up to In progress on the Event-Platform (Sprint 14 A) board.

JArguello-WMF moved this task from In progress to In Review on the Event-Platform (Sprint 14 A) board.May 16 2023, 1:11 PM

JArguello-WMF updated the task description. (Show Details)

tchin merged https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/54

Mock requests response for tests