📊Add a list of instances to filter
Closed, ResolvedPublic3 Estimated Story Points
Actions

Assigned To

Authored By

	gmodena
	Mar 16 2021, 4:28 PM

Description

Establish a filtered content list, and keep it under version control (this will act as a contract and allow change management/tracking

Acceptance criteria

A list of "instance of" items to be filtered is available under version control
All articles that match the filter list have been filtered out
Metrics for items that have been filtered out have been added in the data quality reports

Notes

This will require re-generating production datasets and shipping them to the service instances

Status	Assigned	Task
Resolved	JTannerWMF	T272872 [EPIC] Image Recommendations Android MVP
Resolved	• sdkim	T276137 Exclude unillustrated articles that should not have images
Resolved	None	T276401 SPIKE: Investigate ways to filter out specific articles from image suggestions
Resolved	• Clarakosi	T277557 📊Add a list of instances to filter

• Clarakosi set the point value for this task to 3.

Is this the final ticket for removing disambiguation pages from the results of the api?

In T277557#6956618, @Cparle wrote:

Is this the final ticket for removing disambiguation pages from the results of the api?

@Cparle Yes, it is. There is one more item to complete but you can track the WIP patch here: https://github.com/mirrys/ImageMatching/pull/14

We've implemented the filtering for disambiguation, lists, and year pages. Below you can find the early rough metrics for this filtering.

	wiki	snapshot	Final number of records	Number of records filtered out
0	arwiki	2021-02	628287	11687
1	arzwiki	2021-02	798965	732
2	bnwiki	2021-02	41801	3483
3	cebwiki	2021-02	1360118	159779
4	cswiki	2021-02	202585	21375
5	dewiki	2021-02	137505	30798
6	enwiki	2021-02	2713441	348229
7	eswiki	2021-02	683942	45943
8	euwiki	2021-02	119149	6991
9	fawiki	2021-02	344759	20839
10	frwiki	2021-02	911595	118009
11	hewiki	2021-02	80632	10749
12	huwiki	2021-02	182693	17233
13	hywiki	2021-02	104175	8011
14	itwiki	2021-02	689371	104302
15	kowiki	2021-02	304552	25170
16	plwiki	2021-02	562215	64970
17	ptwiki	2021-02	56195	3798
18	ruwiki	2021-02	578113	56031
19	srwiki	2021-02	136079	22447
20	svwiki	2021-02	1651641	92064
21	trwiki	2021-02	161726	11959
22	ukwiki	2021-02	115388	10742
23	viwiki	2021-02	959365	9476

Thank you! That filters out a substantial number of articles -- it looks like this work will make a big difference!

• sdkim closed this task as Resolved.Jun 1 2021, 5:35 PM