PR at https://github.com/mirrys/ImageMatching/pull/10.

Re-running the algo will take a while. I'll kick-off the job once PR accepted & merged.
I'm currently generating data for a subset of wikis with snapshots 2021-01 and 2021-02.

tl;dr: we are unable to determine the type of ~13% (average) of all unillustrated articles (with and without recommendations), and 15% of unillustrated articles with at least one suggestion.This means that we won't be able to filter out those articles, since we can't determine what they are.

There is going to be some variance per wiki. Below you'll find a breakdown of stats from two samples (January and February 2021). "% missing" denotes the % of articles we won't be able to filter, out of all unillustrated_articles for a given wiki_db. "missing "instance_of" is number of articles we won't be able to filter.

These numbers depict articles we *know* we'll skip. For articles with metadata, we don't have (yet) an estimate of quality and completeness of "instance of" info.

All unillustrated articles

Unillustrated articles with at least one suggestion

We could answer questions about the type of page more comprehensively using the tables categorylinks and templatelinks rather than structured data.

For example if I want to know if page 12345 on enwiki is a disambiguation page I can run the sql query

select cl_from from categorylinks where cl_from=12345 and cl_to='Disambiguation_pages';

Hey @Cparle,

Thats a good point!
To clarify: by categorylinks do you mean wmf_raw.mediawiki_categorylinks on the hive warehouse? Do you maybe have an estimate of how comprehensive it is across languages and categories? I'd expect quite a lot, since it's mediawiki.

In future iterations I'd be happy to revisit the current filtering logic, and augment it/replace it with categorylinks if that would help improve UX.

To give you some context, filtering based on wikidata is the default approach in the upstream code. We investigated alternative approaches to improve filtering, including parsing wikitext and mining wmf_raw.mediawiki_categorylinks.

Before committing to a rewrite we want to get some metrics to understand how well wikidata can satisfy our use case, what the error tolerance is, and build some expertise with the mediawiki_categorylinks data model and access patterns (e.g. efficient ways to handle translations).

Happy to discuss further.

To clarify: by categorylinks do you mean wmf_raw.mediawiki_categorylinks on the hive warehouse?

I don't know much about hive, but if that's a copy/snapshot of the data in the mediawiki categorylinks table then yes. The only complicating factor is that cl_from is a text field with the title of the category, so for each language you'll need to figure out what the title of the "Disambiguation pages" category is. I expect there is one for every language

Correcting stats after fixing https://phabricator.wikimedia.org/T278571

All unillustrated articles

We are unable to determine the type of ~25% (average) of all unillustrated articles (with and without recommendations).

Unillustrated articles with at least one suggestion

The fix did not impact articles for which a suggestion is available.
We are unable to determine the type of ~15% (average) of all unillustrated articles with at least one suggestion.