Page MenuHomePhabricator

Investigaton: how to determine which subcategory of https://de.wikipedia.org/wiki/Kategorie:Sachsystematik a page belongs to?
Closed, DeclinedPublic

Description

Page categories are a non-trivial polyhierarchy, in other words the data is very dirty.

We probably wouldn't make API calls for every article during the scraper run, but would run this only on the smaller sample of articles with similar refs, in a later step.

The technology exists. It's just a question how we can use it efficiently. A trivial approach is to run the search API for every (potential) page combined with every category, like this: https://de.wikipedia.org/wiki/Special:Search/pageid:6646256_deepcat:Literatur. This should be reasonably cheap. If the query returns the page we know it's part of the given category (sub-tree), or not.

Event Timeline

Look into https://meta.wikimedia.org/wiki/PetScan/en , it may be an approach we can use or reverse engineer.

Deprioritized for this scraper run.

WMDE-Fisch subscribed.

Was a nice to have. We're not doing that anymore.