The "webrequest_frontend_text" topic in Kafka we are using to produce the "page_view" stream doesn't have wiki_id, it only has a few fields that can be used to guess the "wiki_id".
These fields are in the current "page_view":
"uri_host": "en.wikipedia.org",
"pageview_info": {
"page_title": "Hyundai_Department_Store",
"project": "en.wikipedia",
"language_variant": "default"
}
"normalized_host": {
"project": "en",
"project_class": "wikipedia",
"project_family": "wikipedia",
"qualifiers": [],
"tld": "org"
}We could use pageview_info.project probably as key, along with page_id, but the standard way to do it in our pipelines is to use "wiki_id", which we don't have here.
Ideally, we should find the way of adding wiki_id into the "page_view" stream, without adding a lot of I/O, as this is a high throughput topic.
There is one interesting endpoint that returns a matrix of wikis, with their url (similar to project) and dbname which seems to be equal wiki_id.
Matrix endpoint: https://meta.wikimedia.org/w/api.php?action=sitematrix&format=json
Other ideas we have considered, but would require more time:
- Find the way of adding wiki_id into webrequest_frontend_text, probably inside the x-analytics field (This would be the recommended way)
- Create a lookup table in Kafka with a log compacted topic. MediaWiki would require to produce events about changes in Wikis, so we can keep the lookup table updated. (This will require more work in different places, so it is discarded for now)
The simplest solution looks to simply call the Matrix endpoint at startup and load all the wikis (url - dbname) in a key value store that can be shipped to all Flink workers.