Page MenuHomePhabricator

Support new fields: (1) namespace textual prefix and (2) page title without prefix
Open, Needs TriagePublic

Description

Presently event.mediawiki_page_html_content_change_v1 provides the page_title field as <prefix>:<page_title> when the namespace is not the main namespace.

spark-sql (default)> select page from event.mediawiki_page_html_content_change_v1 where year = 2026 and month = 5 and day = 9 and hour = 1 and changelog_kind = 'update' and wiki_id  = 'frwiki' and page.page_id = 13459570 limit 1;
page
{"is_redirect":false,"namespace_id":6,"page_id":13459570,"page_title":"Fichier:Logo_Rugby_Club_Savoie_Rumilly_2018.png","redirect_page_link":{"interwiki_prefix":null,"is_redirect":null,"namespace_id":null,"page_id":null,"page_title":null},"revision_count":null}

This is slightly asymmetric with the page_title found in other places like the page.page_title field in MediaWiki.

[commonswiki]> select page_id, page_namespace, page_title from page where page_id = 12444991;
+----------+----------------+-----------------------------------------+
| page_id  | page_namespace | page_title                              |
+----------+----------------+-----------------------------------------+
| 12444991 |              6 | KirstenboshBotGard09292010OverviewN.JPG |
+----------+----------------+-----------------------------------------+

It's handy to join on the page_title without the namespace prefix, particularly if the page_id is many hops away in a join, or someone is operating on a list of page titles and didn't have the page_id readily available. That said, the namespace prefix can indeed be handy sometimes, especially when one struggles to determine the name of the namespace.

This is a request to add two fields for ergonomics:

  • namespace_prefix: the plaintext prefix string for the namespace (in the example above this may be Fichier or File, for example, maybe good to have the main namespace localized name as well instead of implicit as an empty string, too?).
  • page_title_noprefix: the page title stripped of its namespace prefix string. This will also make it so that titles that happen to contain a : character can't be as easily visually confused as potentially bearing a namespace (e.g., enwiki's CD:UK).

Maybe there's a more straightforward way, but after talking with @Ottomata it was suggested to file a task to see what could be workable.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Hm, I'm not opposed! Quick name bikeshed:

  • How about just namespace (since we already have the id as namespace_id) or uh namespace_name? I think MW calls this namespaceName in various places internally.
  • title: Mediawiki seems to call this 'dbkey' and also title 'text' in various places. So maybe page_title_dbkey or page_title_text?

To do this, we'd need to:

This will get the fields in the mediawiki.page_change.v1 stream. If we want it in downstream streams like the content and / or html stream and / or html_feature_counts streams, we need to

  • Bump the $refs in output schemas used by those pipelines to get the new fields in the output schemas.
  • Bump the input and output schema versions used by those pipelines
  • redeploy.

Since the streaming enrichment pipelines just copy the input data to the output data and add a extra stuff, I don't think there should be any code changes needed.

namespace_name is usually the English canonical version. Maybe do both of these?

  • namespace_name_canonical
  • namespace_name_local

If you think it preferential to have only one of those, then I'd suggest namespace_name_local (or maybe namespace_name_interwiki; something to convey that it is whatever the wiki's normal string representation of the thing).

And then:

page_title_dbkey seems reasonable enough if that seems more fit than page_title_noprefix. It's a little MW-ish, but dbkey will be intuitive enough from code lookup (in addition to the schema's documentation, obviously). There's one discovery table using dbkey as a number field, but I don't think that constitutes a pattern and was not meant in the same way we mean it here, so it seems safe enough here

dr0ptp4kt renamed this task from Support namespace_prefix and page_title_noprefix to Support new fields: (1) namespace textual prefix and (2) page title without prefix.May 14 2026, 5:11 PM

(Post message send thought: namespace_name could also simply be the canonical one, and then we could let namespace_name_local be the one that is distinctive for the given wiki; that way we get both conformance to other schemata, but the desirable property to have the bona fide prefix for the given wiki.)

no strong feelings -- it's true that e.g., mediawiki history drops the namespace prefix and so that seems to be the standard we've adopted in the data world. In that sense, just using page_title without additional qualifiers would seem to map to the non-prefixed version in most places. I guess my question would be: why include the namespace name if it's not relevant for any data joins? I guess it allows you to quickly construct a full URL if that's what's desired but that seems to be in the meta field already?

why include the namespace name if it's not relevant for any data joins?

That is a very good point. If MW stores page_title without prefix, why are events emitting it with prefix? I don't know! Historical baggage? I'm sure I get my phab archeology tools out and find a decision, but I don't have time for that now! ;)

I wonder if anything would break if we just stopped emitting prefixed page_title in events? ...Probably.

In terms of guarantees on the data, changing the present emission behavior on page_title constitutes a breaking change.

For URL composition that is not to make the canonical URL, but still involves the namespace name as well as the title (e.g., api.php calls), namespace_name_canonical (or namespace_name) and the namespace_name_local (coupled with page_title_dbkey or page_title_noprefix) are beneficial.

It's true that in most cases the namespace_name_canonical (or namespace_name) is legal in the sense that MediaWiki will honor the English version even if there's a localized name present for the wiki, although also having the true namespace_name_local is more conventional in practice...and having it at the ready sidesteps needing to parse the meta property.

I thought I had seen that we were hoping to avoid certain aspects of the meta segment of events (but determined we couldn't mess with it too much), but I'm not sure about where that may sit these days. In the context of the HTML page change events (which was the thing that spurred me to open this here task), I'm unsure how to reason about it cleanly, if for example there's a page change move...although ultimately one has to just deal with that anyway for any downstream pipeline. This is to say that I see the utility of the meta field, but also wasn't sure if we had a thought about it for data analysis when the rest of the payload bears the the data.

I think @Isaac you were speaking to the notion that the inclusion of the localized namespace string present state in page_title seems a little unusual given how page_title is bare without that localized namespace string and colon in other datasets; yeah, same same. As far as whether we ought to have additional bespoke fields to make the distinction clear: for those, I think the additional fields will be useful in practice (and hopefully folks would switch over to them in due time when they're ready...and where it is appropriate, I guess; obviously the schemata and fragment schemata versioning in theory handles this, but I worry a little people would see the semantics as being problematic without a new field...but am of course guessing some).

changing the present emission behavior on page_title constitutes a breaking change.

Indeed. Okay new field.


For namespace I think I prefer:

  • namespace_name - the 'canonical one'
  • namespace_name_localized - the 'local wiki one'. Or um, it looks like namespaces can be localized into any language? Maybe we shouldn't try to emit the localized one at all if there can be many different localized versions, or if it can change?

For page_title I can't decide. 'dbkey' is too weird. Maybe just page_title_key? Not very clear. page_title_main? page_title_original? page_title_standalone? Other ideas?

changing the present emission behavior on page_title constitutes a breaking change.

Indeed. Okay new field.

SGTM

For namespace I think I prefer:

  • namespace_name - the 'canonical one'

Cool.

  • namespace_name_localized - the 'local wiki one'. Or um, it looks like namespaces can be localized into any language? Maybe we shouldn't try to emit the localized one at all if there can be many different localized versions, or if it can change?

I'd prefer we have the string that's considered standard for the given wiki at that time (useful for history in case the prefix does change; it's probably rare, but still). But if you feel iffy, I think we can skip it. The slightly awkward thing if trying to construct URLs is they'd need to be regex extracted from the meta field or they'd need to rely upon tacit knowledge of the localized version (or rely upon the English version if known, which is legal, but at odds with URL reconstruction)...I don't know, though, do we have a table against which one could join? Feels cheaper to capture the thing at the time the event is emitted than to require a join on some table. But again, if in doubt, I think this localized version field string (when there is a localized version of the namespace name) could probably be foregone.

For page_title I can't decide. 'dbkey' is too weird. Maybe just page_title_key? Not very clear. page_title_main? page_title_original? page_title_standalone? Other ideas?

page_title_key looks okay. page_title_noprefix was my earlier idea, but this is at odds with the terminology of "prefix" at https://en.wikipedia.org/wiki/Special:PrefixIndex , and a more correct page_title_sans_namespace (I kid on the sans, but it's short!) feels long, so page_title_key feels like a good enough compromise. The schema documentation could note that its friend in MediaWiki is dbkey, it's the title free of any namespace prefix, etc.

page_title_unprefixed ? I like either that or page_title_key I think.

Oooh...maybe page_title_unnamespaced or page_title_denamespaced would get it accurately (even if it isn't technically de-namespacing, let's say), but avoid any other ambiguity. I don't have particular serious problem with page_title_unprefixed...just the mild reservation about the term 'prefix' being prone to be conflated with that Special:PrefixIndex meaning, its dropdown lets one choose the namespace. (Perhaps comically one might note that if one leaves the namespace at Article on that special page and types in a known namespace string, well, because the main namespace implicitly lacks a namespace string there, one can get things like, for example: https://en.wikipedia.org/wiki/Talk:A%26M ; but that does not impugn the form's intent of focusing upon a particular namespace; it just has that side effect.). Your call @Ottomata! This is proof yet again naming things is hard.