24% of all wikimedia pageviews are for Wikidata? I find that extremely surprising. Is that on the app servers, or does it include CDN cache hits?

The second instance involves the TitleMoveComplete hook. Moving a page creates a dummy revision, which probably has not yet been replicated to all replicas when the hook handler is called. The hook handler, FlaggedRevsHooks::onTitleMoveComplete, may have to force the master database to be used, via the appropriate query flags. This should be safe, since the hook is only fired in requests that write to master anyway.

The first instance looks like a race condition including a 3rd party client (bot). E.g. bot makes and edit, and immediately tries to load the revision. Or a bot lists revisions, and then tries to load their content by ID, in a different request, hitting a different replica that is more lagged.

Option "a" could cause an outage on the master if for any reason all the replicas are lagging

To go out on a limb here, this looks like it's caused by replication lag: some bit of code is trying to load the vision from a replica right after it has been saved, and it hasn't been replicated yet. This can be avoided by a) falling back to loading from master if the revision isn't found on the replica, and b) making the RevisionRecord that was created when the edit was made directly available to the code that needs it. (b) is preferable, but often rather hard to do.

The reason this is a dedicated service is the language it is written in (typescript), which was chosen because it allows us to create an implementation which can be compiled/transpiled to work on both server and client - something not possible with PHP.

A quick note on how Wikidata is different, particularly from Wikipedia:

In T213453#4870354, @Addshore wrote:

Getting the whole old revision in the form expected is easy, AbuseFilter::revisionToString
We need to get from the Content / string text to the new revision though. So we need to figure out what slot has been edited so that we can make a "pretend" whole revision to use for extracting the whole text, or just have that whole revision passed all the way down, instead of just the content / text.

@Bawolff my assumption was that project membership was intended to reflect current staff. Unfortunately, there seems to be no documentation for this. Perhaps this should be reviews as part of the upcoming revamp of the +2 policy for gerrit.

Did the chat with @Joe happen? What was the outcome?

This RFC is done. The storage layer of MCR is implemented and deployed. Further refactoring and new features building on top of that will be discussed in separate RFCs if needed.

Special:PageData is intended to provide a canonical way to get the raw content of a page, in machine readable form. The syntax depends on the page's content model - on most pages, it will be WikiText. Note that raw content is different from the page's meta-data. In case of media files, this line is somewhat blurry. EXIF data is currently treated as meta-data, not content. And the actual media blob is treated as a raw file, not page content either. Both could potentially be treated as page content in the future, but there are no concrete plans for this.

Moving this to "backlog" for now. This should have happened right after last week's TechCom meeting, but I forgot to do it.

Quick meta-point:

Hi Jan! I removed you (and a few other former employees, IIRC) when looking over group membership in github; We are trying to consolidate onboarding/offboarding procedures, and keep groups on different services in sync. You being in that group seemed like an oversight. As far as I'm aware, membership in the wikimedia organization on github is intended to reflect current staff, with the exception off external people actively involved in ongoing projects. The cleanup is still incomplete, I just took care of the people from the wikidata team.

It's possible that this was caused by https://gerrit.wikimedia.org/r/c/mediawiki/core/+/452708/. Shouldn't happen, but who knows. Perhaps we should just revert the patch and see what happens.

@Krinkle said:

Rather, it starts out on the assumption that we're going to have UI code in production (based on Vue.js) written in a way that contains too much business logic in its templating code.

As far as I can see, the situation is this: Option 1 is ruled out already, since data access from wikitext doesn't work with that approach. Options 2 would need community consensus. Option 3 (default repo per entity type, with prefixes used internally) is still on the table. Option 4, proposed by Leszek (federation directly based on entity type, with no prefixes used internally) is made unattractive by by Lydia's statement that we cannot guarantee that we will not need prefixes in the future, which would put us back into the position we are now, giving us the choice between option 2 and 3.

I pushed a new version of https://gerrit.wikimedia.org/r/481015. The reference to SlotRoleHandler is simply not needed in 1.32. I just removed it. No need to drag in the new service.

@Joe said:

the SSR service should not need to call the mediawiki api. It should accept all the information needed to render the termbox in the call from mediawiki.

@Reedy this is really more of a conflict than a dependency. I would recoommend against backportinng SlotRoleHandler to 1.32. I'll touch up the original patch.

It should not be possible for $this->mRecord to be null in line 652 of /srv/mediawiki/php-1.33.0-wmf.8/includes/Revision.php. I made a patch that should help us to find out how it does become null.

Moving to the RFC inbox, so TechCom will look at it during the next meeting. Since DBA have approved the plan, TechCom will probably put the RFC on last call (per T200297#4787259) during its next meeting, which will be on January 3rd.

@Tgr No, that error is specific to forced updates that are not edits.

For the record, the revert should be temporary, on the deployment branch, not on master. This is exposing a problem in the calling code, which needs to be addressed. The revert is not a proper fix, just a stop-gap.

Pinging @Jakob_WMDE and @Pablo-WMDE, since the rest of the Wikidata team is already out for the holidays. Pinging @Lucas_Werkmeister_WMDE, since he probably knows most about wbcheckconstraints.

You first need to render the page on the server before you know whether the client supports JS/SW or not, so it will need to be rendered on the server irrespective of the client's capabilities in case where MW calls the service directly before handing out the page.

When rendering the page, index.php knows the exact data that needs to be rendered already, correct?

In T212189#4837780, @mobrovac wrote:

When rendering the page, index.php knows the exact data that needs to be rendered already, correct? If so, it can send that to the client, and then based on whether JS/SW is available or not, the client either renders it or sends a request to the service which sits behind Varnish. Or am I missing something.

@Milimetric wrote:

In the simplest case, this code would be almost identical client and server-side. No matter where it's running, nodejs or the browser, it would request data, receive it, and render it as html.

It would be nice to collect stats about number of pages, users, edits, etc; and info about whether the wiki is open (allows anon edits) or private (allows anon reads).

ping Lydia

Well, I consider calling the MW api from a service called by MediaWiki an antipattern that we should absolutely avoid.

In T211800#4831696, @WMDE-leszek wrote:

As I am gone for two more weeks, it would probably the best that I shortly answer here: to me it seems both kinds of federation are completely separate and possibly even mutually exclusive. So I would rather not mix these, neither conceptually nor in terms of code. So depending on how you look at it, the difference might be either "a lot" or "not at all".

I agree with Joe that it would be better to have the service be internal, and be called from MW. It doesn't have to be that way, but it's preferable because:

• we would not expose a new endpoint
• we should in general avoid (more) services calling MediaWiki, because:
  • PHP has high startup time, and also for reasons of general hygiene of the architecture
  • we don't want MW and external services calling each other, back and forth
• "pure functional" services that do not interact with storage are easier to reason about, and easier to run and maintain.

Using accept-language is not an option, at least not the accept-language from the browser. The relevant list of languages comes from user preferences and the Babel extension. This also means that it typically contains more than one language, and there are many, many combinations to consider.

Calls to getMainObjectStash: https://codesearch.wmflabs.org/search/?q=getMainObjectStash&i=nope&files=&repos=

@mobrovac Please note that the term box is shown based on user preferences (languages spoken), the initially served DOM however needs to be the same for all users, so it can be cached. Also note that the language specific data that goes into the term box has to be loaded from the wikibase entity. So the only way to make it work as you suggested would be to always send all the terms in all languages, which, for some items, would be quite a bit of data.

A detail to consider when going for "federation without prefixes": does this mean no prefixes just for user input, or also in the JSON serialization? Using no prefixes there either may seem intuitive, but it make the data more brittle, and harder to exchange between instances. Other repos consuming that data will then need to replicate the data-type mapping instead of a prefix mapping. That seems error prone to me.

In T211800#4830605, @WMDE-leszek wrote:

• Having two not-necessarily same config flying around and parts of code arbitrarily picking one config, and other parts picking up the other seems like a bug/unfinished implementation to me. I am surprised it only surfaces now (it has been for 99% me who had messed that up), but should be fixed. If the current "broken" state actually makes commons work the fixing schedule can be of course postponed :)

One source of confusion is the fact that the CachingPropertyInfoLookup used by client code and repo code are hitting the same cache entry. That cache entry ends up containing both prefixed (wikidata:P1) and unprefixed (just P1) property IDs, side by side. This in turn causes API modules, which use the repo service instances, to also accept unprefixed property IDs, which causes data corruption: MediaInfo entities can have some Statements that use prefixed property IDs, and some that use unprefixed property IDs, and these are treated as different and incompatible (correctly - according to the configuration, they come from different repos). Repo side code (in "split brain" operation) should not accept unprefixed property IDs or unprefixed item IDs, since there are no properties or items on the local repo.

I have confirmed locally that the current situations prevents access to MediaInfo entities using the {{#statements}} (nor presumably from Lua, which uses the same underlying mechanism). This means that MediaInfo cannot be used on file description pages as intended, until this is resolved.

Superseded by MCR.

I just realized that this probably blocks access to MediaInfo from wikitext on commons. I have not confirmed this, but if my mental model is right, this needs to be fixed before we'll be able to access MediaInfo from wikitext.

This should probably be in the 1.32 release.

As far as I can tell, all patches are merged. Can this be closed now?

In T191922#4540021, @Jdforrester-WMF wrote:

Let's do it.

What's still missing here? The ability for extensions to require a php version, as requested by this ticket, appears to have landed, plus the ability to require php extensions. The patch that is still open seems to be for specifying platform capabilities. That seems to go well beyond the scope of this ticket. Should that really block the release? Am I missing something?

Is anyone committing to implementing this? It's in the backlog for the release, but in no team's backlog, and not assigned to anyone. If nobody committed to doing this, it probably should not block the release.

This is marked as a release blocker. What's still needed here?

If I read the discussion on the patch correctly, this ticket currently blocks the release of 1.32 on the question of whether the API should issue a warning when producing a diff against a revision that does not exist. This seems disproportionate. I think neither solution (having or not having the warning) is so problematic that it should be prevented from being in the release. I would default to restoring the old behavior as-is, even if I didn't particularly like it. On the other hand, if the patch author insists on having the warning, I wouldn't block on that either. In theory, this should be decided by the product owner of the API - which is the job that @EvanProdromou just started. But really, it just doesn't seem big enough of a deal to me to block on it.

@ArielGlenn there is a lot of stuff here, I lost track at some point. Can you tell me whether anything urgent remains to be done here? Also, do we have tickets for the things that are not urgent, but should be addressed or investigated?

For option three, we'd have to decide whether the per-entity-type-defaults should be defined separately from the prefix mapping, or whether it should use some kind of special syntax., I'd suggest the hybrid approach we have also been using for including the target slot in EntityNamespaceLookup: use separate arrays internally (cleaner), but use some special syntax in the config (simpler). So for commons, you would have a mapping like [ '@item' => 'wikidata', '@propert' => 'wikidata' ], with the @ indicating that the mapping is not for prefixes, but for entity types.

This needs a (minor) release of wikibase/data-model-serialization.

Pull request: https://github.com/wmde/WikibaseDataModelSerialization/pull/250

Thanks for testing, James!

As it turns out, I missed to crucial line of configuration when replicating the setup locally:

In T204748#4820156, @Cparle wrote:

So should MediaInfo items have the page namespace, or their own namespace?

In T204748#4820069, @Cparle wrote:

If MediaInfo items only exist as slots, then I don't really understand why they need a namespace

In T204748#4819946, @daniel wrote:

I don't think this is the reason that MediaInfo doesn't work on beta, though. That still needs more investigation.

Looking at T204748#4783220 and poking at the code, it seems that the problem is that the entityNamespace setting for MediaInfo is missing. However, this should be impossible, since that setting is forced by the onWikibaseRepoEntityNamespaces hook.

In T204748#4784693, @gerritbot wrote:

Change 476403 merged by Ladsgroup:
[mediawiki/extensions/WikibaseMediaInfo@master] Don't check for integer namespace in MediaInfoIdLookup

https://gerrit.wikimedia.org/r/476403

ping @LucasWerkmeister @Lydia_Pintscher

Is this the same as T196042: Configurable RDF prefixes for WDQS?

I have a suspicion about this, but I want to dig some more to confirm. But if I am correct, this never worked on beta, at least not with MediaInfo in slots, with the config that is there now.