In T176312#3626696, @Lucas_Werkmeister_WMDE wrote:

RE2 sounds nice… we could probably live without backreferences. But we’d need some way to use RE2 in PHP.

@jcrespo After some discussion on https://www.mediawiki.org/wiki/Talk:Multi-Content_Revisions/Database_Schema, on https://gerrit.wikimedia.org/r/#/c/378724/4, and on the mailing list, there are a few questions that require a judgement call from a DBA. Ideally of course we would test and measure. But I'm afraid at least some of these decisions need to be made beforehand, so an assessment from experience would be very helpful.

This RFC was discussed on IRC on September 20th. Log: https://tools.wmflabs.org/meetbot/wikimedia-office/2017/wikimedia-office.2017-09-20-21.01.log.html

In T121274#3619944, @Smalyshev wrote:

My point was: expanding IDs to resource references is not really the same thing as normalizing values

That depends on normalization I'd guess. But that's a good point. In any case, how we make a decision here?

I'm still very confused about the name. "InterruptMutexManager" sounds like a manager of mutexes of interrupts. If I understand correctly, "Interrupt" in the name refers to the guarantee that the lock will be released automatically when the current request terminates, no matter how it terminates. So perhaps "AutoReleaseMutexManager" would be more descriptive. But then, will we have MutexManagers that do not guarantee this? Why not just call the interface "MutexManager"?

@Krinkle thank you for the overview!

In T151717#3623996, @Lydia_Pintscher wrote:

I assume the latter also means we are not decreasing the number of irrelevant changes shown in the watchlist and recent changes yet?

@GWicke suggest looking into RE2 https://en.wikipedia.org/wiki/RE2_(software)

All relevant patches are merged now.

@Krinkle wrote a while back:

Reopening, because the patches failed to merge. CI is failing due to a problem in the Cirrus extension T174654.

@jcrespo please note that the proposed change is for all wikibase client wikis (that is, almost all wikis). Wikidata itself also has wbc_entity_usage since it is a client of itself, but the proposed change is much more relevant for large client wikis, such as enwiki and commonswiki.

I didn't know the dashboard would save the time range in admin mode. That's kind of odd :D

In T121274#3614183, @Lucas_Werkmeister_WMDE wrote:

Copying Daniel’s comment from Gerrit:

The normalized value of an identifier is not the same as its URI expansion. E.g. ISBNs have a normalized form and a URN, they are not the same.

I think this means that for an ISBN-10 052122151x, the normalized form is 0-521-22151-X while the URN/URI is urn:isbn:0-521-22151-X. But do we have plans to do such external identifier normalization in Wikibase itself? I was under the impression that we expect them to be saved in normalized form in Wikidata itself (for instance, many “external identifier” properties have format constraints to ensure this).

This RFC has been scheduled for public discussion on IRC next Wednesday, September 20.

After revisiting the IRC discussion, TechCom decided to decline this RFC as proposed. However, we acknowledge that the problems listed are worth looking into. Also, the discussion turned up some interesting proposals and alternatives.

@Lucas_Werkmeister_WMDE, @Smalyshev: can you guys agree on something and write it down, so @Ladsgroup can implement it? I don't know which was is actually the easiest to use or the most semantically sound in RDF, and I'm busy with the db schema for MCR.

For now, let's just try to get the size of the jobs below the 4MB mark? :)

@mobrovac how about a very large number of very small jobs? e.g. a million jobs to purge a million pages from cdn?

@Pchelolo actually, can you confirm how many entries there were in the "pages" parameter? With the latest patches deployed, there should be no more than 20. Perhaps this is an old job getting retried, because it failed earlier?

@Pchelolo disabling escaping of non-ascii characters would probably reduce the size to a fourth...

If we still want to cover the "back button" use case, I think we should try and detect that situation on the client side.

Fix above, backport applies cleanely. Note that the fix must be merged into the wikidata build, which then has to be re-deployed (yes we know that this sucks). Perhaps @aude or @Addshore or @hoo can help. @Legoktm should also know how it works. And I guess I could manage in a pinch. Relevant documentation can be found at https://wikitech.wikimedia.org/wiki/How_to_deploy_Wikidata_code

Note that T174422: Make dbBatchSize in WikiPageUpdater configurable is related, but would not change the fact that the entire set of titles would be put into a single InjectRC job, at the moment.

InjectRCRecords batches inserts when running the job, but doesn't chop the batch up before scheduling the job.
I can easily fix that. The patch should be back-portable, too. Give me a minute...

Historical justification, as per Brion on IRC:

Afaics, this task as described in the summary is complete, so this ticket can be closed. Or shall we keep it open to discuss more fine grained configuration, as per my proposal above?

As per https://gerrit.wikimedia.org/r/#/c/377458/, WikiPageUpdaterDbBatchSize is now set to 20. That's probably still to high for RefreshLinksJob, and way low for UpdateHtmlCacheJob and InjectRCRecordsJob.

The batch size is already configurable, that was deployed a few weeks go. The relevant setting is WikiPageUpdaterDbBatchSize.

In T151717#3598520, @Bawolff wrote:

My hope is that by using fine grained tracking on wbc_entity_usage, only when someone edits the actual property in use, will a wikidata edit be propogated to the local recentchanges table, which would significantly reduce the size of that table.

IRC meeting notes: https://tools.wmflabs.org/meetbot/wikimedia-office/2017/wikimedia-office.2017-09-06-21.01.html

This RFC has been scheduled for a public discussion on IRC tonight, September 6.

In T152540#3582851, @bearND wrote:

When I first read this I thought you meant the <head> or <header> tags. After reading the transcript I think you mean the heading tags (<h1>, <h2>, ...) instead.

It would be very nice if we had a setup that would allow us to compare different search profiles side by side, similar to the way we can now compare sql vs cirrus. Can we choose the search profile via a url parameter?

Triaging as "high" to reflect the outcome of the post-mortem with @VColeman, @demon and @greg re https://wikitech.wikimedia.org/wiki/Incident_documentation/20170721-Train-Wikidata.

IRC meeting happend on August 30. Log: https://tools.wmflabs.org/meetbot/wikimedia-office/2017/wikimedia-office.2017-08-30-21.02.log.html

Result: the number of clashes is minimal, and the majority of clashes seem to be unintentional. There seems to be no use case for two clashing titles to exist on a single Wiktionary. This validates the approach taken by cognate, namely, to treat certain titles as equivalent with respect to interwiki-linking.

Do we have the same issue with Fahrenheit as we do with Celsius?

IRC meeting on this happening NOW in the #wikimedia-office channel

Quick reality check: based on the above numbers, less than 2% of enwiki subscriptions are stale, and about 5% of ruwiki subscriptions are stale. That's worth investigation, but probably doesn't have a huge impact.

Pruning stale subscriptions would be nice indeed. Finding out where they come from, and fixing that, would be even better :)

@aude can you help with manually testing Matt's patch? It's looking good, but it needs verification. You have central auth set up, right?

Now let's see what the reduced batch size does. It may actually make the problem worse, but increasing the total number of jobs. Let's hope it makes it better, by reducing the time job runners are blocked...