In T220826#5183700, @WMDE-leszek wrote:

Looking at the query in P8524, do I interpret it right that oldid use is only counted when the revision parameter is not used? Do we care about cases when both revision and oldid are part of the input?

For things added my wmf.3, it looks like there were indeed some chronology protector changes https://github.com/wikimedia/mediawiki/compare/wmf/1.34.0-wmf.1...wmf/1.34.0-wmf.3

My name will be back on the list once I am fully back from vacation

Going to mark this as resolved now

Isn't there some way to be more concise in these entries? So far there's only around 250 of them, but each one of them is over 1GB of data for all of its revisions, *compressed*. We kind of expect articles to take their time to get huge...

In most situations you will just want to pull the image from Docker hub rather than building it yourself too.
Unless you want to make modifications to the build.

I guess a way forward for now would be to introduce the extra configuration option, and then next time we want to add a new language code for monolingual text, we can think whether it belongs in the hard-coded list or in the new config variable…?

Looks like this has now not happened since March 1st

Maybe use a log channel like Wikibase.TermboxSSR. c.f. Wikibase.NewItemIdFormatter extensions/Wikibase/repo/WikibaseRepo.datatypes.php:383

Some batching could be added to RdfDumpGenerator::preBatchDump rather than just doing the array_map there with Title::getTitleForId.

I took a look at the usage on a current day:

That all looks fine.
I added a new panel to https://grafana.wikimedia.org/d/000000574/t204083-investigation?orgId=1 showing the stats for WANObjectCache and these keys.

So, I did some really crappy analysis of the hit rate in varnish before and after this change, looking at the 5th of april and the 5th of may (one before and one after as far as I can tell).

As this is just test wiki (and not a real wiki), and that this can no longer occur, perhaps we should just totally nuke the page from the db

Looking at the patch that was merged, I think we can close this, as it indeed looks like it was just a test issue.

For wikidata this is done in

Quickly looking over this ticket:

When you changed from http to https did you change anything in the sites table of the wikibase?
Is there a redirect in place?

All of the options seem like they would work, looks like 1 was the easiest and it seems you've already done it?

In T220823#5135899, @Nuria wrote:

pinging @Fjalapeno from your comments the other day I understand Wikidata is going to use cassandra for these use cases at the end? cc @Addshore

Sounds okay to me!

Partly done, but not actively being worked on, so moved back to ready to go from doing

That sounds like a great idea :)

Exact duplicate of T221214 @Tarrow ?

I removed "development" from the title as the images we are talking about here are not intended for development use.

For "Iteration 1 Acceptance Criteria" rather than specifically counting, I think logging these would be enough, then we can count these in logstash.

In T221602#5164087, @Tarrow wrote:

I'm not clear that this is necessary. If we remove the "dead languages" at the time of discontinuing them then it shouldn't be something we need to ignore. If people are trying to make edits that include now-defunct languages I think it might be better that we error and let them know rather than silently ignoring them and leaving them to wonder why the edit they asked for wasn't made; I think this would only happen in the end someone was trying to edit with outdated-data or old code that needs updating.

I just flicked back to https://grafana.wikimedia.org/d/000000574/t204083-investigation today and noticed that as of 1.34.wmf.3 a pattern similar to what we were seeing before is back?
https://tools.wmflabs.org/sal/log/AWp6A7RAEHTBTPG-zSOo

I don't think we will be able to expose the bits of WANCache that we need in some places through an interface such as PSR6 or bagostuff.
For the places where we do need such complex set get delete purge etc logic for caching for a multi DC situation we will have to write the code to use the WANCache interface, or something like it.
As we already have to write that logic, I'm not sure I see the value in then also allowing either a regular bagostuff or psr6 thing here.
At the end of the day one can create a WANObjectCache instance that just contains a single bagostuff as far as I know, rather than having code in wikibase do different types of caching based on the interface passed in.

CacheAwarePropertyInfoStore is solved with T218197

I month on, is there a decision then? or? :)

My only thought when reading this is that when we get onto point 3 for items, data in wb_terms will remain and slowly become out of date? (unless there are some deletes happening during these writes?)

A couple of comments from IRC so we don't forget them in the coming months:

I guess we need to update operations/dns then?

I had a quick look around and found a few examples of legitimate aliases or labels with |s in:

My final thought here before heading to vacation is, when terms are removed from an entity, we already know what terms therefor might need to be removed from the other tables, so we should probably use that information rather than have some secondary process cycle through the millions of terms trying to find what to remove.

Ping @MasinAlDujailiWMDE ? :)

It sounds like we shouldn't really be removing things from WikibaseContentLanguages::getDefaultMonolingualTextLanguages().
For wikidata.org we should have some configuration which can add language codes to that list.
If we want them to be available for all wikibases then of course they should be added to the main list.

I have no idea how these queries would perform, but they are scary and I don't think we should be running them on the production master DB.
If cleanup is wanted from some sort of maintenance script it should probably use batches of deletes and or selects.

Are we really sure disallowing aliases with |s in is the right path forward?
What are we going to do with the 56k aliases that currently have pipes in them? (Note this is 56k alias strings, not 56k items (it will be more items I expect).

After I had a look at the code I don't think this one needs a security review.
So i went ahead and merged the patch merging it into mediawiki-vendor.

I'm slightly confused with the description 2 comments up.

This will likely get worked on, at least for Api output with the work on federation that should happen toward the end of this year.

I'm pretty sure no current users of the thing use hhvm

Amazing, I think we can resolve this then!

We should see changes in 1.33.0-wmf.24.
It looks like the train didn't make it past group0 this week, so no change really spotted yet.

It looks like the train didn't make it past group0 this week, so no change really spotted yet.

As pointed out at some point we probably want to remove the strings from the table fairly sharpish, as content my be removed, revdeled, and should not continue to appear in public places such as labs dB replicas. So that rules out maintenance scripts run periodically .

I believe analytics-wmde is enough (as it gives you sudo on the analytics wmde users, but should be double checked if one can actually SSH into the machines with this.

In T217897#5066900, @Smalyshev wrote:

WDQS does know what the latest version of the entity that it is trying to get updates for is,

But "last version that WDQS knows of" can be very different from "last version that Wikidata has". That's the whole issue.

I had an idea recently though. Maybe we could work it in two modes - if the stream is lagged sufficiently, we use the "latest available" mode - to jump to the front, but if we're more or less current, the probability of our change being current is high, so we could use "by revision ID" mode. Need to look at edit timings to see if it's workable but may be splitting two cases - catching up from large lag and keeping current - would be more efficient and allows us to use cache for the most frequent case (which is "keeping current"). That would be easy to implement and test - just a couple of if's in proper places.

In T217897#5062728, @Smalyshev wrote:

the cache we are talking about there would be unnecessary if the wdqs just hit varnish.

It is problematic for WDQS to "just hit varnish", because varnish does not know if certain revision is the latest one available or not. Wikidata on the other hand does.

So, the mock suggests that "|" will be used for inputting multiple aliases.
Currently in the wikidata UI one can enter aliases with |s in them
Example: https://test.wikidata.org/w/index.php?title=Q214&diff=489761&oldid=2181
The special page is actually destructive here and loading the page for the entity and clicking save results in loosing data: https://test.wikidata.org/w/index.php?title=Q214&diff=489762&oldid=489761
I filed a task for that @ T219499

In T219438#5065571, @hashar wrote:

Note that you can get a 90 days Windows VM directly from Microsoft: https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/ They provide a VirtualBox image.

Those images are intended to provided web browser to web developers so they can verify websites work as expected on IE / Edge. So it is a narrow focus, but maybe the VM ship enough material to actually replay the install of local charts.

The machines expires after 90 days.

presumably the federation which we now disabled was used for something (“depicts” statements, right?), so the Structured Data on Commons team might need to communicate that it’s currently unavailable

Reproducible on testcommons