Software developer on the Wikidata team at Wikimedia Germany. Private account: @LucasWerkmeister.
I think all the items that were updated during the breakage, and were not updated since then, would be broken now – their data was imported using the swapped Ps and Qs, and then after the fix it’s swapped the wrong way. We should be able to get a list of all potentially affected items from recentchanges – @Smalyshev is it possible to tell the updater to re-import those specific entities?
Thu, Aug 15
Scholia also looks fine again – lowering priority since only the investigation is missing. Thanks a lot for the quick response Stas!
?: Podcasts. I ask Wikimedians questions like: What would be your first orders as King (or Queen) of Wikipedia?
Those errors usually happen when the query returns 0 results and the visualization is not able to handle it, and that seems to be what’s happening.
Mon, Aug 12
In EntitySchema (package.json) we’ve been very happy with husky and lint-staged – I think @Michael is the expert there. (Unfortunately, PHPCS/PHPCBF’s exit codes don’t play nicely with it, but for the JS part it’s really wonderful.)
Well, there’s no train this week (Wikimania), but we could add it to next week’s train blockers already.
Thu, Aug 8
Even Pywikibot, which still calls Statements “Claims”, doesn’t read the type when deserializing a Statement (Claim.fromJSON), although it does emit a type when serializing it (Claim.toJSON).
Hm, that’s tricky… KrBot automatically fixes the format, at least sometimes (example edit), but I don’t know if it also does that in reference snaks (or would Citoid create new items for each ISBN in a reference, so that it would always be a main statement?), and I’m not sure if it’s acceptable for Citoid to make edits that need to be cleaned up by a bot later.
Wed, Aug 7
I assume this refers to the final TODO in init.js / “shows the current targetValue”, where we currently don’t check that the value is what we expect?
As far as I’m aware, this is done.
Tue, Aug 6
The petscan3 instance is one of those brought down due to Cloud VPS maintenance yesterday and today (announcement) – if the VM is set up such that it doesn’t need any manual action after a reboot, then this might resolve itself within a few hours.
Actually, let’s split this task up after all. Disregard the gerritbot comments, I’ll move those patches to one of the subtasks.
Mon, Aug 5
To clarify: for Wikidata Bridge we likely only need support in wbeditentity, but the most reasonable implementation (as far as I can tell) ends up adding support to almost all other modules as well. The exception are modules that create a redirect rather than a regular entity edit – I’ll leave those aside for now.
Late update – I just found out that MediaWiki allows you to suppress the external link marker using the plainlinks class, so we could’ve gone with the external link approach after all. We don’t have to change anything for that now, but it might be useful if we ever need to come back to this for another reason.
Fri, Aug 2
The issue persisted for ca. 10 minutes after the scap finished, for reasons I don’t understand, but now it does seem to be resolved. Leaving the task open for verification by others.
Both of the changes attached to this task are merged, so I assume this can go to verification.
Tested with multiple instances across two projects (wikidata-dev and openrefine).
Done – I have backups on my laptop in case anybody needs them:
Thu, Aug 1
The wikibase.api module in wdio-wikibase has createItem() and getProperty() functions that we can use for this.
However, it is not clear why this should make a difference as our code is not supposed to modify $, even when called in the global scope.
Well, for the DB it wouldn’t make a difference because it always gets 2500 results anyways (and then sorts and limits them in PHP – I guess applying an offset here would save a bit of memory, discarding the unneeded terms earlier), but we definitely should pass the offset to ElasticSearch, yeah.
Wed, Jul 31
I looked a bit more into this, and it turns out that SearchEntities doesn’t support continuation all that well – basically, it asks the underlying search backend for offset + limit + 1 results, then returns the [offset, offset+limit) slice of that. Clearly, this isn’t very efficient for larger and larger offsets, which is why the API won’t return offsets higher than the standard API limit (50) for continuation. However, it won’t stop you from specifying larger limits yourself, potentially asking the search backend for arbitrarily large numbers of results.
I’ve proposed the API change in T229460: Support standard MediaWiki API continuation in wbsearchentities module / wbsearch list/generator.
Indeed – this seems to have been fixed in codemirror/CodeMirror#5936 (using the approach I mentioned above: hard-code the ranges of all Unicode letters into the regexp), which was released in 5.48.2 12 days ago, and I guess we installed that version?
It looks like the redirect is generated in Varnish, see text-frontend.inc.vcl.erb::mobile_redirect. We could try to disable it in some ways:
Well, jQuery would try to follow the redirect, but it doesn’t have the Access-Control-Allow-Origin header set, so it’s blocked by the Same Origin Policy. (That is, https://de.m.wikipedia.beta.wmflabs.org is not allowed to read the https://wikidata.beta.wmflabs.org redirect.)
It looks like we request https://wikidata.beta.wmflabs.org/wiki/Special:EntityData/Q11.json, and instead of directly returning the entity JSON, that redirects to https://m.wikidata.beta.wmflabs.org/wiki/Special:EntityData/Q11.json (mobile subdomain), and we don’t follow that redirect. I don’t really like Special:EntityData redirecting to a different domain, but fixing that sounds like it would be complicated (and involve changes in MobileFrontend, not just Wikibase), so it’s probably best to just follow the redirect.
Tue, Jul 30
I think we expect it to stay around for now.
Updater.java:289 is this line:
So if we purge the page via action=purge (probably with forcelinkupdate=1, though tbh I’m not quite sure what that controls) before reloading it, we shouldn’t have to worry about dispatch lag. That still leaves the issue of replication lag on the repo wiki, though: if the client wiki page is purged and re-rendered based on data from a replica database on the repo wiki that hasn’t seen the edit yet, there will still be stale data.
There currently seem to be at least two workarounds for this missing feature:
Alternatively, perhaps wbsearchentities should be changed to do continuation the usual way? That would probably improve compatibility with other ways to access the API as well (“continue” button on Special:ApiSandbox, continuation=True in python-mwapi, etc.).
Mon, Jul 29
Well, the original query doesn’t tweak the parameters, and by default we don’t want to follow too many empty continuations. This query returns results, though: