In T277862#7093619, @Michael wrote:

Can we robustly detect replication lag via the API? It might make sense to skip tests (or assertions) in those cases, rather than accepting them to flake because that will train us to ignore the real failures as well.

https://github.com/wmde/wdio-wikibase/pull/33

https://github.com/wmde/wdio-wikibase/pull/33

In T281587#7087337, @Ladsgroup wrote:

Technically this is done but it's test renable patch got reverted. I let you decide how to proceed here.

Moving this back into TODO to give someone else the chance to pick this up (as I wont work much more this week).

I superficially looked at all tests in question, and I think they all serve a value. The only test that is a little redundant is Lexeme:Lemma can be edited (which is mostly redundant with can be edited multiple times).

This led to the discovery of T282440: ImageLinksDataUpdater::updateParserOutput confuses SHA1 and timestamp when calling ParserOutput::addImage.

The functionality that caused the problems here is covered in ImageLinksDataUpdaterTest (and additionally partly in FullEntityParserOutputGeneratorTest). That currently doesn't test the RepoGroup interaction, thus I will add another test case for that. Once that is done, I consider this fixed.

Probably caused by 83e184670c24470223f593babfa7b5d05bee5cda.

The runs last week failed (and no one noticed in time, see T281267: various weekly and daily dumps run from systemd timers are broken for that). The JSON dumps should appear as usual this week.

Just wanted to note that using a template for all the properties on the page surely wont work (these are just to many), but IMO this is probably still useful for highlighting specific properties and for wikis which have fewer entries then Wikidata.

I've merged https://github.com/wmde/wikibase-release-pipeline/pull/163 (as there were no objections). Putting this back to to do, due to T281472: Remove latest, latest-bundle, latest-base tags from dockerhub.

Fixed: https://github.com/wmde/wikibase-release-pipeline/actions/runs/795896779

Fixed with https://gerrit.wikimedia.org/r/c/683574

I just grepped git diff origin/wmf/1.36.0-wmf.30..origin/wmf/1.37.0-wmf.3 client/ lib/ data-access/ as well and that seems to contain no further relevant changes, so once the three mentioned at the top are addressed we should be good here.

Addressed by https://github.com/wmde/wikibase-release-pipeline/pull/152

https://github.com/wmde/wikibase-release-pipeline/pull/153

Such a check can (only) be done trivially if the job queue that is being pushed into belongs to the same wiki.

In T248984#6089155, @Addshore wrote:

This could happen when an insert to wb_changes fails?

No idea, but that might be the cause. No idea if this is even happening still.

Not sure how much this is worth digging into or even trying to remove?

Not much IMO… but then again, having this around in a broken state could cause trouble later on (eg. if something starts using that field).

How hard do you think it would be to remove? I guess this would only be removal of the id from the data stored in the recentchanges row?

Trivial, but we need to make sure that nothing reads from this again (the ticket is ~1y old by now).

In T277300#6990365, @Addshore wrote:

Seemingly might be done?

Icinga sends alerts, and those would come to me I guess, which is probably not the best outcome :-)

We could use the wikidata contact group for that.

In T44325#6976484, @Addshore wrote:

So assuming we are looking at the immediate time after item creation, and before the sitelinks enter the secondary index, we could add an extra layer of checks for this specific case.
This could for example be storing newly created item sitelinks in memcached for a short period of time (covering the time before they will be in the sql index).
This would then also be checked when new sitelinks are added.

Confirmed fixed now (after running composer update). Updating vendor will be done as part of T274821: Update data-values/* in mediawiki-vendor.

I just +2ed the last backports here, and except for the question posed above and the missing release notes, this is done.

@Samantha_Alipio_WMDE Do we also want 97fb57717b5c858d41df60842475958fef905e1e (fixes undoing changes that turn entities from redirects into normal entities again, thus making them redirects again)?

In T278824#6956622, @hoo wrote:

Addressed in https://github.com/wmde/Number/pull/135.

Addressed in https://github.com/wmde/Number/pull/135.

These bad entries were added in October 2013 (fixed now), probably due to a broken input validator.

In T277817#6931307, @Lydia_Pintscher wrote:

@hoo Is this just for stats (and if so which ones?) or also usage tracking?

In T277662#6922286, @Addshore wrote:

@hoo I think I saw another ticket realting to something odd going on in dump recently?
Is this also related?

I just +2ed all remaining patches. I suggest we go over the ones that are harder to backport with @Samantha_Alipio_WMDE and decide which ones are worthwhile doing.

In T275866#6916597, @toan wrote:

2cfceb27e5023af0782993797524fe6de2c3738f <--- this is only improving the performance of tests, do we care?

The fixed 20210301 JSON dumps (very same content, only the structure was fixed) can now be found on dumps.wikimedia.org (and the old broken ones are in /bad).

The above look good to me. I also just went through git log --no-merges origin/REL1_35..origin/master and found the following that looked interesting:
(in no particular order)

I wrote a small script to fix these dumps (wikibase-json-dump-double-entry-fix) and that is currently running over the invalid dump (on stat1007). Once it is done, and we re-compressed the dump in both gzip and bzip2, we will provide a fixed up 20210301 JSON dump.