I assume a mixture of opportunistic (limited row count) purging of rows during existing watchlist row changes along with WHERE clause filtering on SELECT to ignore expired rows would work (e.g. similar to blocks and page protections).

This was likely due to an APC change. Filing a separate task for the 4/20 group 2 regression (which seems out of band for deployments).

Some sort of meeting sounds reasonable.

I assume the slow part would be $ketama_test( 1e5 ). It could probably use 1e4 if there is a speed problem.

In T203786#5249703, @elukey wrote:

After the last round of rolling restarts (happened days ago) I didn't see any TKO event related to mc1* shards. The only ones that I can see now, very sporadically, are related connection timeouts related to codfw proxies (causing very little impact but worth to investigate in a separate task).

I think that we reached the end of this infinite task, and we can close it. @Krinkle or @aaron let me know your thoughts :)

This looks OK again.

This can be done in the Prometheus migration, but it is already available in Hive and turnilo.

What is the status of this after the above patches are applied?

In T219528#5216604, @daniel wrote:

While having an atomic incr() method would certainly be nice, this task doesn't make sense to me for two reasons:

1. The BagOStuff interface doesn't require or even suggest incr() to be atomic, and non-atomic implementations exist (e.g. in RedisBagOStuff). Because of this, there should be no callers that require atomic semantics.
2. RESTBagOStuff is designed to work with any RESTish CRUD service. There's a variety of ways to implement atomic increment on top of such an interface, but all of them require some level of cooperation from the backend, and all of them come with cost in terms of latency or failure modes. It's unclear which mechanism RESTBagOStuff should use, this would have to be left to a subclass.
   
   Overall, BagOStuff is overloaded and under-specified. We should introduce new, more narrow interfaces that provide more guarantees about the semantics of the methods they define. An interface that is designed to fit any key-value-store will lack the guarantees that would make it useful in any but the most trivial use cases.

Note that the above patches should be live.

They preferably would happen in the main DC via POST, but that is not a hard requirement. This is partly why there are WRITE_SYNC and READ_LATEST flags in BagOStuff. A write should work in any DC and will should replicate asynchronously. If the backend supports WRITE_SYNC, then it should make the write best-effort synchronous.

What version (or git hash) of MediaWiki are you currently running and do you still have unresolved problems? I've since made patches like 9df277a4ba3b1dc86e11f469d742c3781141da70 and some mediawiki.org documentation changes.

In T215850#5050967, @Markdoliner wrote:

Thanks for working on this, @aaron. I'm not familiar with the Mediawiki source code but your change seems reasonable to me. Assuming I understand this correctly, could I suggest updating https://www.mediawiki.org/wiki/Manual:$wgDBservers and/or https://www.mediawiki.org/wiki/Manual:$wgDBname and mentioning that $wgDBname should be set even if $wgDBservers is used? And that they should have the same value? I think that would have helped me. It looks like I probably have access to update those pages, but I think you'd probably do a better job since you're familiar with the rationale behind the requirement.

In T218511#5185892, @IKhitron wrote:

Group 1 deployed, the problem continues.

In T223310#5185185, @Addshore wrote:

For things added my wmf.3, it looks like there were indeed some chronology protector changes https://github.com/wikimedia/mediawiki/compare/wmf/1.34.0-wmf.1...wmf/1.34.0-wmf.3

https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/505340/
https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/504815/
https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/505864/

In T218511#5170113, @Jonesey95 wrote:

Thanks for the patch. Things are much better today, but still not entirely fixed.

On my watchlist, for example, I haven't looked at any pages for about two hours. One page has had 17 changes today, including three in the past two hours. It is marked in my Watchlist as completely read, and is not bold, with the following notation:

(17 changes | history)

It should instead be bold and should say:

(14 changes | 3 since last visit | history)

Most other pages are displayed correctly (for me). I know that makes it frustrating to track down the cause of the problem. I hope that this help you track down the cause. Post here if you need further information.

It's the new use of DeferredUpdates::attemptUpdate/addCallableUpdate in that change instead of plain doUpdate() calls that gives them outer scope.

Confirmed via xhgui on page that is known to log "MediaWiki\Revision\RenderedRevision::outputVariesOnRevisionMetaData: Prepared output has vary-revision"

Yes, even Oracle can go under /libs once the foreign key mess is dealt with.

After struggling to reproduce anything, I saw T222431 mentioned the old non-default watchlist (that doesn't use JS). I can reproduce some problems when setting that preference on.

In T221577#5154583, @Jdforrester-WMF wrote:

Is it worth back-porting that patch? Or is it too risky?

https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/497537/ might fix that (by giving each update it's own transaction round)...but that patch still has some CI issues.

In T211721#5009838, @aaron wrote:

The SET metric for redis is very slow, so wouldn't use 10x that figure.

CentralAuth (CA) logins involves HTTP POST, redirect => HTTP GET, redirect HTTP GET. In terms of session store writes there is:
a) POST localwiki: session backend set + CA start-token set
b) GET centralwiki: session backend set + CA start-token delete + CA completion-token set
c) GET localwiki: session backend set + CA completion-token delete

The original cause of this task was deploy related and was probably fixed in d256b472f73956ee8e2503e0254a1107baa1f00a . The later timing increase is probably from something else on-wiki.

Did you try sqlite LCStore (with journal_mode = WAL and synchronous = NORMAL? like the installer uses)?

Right, so this isn't really about LazyImageTransform, but rather than is just one thing used the parsed DOM object. I guess the summary through me off.

What other things are there other than <img> tags? Maybe you could start moving off those things that can be changed to avoid the DOM and document the rest. As I described above, I don't mind hybrid regex/DomElement logic. You could also have a simple state machine that reads the HTML body string, looking for certain tags and when the corresponding terminating tag is found, recording the positions, and using those to get chunk of strings to parse as DOMElements. That would require some assumptions about HTML being balanced/correct (I assume our Parser should handle that already with Tidy/Remex). I'm thinking of something simple kind of like Language::truncateHtml().