I don't think so.

Does this still occur?

Per my comment above, this is the expected behavior.

It's an optional table, not installed by update.php.

They were obsoleted by flaggedrevs_statistics.

The remaining vary-revision instances are basic self-transclusions (https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/526157/ should handle those).

Is https://phabricator.wikimedia.org/T212881#5195101 the error that still happens or is it the read-only one too?

Jobs are fine...though this case is complicated since people want their "latest views" to be immediately reflected...so it would have to do something like WatchedItemStore.

How much of this is unique from T205936 ?

I wonder if this is fixed in https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/519565/

The logs for doSelectDomain() look quite for the last 7 days.

959daa2ca44c039e72c8a9a5199d4c74dd05caba added the << $status->value = [ 'warnings' => $upload->checkWarnings() ]; >> line. It seems like checkWarnings() has all kinds of File objects inside of it potentially. Some callback could easily slip in given that.

ObjectCache always mentioned getMainStashInstance() as "Ephemeral global storage". It was just supposed to *try harder* to be persistent than memcached (rdb snapshots, expectation that stuff can *probably* still be there a week later or so). The existence of redis evictions and consistent re-hashing on host failure making data disappear or go stale was well known at the time it was picked as the original "stash".

In T228465#5352272, @tstarling wrote:

Fixed and confirmed

[0450][www-data@labweb1001:/]$ time /usr/local/bin/mwscript extensions/TorBlock/maintenance/loadExitNodes.php --wiki=labswiki --force
Successfully loaded 1206 exit nodes.
real	0m1.958s
user	0m0.572s
sys	0m0.212s

I guess the cron jobs should be removed now, is there a bug for that?

JobQueueException should be thrown from push(), with nothing catching it other than MWExceptionHandler or site-specific caller. Typically, push() should be used pre-send, before preOutputCommit, so everything would just rollback anyway. Jobs pushed after than are enqueued during DeferrableUpdates (directly or indirectly via lazyPush()); in that case, DeferredUpdates should (already) catch any exceptions (not just job queue ones) and rollback on an update-by-update bases. The exceptions are logged in the DeferredUpdates channel (previously the Exception channel).

Also, the timeout exceptions themselves where redis, not LBFactory. The later seemed to just have errors related to the improper shutdown.

In T225103#5348415, @Jdforrester-WMF wrote:

Is this still a train blocker?

Dropping the field doesn't make sense, but dropping the whole table does. We do not use that class in production (and it is optional within MW core).

The redis bug is at T228303

The timeouts correspond with the redis problems:

The timeout aspect seems strange. The huge "idle" time increase at https://grafana.wikimedia.org/d/000000273/mysql sounds like the PageEditStash::parseAndCache() has an infinite timeout instead of 0 seconds (bug, it should be 0 as in non-blocking) and the parsing may have been slowed down for some reason, making more threads wait on the lock. Maybe the concurrent nutcracker issues were also affecting mcrouter (since the same hosts are used). Could also be something adding memcached write load: https://grafana.wikimedia.org/d/000000316/memcache?orgId=1&from=1563458818482&to=1563464680644 looks a little unusual, though not unlike the result of key version changes that happen from release to release (including the slow return to normal set() rate).

OK, replication for SET/DELETE seems fine on mw1261/mw2224 for me and the STORED/NOT_STORED and FOUND/NOT_FOUND replies are what I expect when using (no prefix, /otherdc/mw-wan, and /thisdc/mwwan).

Err, more PEBCAK . I put the * in the wrong spot...

So, I've noticed that on mw1261/mw2224 as *well* as plain old mwmaint1002,mwmaint2001, that broadcasting keys doesn't seem to work, e.g.:

I guess it can go on our backlog.

Is there a codfw host with the patch applied?

In terms of what MediaWiki actually queries, you have the following cases from both eqiad and codfw:

a) Local getWithSetCallback() requests for (regular) value keys:
"get WANCache:v:elukey-test"
"add WANCache:v:elukey-test"
"cas WANCache:v:elukey-test"

The relevant getWithSetCallback() call uses pcTTL, so there still shouldn't be many of these queries. Unless a large number of distinct connections were acquired. Not just that, but connections to different load balancer clusters.

Is $wgMainCacheType set to CACHE_NONE ?

In T214275#5328625, @elukey wrote:

In T214275#5313635, @Andrew wrote:

In T214275#5307951, @elukey wrote:

The latter should be doable, but the former seems a bit more complicated. Is there any plan to deprecate the labswiki infra and fold it into the appserver layer?

There definitely is such a plan, although it will be quite a while (maybe end of this FY?) before we're able to move forward.

Thanks! I think this is totally fine, there is not real rush to move everything away now from nutcracker.

In T214275#5313719, @jijiki wrote:

@elukey I can do thumbor, not sure when yet though. I opened T221081 a while a go for it

Thanks a lot :)
@aaron what do you think? Would it be ok in your opinion to close this task?

For generic key testing, there is always:

Not seeing this in the logs lately.

I think (1) is more useful and fills a needed gap of writes on GET/HEAD. What I've been doing in T227376 is trying to move things off the Stash that can easily enough use some other store. This narrows down the "problem space".

I strongly prefer 0-based.

In T184529#5308021, @daniel wrote:

In T184529#3888432, @aaron wrote:

I see wiki IDs as a type of "domain ID" that just uses two ASCII components, (dbname,prefix), neither using slashes to avoid the ugliness of using things like "mysite?hnewswiki-en" have to appear on config or in "table_wiki" DB fields.

I think using strings for this at all is a big problem. And encoding any "real" information into these strings makes it even worse. Wiki IDs should be totally opaque identifiers, and we should have a class to model them. @Tgr and I discussed this at the hackathon, but it seems we didn't write it down. I made a ticket now, see T227305: Define a WikiID class for uniquely identifying wikis.

In T35409#5306122, @ArielGlenn wrote:

Isn't CDB a thing of the past? I.e. can't we close this now?

Can this be closed now?

The current code will never trigger this path if $wgMiserMode is set, which it is on production.

Is this still a problem?

In T221987#5257554, @EvanProdromou wrote:

So, one way we could do this is with siege or something similar, hitting a Web page or API endpoint with a session cookie set. We could do a baseline comparison against production (?) and then get similar data for a staging service we set up for T222099.
@aaron we pulled you in for performance discussions before. Do you have any tools that you prefer or recommend for this kind of evaluation?

Not sure what to do with this. It seems like some kind of connectivity problem, not just the server being read-only.

In T225103#5271836, @Krinkle wrote:

Seems related: rMW143333b172cd: rdbms: do not close the connection in LoadBalancerSingle::__destruct / https://gerrit.wikimedia.org/r/517350.
Not exactly the same though (commit is LoadBalancer, this bug is LBFactory), but might need a similar change?

I assume a mixture of opportunistic (limited row count) purging of rows during existing watchlist row changes along with WHERE clause filtering on SELECT to ignore expired rows would work (e.g. similar to blocks and page protections).

This was likely due to an APC change. Filing a separate task for the 4/20 group 2 regression (which seems out of band for deployments).

Some sort of meeting sounds reasonable.