In T211661#4928637, @ori wrote:

It seems that Swift has built-in support for object expiration, which can be requested by setting a header (either X-Delete-After or X-Delete-At).
It also looks like the expiry can be re-set, either by first removing it via X-Remove-Delete-At, and then setting it anew, or by updating the metadata in-place.
Is there a reason why these mechanisms are not under consideration?

A straw-man proposal:

• On thumbnail creation, set X-Delete-After: 2592000 (one month).
• Each time a thumbnail is retrieved, there's a 0.01% chance we also used the opportunity to reset the expiry to one month.

In T146257#4959069, @Legoktm wrote:

Nope. What do you want to call it? ObjectCache or BagOStuff?

So is anything other than EventRelayer blocking this?

I'd prefer we didn't take preferences and bits of data like this onto sessions (we used to many years ago and that was lame), especially given the cross-DC latency. I worry people will go overboard with such features that cause slow writes.

I think Timo backported the change, so it should be live:

In T151903#4955370, @EBernhardson wrote:

In T151903#4955106, @aaron wrote:

In T151903#4946443, @Joe wrote:

Wouldn't it be feasible to have the search request generate a simple job that saves that preference asynchronously?

The jobqueue is already capable of listening to the events in both DCs, and execute them in the primary one only, so most things that can happen post-send can probably be deferred to the jobqueue.

I generic preference setting job would be useful for this and similar cases.

A delay in making the new preference available might be unintuitive for users. In particular if they perform another search a few seconds after the one where they saved they will expect the new setting to load. I'm not entirely clear on things, but i think ChronologyProtector ensures this with the current setup, but if the preference pushed into a job there would be no guarantee? It could also be that simple UI affordances to keep everything selected will negate any concern about reading the preference.

$wgDBname and $wgDBprefix are used by methods like wfWikiId() and "DB domain" methods from WikiMap. The former is a very old method. They are assumed to contain the DB name/prefix of the current wiki (wiki farms will set this depending on the vhost or URL or something).

In T215611#4950057, @bd808 wrote:

In T151903#4946443, @Joe wrote:

Wouldn't it be feasible to have the search request generate a simple job that saves that preference asynchronously?

The jobqueue is already capable of listening to the events in both DCs, and execute them in the primary one only, so most things that can happen post-send can probably be deferred to the jobqueue.

In T215611#4951753, @jcrespo wrote:

^I don't have enough context for this patch, is configuration for regular servers setup to nor produce DEBUG lines outside? Otherwise, I don't understand how the patch can produce the desired output.

In T203786#4938798, @elukey wrote:

@aaron Any more ideas about what could be tuned to make this work? I am also wondering if https://gerrit.wikimedia.org/r/487622 should work out of the box or if a change is in the translation extension is needed to make the lockTSE logic working?

I updated the "Database transactions" and "Performance guidelines" pages, which I'd hope most people that work on MediaWiki will encounter when looking for generic guidelines. Things can be better linked, though that's a broader matter...

I recall something like this when testing around with the old python-memcached-relay daemon expirement (before we decided on mcrouter instead). That used non-blocking checks and conditional sleep (polling), but since there is only one server here, the message() timeout could work. Something like https://github.com/andymccurdy/redis-py/issues/631 with try/catch for redis.TimeoutError and resubscribe logic should be doable.

Also note that even the old LRU algorithm avoided bumping things in a slab more than once a minute (https://memcached.org/blog/modern-lru/), so that perhaps make it more likely that some other keys are flooding out the stab, since being "hot" does not mean being at the top of the list so much (if there are other things like prepared edit parse blobs and parsoid serialization blobs coming in on every edit).

In T203786#4916848, @elukey wrote:

@aaron, @Nikerabbit - if you guys have time during the next days can we chat about the next steps for this task?

Is this still reproducible in master?

Things needed here:

• Use only mcrouter in deployment-prep (no multiwrite) from MW
• Remove puppet code for deployment-prep
• Install mcrouter on the memached servers used by labswiki
• Make MW use mcrouter on labswiki
• Remove "memcached-pecl" cache entry from config
• Remove labswiki nutcracker code from puppet

I don't think ChronologyProtector is involved.

From a perspective of layered architecture and separation of concern, I'm not sure I like the idea of a MediaWiki script. But some script that reads from etcd and does the updates to the table seems reasonable.

To be useful, the tables.sql and other bits need to also handle idempotence or have some script parameter to skip them though...

I see

Error: 1050 Table 'blobs_cluster24' already exists (10.64.32.184)

Did it finish?

Another option, is using the existing 'ChronologyClientId' header that MediaWiki supports for acting on behalf of a client (e.g. no need to forward the agent/IP).

In T203786#4877223, @elukey wrote:

@Nikerabbit looking forward to see it deployed, thanks to both you and Aaron for this work.

One question - would it be possible, as long term view, to see this key being broken down into smaller pieces to avoid a never ending increase? I am a bit worried that we are now resolving this issue but leaving another one (a big key that can generate a lot of traffic) aside.. It is fine if the answer is No, I am just wondering what's possible for the future :)

In T213398#4872087, @Krinkle wrote:

In T202715#4865823, @jcrespo wrote:

I assume the 6 errors correlate to the 6 edits in the same minute given the update happens post-send (it is already biased towards later), As such, it would seem that 6 out of 6 timed out.

From that I would say the locking of the row happens previously, on the same piece of code for another user or more likely at a different piece of code that locks the same rows, and just happens that the editing of the edit count is frequent enough to be only the "sufferer".

We could setup some monitoring so that we identify what is locking the rows when that happens for debugging.

This was probably fixed alongside T207979.

I don't recall an overriding reason it has to be there. Seems reasonable to experiment with changing it as long as the common desktop case (fuller window) looks the same.

That sounds right.

I see EntityRevisionCache and CacheAwarePropertyInfoStore seems to use set() on invalidation. Also, EntityRevisionCache
and CachingEntityRevisionLookup, and PopulateInterwiki seems to call delete() on a non-WAN cache instance.

The current callers don't assume the level of durability as with mysql, just that the data will likely not be randomly removed (e.g. high eviction rate, power outage, network blips). The WAN cache callers can handle a fair amount of that on the other hand.

In T211721#4822917, @Joe wrote:

To add to what @Tgr found, we have to search for usage of MediaWikiServices::getInstance()->getMainObjectStash(); as that's what that method uses under the hood.

There is quite a few uses of it here:

https://codesearch.wmflabs.org/search/?q=getMainObjectStash&i=nope&files=&repos=

I would propose that whatever is not extremely valuable (so anything that's not in the session store) should be stored on memcached via mcrouter, so that we can have multi-dc writes and broadcast both writes and evictions if needed. We can't realistically support sub-millisecond latencies in the service we're desinging and most uses of this cache are for caching purposes indeed.

@aaron do you think using mcrouter for MainObjectStash is feasible?

We need persistence and replication. The plan is to use the same store as session for the rest of the object stash usage (probably Cassandra). Flags like WRITE_SYNC might be used in a few callers, and should use appropriate backend requests (e.g. QUOROM_* settings in Cassandra). The callers of the main object stash all need persistence and replication though (callers have already been migrated to stash vs WAN cache and such).

I'm not sure why the recache() calls would cause many CAS commands though. The only threads doing the regeneration (and CAS) would be those of requests doing updates...which should not be that frequent.

In T203786#4807458, @aaron wrote:

In T203786#4805938, @elukey wrote:

Today another mediawiki alert from ~12:24 to ~12:27 UTC. @Nikerabbit, @aaron - do you think that we can narrow down specific events (beside TTL expiring that may cause this?

Actually, from TranslatePostInitGroups, I only see global variable and config file timestamp dependencies, so those are not actually user controlled purges. We don't use TranslateSVG, which calls delete().

In T210992#4793399, @Marostegui wrote:

@Krinkle can you confirm whether those are the two reverts we have to do?
I have been looking around and I haven't found anything else to revert
Thank you!

In T203786#4805938, @elukey wrote:

Today another mediawiki alert from ~12:24 to ~12:27 UTC. @Nikerabbit, @aaron - do you think that we can narrow down specific events (beside TTL expiring that may cause this?

To clarify, the $useMutex logic in WAN cache never triggers due to minAsOf=INF, resulting in stampedes when someone invalidates the cache. Instead, this should be treated like a regular TTL expiration and have one thread at a time doing regeneration.

In T203786#4791225, @Nikerabbit wrote:

Is the value so big that every time it is SET it causes a TKO? Besides the 24h TTL, the cache value is updated when there are changes to the underlying data. In other words: when people create or remove translatable pages, aggregate message groups or other similar stuff.

@Gilles: Comcast only has cable infrastructure in terms what the ISP provides itself. For customers with cable, they can also get XFinity Mobile (https://www.tomsguide.com/us/xfinity-mobile-faq,news-25223.html) . That's basically just a bunch of Wi-Fi hotspots build off of Verizon. I don't know how many people are using that and it seems new-ish. Also, the latency figures are quite low, which makes me doubt that it is XFinity Mobile and more likely regular wireless/xfinity.

It looks sane, though I wonder why Comcast is so high in usage for mobile? Is that mostly from touchpad devices instead of smartphones?

In T196378#4550382, @jcrespo wrote:

It definitely seems like something worth doing. Having the potential for high use cache keys becoming unusable for undefined periods of time is too much of a stability concern.

In T157651#3018942, @Tgr wrote:

Indeed. The updater calls the LoadExtensionSchemaUpdates hook from the constructor (bleh) and Echo and SecurePoll use dropTable/modifyField (which are executed immediately) instead of dropExtensionTable/modifyExtensionField (which would just push an entry to the update list).

This is a very, very ugly accident waiting to happen. @aaron any preference how to prevent it? We could make getSchemaVars static, or split out the extension loading part from the constructor, or add some flag that prevents calling non-extension methods on the updater. Maybe even a unit test which calls the hook with a fake updater which fails the test if any non-extension methods are called on it.

Since CategoryMembershipChangeJob runs via the job queue, wouldn't that have little effect on save timing itself? I guess it wouldn't hurt to optimize.

In T203786#4730954, @elukey wrote:

In T203786#4729548, @aaron wrote:

Keys are set by add/cas normally, so it seems like some key that takes a long time to regenerate might have expired (there are two data points at the elevated value over more than just a few seconds) or a class of many keys expired. The other possibility is some sudden change in access patterns for keys, which seems less likely, especially the more periodic this is.

Checking keys via tcpdump is not super easy, so I tried to order the packets by size inspecting the keys with bigger size and lower TTL. I keep seeing this pattern of SETs for metawiki:translate-groups (TTL 2 hours) followed by gets during the timeframe in which timeouts happen, that given the size (~380k) could surely aggravate the bandwidth problem that we are seeing. Would it be possible to increase this TTL to say a 24h to see if anything changes? Or possibly migrate the code to something more like gadget-definition, namely setting the new key only if really needed and not every two hours by default. It would help removing a (big) variable from the problem..

To answer the point brought up by @Nikerabbit about the code not changed in ages - I suspect that in the past nutcracker might have masked problems like this one (see T208934), and that the change to mcrouter caused more errors raised due to bandwidth/latency variations. I am probably not right about the metawiki:translate-groups key as culprit, but I'd just need to see if removing a big key makes any difference in what we see in the graphs, of course if this is not going to affect users in any way.

wl_notificationtimestamp is not meant to store the time the article was watched but the last revision the user saw on the page (NULL if they saw the latest revision). This would require a new column. Ideally, if watchlist sizes were limited, this woudn't need an index, but they are not.

Keys are set by add/cas normally, so it seems like some key that takes a long time to regenerate might have expired (there are two data points at the elevated value over more than just a few seconds) or a class of many keys expired. The other possibility is some sudden change in access patterns for keys, which seems less likely, especially the more periodic this is.

Fixed in bf30fcb71427d673f7c83a067b3241040d3470b6. Rollback is used instead and uses $ignoreErrors so as not to trigger the exception in reportQueryError().

Cleaned up in 633eb437a3b808518469c6eaf4e86a436941d837

In T194299#4714630, @daniel wrote:

In T194299#4714614, @aaron wrote:

openConnection is badly named and still reuses connections. You'd probably want getConnection with CONN_TRX_AUTO

I hate this hack. This may *still* re-use connections, if anything else used CONN_TRX_AUTO. We should have CONN_NEW.

openConnection is badly named and still reuses connections. You'd probably want getConnection with CONN_TRX_AUTO

What about our use of register_postsend_function? Is there anything equivalant?

In T207223#4703802, @Alexia wrote:

The recommend fix does work for our extension. I apparently had configured Echo properly in the past so it was working properly. For AbuseFilter I had to patch it to use the same pattern since it only specifies the database and not the cluster to use.

In T207809#4701489, @Krinkle wrote:

@aaron The fix LGTM, but do we know why this started happening? I'd be nice to know what commit or task prompted it so that we can learn why it wasn't prevented by our tests and/or Jenkins.

In theory, this kind of warning can be triggered in tests and would be captured by Jenkins in a way that fail the build. That definitely worked at some point.

Closing, per " The Error Occurs if the memcache is too slow".

This will be better with a3d6c1411dad3e057b if there are many message pages that exists for extension use.

4b1db1190bb8f2a115c6a81a5ee487b7d18cd303 seems more likely.

Note that git master (19dd28798163) installs fine with postgres, which has the same DB domain patches as 1.32.

It looks like the errors come from some tool (JS?) that fires a bunch of API requests from a Special:Search tab to edit numerous pages in parallel. Each burst always for a certain user ID with a single referrer URL.

Does this really need to call commitAndWaitForReplication() when there is only one batch? Is it ever called thousands of times in a row?

In T202715#4688919, @jcrespo wrote:

COMMIT takes a few seconds

It is unlikely for a commit to take a lot, as in, the actual physical changes that happen then- at commit time there is only a metadata change + flush to disk cache, which should be very fast. Look for contention somewhere as a first option when you see commit taking a long time (e.g. large transactions blocking each other).

In the getMasterDatabase() method posted above, I noticed that the database domain (e.g. DB/schema/prefix) is missing from getConnection(). Instead that should be: