Where are the jenkins jobs defined?

Can this task be closed?

Something like that approach seems worth trying.

Is it possible to just update pagetriage_page_tags on page saves (and other relavent POST requests) when there are already master connections? For anything that depends on things updated via the job queue (like backlinks), those would have to be attached such LinksUpdates (which already run in POST/jobs). Why do things have to be updated on page views?

I noticed that regular memcached counts ADD as it does SET (cmd_set). There is no cmd_add. However, mcrouter does seem to expose a cmd_add counter. Perhaps there can be a mcrouter dashboard similar to the Memcache on in Grafana?

Are there any tasks here that remain and are blockers to multi-DC?

Regression from fb51330084b4bde1880c76589e55e7cd87ed0c6d I assume

In T200471#4454643, @Ciencia_Al_Poder wrote:

This may be caused by rMW14ee3f210782 self-merged by @aaron

From https://logstash.wikimedia.org/goto/0b9191830a12ab3d15bce062cdb36a93, this seemed to be better. But we should wait longer.

From a glance, it looks like xtradb cluster is build on Galera (which is something itself to consider in the future). Use of GET_LOCK is tricky there since it would have to use wsrep or have such queries directed to dedicated master (perhaps with some HA in front that doesn't split brain).

Ah, right, I read that ternary backwards, <<$maxTime < PHP_INT_MAX ? PHP_INT_MAX : 1>>.

In T200420#4453134, @Addshore wrote:

Something to note, because the locks are no longer in the DB, we end up selecting the same 15 or so wikis that are locked all of the time.
It could be that the other wikis actually don't have locks:

before using the redis lock manager the status of the lock from the db was also in the select so that locked dbs would not be selected at all.

Normally, it would be odd to let jobs pile up but not execute them, though the multi-DC use case of $wgReadOnly in one of the DCs wasn't considered in T130795. Ideally, jobs enqueued on GET/HEAD wouldn't be a thing...but that's not going away anytime soon.

My first inclination is to try to reduce the refreshCounts() calls.

This was fixed by the 61a7e1acd0af4a5386df03335733accfde179fa1 backport.

Fixed with the 61a7e1acd0af4a5386df03335733accfde179fa1 backport.

In T194403#4415063, @gerritbot wrote:

Change 445110 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] rdbms: fix value of ChronologyProtector::POSITION_COOKIE_TTL

https://gerrit.wikimedia.org/r/445110

Given how low server_failure_limit is, it might help to lower server_retry_timeout from 30s to something < 5s. Consistent hash ejections seem like the most obvious thing that could cause an acknowledged write to be seen as not being there for any of the next 5 seconds.

The configuration change was reverted. It will be fine to re-apply it once 4ad6b70ba132c66e14a706eae240887885946a42 is merged (I thought that 13 day old change landed already).

In T198483#4408326, @Tgr wrote:

I wonder why this (or even the previous issue) does not show up on the edit stash dashboard? This should be significantly affecting stash hit ratios.

In T198483#4393585, @aaron wrote:

Judging from https://performance.wikimedia.org/xenon/svgs/daily/2018-06-26.api.svgz (pre-28th) and https://performance.wikimedia.org/xenon/svgs/daily/2018-07-01.api.svgz , perhaps the prepared parser output is not getting reused between SpamBlacklist and doEditContent() as much as before.

In T198239#4399669, @gerritbot wrote:

Change 440469 merged by jenkins-bot:
[operations/mediawiki-config@master] Make test wikis just write to both nutcracker and mcrouter

https://gerrit.wikimedia.org/r/440469

LCStoreDB should use a separate DB file, as does objectcache (by default, via the installer), if it is to be used at all. Better yet, is that $wgCacheDirectory can be set so that LCStoreCDB can be used.

In T198156#4320080, @Krinkle wrote:

In T198156#4320057, @MarcoAurelio wrote:

@Krinkle Do you think some eval.php magic could do it or even if we do that via that script transaction limits would still apply? Regards.

The method some people use via eval.php would use the same methods as action=delete or deleteBatch.php would, and is also subject to transaction limits. While one could manually make changes in the database, these limits exist for a reason.

At this point, I want to just go ahead and deploy https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/440469/ .

In T198239#4378887, @Joe wrote:

+1 to the overall plan; I'd like to see dates attached to the various steps now, so that we can have a clear schedule.

Judging from https://performance.wikimedia.org/xenon/svgs/daily/2018-06-26.api.svgz (pre-28th) and https://performance.wikimedia.org/xenon/svgs/daily/2018-07-01.api.svgz , perhaps the prepared parser output is not getting reused between SpamBlacklist and doEditContent() as much as before.

Testing around, I see that https://test2.wikipedia.org/wiki/MediaWiki_talk:Msg-override doesn't use the {{msg-override}} message from test.wkipedia.org. Is this task still named appropriately?

In T190260#4304894, @Krinkle wrote:

Still seeing ~ 200 hits per day in Logstash for (DBTransactionSizeError) just within wiki:commonswiki AND method:POST.

As of now, I get (as expected):

In T198350#4321988, @daniel wrote:

The DBPerformance log shows a spike during the time wmf-10 was deployed on group1: https://logstash.wikimedia.org/goto/7c86a7d63a305c220a37a3a49844ef2c. The vast majority of entries are for commonswiki. Here are a few examples:

Sub-optimal transaction on DB(s) [10.64.48.23 (commonswiki) (TRX#26770d)]: 
0	0.000436	query-m: INSERT IGNORE INTO `page` (page_namespace,page_title,page_restrictions,page_is_redirect,page_is_new,page_random,page_touched,page_latest,page_len) VALUES ('X') [TRX#26770d]
1	0.001476	query-m: INSERT INTO `blobs_cluster2N` (blob_text) VALUES ('X')
2	0.000313	query-m: INSERT INTO `text` (old_id,old_text,old_flags) VALUES (NULL,'X') [TRX#26770d]
3	0.000497	query-m: INSERT INTO `comment` (comment_hash,comment_text,comment_data) VALUES ('X',NULL) [TRX#26770d]
4	0.000425	query-m: INSERT INTO `revision` (rev_page,rev_parent_id,rev_text_id,rev_minor_edit,rev_timestamp,rev_deleted,rev_len,rev_sha1,rev_comment,rev_user,rev_user_text,rev_content_model,rev_content_format) VALUES ('X',NULL,NULL) [TRX#26770d]
5	15.847574	query-m: INSERT INTO `revision_comment_temp` (revcomment_rev,revcomment_comment_id) VALUES ('X') [TRX#26770d]

This is from https://commons.wikimedia.org/w/index.php?title=File:Portrait_of_Maria_van_Rijswijk_Dutch_School_Rijksdienst_voor_het_Cultureel_Erfgoed_B670.jpg&action=edit

I'm surprised to see blobs_cluster2N in there - ExternalStore's blob tables are generally on a different DB server, no? I'll check whether this may be the cause of the problem somehow.

I get:

Maybe for certain keys, when the mc server used by nutcracker matches that of the mcrouter one, add() fails since it has to succeed on *all* backends for MultiWriteBagOStuff to return true and the second write would see the first one.

How is $cache defined?

I can't reproduce this in eval.php (after https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/442245/).

In T118893#4309605, @Krinkle wrote:

Looking at https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/441601/

I assume the reason they're not fine for APC is not because of access control (e.g. vary by user or session), but rather that the content pre-filled there might change without a purge strategy?

For the case of message with no page definitions but rather hook-based definitions:
Looking at MWOAuthUIHooks::onMessagesPreLoad, I see that the use of MWOAuthDAOAccessControl would be problematic for APC use, though a process cache could still be used.

I believe that is the script, yes, though I can't seem to figure out how to it to format properly (the histogram bars are the same size).

Fixed in daf0514345f03189187606ba2323794588c79dc9 .

For web requests, the lock timeout should be 5 min:

That said, from mc1019, I see:

In T194403#4278157, @jcrespo wrote:

Please excuse my ignorance, but you are talking redis for sessions, not for the jobqueue (which is, or is close to be, deprecated, right?).

I suggest that cookie set/receive round-tripping should be tested for encoding/truncation issues with @ or # for these apps, as well as letter case changes or such. The above patch simply discards cookie headers for cpPosIndex that are botched.

There are now only ~10/min of these now. I still see no 'redis' channel errors, but I wonder if the random eviction model of redis is at play. redis 3.0 is a bit better at LRU per https://redis.io/topics/lru-cache than our 2.8.

Does this still happen after https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/440031/ ?

Seems to gotten better after https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/440031/ was deployed and likewise for logstash (+"ChronologyProtector::initPositions").

In T95799#4274424, @tstarling wrote:

I don't understand how we can implement the task as described. It's intentional that write-heavy maintenance scripts go at the speed of the slowest slave. If you only wait for a majority then you could have 50% of slaves permanently lagged, potentially by days or weeks.