Change 298554 merged by jenkins-bot:
Troubleshoot why Echo is still triggering CA failures

https://gerrit.wikimedia.org/r/298554

Mentioned in SAL [2016-07-12T19:30:11Z] <mattflaschen@tin> Synchronized php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php: T119736: T140144: Troubleshoot why Echo is still triggering CA failures (duration: 00m 39s)

The Echo fix didn't work as intended, so I put up a troubleshooting patch. Please follow T140144: Echo triggering CentralAuth "Can only obtain a centralauthtoken when using CentralAuth sessions" error for the Echo-specific part going forward.

In T119736#2451232, @MZMcBride wrote:

Reading through this task, we've failed in a number of unacceptable ways here. Once the immediate issue is resolved and we're no longer throwing thousands of exceptions per day, this task should have a full postmortem to ensure that we identify and learn from what went wrong here. I'll file a separate task about doing this.

Agreed. I've set a meeting for 20-July-2016 at 11:30-12:30 SF time (subject to change, but wanted to get something on people's calendars before their schedules fill up). Anyone who wants to join, please email me directly well ahead of time so I can add you to the Hangout.

In T119736#2449636, @bd808 wrote:

In T119736#2445005, @JJMC89 wrote:

Special:CentralAuth for Acastrohowell gives Fatal exception of type "Exception" after getting To avoid creating high replication lag, this transaction was aborted because the write duration (6.1167709827423) exceeded the 5 seconds limit. If you are changing many items at once, try doing multiple smaller operations instead. when attempting to create at enwiki.

Error was "Could not find local user data for Acastrohowell@enwiki" which interestingly was the "new" wiki for the account. Manually deleted the dangling reference and https://en.wikipedia.org/w/index.php?title=Special%3ACentralAuth&target=Acastrohowell works now.

Not sure if this will be helpful to you or not, but all of the exceptions that I have reported in this task were new accounts created via Special:CreateAccount at enwiki. @ori provided a trace for the first one that I encountered and reported on IRC which did not point to Echo.

In T119736#2458586, @JJMC89 wrote:

Not sure if this will be helpful to you or not, but all of the exceptions that I have reported in this task were new accounts created via Special:CreateAccount at enwiki. @ori provided a trace for the first one that I encountered and reported on IRC which did not point to Echo.

The real cause of the problem for Optimus24 was very likely

2016-06-29 04:26:37 [V3NN9ApAIDEAAAuRbAgAAAAL] mw1179 enwiki 1.28.0-wmf.7 exception ERROR: [V3NN9ApAIDEAAAuRbAgAAAAL] /w/index.php?title=Special:CreateAccount&returnto=[REDACTED]   DBTransactionError from line 235 of /srv/mediawiki/php-1.28.0-wmf.7/includes/db/loadbalancer/LBFactory.php: To avoid creating high replication lag, this transaction was aborted because the write duration (6.002995967865) exceeded the 5 seconds limit.
If you are changing many items at once, try doing multiple smaller operations instead. {"exception_id":"V3NN9ApAIDEAAAuRbAgAAAAL"} 
[Exception DBTransactionError] (/srv/mediawiki/php-1.28.0-wmf.7/includes/db/loadbalancer/LBFactory.php:235) To avoid creating high replication lag, this transaction was aborted because the write duration (6.002995967865) exceeded the 5 seconds limit.
If you are changing many items at once, try doing multiple smaller operations instead.
  #0 /srv/mediawiki/php-1.28.0-wmf.7/includes/db/loadbalancer/LoadBalancer.php(1310): Closure$LBFactory::commitMasterChanges(DatabaseMysqli)
  #1 /srv/mediawiki/php-1.28.0-wmf.7/includes/db/loadbalancer/LBFactory.php(237): LoadBalancer->forEachOpenConnection(Closure$LBFactory::commitMasterChanges;240668706)
  #2 /srv/mediawiki/php-1.28.0-wmf.7/includes/db/loadbalancer/LBFactoryMulti.php(411): Closure$LBFactory::commitMasterChanges#2(LoadBalancer)
  #3 /srv/mediawiki/php-1.28.0-wmf.7/includes/db/loadbalancer/LBFactory.php(238): LBFactoryMulti->forEachLB(Closure$LBFactory::commitMasterChanges#2;240668706)
  #4 /srv/mediawiki/php-1.28.0-wmf.7/includes/MediaWiki.php(562): LBFactory->commitMasterChanges(string, array)
  #5 /srv/mediawiki/php-1.28.0-wmf.7/includes/MediaWiki.php(540): MediaWiki::preOutputCommit(RequestContext)
  #6 /srv/mediawiki/php-1.28.0-wmf.7/includes/MediaWiki.php(753): MediaWiki->doPreOutputCommit()
  #7 /srv/mediawiki/php-1.28.0-wmf.7/includes/MediaWiki.php(520): MediaWiki->main()
  #8 /srv/mediawiki/php-1.28.0-wmf.7/index.php(43): MediaWiki->run()
  #9 /srv/mediawiki/w/index.php(3): include(string)
  #10 {main}

The immediate culprit for this one is @aaron's $wgMaxUserDBWriteDuration arbitrarily rolling back the creation of the user because the request took more than 5 seconds to process for some unknown reason. Mintudazel, Mtrmananes, and Agusteknik2016 that you reported all look like the same thing.

For anyone else with access to the logs who wants to try to track down causes, this is the procedure:

1. Determine the wiki with the problem, i.e. the "{wiki}" in "Could not find local user data for {Username}@{wiki}".
2. Find the request ID for the creation of the user on the wiki. Generally, this will be in CentralAuth.log as "Attaching local user {Username}@{wiki}"
3. Look for other log messages, particularly exception logs, for that request ID.
   1. If the exception found would have caused a database rollback, you've probably found the culprit.
   2. If the exception was before yesterday and the creation in step 2 was "by 'login'" chances are pretty good that https://gerrit.wikimedia.org/r/298531 fixed it already. Look for more recent examples.

In T119736#2458586, @JJMC89 wrote:

Not sure if this will be helpful to you or not, but all of the exceptions that I have reported in this task were new accounts created via Special:CreateAccount at enwiki. @ori provided a trace for the first one that I encountered and reported on IRC which did not point to Echo.

The stack traces for the end user visible "Could not find local user data for ..." exceptions actually aren't very interesting. They will all lead back to one of two or three methods in CentralAuthUser where the data from the localuser table is checked against the attached wiki user table.

The interesting traces are the ones that caused the transaction creating the user row to be rolled back after the CentralAuth association was made initially. These however are pretty tricky to find and actually correlate with the creation of the localuser record. I mostly stumbled upon the correlation mentioned in T119736#2451220 that lead to @Catrope and @Mattflaschen-WMF's work in T140144.

@Anomie's patch at rMW83c66caa0831: AuthManager: Commit transaction after auto-creating a user should help quite a bit with narrowing the types of problems that can cause the local wiki's User record to be rolled back. It forces a transaction commit just before the end of AuthManager::autoCreateUser. The window for a rollback error is now limited to unhandled exceptions and fatal errors that happen via:

• autoCreatedAccount calls on primary and secondary authentication providers
• AuthPluginAutoCreate and LocalUserCreated hook handlers
• User::saveSettings
• User::addWatch
• ManualLogEntry::insert

The Flow related bug was causing an unhandled exception during User::saveSettings via a UserSaveSettings hook call.

In T119736#2458913, @bd808 wrote:

The window for a rollback error is now limited to unhandled exceptions and fatal errors that happen via:

Note this is the window for the error when it happens during auto-creation due to CentralAuth sessions or the like. There's also auto-creation when logging in (which the patch might help with too) and normal creations.

In T119736#2458913, @bd808 wrote:

The Flow related bug was causing an unhandled exception during User::saveSettings via a UserSaveSettings hook call.

I think you mean Echo.

In T119736#2458851, @Anomie wrote:

The immediate culprit for this one is @aaron's $wgMaxUserDBWriteDuration arbitrarily rolling back the creation of the user because the request took more than 5 seconds to process for some unknown reason.

This appears to still be happening, BTW. Looking at timestamps on log messages, whatever is being slow is somewhere in this code. Unfortunately, that's not hugely helpful, since it includes a lot of stuff:

• A 'UserSaveOptions' hook call
• A 'LocalUserCreated' hook call
• Another 'UserSaveOptions' hook call
• A 'UserSaveSettings' hook call
• Publishing a log entry (which unfortunately gets rolled back too)
• Various SecondaryAuthenticationProviders' calls, including CampaignsSecondaryAuthenticationProvider, MediaWiki\Auth\ResetPasswordSecondaryAuthenticationProvider, and MediaWiki\Auth\EmailNotificationSecondaryAuthenticationProvider.

Change 298817 had a related patch set uploaded (by Anomie):
Add timing data logging for T119736

https://gerrit.wikimedia.org/r/298817

Overview of logs for the Echo part: P3422

Keep in mind the dates of the files are the date of log rotation, not necessarily date of occurrence.

Change 298820 had a related patch set uploaded (by Anomie):
Add timing data logging for T119736

https://gerrit.wikimedia.org/r/298820

Change 298821 had a related patch set uploaded (by Anomie):
Add timing data logging for T119736

https://gerrit.wikimedia.org/r/298821

Change 298820 merged by jenkins-bot:
Add timing data logging for T119736

https://gerrit.wikimedia.org/r/298820

Change 298821 merged by jenkins-bot:
Add timing data logging for T119736

https://gerrit.wikimedia.org/r/298821

Mentioned in SAL [2016-07-13T19:23:03Z] <anomie@tin> Synchronized php-1.28.0-wmf.8/includes/auth/AuthManager.php: Add timing data logging for T119736 (duration: 00m 27s)

Mentioned in SAL [2016-07-13T19:23:56Z] <anomie@tin> Synchronized php-1.28.0-wmf.10/includes/auth/AuthManager.php: Add timing data logging for T119736 (duration: 00m 28s)

Change 298823 had a related patch set uploaded (by Aaron Schulz):
[WIP] Revert "Wrap CentralAuthUser::attach in an explicit transaction"

https://gerrit.wikimedia.org/r/298823

In T119736#2458851, @Anomie wrote:

The immediate culprit for this one is @aaron's $wgMaxUserDBWriteDuration arbitrarily rolling back the creation of the user because the request took more than 5 seconds to process for some unknown reason. Mintudazel, Mtrmananes, and Agusteknik2016 that you reported all look like the same thing.

Did this timeout change recently, or is it just that something recently started exceeding the 5 second limit?

In T119736#2459311, @Stashbot wrote:

Mentioned in SAL [2016-07-13T19:23:03Z] <anomie@tin> Synchronized php-1.28.0-wmf.8/includes/auth/AuthManager.php: Add timing data logging for T119736 (duration: 00m 27s)

Results: There were four hits so far, V4abdApAEKsAAHuBLOwAAACB, V4afkgpAMEwAAAXlT5kAAAAS, V4agfgpAADsAASl1QIMAAAAT, and V4abdApAEKsAAHuBLOwAAACB . For all four, it's $user->addToDatabase() that took a long time.

In T119736#2458851, @Anomie wrote:

For anyone else with access to the logs who wants to try to track down causes, this is the procedure:

I took a stab at this for a few, but I wasn't able to find any of the exceptions. Maybe I'm doing something wrong:

At least Clauricekoi doesn't seem to have been a failure. Instead the CentralAuthCreateLocalAccountJob on metawiki started at 21:04:56, the errors showed up from 21:04:57 to 21:05:42, and the CentralAuthCreateLocalAccountJob completed on 21:06:00. It seems like something in this block of code (based on the account attachment in CentralAuthPrimaryAuthenticationProvider::autoCreatedAccount() being logged at 21:04:56 and the on-wiki log message at 21:06) took a strangely long amount of time; this might be related to T121161.

@bd808: What would https://gerrit.wikimedia.org/r/#/c/297936/ do in this situation?

In T119736#2460239, @Anomie wrote:

At least Clauricekoi doesn't seem to have been a failure. Instead the CentralAuthCreateLocalAccountJob on metawiki started at 21:04:56, the errors showed up from 21:04:57 to 21:05:42, and the CentralAuthCreateLocalAccountJob completed on 21:06:00. It seems like something in this block of code (based on the account attachment in CentralAuthPrimaryAuthenticationProvider::autoCreatedAccount() being logged at 21:04:56 and the on-wiki log message at 21:06) took a strangely long amount of time; this might be related to T121161.

In this case, I don't think it's related to T121161: CentralAuthCreateLocalAccountJob failing on meta due to Echo deadlocks.

I looked at a couple possibilities:

• Direct usage by Echo of one of those hooks. It doesn't listen to AuthPluginAutoCreate, and listens to LocalUserCreated, but does nothing if it's auto-created. There is also an Echo UserSaveSettings hook, but it just adjusts the user's count, it doesn't send a notification.
• I grepped through on GitHub for others that could directly or indirectly trigger a notification on auto-create. For other wikis, it could be indirectly from NewUserMessage, which listens to AuthPluginAutoCreate and LocalUserCreated. However, NewUserMessage is not enabled at all on meta. (It does trigger notifications on auto-create if enabled and configured to do so (wgNewUserMessageOnAutoCreate)). I didn't find any others that trigger on signup.

So, my understanding from your post is, this is a brief window where we might expect the error (would the user see an exception here?), but that the window is too long in this case.

In T119736#2460239, @Anomie wrote:

At least Clauricekoi doesn't seem to have been a failure. Instead the CentralAuthCreateLocalAccountJob on metawiki started at 21:04:56, the errors showed up from 21:04:57 to 21:05:42, and the CentralAuthCreateLocalAccountJob completed on 21:06:00. It seems like something in this block of code (based on the account attachment in CentralAuthPrimaryAuthenticationProvider::autoCreatedAccount() being logged at 21:04:56 and the on-wiki log message at 21:06) took a strangely long amount of time; this might be related to T121161.

@bd808: What would https://gerrit.wikimedia.org/r/#/c/297936/ do in this situation?

That error was raised on enwiki, so my proposed patch would have ignored the missing metawiki account and submitted a CentralAuthUnattachUserJob job. What happens next would depend on whether or not the CentralAuthCreateLocalAccountJob had finished by the time that the CentralAuthUnattachUserJob executed. If the local account on metawiki exists when the CentralAuthUnattachUserJob job runs then it will leave everything alone. If however the user still did not exist it would remove the localuser entry in the CA database.

[V4b4RwpAADsAAED4S-gAAAAR] 2016-07-14 02:26:16: Fatal exception of type "Exception" for Meatballs123

[V4b9FApAADsAAFbXlS8AAABM] 2016-07-14 02:46:44: Fatal exception of type "Exception" for Er_Sahaj_Arora

In T119736#2334219, @csteipp wrote:

https://gerrit.wikimedia.org/r/#/c/289778/ with https://gerrit.wikimedia.org/r/#/c/289780/ fixes this particular case. I'll see if I can get those merged today.

These are still unmerged. Are they still relevant? They will need a shepherd now.

This task needs someone in the "assigned to" field. I'm looking at @Anomie given his responses thus far on this task (especially lately) and given that @Tgr is still out. This issue needs complete follow through now. It's lasted long enough.

In T119736#2460310, @Mattflaschen-WMF wrote:

There is also an Echo UserSaveSettings hook, but it just adjusts the user's count, it doesn't send a notification.

It adjusts the users count, after firing a bunch of HTTP requests to other wikis and waiting for them to complete. Even if it's not T121161, I still suspect that as a source of slowness.

So, my understanding from your post is, this is a brief window where we might expect the error (would the user see an exception here?), but that the window is too long in this case.

Yes. https://gerrit.wikimedia.org/r/#/c/298823/ should help with that by not making there be as much of a window where the CA transaction is committed while the local transaction isn't.

In T119736#2460468, @bd808 wrote:

If however the user still did not exist it would remove the localuser entry in the CA database.

That's the case I was worried about, and I think that's right. We should probably wait for https://gerrit.wikimedia.org/r/#/c/298823/ before merging your patch then to avoid this race :( On that task, I'm waiting on a reply from @aaron about the potential for it to cause the opposite bug in fixing this one.

In T119736#2460941, @greg wrote:

In T119736#2334219, @csteipp wrote:

https://gerrit.wikimedia.org/r/#/c/289778/ with https://gerrit.wikimedia.org/r/#/c/289780/ fixes this particular case. I'll see if I can get those merged today.

These are still unmerged. Are they still relevant? They will need a shepherd now.

Those seem mostly not relevant to this bug (they might be relevant for other bugs though). The only relevant bit is covered by @bd808's patch now, which I think is good to go right after https://gerrit.wikimedia.org/r/#/c/298823/.

In T119736#2460942, @greg wrote:

This task needs someone in the "assigned to" field. I'm looking at @Anomie given his responses thus far on this task (especially lately) and given that @Tgr is still out. This issue needs complete follow through now. It's lasted long enough.

The reward for work is more work? ;)

I think we're close on fixing the UBN nature of this task, @bd808's patch should reduce it from "OMG people can't log in!" to "logspam".

In T119736#2463092, @Anomie wrote:

It adjusts the users count, after firing a bunch of HTTP requests to other wikis and waiting for them to complete. Even if it's not T121161, I still suspect that as a source of slowness.

I agree it's a cause of slowness. It's an issue, but a planned and temporary one. We should be able to turn it off soon, as soon as our backfill scripts finish on all wikis.

Change 298817 abandoned by Anomie:
Add timing data logging for T119736

Reason:
Served its purpose.

https://gerrit.wikimedia.org/r/298817

In T119736#2463092, @Anomie wrote:

I think we're close on fixing the UBN nature of this task, @bd808's patch should reduce it from "OMG people can't log in!" to "logspam".

I couldn't determine with a quick look which specific patch this was referring to: https://gerrit.wikimedia.org/r/#/c/297946/ (2 +1s already) or https://gerrit.wikimedia.org/r/#/c/297936/ ?

Can we just get that done? If there's something that can get us from UBN! to High or Normal, let's do it ASAP.

The latest run of the script is done. This should cover any cases caused by Echo:

[07/14/16 16:02] <legoktm> matt_flaschen: the script finished
[07/14/16 16:02] <legoktm> legoktm@terbium:~$ grep "not found" clurm-20160713.log -c
[07/14/16 16:02] <legoktm> 134

Change 299094 had a related patch set uploaded (by Aaron Schulz):
Make pre-commit DB callbacks more atomic with multi-DB updates

https://gerrit.wikimedia.org/r/299094

Change 299087 had a related patch set uploaded (by Aaron Schulz):
Make LocalFile::lock() initialize DBO_TRX transactions

https://gerrit.wikimedia.org/r/299087

Status update? How many users are experiencing this still? Are people newly getting bit by it?

In T119736#2465099, @greg wrote:

Status update? How many users are experiencing this still? Are people newly getting bit by it?

Meatballs123 is unresolved.

In T119736#2460486, @JJMC89 wrote:

[V4b4RwpAADsAAED4S-gAAAAR] 2016-07-14 02:26:16: Fatal exception of type "Exception" for Meatballs123

'+channel:exception +message:"* local user data *"' shows about 96 users in the last 6 hours.

@greg: Status update: I +2ed the patches that should turn this from "OMG people can't log in!" to "logspam". Current status is that they'll go with the next train.

Change 299094 merged by jenkins-bot:
Make pre-commit DB callbacks more atomic with multi-DB updates

https://gerrit.wikimedia.org/r/299094

Change 299087 merged by jenkins-bot:
Make LocalFile::lock() initialize DBO_TRX transactions

https://gerrit.wikimedia.org/r/299087

Change 298823 merged by jenkins-bot:
Revert "Wrap CentralAuthUser::attach in an explicit transaction"

https://gerrit.wikimedia.org/r/298823

Change 297946 merged by jenkins-bot:
Detach accounts during global rename

https://gerrit.wikimedia.org/r/297946

Change 297936 merged by jenkins-bot:
Cleanup localuser records when wiki is missing data

https://gerrit.wikimedia.org/r/297936

In T119736#2465899, @Anomie wrote:

@greg: Status update: I +2ed the patches that should turn this from "OMG people can't log in!" to "logspam". Current status is that they'll go with the next train.

Thanks @Anomie (and everyone else) :) Have a good weekend.

In T119736#2463287, @Mattflaschen-WMF wrote:

In T119736#2463092, @Anomie wrote:

It adjusts the users count, after firing a bunch of HTTP requests to other wikis and waiting for them to complete. Even if it's not T121161, I still suspect that as a source of slowness.

I agree it's a cause of slowness. It's an issue, but a planned and temporary one. We should be able to turn it off soon, as soon as our backfill scripts finish on all wikis.

The backfill scripts are done, and the temporary transition config has been removed, so these Echo HTTP requests to get counts from other wikis are no longer happening.

Change 299896 had a related patch set uploaded (by Gergő Tisza):
Cleanup localuser records when wiki is missing data

https://gerrit.wikimedia.org/r/299896

Change 299896 merged by jenkins-bot:
Cleanup localuser records when wiki is missing data

https://gerrit.wikimedia.org/r/299896

Exception frequency is now back to pre-Echo levels, which is to say, still high, only 100/day instead of 1000/day (there is probably no user impact anymore due to Bryan's patch, though). Do we have any idea what causes that? Previously I have assumed T137973 but that does not match the timeline well (login errors were very rare till end of June, hundreds a day from July 1 to 5, thousands a day from 6 to 19, back to hundreds a day now; renames were broken mid-June and fixed on July 11).

And why is the logstash data so different from P3370?

In T119736#2481807, @Tgr wrote:

which is to say, still high, only 100/day instead of 1000/day (there is probably no user impact anymore due to Bryan's patch, though). Do we have any idea what causes that?

Bryan's fix doesn't land everywhere until the train tomorrow, as far as I know.

I still see a fair number of the rollbacks for "To avoid creating high replication lag" in CreateAccount. I tracked that as far as the call to User::addToDatabase() being slow, I suppose the next step is to push timing logging down into User unless @aaron has a better idea.

Agreed. I've set a meeting for 20-July-2016 at 11:30-12:30 SF time (subject to change, but wanted to get something on people's calendars before their schedules fill up). Anyone who wants to join, please email me directly well ahead of time so I can add you to the Hangout.

Notes posted at https://wikitech.wikimedia.org/wiki/Incident_documentation/20160712-EchoCentralAuth/Retrospective

In T119736#2481811, @Tgr wrote:

And why is the logstash data so different from P3370?

The logstash data doesn't look much different to me for the days that are actually currently in the backing indices. We dropped everything before 2016-07-01 as part of the Elasticsearch and Kibana upgrade this week and 2016-07-04's data is still being imported by my recovery script.

The relevant counts from @Legoktm's grep are:

exception.log-20160701.gz:420
exception.log-20160702.gz:352
exception.log-20160703.gz:196
exception.log-20160704.gz:246
exception.log-20160705.gz:298
exception.log-20160706.gz:872
exception.log-20160707.gz:3284
exception.log-20160708.gz:4056
exception.log-20160709.gz:4686
exception.log-20160710.gz:4084
exception.log-20160711.gz:4730

Note that the timestamp of an exception.log archive file is the date that the rotation happened, not necessarily the date that the events in the file occurred.

The last 24 hours look much much better:

Change 298527 abandoned by Mattflaschen:
Add single-user option to checkLocalUser.php

Reason:
This branch is not deployed anymore.

https://gerrit.wikimedia.org/r/298527

Change 298528 abandoned by Mattflaschen:
Add single-user option to checkLocalUser.php

Reason:
This branch is not deployed anymore.

https://gerrit.wikimedia.org/r/298528

In T119736#2481988, @Anomie wrote:

I still see a fair number of the rollbacks for "To avoid creating high replication lag" in CreateAccount. I tracked that as far as the call to User::addToDatabase() being slow, I suppose the next step is to push timing logging down into User unless @aaron has a better idea.

Oh yeah, sorry, you already wrote that earilier. The volume of those errors seems to match the volume of the login errors. If I interpret these messages right, most of the time is spent waiting for the INSERT IGNORE INTO user query to finish. Something gap locks user and the insert times out waiting for it?

(If that's the case the whole "to avoid creating high replication lag" thing seems counterproductive - if the query is only slow due to locking and not by its own right, replicating it would not be slow and there is no need to roll it back and potentially break DB consistency in the process.)

In T119736#2482647, @Tgr wrote:

In T119736#2481988, @Anomie wrote:

I still see a fair number of the rollbacks for "To avoid creating high replication lag" in CreateAccount. I tracked that as far as the call to User::addToDatabase() being slow, I suppose the next step is to push timing logging down into User unless @aaron has a better idea.

Oh yeah, sorry, you already wrote that earilier. The volume of those errors seems to match the volume of the login errors. If I interpret these messages right, most of the time is spent waiting for the INSERT IGNORE INTO user query to finish. Something gap locks user and the insert times out waiting for it?

(If that's the case the whole "to avoid creating high replication lag" thing seems counterproductive - if the query is only slow due to locking and not by its own right, replicating it would not be slow and there is no need to roll it back and potentially break DB consistency in the process.)

It looks to me that you're right.

Ten days later, are there any updates to share?
Is this pretty much solved? Or not?

Twenty days later, are there any updates to share?
Is this pretty much solved? Or not?

Change 289778 abandoned by Chad:
Safely handle policies for local groups

https://gerrit.wikimedia.org/r/289778

Change 289780 abandoned by Chad:
Redo local password enforcement

https://gerrit.wikimedia.org/r/289780