Page MenuHomePhabricator
Create Task
Maniphest T124414

MassMessage Fatal error: CAS update failed on user_touched
Closed, ResolvedPublic
Actions

Description

Trying to use MassMessage yields the error above. I've tried with several test messages and target lists.

2016-01-22 13:14:39 mw1257 metawiki exception ERROR: [079ab5da] /wiki/Special:MassMessage   MWException from line 3662 of /srv/mediawiki/php-1.27.0-wmf.11/includes/user/User.php: CAS update failed on user_touched for user ID '3580349' (read from slave); the version of the user to be saved is older than the current version. {"exception_id":"079ab5da"} 
[Exception MWException] (/srv/mediawiki/php-1.27.0-wmf.11/includes/user/User.php:3662) CAS update failed on user_touched for user ID '3580349' (read from slave); the version of the user to be saved is older than the current version.
  #0 /srv/mediawiki/php-1.27.0-wmf.11/includes/session/SessionManager.php(534): User->saveSettings()
  #1 /srv/mediawiki/php-1.27.0-wmf.11/includes/user/User.php(679): MediaWiki\Session\SessionManager->preventSessionsForUser(string)
  #2 /srv/mediawiki/php-1.27.0-wmf.11/extensions/MassMessage/includes/MassMessage.php(44): User::newSystemUser(string, array)
  #3 /srv/mediawiki/php-1.27.0-wmf.11/extensions/MassMessage/includes/SpecialMassMessage.php(291): MassMessage::getMessengerUser()
  #4 /srv/mediawiki/php-1.27.0-wmf.11/extensions/MassMessage/includes/SpecialMassMessage.php(189): SpecialMassMessage->preview(array)
  #5 /srv/mediawiki/php-1.27.0-wmf.11/includes/htmlform/HTMLForm.php(607): SpecialMassMessage->callback(array)
  #6 /srv/mediawiki/php-1.27.0-wmf.11/includes/htmlform/HTMLForm.php(504): HTMLForm->trySubmit()
  #7 /srv/mediawiki/php-1.27.0-wmf.11/extensions/MassMessage/includes/SpecialMassMessage.php(71): HTMLForm->tryAuthorizedSubmit()
  #8 /srv/mediawiki/php-1.27.0-wmf.11/includes/specialpage/SpecialPage.php(384): SpecialMassMessage->execute(NULL)
  #9 /srv/mediawiki/php-1.27.0-wmf.11/includes/specialpage/SpecialPageFactory.php(564): SpecialPage->run(NULL)
  #10 /srv/mediawiki/php-1.27.0-wmf.11/includes/MediaWiki.php(280): SpecialPageFactory::executePath(Title, RequestContext)
  #11 /srv/mediawiki/php-1.27.0-wmf.11/includes/MediaWiki.php(738): MediaWiki->performRequest()
  #12 /srv/mediawiki/php-1.27.0-wmf.11/includes/MediaWiki.php(510): MediaWiki->main()
  #13 /srv/mediawiki/php-1.27.0-wmf.11/index.php(43): MediaWiki->run()
  #14 /srv/mediawiki/w/index.php(3): include(string)
  #15 {main}

Details

SubjectRepoBranchLines +/-
Introduce User::INVALID_TOKENmediawiki/coremaster+21 -18
User: Fix loading of user_touchedmediawiki/coremaster+6 -1
User: Fix loading of user_touchedmediawiki/corewmf/1.27.0-wmf.11+6 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Johan created this task.Jan 22 2016, 1:43 PM
Johan raised the priority of this task from to Needs Triage.
Johan updated the task description. (Show Details)
Johan added a project: MassMessage.
Johan subscribed.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptJan 22 2016, 1:43 PM
Johan added a subscriber: Trizek-WMF.EditedJan 22 2016, 1:44 PM

@Trizek-WMF got the same type of error message when he tried.

Reedy updated the task description. (Show Details)Jan 22 2016, 1:44 PM
Reedy set Security to None.
Reedy added a project: MediaWiki-General.
Johan triaged this task as Unbreak Now! priority.Jan 22 2016, 2:03 PM
Aklapper added subscribers: Legoktm, greg.Jan 22 2016, 4:29 PM

This issue might likely block Tech News delivery on Monday, according to Johan.

Aklapper renamed this task from MassMessage error: [079ab5da] (timestamp) Fatal exception of type "MWException" to MassMessage Fatal error: CAS update failed on user_touched.Jan 22 2016, 4:29 PM
Glaisher added subscribers: Anomie, Tgr, Glaisher.Jan 22 2016, 4:30 PM

@Anomie @Tgr Is this related to the new session stuff?

• Elitre subscribed.Jan 22 2016, 4:52 PM
• PEarleyWMF subscribed.Jan 22 2016, 4:53 PM
Anomie added a comment.EditedJan 22 2016, 5:04 PM

Smells like lag/caching to me. We could try manually calling $user->load( User::READ_LATEST ) at SessionManager.php line 532-ish to see if that fixes it, but that's mostly a guess.

matmarex subscribed.Jan 22 2016, 5:35 PM

Maybe related? T124441: Not all MassMessage sent (the task number looks confusingly similar but it's different)

gerritbot subscribed.Jan 22 2016, 5:50 PM

Change 265775 had a related patch set uploaded (by Anomie):
User: Fix loading of user_touched

https://gerrit.wikimedia.org/r/265775

gerritbot added a project: Patch-For-Review.Jan 22 2016, 5:50 PM
Anomie added a comment.Jan 22 2016, 5:51 PM

Turns out the user_touched timestamp wasn't even being loaded here, due to metawiki's user table having user_email declared as nullable and a bogus test in User::loadFromRow().

gerritbot added a comment.Jan 22 2016, 5:52 PM

Change 265776 had a related patch set uploaded (by Anomie):
User: Fix loading of user_touched

https://gerrit.wikimedia.org/r/265776

gerritbot added a comment.Jan 22 2016, 5:57 PM

Change 265775 merged by jenkins-bot:
User: Fix loading of user_touched

https://gerrit.wikimedia.org/r/265775

gerritbot added a comment.Jan 22 2016, 6:00 PM

Change 265776 merged by jenkins-bot:
User: Fix loading of user_touched

https://gerrit.wikimedia.org/r/265776

Anomie closed this task as Resolved.Jan 22 2016, 6:03 PM
Anomie claimed this task.

Fixed and backported.

matmarex mentioned this in T124441: Not all MassMessage sent.Jan 22 2016, 6:42 PM
ReleaseTaggerBot added projects: MW-1.27-release-notes, MW-1.27-release (WMF-deploy-2016-02-02_(1.27.0-wmf.12)), MW-1.27-release (WMF-deploy-2016-01-19_(1.27.0-wmf.11)).Jan 22 2016, 7:00 PM
Tgr added a parent task: T123451: Deploy SessionManager and bot passwords.Jan 22 2016, 7:08 PM
• MZMcBride subscribed.Jan 24 2016, 5:21 AM
Tgr reopened this task as Open.Jan 28 2016, 10:14 PM

We are still getting these.

greg mentioned this in T125143: Blockers to deploying 1.27-wmf.11.Jan 28 2016, 10:21 PM
Anomie added a comment.Jan 28 2016, 10:29 PM

All seem to be on commonswiki now.

Anomie added a comment.Jan 28 2016, 10:31 PM

Also, all are calls from /rpc/RunJobs.php, none through the web interface.

Anomie added a comment.Jan 29 2016, 2:51 AM

Looking more closely at all the exceptions being logged at the time, it's really looking like it is a CAS update failure. I see a bunch of "Lock wait timeout exceeded" database exceptions at about the same time that are all trying to update the same row in the user table, so what it seems is happening is that whatever fires /rpc/RunJobs.php is firing multiple in parallel for MassMessageJob and they're all conflicting.

We could try moving the SessionManager::singleton()->preventSessionsForUser( $user->getName() ); call in User::newSystemUser() inside the if just above where it is now, or figure out some other way to avoid having to update the database every time through.

Tgr added a comment.Jan 29 2016, 3:08 PM

Can't we just have an invalid token value, like the empty string for passwords, and set it to that, and not update the user if it's already set to that?

Anomie added a comment.Jan 29 2016, 3:22 PM

The trick is that we'd also need to make sure that the invalid token value doesn't pass existing token checks, which are all done using === or the like.

Legoktm mentioned this in T125214: Not all mass messages sent out..Jan 30 2016, 8:56 AM

Since 'MediaWiki message delivery' is a reserved username (via the get reserved usernames hook IIRC), can we skip calling SessionManager::singleton()->preventSessionsForUser() entirely? It shouldn't be possible to login using that username, so there should be no active sessions...

Tgr added a comment.Jan 30 2016, 9:32 AM

A horrible hack would be to set the token to <random string>SYSTEM and ignore such tokens when considering an update.

Tgr merged a task: T125214: Not all mass messages sent out..Jan 30 2016, 9:38 AM
Tgr added subscribers: Base, Saqib, Steinsplitter.
Tgr added a comment.Jan 30 2016, 10:05 AM
In T124414#1983677, @Legoktm wrote:

It shouldn't be possible to login using that username, so there should be no active sessions...

What happens to existing sessions (and OAuth grants etc) when you install the MassMessage extension, though?

Steinsplitter added a comment.Jan 30 2016, 7:01 PM

(see https://phabricator.wikimedia.org/T125214#1983892 - posting here again for precaution)

gerritbot added a comment.Feb 2 2016, 4:40 PM

Change 267907 had a related patch set uploaded (by Anomie):
Introduce User::INVALID_TOKEN

https://gerrit.wikimedia.org/r/267907

gerritbot added a comment.Feb 8 2016, 12:55 AM

Change 267907 merged by jenkins-bot:
Introduce User::INVALID_TOKEN

https://gerrit.wikimedia.org/r/267907

Anomie closed this task as Resolved.Feb 8 2016, 4:09 PM
ReleaseTaggerBot added a project: MW-1.27-release (WMF-deploy-2016-02-09_(1.27.0-wmf.13)).Feb 8 2016, 9:02 PM
Tgr added a comment.Feb 8 2016, 11:31 PM

What about T125214#1983892?

In T125214#1983892, @Steinsplitter wrote:

Please delete all failed msg from queue to prevent duplicates.

I don't quite understand what's being asked (delete from the job queue? stuff should be long gone from there I think?), do we have anything further to do about this? @Legoktm I think you did this a previous time?

Aklapper mentioned this in T171003: MassMessage not working for delivery to Suwikiquote or Mlwikiquote: "CAS update failed on user_touched".Jul 19 2017, 9:46 AM
Aklapper mentioned this in T179806: edit.php gives the wrong error message for protected pages.Nov 14 2017, 11:51 AM
Aklapper removed a subscriber: Anomie.Oct 16 2020, 5:38 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL