Page MenuHomePhabricator

Internal api error: CannotCreateActorException
Closed, ResolvedPublic

Description

It is almost impossible today to close deletion request, i am getting tons of such errors:

API request failed (internal_api_error_CannotCreateActorException): [W-66MwpAAE0AAAVLeqgAAACD] Caught exception of type CannotCreateActorException
trace
#0 /srv/mediawiki/php-1.33.0-wmf.6/includes/ActorMigration.php(224): User->getActorId(Wikimedia\Rdbms\DatabaseMysqli)
#1 /srv/mediawiki/php-1.33.0-wmf.6/includes/page/WikiPage.php(2902): ActorMigration->getInsertValues(Wikimedia\Rdbms\DatabaseMysqli, string, User)
#2 /srv/mediawiki/php-1.33.0-wmf.6/includes/page/WikiPage.php(2681): WikiPage->archiveRevisions(Wikimedia\Rdbms\DatabaseMysqli, integer, boolean)
#3 /srv/mediawiki/php-1.33.0-wmf.6/includes/page/WikiPage.php(2624): WikiPage->doDeleteArticleBatched(string, boolean, User, NULL, string, boolean)
#4 /srv/mediawiki/php-1.33.0-wmf.6/includes/FileDeleteForm.php(198): WikiPage->doDeleteArticleReal(string, boolean, integer, boolean, string, User, NULL)
#5 /srv/mediawiki/php-1.33.0-wmf.6/includes/api/ApiDelete.php(171): FileDeleteForm::doDelete(Title, LocalFile, NULL, string, boolean, User, NULL)
#6 /srv/mediawiki/php-1.33.0-wmf.6/includes/api/ApiDelete.php(73): ApiDelete::deleteFile(WikiFilePage, User, NULL, string, boolean, NULL)
#7 /srv/mediawiki/php-1.33.0-wmf.6/includes/api/ApiMain.php(1576): ApiDelete->execute()
#8 /srv/mediawiki/php-1.33.0-wmf.6/includes/api/ApiMain.php(531): ApiMain->executeAction()
#9 /srv/mediawiki/php-1.33.0-wmf.6/includes/api/ApiMain.php(502): ApiMain->executeActionWithErrorHandling()
#10 /srv/mediawiki/php-1.33.0-wmf.6/api.php(87): ApiMain->execute()
#11 /srv/mediawiki/w/api.php(3): include(string)
#12 {main}

Logstash query: message:"Cannot create actor ID for user_id"

Over 30 days:

We had a few occurrences in November. it started spamming heavily 2018-11-26 around 15:20 UTC

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 28 2018, 3:56 PM
exception.file	       	/srv/mediawiki/php-1.33.0-wmf.6/includes/user/User.php:2555
exception.message	       	Cannot create actor ID for user_id=xxxxxxx user_name=XXXXXXXXX

#0 /srv/mediawiki/php-1.33.0-wmf.6/includes/ActorMigration.php(224): User->getActorId(Wikimedia\Rdbms\DatabaseMysqli)
#1 /srv/mediawiki/php-1.33.0-wmf.6/includes/page/WikiPage.php(2902): ActorMigration->getInsertValues(Wikimedia\Rdbms\DatabaseMysqli, string, User)
#2 /srv/mediawiki/php-1.33.0-wmf.6/includes/page/WikiPage.php(2681): WikiPage->archiveRevisions(Wikimedia\Rdbms\DatabaseMysqli, integer, boolean)
#3 /srv/mediawiki/php-1.33.0-wmf.6/includes/page/WikiPage.php(2624): WikiPage->doDeleteArticleBatched(string, boolean, User, NULL, string, boolean)
#4 /srv/mediawiki/php-1.33.0-wmf.6/includes/FileDeleteForm.php(198): WikiPage->doDeleteArticleReal(string, boolean, integer, boolean, string, User, NULL)
#5 /srv/mediawiki/php-1.33.0-wmf.6/includes/api/ApiDelete.php(171): FileDeleteForm::doDelete(Title, LocalFile, NULL, string, boolean, User, NULL)
#6 /srv/mediawiki/php-1.33.0-wmf.6/includes/api/ApiDelete.php(73): ApiDelete::deleteFile(WikiFilePage, User, NULL, string, boolean, NULL)
#7 /srv/mediawiki/php-1.33.0-wmf.6/includes/api/ApiMain.php(1576): ApiDelete->execute()
#8 /srv/mediawiki/php-1.33.0-wmf.6/includes/api/ApiMain.php(531): ApiMain->executeAction()
#9 /srv/mediawiki/php-1.33.0-wmf.6/includes/api/ApiMain.php(502): ApiMain->executeActionWithErrorHandling()
#10 /srv/mediawiki/php-1.33.0-wmf.6/api.php(87): ApiMain->execute()
#11 /srv/mediawiki/w/api.php(3): include(string)
#12 {main}

Change 476508 had a related patch set uploaded (by Anomie; owner: Anomie):
[mediawiki/core@master] User: Bypass repeatable-read when creating an actor_id

https://gerrit.wikimedia.org/r/476508

greg triaged this task as Unbreak Now! priority.
Restricted Application added subscribers: Liuxinyu970226, TerraCodes. · View Herald TranscriptNov 29 2018, 6:32 PM
greg assigned this task to Anomie.Nov 29 2018, 6:34 PM
hashar updated the task description. (Show Details)Nov 29 2018, 8:48 PM

I have looked at logstash over 30 days, there were a few occurrences in November that pre date 1.33.0-wmf.6 so it is no more a blocker.

The log raised starting 2018-11-26 around 15:20 UTC which would correspond to:

15:24 <anomie@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Setting actor migration to write-both/read-old on group 1 (T188327) (duration: 00m 46s)

greg lowered the priority of this task from Unbreak Now! to Normal.Nov 29 2018, 10:25 PM
greg added a subscriber: greg.

I have looked at logstash over 30 days, there were a few occurrences in November that pre date 1.33.0-wmf.6 so it is no more a blocker.

This comment was removed by Dzahn.

Change 476508 merged by jenkins-bot:
[mediawiki/core@master] User: Bypass repeatable-read when creating an actor_id

https://gerrit.wikimedia.org/r/476508

Change 478363 had a related patch set uploaded (by Paladox; owner: Anomie):
[mediawiki/core@REL1_32] User: Bypass repeatable-read when creating an actor_id

https://gerrit.wikimedia.org/r/478363

Change 478365 had a related patch set uploaded (by Paladox; owner: Anomie):
[mediawiki/core@REL1_31] User: Bypass repeatable-read when creating an actor_id

https://gerrit.wikimedia.org/r/478365

Change 478685 had a related patch set uploaded (by Anomie; owner: Anomie):
[mediawiki/core@wmf/1.33.0-wmf.6] User: Bypass repeatable-read when creating an actor_id

https://gerrit.wikimedia.org/r/478685

Change 478685 merged by jenkins-bot:
[mediawiki/core@wmf/1.33.0-wmf.6] User: Bypass repeatable-read when creating an actor_id

https://gerrit.wikimedia.org/r/478685

Mentioned in SAL (#wikimedia-operations) [2018-12-10T15:05:24Z] <anomie@deploy1001> Synchronized php-1.33.0-wmf.6/includes/user/User.php: Backport fix for T210621 (duration: 00m 46s)

Anomie closed this task as Resolved.Dec 10 2018, 3:08 PM

Fixed and backported.

Mentioned in SAL (#wikimedia-operations) [2018-12-11T18:06:35Z] <anomie@deploy1001> Synchronized php-1.33.0-wmf.6/includes/user/User.php: Backport fix for T210621, for real this time (duration: 00m 53s)

Bad me, I forgot the "git rebase" step so the scap yesterday didn't deploy the actual fix.

Change 478363 merged by jenkins-bot:
[mediawiki/core@REL1_32] User: Bypass repeatable-read when creating an actor_id

https://gerrit.wikimedia.org/r/478363

Change 478365 merged by jenkins-bot:
[mediawiki/core@REL1_31] User: Bypass repeatable-read when creating an actor_id

https://gerrit.wikimedia.org/r/478365