Page MenuHomePhabricator

Users with detached local accounts after rename during train deployment of 1.27.0-wmf.11
Closed, ResolvedPublic

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Steinsplitter closed this task as Resolved.Jul 21 2016, 6:00 PM
Steinsplitter claimed this task.

Likely related:
On the renamer mailinglist it has been reported that this account (nsfw) (locked and locally) blocked has an unattached account (where blocked) as well.

Steinsplitter reopened this task as Open.Jul 21 2016, 6:00 PM
Steinsplitter removed Steinsplitter as the assignee of this task.
Tegel added a subscriber: Tegel.Jul 21 2016, 6:08 PM

For https://commons.wikimedia.org/w/index.php?title=Special%3ACentralAuth&target=MartyRus it looks to me like all of the unattached wikis are group2 wikis (i.e. Wikipedia projects). The new renaming job code that will unattach at the start of a rename and reattach at the end is included in 1.28.0-wmf.11 which would have hit the group2 wikis at 2016-07-21T19:27Z. This is after the rename jobs ran on the group2 wikis (16:15 to 16:45).

MarcoAurelio triaged this task as Unbreak Now! priority.Jul 21 2016, 7:44 PM

Due to the fact that new account creations not being SUL-merged breaks the existing SUL finalization feature and delay in looking after this will cause much more work, among possible name clashes, etc.

Restricted Application added subscribers: Luke081515, TerraCodes. · View Herald TranscriptJul 21 2016, 7:44 PM

Any rename done between 2016-07-20T20:06Z and 2016-07-21T19:27Z would show this pattern of local accounts on the Wikipedia family wikis being left unattached. When 1.28.0-wmf.11 became the active version on metawiki it would have started using the new logic introduced in rECAUb4c002c to detach all local accounts before the rename jobs were enqueued. The group2 (Wikipedia) wikis would not have had the new logic from the same commit to re-attach the accounts until 1.28.0-wmf.11 became active around 2016-07-21T19:27Z.

It shouldn't be too hard to find the set of effected users and reattach their group2 accounts. I will get started on that.

This behavior should not be continuing now that 1.28.0-wmf.11 is on all wikis. We should keep an eye out for that for a day or two to ensure that there isn't a larger problem that needs to be addressed.

Restricted Application added a project: User-bd808. · View Herald TranscriptJul 21 2016, 7:45 PM
Tgr added a comment.Jul 21 2016, 7:58 PM

T140074 is from a week earlier so there might be other affected accounts as well.

T140074 is from a week earlier so there might be other affected accounts as well.

T140074 looks to be a different root cause. The new rename detach/attach process wouldn't have started until 1.28.0-wmf.11 was live on metawiki at 2016-07-20T20:06Z.

K6ka added a subscriber: K6ka.Jul 21 2016, 8:05 PM

According to logstash, these are the rename jobs for 1.27.0-wmf.10 that have run in the last 48 hours:

message 	Count
GlobalRename: Starting rename of Leefeni,de Karik to Leefeni de Karik	95
GlobalRename: Starting rename of MartynovRussia to MartyRus	54
GlobalRename: Starting rename of செலின் ஜார்ஜ் to செலின் சார்ச்	38
GlobalRename: Starting rename of 流しのインストーラ to 流しの	33
GlobalRename: Starting rename of Borikén to Seriesphile	30
GlobalRename: Starting rename of Reece Leonard to Anonymous5454	25
GlobalRename: Starting rename of 13tobe13 to Gladystti	24
GlobalRename: Starting rename of Ozer yosef to Ozer Yosef	20
GlobalRename: Starting rename of Ayoub Favorite to AlfieDesu	15
GlobalRename: Starting rename of Mikethecrilly to Pijun99	9
GlobalRename: Starting rename of Gopatholabs to Rajatheexplorer	8
GlobalRename: Starting rename of Moinul04 to মঈনুল হাসান	8
GlobalRename: Starting rename of Rafmelorio to Werjako	8
GlobalRename: Starting rename of Nationalmedals to SaraatNSTMF143	6
GlobalRename: Starting rename of Аеропорт Бориспіль to Дмитро Дьошин	6
GlobalRename: Starting rename of Bingoko to Bingo0718	5
GlobalRename: Starting rename of Cristiano Evaristo da Rosa Alves to Crisalves9821	5
GlobalRename: Starting rename of Gravitartech to Gravitar SAS	5
GlobalRename: Starting rename of Hepa-merz to Hepa-wiki	5
GlobalRename: Starting rename of Marizafoods to Mjhorista	5
GlobalRename: Starting rename of Omarrasydin to Pencintadramatempatan	5
GlobalRename: Starting rename of PSR-Recherche to Machtbewusst	5
GlobalRename: Starting rename of Sasnfbi1234 to Sasattack	5
GlobalRename: Starting rename of Alfredceramicartmuseum to Brownrosecaitlin	4
GlobalRename: Starting rename of Fredling Darry to Darry Fredling	4
GlobalRename: Starting rename of Fundacja ECCC to Pysio1410	4
GlobalRename: Starting rename of Bridgetrlg to LeadLemon527	3
GlobalRename: Starting rename of Dga7696 to Laza42	3
GlobalRename: Starting rename of Gatewayschoolnyc to Gwadv2016	3
GlobalRename: Starting rename of HortenseLax to HortenseLac	3
GlobalRename: Starting rename of Isaw~eswiki to Isaw29	3
GlobalRename: Starting rename of Jacobwg to ReedqLEbQfcg27aTzKvxtCV	3
GlobalRename: Starting rename of Kat.zielinska to Kat.Zielinska	3
GlobalRename: Starting rename of Kiran.k.hayer to Kiran. K. Hayer	3
GlobalRename: Starting rename of Nanthu.snk to Nanthakumar Snk	3
GlobalRename: Starting rename of Neogrowth123 to Vikas at NeoGrowth	3
GlobalRename: Starting rename of Shrotevarsha to Rknec	3
GlobalRename: Starting rename of Thebandcamino to The Band CAMINO	3
GlobalRename: Starting rename of Ayuntamiento de La Iglesuela to La Iglesuela	2
GlobalRename: Starting rename of Ecom.office to GutBedacht	2
GlobalRename: Starting rename of Gothic Jazz Orchestra to Midnight Story Orchestra	2
GlobalRename: Starting rename of Rawalshreya to Libreknowledge	2
GlobalRename: Starting rename of दामिनि निरंजन् to Mind2016	2
GlobalRename: Starting rename of はらだげんき to ボンおじさん	2
GlobalRename: Starting rename of Bwrl to Kabali1947	1
GlobalRename: Starting rename of Capitolcrossingdc to Alexchow17	1
GlobalRename: Starting rename of Judith Antonie Anderssen to Judith Antonie Anderssen (WMDE)	1
GlobalRename: Starting rename of Mattdho to VuseWare	1
GlobalRename: Starting rename of RY9874 to Scavtin	1
GlobalRename: Starting rename of Ravinda.mahagamasekera to Ravinda Mahagamasekera	1
GlobalRename: Starting rename of Ryneweir01 to JWStillwater	1
GlobalRename: Starting rename of SIContent to Searchinit	1
GlobalRename: Starting rename of SarahInflued to Justducky	1
GlobalRename: Starting rename of Skinsfan2512 to Guybrush Thriftweed	1

I suspect T140074 was because the account was stuck in a partially renamed state from T137973 for so long, some manual action probably screwed it up even more.

Tgr added a comment.Jul 21 2016, 10:28 PM

T140074 looks to be a different root cause.

Right, I just thought accounts that have been messed up by earlier errors could be fixed as well at the same time. Not sure how to find them though - I tried comparing localuser and localnames, but the query takes forever and due to the recent bugs neither table is exactly reliable...

I made a new maintenance script to attach the dangling local accounts. This script can be found at ~bd808/projects/T141020-rename-unattached/attachAccount.php on terbium and this paste:

1<?php
2$IP = getenv( 'MW_INSTALL_PATH' );
3if ( $IP === false ) {
4 $IP = __DIR__ . '/../../..';
5}
6require_once( "$IP/maintenance/Maintenance.php" );
7
8class AttachAccount extends Maintenance {
9 public function __construct() {
10 parent::__construct();
11 $this->mDescription = "Attaches the specified usernames to a global account";
12 $this->start = microtime( true );
13 $this->partial = 0;
14 $this->migrated = 0;
15 $this->total = 0;
16 $this->dbBackground = null;
17 $this->batchSize = 1000;
18
19 $this->addOption( 'userlist', 'List of usernames to migrate', false, true );
20 }
21
22 public function execute() {
23 $this->dbBackground = CentralAuthUtils::getCentralSlaveDB();
24
25 if ( $this->getOption( 'userlist', false ) === false ) {
26 $this->output( "ERROR - No list of usernames given\n" );
27 exit( 1 );
28 }
29 $list = $this->getOption( 'userlist' );
30 if ( !is_file( $list ) ) {
31 $this->output( "ERROR - File not found: $list" );
32 exit( 1 );
33 }
34 $file = fopen( $list, 'r' );
35 if ( $file === false ) {
36 $this->output( "ERROR - Could not open file: $list" );
37 exit( 1 );
38 }
39 while( strlen( $line = trim( fgets( $file ) ) ) ) {
40 $values = explode( "\t", $line );
41 switch( count( $values ) ){
42 case 1:
43 $this->migrate( $values[0] );
44 break;
45 default:
46 $this->output( "ERROR: Invalid account specification: '$line'\n" );
47 continue;
48 }
49 if ( $this->total % $this->batchSize == 0 ) {
50 $this->output( "Waiting for slaves to catch up ... " );
51 CentralAuthUtils::waitForSlaves();
52 $this->output( "done\n" );
53 }
54 }
55 fclose( $file );
56
57 $this->migratePassOneReport();
58 $this->output( "done.\n" );
59 }
60
61 function migrate( $username ) {
62 $this->total++;
63 $this->output( "CentralAuth account attach for: {$username}\n");
64 $central = new CentralAuthUser( $username, CentralAuthUser::READ_LATEST );
65 try {
66 $unattached = $central->queryUnattached();
67 } catch ( Exception $e ) {
68 // This might happen due to localnames inconsistencies (bug 67350)
69 $this->output( "ERROR: Fetching unattached accounts for {$username} failed." );
70 return;
71 }
72
73 if ( $central->exists() ) {
74 foreach ( $unattached as $wiki => $local ) {
75 $this->output( "ATTACHING: $username@$wiki\n" );
76 $central->attach( $wiki, 'login', false );
77 }
78 } else {
79 $this->output( "ERROR: No CA account found for: $username\n" );
80 return;
81 }
82 $unattachedAfter = $central->queryUnattached();
83
84 if ( count( $unattachedAfter ) == 0 ) {
85 $this->migrated++;
86 return;
87 } elseif ( count( $unattachedAfter ) > 0 && count( $unattachedAfter ) < count( $unattached ) ) {
88 $this->partial++;
89 $this->output( "INFO: Incomplete migration for '$username'\n" );
90 }
91 }
92
93 function migratePassOneReport() {
94 $delta = microtime( true ) - $this->start;
95 $this->output( sprintf( "%s processed %d usernames (%.1f/sec), %d (%.1f%%) fully migrated, %d (%.1f%%) partially migrated\n",
96 wfTimestamp( TS_DB ),
97 $this->total,
98 $this->total / $delta,
99 $this->migrated,
100 $this->total > 0 ? ( $this->migrated / $this->total * 100.0 ) : 0,
101 $this->partial,
102 $this->total > 0 ? ( $this->partial / $this->total * 100.0 ) : 0
103 ) );
104 }
105}
106
107$maintClass = "AttachAccount";
108require_once( RUN_MAINTENANCE_IF_MAIN );

I ran the script against the list of 54 usernames from my logstash search in T141020#2485067. The script found unattached accounts for 18 of the users and re-attached them.

bd808 lowered the priority of this task from Unbreak Now! to High.Jul 21 2016, 11:23 PM

Changing priority from UBN! to High on the presumption that the full deployment of 1.27.0-wmf.11 will keep this from happening again (at least for this particular reason).

I will check back tomorrow morning with a new logstash search and see if there continue to be any dangling accounts left by the new global rename task procedure.

bd808 renamed this task from Users with not attached accounts after rename to Users with detached local accounts after rename during train deployment of 1.27.0-wmf.11.Jul 21 2016, 11:24 PM
greg added a subscriber: greg.Jul 21 2016, 11:32 PM

I have also attached the dangling accounts for the user from T140074 and the blocked user from T141020#2484508. The reason for the detachment of local wikis for these accounts was something different from the primary bug tracked here.

a long wiki break ends. Thanks @bd808

To audit this further, I grepped the Renameuser logs on fluorine for the entire month of July and extracted a list of 748 accounts that were renamed.

  • 744 of these accounts were completely attached to their local wiki accounts.
  • Sandra Müllrick (WMDE) (usurped) does not have a CA account but does have local accounts on commons, mw.o, meta, and wikidata.
  • Rehakles@dewiki was unattached. Interestingly I can't find any evidence in the logs that a LocalRenameUserJob job was even processed for this account on dewiki. The jobs that were logged ran at 2016-07-21T07:18Z which is in the window for being left unattached due to 1.28.0-wmf.10/11 differences however.
  • Hari Shree was unattached on 20 wikis. Rename jobs ran around 2016-07-13T15:00Z so caused by some other issue. Logstash has multiple entries for jobs running on the wikis that were left unattached.
  • Akhemen was unattached on 7 wikis. Jobs processed around 2016-07-14T15:29 (outside wmf.10/11 window) and multiple job entries for the unattached wikis.
Tgr added a comment.Jul 22 2016, 10:13 PM

Here: https://meta.wikimedia.org/wiki/Special:CentralAuth/%E6%97%A5%E6%9C%AC%E8%84%B1%E7%A8%8E%E5%85%9A
is another newly created account with one unattached account.

That account was never renamed; opened T141160 about it.

Change 305428 had a related patch set uploaded (by BryanDavis):
New attachAccount maintenance script

https://gerrit.wikimedia.org/r/305428

Change 305428 merged by jenkins-bot:
New attachAccount maintenance script

https://gerrit.wikimedia.org/r/305428

bd808 moved this task from To Do to Archive on the User-bd808 board.Sep 1 2016, 5:14 PM