Page MenuHomePhabricator

Adding a user to a project results in a blank page with the user added to the project but no shell access
Closed, ResolvedPublic

Description

I added the users BengaliHindu and Nedol to the Toolforge project by visiting https://wikitech.wikimedia.org/w/index.php?title=Special:NovaProject&action=addmember&projectname=tools&member=BengaliHindu and https://wikitech.wikimedia.org/w/index.php?title=Special:NovaProject&action=addmember&projectname=tools&member=Nedol and clicking "Submit".

After a while, the resulting page (https://wikitech.wikimedia.org/wiki/Special:NovaProject) was empty. I looked at "Manage Projects" (same URL, but not reload with POST parameters, but a new request) and verified that the two users were now members of the Toolforge project and marked the requests as done.

But subsequent inspection showed that they hadn't been given their shell access rights ("shell" at https://wikitech.wikimedia.org/wiki/Special:UserRights). I have fixed that for the two users in question since.

Details

Related Gerrit Patches:
mediawiki/extensions/OpenStackManager : wmf/1.27.0-wmf.2Don't record members on bastion project page
mediawiki/extensions/OpenStackManager : masterDon't record members on bastion project page

Event Timeline

scfc created this task.Sep 30 2015, 11:53 AM
scfc raised the priority of this task from to Unbreak Now!.
scfc updated the task description. (Show Details)
scfc added a project: wikitech.wikimedia.org.
scfc added a subscriber: scfc.
Restricted Application added a project: Cloud-Services. · View Herald TranscriptSep 30 2015, 11:53 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Can you let me know if this happens again and at what time?

scfc added a comment.Sep 30 2015, 7:19 PM

Will do in the future. If you want to look at the existing logs, the times should be roughly the modifications in https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Tools&action=history, i. e. 2015-09-30T11:14:53‎ and 2015-09-30T09:06:17‎.

scfc added a comment.Oct 1 2015, 1:59 PM

Happened just now again with user Steini.

scfc added a comment.Oct 1 2015, 2:00 PM

(And I have just given him manually the shell access right.)

I don't see any logs of this. Are you just getting a completely blank document back, as you would with some PHP errors?

scfc added a comment.Oct 1 2015, 6:01 PM

I just tried to replicate it. When I removed "Tim Landscheidt (Test 2)" from Toolforge, the user retained his shell right. I added him back as a member of Toolforge without any problems and in considerably less time than in the previous failures.

I then created "Tim Landscheidt (Test 3)" and tried to add him to the Toolforge project. This failed after a longer period of time (consistent with wikitech trying to give him his shell right) with an empty page and no shell right. Empty page means here: No source at all, after saving to disk the file has a length of 0, Firefox displays the URL as the tab title, but also under page information (?; right click, "Seiteninformationen anzeigen") lists a modification date of "01 Okt 2015 17:45:11 UTC" and a "size" ("Größe") of "13.609 Byte"; the latter seems to be an artifact of the last successful request of the URL, i. e. when I opened https://wikitech.wikimedia.org/wiki/Special:NovaProject in another tab, the "size" for the empty page went to 13611 bytes.

I have added you as a project admin for Toolforge so you can test the failure as well (unless it is specific to my browser).

Krenair added a comment.EditedOct 4 2015, 10:59 PM

I got an HTTP 500 when I tried to add Alex Monk (Test 1). Going to ask if I can get access to silver.wikimedia.org:/var/log/apache2/error.log (this file is not readable to me :/)

Krenair added a subscriber: coren.EditedOct 5 2015, 12:25 AM

@coren found the log entry for me, as predicted it was a "Maximum execution time of 30 seconds exceeded".

I've been fiddling around with wfDebugLog statements on tin (syncing them to silver only) and have narrowed this down to our addUserToBastionProject hook into UserAddGroup (called to grant shell rights). It seems that adding users to (OpenStackNovaProject::addMember)/removing users from (OpenStackNovaProject::deleteMember) this group is ridiculously slow. When calling it from the console on silver my buffer is flooded with these warnings:
Notice: Array to string conversion in /srv/mediawiki/php-1.27.0-wmf.1/extensions/SemanticMediaWiki/includes/storage/SQLStore/SMW_SQLStore3_Writers.php on line 384

It's somewhere beneath OpenStackNovaArticle::editArticle - I'm guessing the edit triggers slow+buggy SMW hooks?

krenair@tin:/srv/mediawiki-staging/php-1.27.0-wmf.1/extensions/OpenStackManager ((fc2cbbc...))$ git diff
diff --git a/nova/OpenStackNovaProject.php b/nova/OpenStackNovaProject.php
index 6585bef..0981046 100644
--- a/nova/OpenStackNovaProject.php
+++ b/nova/OpenStackNovaProject.php
@@ -939,7 +939,9 @@ RESOURCEINFO;
                        implode( ',', $admins ),
                        implode( ',', $members )
                );
+               wfDebugLog( 'T114229', 'OpenStackNovaArticle::editArticle called' );
                OpenStackNovaArticle::editArticle( $this->getProjectName(), $text, $wgOpenStackManagerProjectNamespace );
+               wfDebugLog( 'T114229', 'OpenStackNovaArticle::editArticle returned' );
                if ( $wgOpenStackManagerCreateProjectSALPages ) {
                        $pagename = $this->getProjectName() . "/SAL";
                        $id = Title::newFromText( $pagename, $wgOpenStackManagerProjectNamespace )->getArticleId();
2015-10-05 00:56:02 silver labswiki T114229 INFO: OpenStackNovaArticle::editArticle called  
2015-10-05 00:57:06 silver labswiki T114229 INFO: OpenStackNovaArticle::editArticle returned

I kind of wonder whether can just hack the OpenStackNovaProject::editArticle caller to not record members for bastion, but I have no real proof it's the huge number of members which causes this issue (for reference, bastion project has 4,692 members).

Change 243603 had a related patch set uploaded (by Alex Monk):
Don't record members on bastion project page

https://gerrit.wikimedia.org/r/243603

The patch above is live hacked, uncommitted, on tin and synchronised to silver. You should be able to add people to projects without failures now though.
/me goes to sleep

scfc added a comment.Oct 5 2015, 2:10 AM

Thanks. Just to recap: The slowness comes not from adding the user to LDAP, but the "dump" of the new LDAP data to https://wikitech.wikimedia.org/wiki/Nova_Resource:Bastion that triggers via https://wikitech.wikimedia.org/wiki/Template:Nova_Project the creation/update of the various SMW "properties"?

Because nobody reviewed the patch, this will have broken again as wikitech has moved to a different version without the live hack applied.

Change 243603 merged by jenkins-bot:
Don't record members on bastion project page

https://gerrit.wikimedia.org/r/243603

Change 244707 had a related patch set uploaded (by Alex Monk):
Don't record members on bastion project page

https://gerrit.wikimedia.org/r/244707

Change 244707 merged by jenkins-bot:
Don't record members on bastion project page

https://gerrit.wikimedia.org/r/244707

Krenair closed this task as Resolved.Oct 9 2015, 6:16 PM

Should work now.