Page MenuHomePhabricator

on-request job queue running change to shell, still executing synchronously, blocking the request
Closed, ResolvedPublic

Description

https://gerrit.wikimedia.org/r/#/c/59797/ changed the way jobs are executed on page loads, starting a new php shell to launch runJobs.php. The idea was to execute it asynchronously, so the server can send the page to the client and end the request without waiting for runJobs.php to end. However, this doesn't work.

I've just tested it, changing $wgPhpCli to a shell script with only a sleep command, and no other page would load until the sleep command ends.

According to http://php.net/manual/en/function.passthru.php

*Note:* If a program is started with this function, in order for it to
continue running in the background, the output of the program must be
redirected to a file or another output stream. Failing to do so will cause
PHP to hang until the execution of the program ends (!).

It does this:

$cmd = wfShellWikiCmd( "$IP/maintenance/runJobs.php", array( '--maxjobs', $n ) );
wfShellExec( "$cmd &", $retVal );

But it isn't redirecting stdout nor stderr to another stream (/dev/null for example), so PHP gets stuck on that command until it finishes.

This could cause issues like bug 47375 comment 9

The odd thing is that it apparently causes the entire server to hang. For example, in the test shell script I wrote, i put "sleep 5", and then I open 3 different links of the wiki consecutively, and in a top command there's only one script executing at a time. The last page loads at about 15 seconds after I open the first link (5 + 5 + 5). My environment is PHP 5.3.15 (apache2handler)

If this is really happening, that's not something desirable, even if the process is executed in the background, not blocking the request.


Version: 1.22.0
Severity: major
OS: Linux

Details

Reference
bz60231

Event Timeline

bzimport raised the priority of this task from to High.Nov 22 2014, 3:06 AM
bzimport set Reference to bz60231.

(In reply to comment #0)

According to http://php.net/manual/en/function.passthru.php

Note that that's entirely irrelevant since wfShellExec doesn't use passthru. It uses proc_open.

Now, whether proc_open has a similar issue when directed to use php://stdin and/or php://stderr in certain configurations, I have no idea. It seems to background fine in my local setup (Apache 2.4.7, PHP 5.5.8, using mod_php).

(In reply to comment #1)

Note that that's entirely irrelevant since wfShellExec doesn't use passthru.
It uses proc_open.

I see a shiny passthru call on that function :P

http://git.wikimedia.org/blob/mediawiki%2Fcore.git/REL1_22/includes%2FGlobalFunctions.php#L2807

(In reply to comment #2)

(In reply to comment #1)

Note that that's entirely irrelevant since wfShellExec doesn't use passthru.
It uses proc_open.

I see a shiny passthru call on that function :P
http://git.wikimedia.org/blob/mediawiki%2Fcore.git/REL1_22/
includes%2FGlobalFunctions.php#L2807

Sorry, I didn't notice you were reporting a bug against 1.22. The passthru call has already been removed from later versions in Gerrit change 91115.

Glad to hear it was fixed on master.

Would it be possible to backport to 1.22? It wrecks havoc in some installations

https://www.mediawiki.org/wiki/Thread:Project:Support_desk/Mediawiki_1.22%2B_Causes_two_of_my_servers_to_hang_indefinitely%2E

Change 108740 had a related patch set uploaded by Aaron Schulz:
Various fixes to job running code in Wiki.php

https://gerrit.wikimedia.org/r/108740

Change 108740 merged by jenkins-bot:
Various fixes to job running code in Wiki.php

https://gerrit.wikimedia.org/r/108740

Change 110923 had a related patch set uploaded by Nemo bis:
Various fixes to job running code in Wiki.php

https://gerrit.wikimedia.org/r/110923

Change 110923 abandoned by Nemo bis:
Various fixes to job running code in Wiki.php

https://gerrit.wikimedia.org/r/110923

backport still pending, but no patch to review

Change 110923 restored by MarkAHershberger:
Various fixes to job running code in Wiki.php

Reason:
I want it.

https://gerrit.wikimedia.org/r/110923

Change 110923 merged by jenkins-bot:
Various fixes to job running code in Wiki.php

https://gerrit.wikimedia.org/r/110923

Backported as per comment 10 & comment 11 to 1.22.3 - setting flag from ? to +.

bd808 moved this task from Tag to Archive on the Performance Issue board.Mar 11 2015, 7:30 PM