Can we also clarify what protected $timeout = 'default'; resolves to?

According to https://curl.haxx.se/libcurl/c/CURLOPT_TIMEOUT.html, default is no timeout. Anyway, that class isn't used here, it goes through MultiHttpClient, which says protected $reqTimeout = 300;. That's five minutes, which doesn't look like enough.

In T226979#5298009, @Urbanecm wrote:

Can we also clarify what protected $timeout = 'default'; resolves to?

According to https://curl.haxx.se/libcurl/c/CURLOPT_TIMEOUT.html, default is no timeout. Anyway, that class isn't used here, it goes through MultiHttpClient, which says protected $reqTimeout = 300;. That's five minutes, which doesn't look like enough.

Mine took quite a bit longer than 5 minutes and succeeded, yours took only just over 5 minutes and failed...

True, but importUpload.php takes some time for prep stuff, before it actually starts a request. No idea how long that stuff takes. In T226953, I tried to change reqTimeout to 3000 (50 minutes) and it instantly succeeded. Yes, it's possible that it'd succeed anyways, but, well, we can't be 100% sure I think.

In T226979#5298109, @Urbanecm wrote:

True, but importUpload.php takes some time for prep stuff, before it actually starts a request. No idea how long that stuff takes. In T226953, I tried to change reqTimeout to 3000 (50 minutes) and it instantly succeeded. Yes, it's possible that it'd succeed anyways, but, well, we can't be 100% sure I think.

We don't want to unconditionally raise it ideally... I'd suggest upto 10 or 15 minutes is probably enough of an increase, if it can succeed at 5 minutes sometimes, doubling or tripling it seems enough for the time being.

That wasn't mean as a permanent solution, just wanted to make sure i found the right timeouting place.

Sure, but it does seem like we probably need to increase it

Change 520135 had a related patch set uploaded (by Urbanecm; owner: Urbanecm):
[mediawiki/core@master] Raise default reqTimeout in MultiHttpClient

https://gerrit.wikimedia.org/r/520135

Was trying to find a way how to raise only for the script, but couldn't understand the method call flow. Might try that tomorrow once more, if you think it's necessary.

I don't think there's an easy way without building it in...

$backend = MediaWiki\MediaWikiServices::getInstance()->getRepoGroup()->getLocalRepo()->getBackend();
$c = new ReflectionClass( $backend );
$p = $c->getProperty( 'backends' );
$p->setAccessible( true );
$backends = $p->getValue( $backend );
foreach( $backends as $b ) { var_dump( get_class( $b ) ); }

string(16) "SwiftFileBackend"
string(16) "SwiftFileBackend"

But basically, MediaWiki\MediaWikiServices::getInstance()->getRepoGroup()->getLocalRepo()->getBackend()->backends contains the SwiftFileBackend instances if you want a starting point...

In T226979#5298668, @Reedy wrote:

I don't think there's an easy way without building it in...

$backend = MediaWiki\MediaWikiServices::getInstance()->getRepoGroup()->getLocalRepo()->getBackend();
$c = new ReflectionClass( $backend );
$p = $c->getProperty( 'backends' );
$p->setAccessible( true );
$backends = $p->getValue( $backend );
foreach( $backends as $b ) { var_dump( get_class( $b ) ); }

string(16) "SwiftFileBackend"
string(16) "SwiftFileBackend"

But basically, MediaWiki\MediaWikiServices::getInstance()->getRepoGroup()->getLocalRepo()->getBackend()->backends contains the SwiftFileBackend instances if you want a starting point...

Tried, giving up for this. Do you think we can just increase the default timeout for this timebeing, so it's at least possible to upload the files? This can be left opened then for somebody to provide a better solution sometime in the future.

@Reedy Ping?

I have not forgotten about T202352, which converts MultiHttpClient to use Guzzle internally. If no one else gets to that first, I'll eventually get back to it, and will update that patch for this timeout change, plus any additional changes that have occurred since I last worked on that. Guzzle (on our stack) will ultimately use curl anyway, and the timeout change will still be relevant.

Change 520135 merged by jenkins-bot:
[mediawiki/core@master] Raise default reqTimeout in MultiHttpClient

https://gerrit.wikimedia.org/r/520135

Change 522951 had a related patch set uploaded (by Urbanecm; owner: Urbanecm):
[mediawiki/core@wmf/1.34.0-wmf.13] Raise default reqTimeout in MultiHttpClient

https://gerrit.wikimedia.org/r/522951

Change 522951 merged by jenkins-bot:
[mediawiki/core@wmf/1.34.0-wmf.13] Raise default reqTimeout in MultiHttpClient

https://gerrit.wikimedia.org/r/522951

Mentioned in SAL (#wikimedia-operations) [2019-07-15T11:24:11Z] <urbanecm@deploy1001> Synchronized php-1.34.0-wmf.13/includes/libs/http/MultiHttpClient.php: SWAT: [[:gerrit:522951|Raise default reqTimeout in MultiHttpClient]] (T226979) (duration: 00m 51s)

An alternative way to fix this would be to have importImages.php pass reqTimeout/connTimeout as an option to LocalFile::publish(). The options are passed down to FileRepo::publishBatch(), which requires a patch to forward them to FileBackend. From there, the options array will apparently be passed down to SwiftFileBackend::doCreateInternal(), requiring only doc comment changes along the way. Parameters set in the request arrays in doCreateInternal() will be passed to MultiHttpClient.

It could also be addressed using constructor configuration. This would affect all callers of SwiftFileBackend, not specifically importImages.php. Timeout options would be added to the constructor options in $wgFileBackends which is set in WMF's filebackend.php. Then a patch to SwiftFileBackend::__construct() would pass the options through to MultiHttpClient::__construct().

Change 596535 had a related patch set uploaded (by Tim Starling; owner: Tim Starling):
[mediawiki/core@master] In SwiftFileBackend allow HTTP timeouts to be set in the constructor

https://gerrit.wikimedia.org/r/596535

Change 596535 merged by jenkins-bot:
[mediawiki/core@master] In SwiftFileBackend allow HTTP timeouts to be set in the constructor

https://gerrit.wikimedia.org/r/596535