Page MenuHomePhabricator
Create Task
Maniphest T332397

Frequent flakiness from UploadFromUrlTest::testSyncDownload
Closed, ResolvedPublic
Actions

Description

Happening in main, REL1_40, probably elsewhere.

00:08:31.613 1) UploadFromUrlTest::testSyncDownload
00:08:31.613 fopen(/tmp/URL5f387cba6cb4.urlupload_): Failed to open stream: No such file or directory
00:08:31.621 
00:08:31.621 /workspace/src/includes/upload/UploadBase.php:1303
00:08:31.621 /workspace/src/includes/upload/UploadBase.php:553
00:08:31.621 /workspace/src/includes/upload/UploadBase.php:474
00:08:31.621 /workspace/src/includes/upload/UploadBase.php:396
00:08:31.621 /workspace/src/includes/api/ApiUpload.php:636
00:08:31.621 /workspace/src/includes/api/ApiUpload.php:115
00:08:31.621 /workspace/src/includes/api/ApiMain.php:1903
00:08:31.621 /workspace/src/includes/api/ApiMain.php:849
00:08:31.621 /workspace/src/tests/phpunit/includes/api/ApiTestCase.php:122
00:08:31.621 /workspace/src/tests/phpunit/includes/api/ApiTestCase.php:162
00:08:31.622 /workspace/src/tests/phpunit/includes/upload/UploadFromUrlTest.php:236
00:08:31.622 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:521

Details

SubjectRepoBranchLines +/-
TempFSFile: Keep the WeakMap alivemediawiki/coreREL1_40+14 -11
TempFSFile: Keep the WeakMap alivemediawiki/coreREL1_39+14 -11
TempFSFile: Keep the WeakMap alivemediawiki/coremaster+14 -11
Customize query in gerrit

Related Objects

Event Timeline

Jdforrester-WMF triaged this task as High priority.Mar 17 2023, 2:20 PM
Jdforrester-WMF created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 17 2023, 2:20 PM
Maintenance_bot added a project: Commons.Mar 17 2023, 2:30 PM
Umherirrender subscribed.Mar 17 2023, 8:08 PM

I have only seen it on mediawiki-quibble-composer-mysql-php81-docker maybe a php8.1 issue.
Would be nice if someone see the failure on other versions and can report that.

Jdforrester-WMF added a comment.Mar 17 2023, 9:35 PM
In T332397#8706480, @Umherirrender wrote:

I have only seen it on mediawiki-quibble-composer-mysql-php81-docker maybe a php8.1 issue.
Would be nice if someone see the failure on other versions and can report that.

Hmm, yes, on https://gerrit.wikimedia.org/r/c/mediawiki/core/+/900459 there was a failure on mediawiki-quibble-vendor-mysql-php80-docker but that instead was:

00:08:13.112 1) ApiUploadTest::testUploadChunks
00:08:13.112 ApiUsageException: The file /tmp/localcopy_14095242b63b.jpg does not exist.
00:08:13.119 
00:08:13.119 /workspace/src/includes/api/ApiBase.php:1541

Possibly the same root cause though, with a race condition or similar?

Umherirrender added a comment.Mar 17 2023, 9:57 PM

From the log of a failing run on UploadFromUrlTest::testSyncDownload (https://integration.wikimedia.org/ci/job/mediawiki-quibble-composer-mysql-php81-docker/543 - Using PHP 8.1.16):

[fileupload] Temporary file created "/tmp/URL476feb16cf92.urlupload_"
[fileupload] Starting download from "http://upload.wikimedia.org/wikipedia/mediawiki/b/bc/Wiki.png" <>
[fileupload] Received chunk of 9497 bytes
[fileupload] Download by URL completed successfully.
ApiUpload::verifyUpload about to verify
mime: <> extension: <png>
[PHPUnit] ERROR in test UploadFromUrlTest::testSyncDownload: fopen(/tmp/URL476feb16cf92.urlupload_): Failed to open stream: No such file or directory

[The "download" is a mocked http request and not a real one.]

So the logs says the file was created and also not exists ...

The failure on php80 is on api test ApiUploadTest::testUploadChunks - https://integration.wikimedia.org/ci/job/mediawiki-quibble-vendor-mysql-php80-docker/2142/console

[Mime] MimeAnalyzer::doGuessMimeType: analyzing head and tail of /tmp/localcopy_14095242b63b.jpg for magic numbers.
[Mime] MimeAnalyzer::doGuessMimeType: getimagesize detected /tmp/localcopy_14095242b63b.jpg as image/jpeg
[Mime] MimeAnalyzer::guessMimeType: guessed mime type of /tmp/localcopy_14095242b63b.jpg: image/jpeg
[Mime] MimeAnalyzer::improveTypeFromExtension: improved mime type for .jpg: image/jpeg
[...]
[Mime] MimeAnalyzer::doGuessMimeType: analyzing head and tail of /tmp/localcopy_14095242b63b.jpg for magic numbers.
[Mime] MimeAnalyzer::doGuessMimeType: getimagesize detected /tmp/localcopy_14095242b63b.jpg as image/jpeg
[Mime] MimeAnalyzer::guessMimeType: guessed mime type of /tmp/localcopy_14095242b63b.jpg: image/jpeg
[Mime] MimeAnalyzer::improveTypeFromExtension: improved mime type for .jpg: image/jpeg
[...]
UploadStash::stashFile tried to stash file at '/tmp/localcopy_14095242b63b.jpg', but it doesn't exist
ApiUpload::performStash Stashing temporary file failed: UploadStashBadPathException Path doesn&#39;t exist.
[rdbms] Wikimedia\Rdbms\LoadBalancer::reuseOrOpenConnectionForNewRef: reusing connection for 0/wikidb-unittest_
[rdbms] User::loadFromDatabase [0s] localhost:/workspace/db/quibble-mysql-mf30n3vi/socket: SELECT  user_id,user_name,user_real_name,user_email,user_touched,user_token,user_email_authenticated,user_email_token,user_email_token_expires,user_registration,user_editcount,user_actor.actor_id  FROM `unittest_user` JOIN `unittest_actor` `user_actor` ON ((user_actor.actor_user = user_id))   WHERE user_id = 2  LIMIT 1  
[PHPUnit] ERROR in test ApiUploadTest::testUploadChunks: The file /tmp/localcopy_14095242b63b.jpg does not exist.

Working with it to get the mime and when used a bit later it is deleted (or just not found? - file system cache or such failure?).

Umherirrender added a comment.Mar 17 2023, 10:33 PM

There is a TempFSFile in use, so the WeakMap added in 4e3c6cb25db4423dcc093f63dcbe04d45d8781bb together with gc could already deleted the file while it is still in use, but not referenced. But I have no idea how to verify that it is the case.

tstarling subscribed.Mar 18 2023, 12:34 AM
In T332397#8706684, @Umherirrender wrote:

There is a TempFSFile in use, so the WeakMap added in 4e3c6cb25db4423dcc093f63dcbe04d45d8781bb together with gc could already deleted the file while it is still in use, but not referenced. But I have no idea how to verify that it is the case.

I verified it by inserting gc_collect_cycles() and logging a stack trace from TempFSFile::__destruct(), as I reported at T332461.

gerritbot added a comment.Mar 18 2023, 1:35 AM

Change 900749 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/core@master] TempFSFile: Keep the WeakMap alive

https://gerrit.wikimedia.org/r/900749

gerritbot added a project: Patch-For-Review.Mar 18 2023, 1:35 AM
gerritbot added a comment.Mar 20 2023, 12:47 PM

Change 900749 merged by jenkins-bot:

[mediawiki/core@master] TempFSFile: Keep the WeakMap alive

https://gerrit.wikimedia.org/r/900749

gerritbot added a comment.Mar 20 2023, 12:48 PM

Change 901147 had a related patch set uploaded (by Jforrester; author: Tim Starling):

[mediawiki/core@REL1_40] TempFSFile: Keep the WeakMap alive

https://gerrit.wikimedia.org/r/901147

gerritbot added a comment.Mar 20 2023, 12:48 PM

Change 901148 had a related patch set uploaded (by Jforrester; author: Tim Starling):

[mediawiki/core@REL1_39] TempFSFile: Keep the WeakMap alive

https://gerrit.wikimedia.org/r/901148

gerritbot added a comment.Mar 20 2023, 12:59 PM

Change 901148 merged by jenkins-bot:

[mediawiki/core@REL1_39] TempFSFile: Keep the WeakMap alive

https://gerrit.wikimedia.org/r/901148

ReleaseTaggerBot added projects: MW-1.41-notes (1.41.0-wmf.1; 2023-03-20), MW-1.39-notes.Mar 20 2023, 1:00 PM
gerritbot added a comment.Mar 20 2023, 1:02 PM

Change 901147 merged by jenkins-bot:

[mediawiki/core@REL1_40] TempFSFile: Keep the WeakMap alive

https://gerrit.wikimedia.org/r/901147

Maintenance_bot removed a project: Patch-For-Review.Mar 20 2023, 1:11 PM
ReleaseTaggerBot added a project: MW-1.40-notes.Mar 20 2023, 2:00 PM
TheDJ moved this task from Backlog to Uploading on the MediaWiki-File-management board.Jun 28 2023, 2:09 PM
Umherirrender closed this task as Resolved.Jul 4 2023, 7:51 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits