Page MenuHomePhabricator

Flaky test ApiUploadTest::testUploadChunks()
Open, Needs TriagePublic

Description

Example console triggered by gerrit 900459.

01:19:51 1) ApiUploadTest::testUploadChunks
01:19:51 ApiUsageException: The file /tmp/localcopy_14095242b63b.jpg does not exist.
...
01:19:51 [wfDebug] [debug] UploadStash::stashFile tried to stash file at '/tmp/localcopy_14095242b63b.jpg', but it doesn't exist {"private":false}

I've confirmed that it is possible to trigger deletion of the relevant temporary file while the file is still in use, using gc_collect_cycles(), which causes the TempFSFile to be destroyed.

Concatenation is finished by around line 607541 of the debug log, but it tries to stash again at line 607674. Why? The error indicates that performStash('critical') was called, so it must have been from getChunkResult() or getStashResult().

Event Timeline

The error indicates that performStash('critical') was called, so it must have been from getChunkResult() or getStashResult().

Actually the test fails with the error message backend-fail-notexists, it's not failing in getChunkResult() or getStashResult().

This leads us to a very plausible model for reproducible failure:

diff --git a/includes/libs/filebackend/fileop/StoreFileOp.php b/includes/libs/filebackend/fileop/StoreFileOp.php
index 76cc4b6c494..4c0942ec00f 100644
--- a/includes/libs/filebackend/fileop/StoreFileOp.php
+++ b/includes/libs/filebackend/fileop/StoreFileOp.php
@@ -37,6 +37,8 @@ class StoreFileOp extends FileOp {
        }
 
        protected function doPrecheck( array &$predicates ) {
+               wfDebug( __METHOD__ . "({$this->params['src']})" );
+               gc_collect_cycles();
                $status = StatusValue::newGood();
 
                // Check if the source file exists in the file system and is not too big

Change 900749 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/core@master] TempFSFile: Keep the WeakMap alive

https://gerrit.wikimedia.org/r/900749

Change 900749 merged by jenkins-bot:

[mediawiki/core@master] TempFSFile: Keep the WeakMap alive

https://gerrit.wikimedia.org/r/900749

Change 901147 had a related patch set uploaded (by Jforrester; author: Tim Starling):

[mediawiki/core@REL1_40] TempFSFile: Keep the WeakMap alive

https://gerrit.wikimedia.org/r/901147

Change 901148 had a related patch set uploaded (by Jforrester; author: Tim Starling):

[mediawiki/core@REL1_39] TempFSFile: Keep the WeakMap alive

https://gerrit.wikimedia.org/r/901148

Change 901148 merged by jenkins-bot:

[mediawiki/core@REL1_39] TempFSFile: Keep the WeakMap alive

https://gerrit.wikimedia.org/r/901148

Change 901147 merged by jenkins-bot:

[mediawiki/core@REL1_40] TempFSFile: Keep the WeakMap alive

https://gerrit.wikimedia.org/r/901147

@Bawolff i suspect this particular issue is fixed now, but i noticed "Concatenation is finished by around line 607541 of the debug log, but it tries to stash again at line 607674." which seems like it might be related to some of the issues you have been chasing. If you are aware and think this is no longer in play and not related to what you were looking at, then I think we can close this.