Page MenuHomePhabricator

Allow Mediawiki to store file size greater than 32 bits
Closed, ResolvedPublic

Description

Currently, MediaWiki stores images size as a unsigned 32 bits information. That means a limit of 4 GB (2^32 = 4294967296 bytes).

If we need to allow MediaWiki to store larger files, we need to store this information as 64 bits integers.

The following fields are concerned:

  • image.img_size
  • oldimage.oi_size
  • filearchive.fa_size
  • uploadstash.us_size (to be confirmed, as this uses chunked uploads?)

Event Timeline

 image.img_size
 oldimage.oi_size
 filearchive.fa_size
uploadstash.us_size (to be confirmed, as this uses chunked uploads?)

No, all these are fine. A 32 bit field can store the number 2^32 fine. These fields should work fine as long as the file size is less than 2^(2^32), which we're in no danger of approaching :) [Edit:this is clearly incorrect]

No, all these are fine. A 32 bit field can store the number 2^32 fine. These fields should work fine as long as the file size is less than 2^(2^32), which we're in no danger of approaching :)

Wait what? Don't you need 2^32 bits to store 2^(2^32)?

Bugreporter subscribed.

Don't think it's true.

The fields are currently defined in tables.sql as int unsigned which is a 32-bit type with a range from 0 to 2^32 - 1 (one byte shy of 4 GiB). To record sizes of files of 4 GiB or higher, redefining them as bigint unsigned, a 64-bit type, would be required.

It will also be necessary to ensure that nothing else breaks when interpreting the large numbers as integers (PHP uses C long type for integers, which is 64 bits on Linux x86-64, so this should not be a problem in practice).

Adding DBA to hopefully get a consult on possible implementation steps.

I don't think making the change in the db side is much of a problem, but it might cause issues in swift and so on. I recommend talking to @MatthewVernon first. I also remove the DBA tag because you need a separate ticket for DBA stuff once the schema change patch is merged and deployed (https://wikitech.wikimedia.org/wiki/Schema_changes)

I mean, we should do this change regardless for third parties who don't use swift but just normal file system and want to upload large files.

Swift's issues are discussed in T191802

Change 963332 had a related patch set uploaded (by Brian Wolff; author: Brian Wolff):

[mediawiki/core@master] Store image sizes as 64-bit bigint instead of 32-bit integers

https://gerrit.wikimedia.org/r/963332

Here is a patch for the schema change part (Changing all *_size fields to be bigints). I think the schema change aspect of this is pretty uncontroversial. Actually enabling large files in generic (non-swift) mediawiki probably requires a bit of testing but otherwise should be straight forward.

Actually allowing large files on Wikimedia is a whole big other discussion, but I don't think it should block getting MediaWiki ready for that.

Change 963332 merged by jenkins-bot:

[mediawiki/core@master] Store image sizes as 64-bit bigint instead of 32-bit integers

https://gerrit.wikimedia.org/r/963332

Ok, schema change task at T348183.

What remains to be done on this task:

  • Test that all various file functions actually work with large files
  • Increase FSFileBackend::$maxFileSize to something big (Unclear if we should increase the default value in FileBackendStore or just change it in FSFileBackend. At the very least, discussion on if to increase for SwiftFileBackend should go in T191804 and be handled separately. We should probably keep MemoryBackendStore at 4gb I think)

So initial testing with an 8.8GB file. In total it took about 10m34s to run importImages.php on the 8.8 GB file. Lots of this seems to come down to sha1 generation, which i guess maybe comes down to laptop hardware. My laptop is old, not SSD. openssl sha1 took 2 minutes 45 seconds, so it doesn't seem like a php issue.

Moving the file page also took about 10 minutes. The profile from that was:

99.93% 597553.307      1 - LocalFile::move
99.93% 597550.424      1 - LocalFileMoveBatch::execute
99.82% 596943.484      2 - LocalRepo::skipWriteOperationIfSha1
99.82% 596903.218      2 - FileBackend::doOperations
99.82% 596903.168      2 - FileBackendStore::doOperationsInternal
99.82% 596902.719      2 - section.FileBackendStore::doOperationsInternal-local-backend
99.79% 596725.236      2 - FileOpBatch::attempt
99.71% 596255.870      1 - LocalRepo::storeBatch
99.71% 596255.864      1 - FileRepo::storeBatch
65.60% 392314.029      2 - FileOpBatch::runParallelBatches
65.60% 392313.932      2 - FileOp::attempt
65.52% 391800.739      1 - CopyFileOp::doAttempt
65.52% 391800.731      1 - FileBackendStore::copyInternal
65.52% 391800.681      1 - section.FileBackendStore::copyInternal-local-backend
65.52% 391800.568      1 - FSFileBackend::doCopyInternal
65.51% 391774.245      1 - stream_copy_to_stream
34.18% 204411.148      2 - FileOp::precheck
34.18% 204410.580      1 - CopyFileOp::doPrecheck
34.18% 204410.411      1 - FileOp::precheckDestExistence
34.18% 204407.695      1 - FileOp::fileSha1
34.18% 204407.693      1 - FileBackendStore::getFileSha1Base36
34.18% 204407.670      1 - section.FileBackendStore::getFileSha1Base36-local-backend
34.18% 204407.636      1 - FileBackendStore::doGetFileSha1Base36
34.18% 204407.584      1 - FSFile::getSha1Base36
34.18% 204405.880      1 - sha1_file
 0.12% 721.530    160 - Wikimedia\Rdbms\DBConnRef::__call

So about 2/3 seem to be due to copying the file with stream_copy_to_stream. The other third seems to be generating the sha1 hash.

The copy file source code is as follows:

$copied = false;
// Use fwrite+rename since (a) this clears xattrs, (b) threads still reading the old
// inode are unaffected since it writes to a new inode, and (c) new threads reading
// the file will either totally see the old version or totally see the new version
$fsStagePath = $this->makeStagingPath( $fsDstPath );
$this->trapWarningsIgnoringNotFound();
$srcHandle = fopen( $fsSrcPath, 'rb' );
if ( $srcHandle ) {
        $stageHandle = fopen( $fsStagePath, 'xb' );
        if ( $stageHandle ) {
                $bytes = stream_copy_to_stream( $srcHandle, $stageHandle );
                $copied = ( $bytes !== false && $bytes === fstat( $srcHandle )['size'] );
                fclose( $stageHandle );
                $copied = $copied ? rename( $fsStagePath, $fsDstPath ) : false;
        }
        fclose( $srcHandle );
}

I don't really understand the importance of clearing xattrs, or why it matters new threads reading the file will totally see the old version or the new version, since we are making a copy - the old version isn't going anywhere, and nobody is reading the new version yet. A normal (PHP copy() function or cp command) copy seems only mildly better, taking 4m35seconds (And that could be an artifact of how i tested).

cp --reflink=auto would probably be much faster on file systems that support it. So would just renaming the file, although i imagine we don't want to do that so we can rollback to a consistent state if something goes wrong. I guess maybe you could still rollback with a hardlink instead of a copy.

The generated sha1 hash seems to be mostly not used, except if the destination file already exists in which case it is used to determine whether to exit as already done or exit as failure. There doesn't seem much point to this.


Interestingly, deleting a file seems much more efficient, because it is implemented as a move instead of a copy. The only slow part is doing the unnecessary hash:

99.76% 137390.227      1 - MediaWiki::performAction
99.76% 137387.141      1 - DeleteAction::show
99.72% 137329.599      1 - MediaWiki\Actions\FileDeleteAction::tempDelete
99.72% 137329.596      1 - MediaWiki\Actions\FileDeleteAction::tempExecute
99.64% 137227.789      1 - MediaWiki\Page\File\FileDeleteForm::doDelete
99.53% 137067.970      1 - LocalFile::deleteFile
99.51% 137052.305      1 - LocalFileDeleteBatch::execute
99.31% 136772.590      1 - LocalRepo::deleteBatch
99.31% 136772.588      1 - LocalRepo::skipWriteOperationIfSha1
99.31% 136772.583      1 - FileRepo::deleteBatch
99.31% 136771.698      1 - FileBackend::doOperations
99.31% 136771.673      1 - FileBackendStore::doOperationsInternal
99.31% 136770.938      1 - section.FileBackendStore::doOperationsInternal-local-backend
99.31% 136768.824      1 - FileOpBatch::attempt
99.29% 136749.526      1 - FileOp::precheck
99.29% 136749.440      1 - MoveFileOp::doPrecheck
99.29% 136749.234      1 - FileOp::precheckDestExistence
99.29% 136748.510      1 - FileOp::fileSha1
99.29% 136748.508      1 - FileBackendStore::getFileSha1Base36
99.29% 136748.478      1 - section.FileBackendStore::getFileSha1Base36-local-backend
99.29% 136748.437      1 - FileBackendStore::doGetFileSha1Base36
99.29% 136748.369      1 - FSFile::getSha1Base36
99.29% 136748.191      1 - sha1_file
 0.30% 417.464     61 - Wikimedia\Rdbms\Database::query

Profile of uploading via importImages.php:

100.00% 465884.152      1 - main()
 99.99% 465828.341      1 - MediaWiki\Maintenance\MaintenanceRunner::run
 99.99% 465826.869      1 - ImportImages::execute
 77.85% 362668.923      1 - LocalFile::publish
 77.85% 362668.904      1 - LocalFile::publishTo
 77.84% 362640.470      1 - LocalRepo::publish
 77.84% 362640.468      1 - LocalRepo::skipWriteOperationIfSha1
 77.84% 362640.465      1 - FileRepo::publish
 77.84% 362640.462      1 - LocalRepo::publishBatch
 77.84% 362640.460      1 - LocalRepo::skipWriteOperationIfSha1@1
 77.84% 362640.457      1 - FileRepo::publishBatch
 77.84% 362639.663      1 - FileBackend::doOperations
 77.84% 362639.645      1 - FileBackendStore::doOperationsInternal
 77.84% 362638.330      1 - section.FileBackendStore::doOperationsInternal-local-backend
 77.84% 362622.455      1 - FileOpBatch::attempt
 56.80% 264640.937      1 - FileOpBatch::runParallelBatches
 56.80% 264640.847      2 - FileOp::attempt
 56.80% 264640.839      1 - StoreFileOp::doAttempt
 56.80% 264640.835      1 - FileBackendStore::storeInternal
 56.80% 264640.811      1 - section.FileBackendStore::storeInternal-local-backend
 56.80% 264638.304      1 - FSFileBackend::doStoreInternal
 56.80% 264612.008      1 - stream_copy_to_stream
 42.40% 197518.424      2 - sha1_file
 21.38% 99597.297      1 - MWFileProps::getPropsFromPath
 21.37% 99539.950      1 - FSFile::getSha1Base36
 21.03% 97981.486      2 - FileOp::precheck
 21.03% 97981.233      1 - StoreFileOp::doPrecheck
 21.03% 97981.212      1 - FileOp::precheckDestExistence
 21.03% 97981.199      1 - StoreFileOp::getSourceSha1Base36
  0.59% 2737.227    329 - spl_autoload_call

Change 963862 had a related patch set uploaded (by Brian Wolff; author: Brian Wolff):

[mediawiki/core@master] Allow uploading files up to 32 GB big when using file system backend

https://gerrit.wikimedia.org/r/963862

Change 963860 had a related patch set uploaded (by Brian Wolff; author: Brian Wolff):

[mediawiki/core@master] During FileOp only evaluate SHA1 if it is needed as expensive for big files

https://gerrit.wikimedia.org/r/963860

Change 963862 merged by jenkins-bot:

[mediawiki/core@master] filebackend: Allow uploading files up to 32 GB with FSFileBackend

https://gerrit.wikimedia.org/r/963862

Bawolff claimed this task.

This got merged. Normal files can be up to 32GB and swift can be up to 5GB now ($wgMaxUploadSize has not been changed, so it won't affect anything until that is overriden, except importImages.php)