Currently, MediaWiki stores images size as a unsigned 32 bits information. That means a limit of 4 GB (2^32 = 4294967296 bytes).
If we need to allow MediaWiki to store larger files, we need to store this information as 64 bits integers.
The following fields are concerned:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Dereckson | T191572 Please upload large file to Wikimedia Commons | |||
| Open | None | T191802 [Epic] Determine a strategy to store files between 5 and 100 GB | |||
| Resolved | Bawolff | T191804 Allow to store files between 4 and 5 GB | |||
| Resolved | Bawolff | T191805 Allow Mediawiki to store file size greater than 32 bits | |||
| Resolved | aaron | T348294 FSFileBackend spends a lot of time generating uneeded sha1 hashes that are expensive for large files | |||
| Resolved | ABran-WMF | T348183 Apply schema change for changing img_size, oi_size, us_size, and fa_size to BIGINT |
image.img_size oldimage.oi_size filearchive.fa_size uploadstash.us_size (to be confirmed, as this uses chunked uploads?)
No, all these are fine. A 32 bit field can store the number 2^32 fine. These fields should work fine as long as the file size is less than 2^(2^32), which we're in no danger of approaching :) [Edit:this is clearly incorrect]
The fields are currently defined in tables.sql as int unsigned which is a 32-bit type with a range from 0 to 2^32 - 1 (one byte shy of 4 GiB). To record sizes of files of 4 GiB or higher, redefining them as bigint unsigned, a 64-bit type, would be required.
It will also be necessary to ensure that nothing else breaks when interpreting the large numbers as integers (PHP uses C long type for integers, which is 64 bits on Linux x86-64, so this should not be a problem in practice).
I don't think making the change in the db side is much of a problem, but it might cause issues in swift and so on. I recommend talking to @MatthewVernon first. I also remove the DBA tag because you need a separate ticket for DBA stuff once the schema change patch is merged and deployed (https://wikitech.wikimedia.org/wiki/Schema_changes)
I mean, we should do this change regardless for third parties who don't use swift but just normal file system and want to upload large files.
Swift's issues are discussed in T191802
Change 963332 had a related patch set uploaded (by Brian Wolff; author: Brian Wolff):
[mediawiki/core@master] Store image sizes as 64-bit bigint instead of 32-bit integers
Here is a patch for the schema change part (Changing all *_size fields to be bigints). I think the schema change aspect of this is pretty uncontroversial. Actually enabling large files in generic (non-swift) mediawiki probably requires a bit of testing but otherwise should be straight forward.
Actually allowing large files on Wikimedia is a whole big other discussion, but I don't think it should block getting MediaWiki ready for that.
Change 963332 merged by jenkins-bot:
[mediawiki/core@master] Store image sizes as 64-bit bigint instead of 32-bit integers
Ok, schema change task at T348183.
What remains to be done on this task:
So initial testing with an 8.8GB file. In total it took about 10m34s to run importImages.php on the 8.8 GB file. Lots of this seems to come down to sha1 generation, which i guess maybe comes down to laptop hardware. My laptop is old, not SSD. openssl sha1 took 2 minutes 45 seconds, so it doesn't seem like a php issue.
Moving the file page also took about 10 minutes. The profile from that was:
99.93% 597553.307 1 - LocalFile::move 99.93% 597550.424 1 - LocalFileMoveBatch::execute 99.82% 596943.484 2 - LocalRepo::skipWriteOperationIfSha1 99.82% 596903.218 2 - FileBackend::doOperations 99.82% 596903.168 2 - FileBackendStore::doOperationsInternal 99.82% 596902.719 2 - section.FileBackendStore::doOperationsInternal-local-backend 99.79% 596725.236 2 - FileOpBatch::attempt 99.71% 596255.870 1 - LocalRepo::storeBatch 99.71% 596255.864 1 - FileRepo::storeBatch 65.60% 392314.029 2 - FileOpBatch::runParallelBatches 65.60% 392313.932 2 - FileOp::attempt 65.52% 391800.739 1 - CopyFileOp::doAttempt 65.52% 391800.731 1 - FileBackendStore::copyInternal 65.52% 391800.681 1 - section.FileBackendStore::copyInternal-local-backend 65.52% 391800.568 1 - FSFileBackend::doCopyInternal 65.51% 391774.245 1 - stream_copy_to_stream 34.18% 204411.148 2 - FileOp::precheck 34.18% 204410.580 1 - CopyFileOp::doPrecheck 34.18% 204410.411 1 - FileOp::precheckDestExistence 34.18% 204407.695 1 - FileOp::fileSha1 34.18% 204407.693 1 - FileBackendStore::getFileSha1Base36 34.18% 204407.670 1 - section.FileBackendStore::getFileSha1Base36-local-backend 34.18% 204407.636 1 - FileBackendStore::doGetFileSha1Base36 34.18% 204407.584 1 - FSFile::getSha1Base36 34.18% 204405.880 1 - sha1_file 0.12% 721.530 160 - Wikimedia\Rdbms\DBConnRef::__call
So about 2/3 seem to be due to copying the file with stream_copy_to_stream. The other third seems to be generating the sha1 hash.
The copy file source code is as follows:
$copied = false;
// Use fwrite+rename since (a) this clears xattrs, (b) threads still reading the old
// inode are unaffected since it writes to a new inode, and (c) new threads reading
// the file will either totally see the old version or totally see the new version
$fsStagePath = $this->makeStagingPath( $fsDstPath );
$this->trapWarningsIgnoringNotFound();
$srcHandle = fopen( $fsSrcPath, 'rb' );
if ( $srcHandle ) {
$stageHandle = fopen( $fsStagePath, 'xb' );
if ( $stageHandle ) {
$bytes = stream_copy_to_stream( $srcHandle, $stageHandle );
$copied = ( $bytes !== false && $bytes === fstat( $srcHandle )['size'] );
fclose( $stageHandle );
$copied = $copied ? rename( $fsStagePath, $fsDstPath ) : false;
}
fclose( $srcHandle );
}I don't really understand the importance of clearing xattrs, or why it matters new threads reading the file will totally see the old version or the new version, since we are making a copy - the old version isn't going anywhere, and nobody is reading the new version yet. A normal (PHP copy() function or cp command) copy seems only mildly better, taking 4m35seconds (And that could be an artifact of how i tested).
cp --reflink=auto would probably be much faster on file systems that support it. So would just renaming the file, although i imagine we don't want to do that so we can rollback to a consistent state if something goes wrong. I guess maybe you could still rollback with a hardlink instead of a copy.
The generated sha1 hash seems to be mostly not used, except if the destination file already exists in which case it is used to determine whether to exit as already done or exit as failure. There doesn't seem much point to this.
Interestingly, deleting a file seems much more efficient, because it is implemented as a move instead of a copy. The only slow part is doing the unnecessary hash:
99.76% 137390.227 1 - MediaWiki::performAction 99.76% 137387.141 1 - DeleteAction::show 99.72% 137329.599 1 - MediaWiki\Actions\FileDeleteAction::tempDelete 99.72% 137329.596 1 - MediaWiki\Actions\FileDeleteAction::tempExecute 99.64% 137227.789 1 - MediaWiki\Page\File\FileDeleteForm::doDelete 99.53% 137067.970 1 - LocalFile::deleteFile 99.51% 137052.305 1 - LocalFileDeleteBatch::execute 99.31% 136772.590 1 - LocalRepo::deleteBatch 99.31% 136772.588 1 - LocalRepo::skipWriteOperationIfSha1 99.31% 136772.583 1 - FileRepo::deleteBatch 99.31% 136771.698 1 - FileBackend::doOperations 99.31% 136771.673 1 - FileBackendStore::doOperationsInternal 99.31% 136770.938 1 - section.FileBackendStore::doOperationsInternal-local-backend 99.31% 136768.824 1 - FileOpBatch::attempt 99.29% 136749.526 1 - FileOp::precheck 99.29% 136749.440 1 - MoveFileOp::doPrecheck 99.29% 136749.234 1 - FileOp::precheckDestExistence 99.29% 136748.510 1 - FileOp::fileSha1 99.29% 136748.508 1 - FileBackendStore::getFileSha1Base36 99.29% 136748.478 1 - section.FileBackendStore::getFileSha1Base36-local-backend 99.29% 136748.437 1 - FileBackendStore::doGetFileSha1Base36 99.29% 136748.369 1 - FSFile::getSha1Base36 99.29% 136748.191 1 - sha1_file 0.30% 417.464 61 - Wikimedia\Rdbms\Database::query
Profile of uploading via importImages.php:
100.00% 465884.152 1 - main() 99.99% 465828.341 1 - MediaWiki\Maintenance\MaintenanceRunner::run 99.99% 465826.869 1 - ImportImages::execute 77.85% 362668.923 1 - LocalFile::publish 77.85% 362668.904 1 - LocalFile::publishTo 77.84% 362640.470 1 - LocalRepo::publish 77.84% 362640.468 1 - LocalRepo::skipWriteOperationIfSha1 77.84% 362640.465 1 - FileRepo::publish 77.84% 362640.462 1 - LocalRepo::publishBatch 77.84% 362640.460 1 - LocalRepo::skipWriteOperationIfSha1@1 77.84% 362640.457 1 - FileRepo::publishBatch 77.84% 362639.663 1 - FileBackend::doOperations 77.84% 362639.645 1 - FileBackendStore::doOperationsInternal 77.84% 362638.330 1 - section.FileBackendStore::doOperationsInternal-local-backend 77.84% 362622.455 1 - FileOpBatch::attempt 56.80% 264640.937 1 - FileOpBatch::runParallelBatches 56.80% 264640.847 2 - FileOp::attempt 56.80% 264640.839 1 - StoreFileOp::doAttempt 56.80% 264640.835 1 - FileBackendStore::storeInternal 56.80% 264640.811 1 - section.FileBackendStore::storeInternal-local-backend 56.80% 264638.304 1 - FSFileBackend::doStoreInternal 56.80% 264612.008 1 - stream_copy_to_stream 42.40% 197518.424 2 - sha1_file 21.38% 99597.297 1 - MWFileProps::getPropsFromPath 21.37% 99539.950 1 - FSFile::getSha1Base36 21.03% 97981.486 2 - FileOp::precheck 21.03% 97981.233 1 - StoreFileOp::doPrecheck 21.03% 97981.212 1 - FileOp::precheckDestExistence 21.03% 97981.199 1 - StoreFileOp::getSourceSha1Base36 0.59% 2737.227 329 - spl_autoload_call
Change 963862 had a related patch set uploaded (by Brian Wolff; author: Brian Wolff):
[mediawiki/core@master] Allow uploading files up to 32 GB big when using file system backend
Change 963860 had a related patch set uploaded (by Brian Wolff; author: Brian Wolff):
[mediawiki/core@master] During FileOp only evaluate SHA1 if it is needed as expensive for big files
Change 963862 merged by jenkins-bot:
[mediawiki/core@master] filebackend: Allow uploading files up to 32 GB with FSFileBackend
This got merged. Normal files can be up to 32GB and swift can be up to 5GB now ($wgMaxUploadSize has not been changed, so it won't affect anything until that is overriden, except importImages.php)