Change Details

The current system for [[ https://commons.wikimedia.org/wiki/Help:Server-side_upload | server-side uploads ]] involves getting media files onto the maintenance host, then running [[ https://gerrit.wikimedia.org/g/mediawiki/core/+/master/maintenance/importImages.php | importImages.php ]] to upload them from the local filesystem. On Kubernetes, that won't work. The small reason is that presently mwscript-k8s doesn't support copying files into the MediaWiki pod for maintenance scripts to read. But even when it does (T376230) the big reason is that these media files are too large (sometimes dozens of gigabytes) for that to work. We'd have to get the media file onto the deployment host, then copy it from there to the worker, //then// complete the import -- if we can even set up a big enough volume backed by kubelet-managed disk. So we probably need to invent something different.

The current system for [[ https://commons.wikimedia.org/wiki/Help:Server-side_upload | server-side uploads ]] involves getting media files onto the maintenance host, then running [[ https://gerrit.wikimedia.org/g/mediawiki/core/+/master/maintenance/importImages.php | importImages.php ]] to upload them from the local filesystem. On Kubernetes, that won't work. The small reason is that presently mwscript-k8s doesn't support copying files into the MediaWiki pod for maintenance scripts to read. But even when it does (T376230) the big reason is that these media files are too large (up to 5 GB per file, sometimes up to dozens of gigabytes in a single import) for that to work. We'd have to get the media file onto the deployment host, then copy it from there to the worker, //then// complete the import -- if we can even set up a big enough volume backed by kubelet-managed disk. So we probably need to invent something different.

The current system for [[ https://commons.wikimedia.org/wiki/Help:Server-side_upload | server-side uploads ]] involves getting media files onto the maintenance host, then running [[ https://gerrit.wikimedia.org/g/mediawiki/core/+/master/maintenance/importImages.php | importImages.php ]] to upload them from the local filesystem. On Kubernetes, that won't work. The small reason is that presently mwscript-k8s doesn't support copying files into the MediaWiki pod for maintenance scripts to read. But even when it does (T376230) the big reason is that these media files are too large (up to 5 GB per file, sometimes up to dozens of gigabytess in a single import) for that to work. We'd have to get the media file onto the deployment host, then copy it from there to the worker, //then// complete the import -- if we can even set up a big enough volume backed by kubelet-managed disk. So we probably need to invent something different.