Page MenuHomePhabricator

Move IA Upload from Toolforge to VPS
Closed, ResolvedPublic

Description

In an effort to increase reliability and decrease file-access times (this tool works with hundreds of files on disk), we'd like to move IA Upload to its own VPS.

Event Timeline

Is this complete yet? ia-upload is one of the largest tools NFS directories right now, and we are trying to close up alerts that went off for space growing.

No, I'm still sorting a few things out. Should be done today though, then I want to ask the rest of the Wikisource community to try the new installation (just to make sure it's all fine).

Hopefully we'll soon add some better handling for failed jobs, so the disk space used shouldn't be too much of an issue (although it might still want a separate volume; not sure how big its steady state will be yet). That's a different ticket though.

It's working fine on test, uploading to Beta Commons, but on prod it's saying:

Code: permissiondenied
Message: The action you have requested is limited to users in one of the groups: *, [[Commons:Users|Users]].
Result: {"error":{"code":"permissiondenied","info":"The action you have requested is limited to users in one of the groups: *, [[Commons:Users|Users]].", …

My fault: I didn't add all the required permissions to the OAuth consumer. New one is https://meta.wikimedia.org/wiki/Special:OAuthListConsumers/view/7beb995c81fff6e4716b7cbd8857a515 (it seems weird that 'upload file' and 'edit page' are separate; surely it's not possible to upload a file without also editing its page!).

Both redirects are now in place:

I'm also deleting the backlog in the old Toolforge jobqueue directory (it was 290 GB! 1701 items), and am now implementing a max of 1 week for the backlog (T183338) to prevent it getting that big in the future.