In T258122#6317543, @Urbanecm wrote:

I'm afraid this is hard to solve... IIRC it's impossible to whitelist something smaller than a domain, and we're definitely not going to whitelist Amazon - that would be equivalent to whitelisting anything.

Update: using the default command, the generator takes a 2-3 hours to complete for this large category. Putting step at 1 and it was still unfinished 2 days later when I killed it, so that wasn't really feasible. I can experiment with other lower increments between 1 and 50, but I already know enough to know the only values low enough to prevent truncation and lost pages will make the operation take too long to complete, since 50 was already very slow and still getting truncated.

In T253591#6164399, @Xqt wrote:

The problem is that, for my use case, I really just want page titles, but I guess since Pywikibot wants to generate all the page objects using all the metadata, there is no way around this error currently.

CategorizedPageGenerator does not load the content by default. To get it you have to use content=False as parameter or use an explicit cat.get() later.

In attempting to upload to Wikimedia Commons using chunked upload, I receive a whole series of warnings and errors, including
(1) "WARNING: Unexpected offset.", (2) a large traceback from a read timeout (T253236), (3) series of internal_api_error_DBQueryError and retries, and then finally (4) stashfailed error that fails the upload attempt.

You can upload at a faster rate with a bot account. See https://commons.wikimedia.org/wiki/Commons:Bots

I think it happened for many hours, or maybe more than a day, but is not an ongoing issue.

As @WilliamGraham notes, this is actually breaking some categorization on a large scale, which may take a long time to repopulate once they have been depopulated. It's not just about infoboxes on categories, though. If I am understanding right, this is presumably affecting millions of pages on Wikimedia Commons, considering how widespread the use of templates like Template:Artwork, Template:Institution, Template:Creator in image metadata. It might be used a dozen times in a single file page alone, such as for an artist name, genre, medium, institution, and even the labels for basic terms like "height", "width", "collection", "genre", etc. displayed by the template are pulled in from Wikidata.

It's a bit hard to estimate total size in bytes, especially because the total is a moving target, based on how many partners sign on. @SandraF_WMF I recently gave a webinar on DPLA's work (https://youtu.be/0BSoKSYBcBI). Basically, we are now actively doing outreach to our partners (essentially, and US-based GLAM), and doing uploads on-request for partners that have appropriate rights and can give us the media. There are 37 million items in DPLA's dataset, of which 1.7 million are currently licensed appropriately for Commons upload. An item can have any number of media files, and we don't really know the aggregate number (or byte size), because we don't host them—we're just working with the partners to download the assets they host and reupload them to Commons.

Hi, this is me! 😳If it's easier, I can get on Telegram or IRC to chat with you about my project. Obviously, I've been going at a high rate, but I don't really want to break Wikimedia!

UploadWizard aside, this functionality would be also useful in particular for users relying on the API to do bulk uploads, since it would allow users to upload a file with structured data attached—in the same way as the unstructured file description is currently included in the same request as the media upload. This would reduce the number of API requests necessary to achieve the task of adding structured data statements for new uploads.

I would like to give this a bump. If all of the rationales provided in https://www.mediawiki.org/wiki/PAWS#Why%3F are to believed, then simply telling users to use Toolforge for any task that might need to run for more than an hour isn't the solution. I have made hundreds of thousands of edits via PAWS, using PWB, and it would be an incredible improvement to experience, as a user, to not have to worry about keeping a browser session connected.

@Jdforrester-WMF Since macOS Mojave, Shift-Command-5 is now used for an additional screenshot option, which conflicts with strikethrough. Is it possible to reconfigure? https://support.apple.com/en-us/HT201236

Thanks @Dereckson. I linked to the catalog record, which does have tiled images for dynamic zoom in the UI. However, the download link (and media URLs in API outputs) contains a regular media file (e.g. https://catalog.archives.gov/OpaAPI/media/299685/content/arcmedia/media/images/40/7/40-0678a.gif).