The bot's jamming again; the vast majority of batch jobs submitted yesterday or today've stalled, usually early-to-very-early on, and failed to progress any further, this time without an Internet Archive outage to explain it.

Will the bot's interface be reenabled once the particular service it relies on comes back online, or is there some other reason it's currently disabled?

Whatever service that is, it looks like it's still down.

Internet Archive has services going up and down at the moment, and a critical one for IABot is down right now. It went down yesterday.

Additionally, when killing and rerunning my jammed runs, the reruns jam at the exact same point in the batch job as the original, which would seem to suggest that the culprit's a problem with the bot, the articles, or both, rather than with the Internet Archive (if an inconsistent ability to access the Internet Archive was somehow jamming the bot, you'd expect the jam points to be randomly scattered when rerunning the same list of articles, and if the bot was being jammed by being unable to access the Internet Archive at all, you'd expect it to jam right at the start of every run [unless the first article in the batch job was one with no online citations at all, which is contradicted by what I noted in my previous comment about batch jobs successfully managing to fetch archiveURLs before jamming, which would seem to definitively rule out the culprit being a complete unavailability of the Internet Archive]).

They were doing just fine (and not just running, but successfully fetching archiveURLs) in the days before yesterday, though, when the Internet Archive wasn't in any better of a shape than it is now.

Aaaaand it's happening again, with all my batch jobs since yesterday (with the sole exception of #21566, which I strongly suspect avoided this fate only because it was so short, with just 11 articles) stalling early (sometimes very early) on.

The bot's been stalling again over the past few days, albeit lately "only" for many hours at a time rather than multiple days.

In the specific case of the Graph extension, I think the impact on servers is not very significant (it's mainly ResourceLoader load.php requests, which are cached in Varnish and scale well) but the bandwidth increase is quite large, about 0.3 MB per graph shown on the screen (that's comparable to the total payload size of visiting a Wikipage page with no images on it, so could be a 100% bandwidth increase per graph).

Updated title and description to more accurately describe what the bot seems to be doing.

The Wayback Machine does back up YouTube videos now, though (and has for the past couple years).

Err, it's now throwing a 504 Gateway Time-out error after a minute or two if I try to submit a job with categorization loops; confirming that that's the intended behavior?

Seems that was some sort of stale-OAuth-data issue on my end (at least judging from what fixed the problem for me). Sorry to've bothered you!

NVM, resolved for me now as well. (Going through the high-volume-activity OAuth process again apparently did the trick.)

Huh. I've even tried logging out and back in again, and I'm still running into the same bug...

Are we sure this's resolved, given that there's no way to test for it ATM due to its effect being masked by T334149?

It hasn't been resolved; IABot is still failing to expand category trees for categories entered as part of a batch job.

The bot's back to failing to expand category trees, after several months of expanding category trees properly.

The problem seems to've gotten worse; we've now got jobs stalling mid-run and never moving again, rather than waiting until the end of the run to stall.

Would this potentially be related to why the current five IABot batch jobs (11994, 11995, 11996, 11997, and 11998) are all currently stalled at zero progress, with the most recent batch-job edit being this one from nearly two weeks ago?