I was wondering why all my scripts are failing and thought I was going crazy. Glad it isn't just me.

@zhuyifei1999 any word?

@Kizule @Aklapper My apologies for the delay. I noted it as stalled because the extension isn't maintained and it is thus "stalled" getting it merged. It doesn't appear that the maintainer of that project is on here.

This seems to have stalled. It is currently an issue I am running into as well that is preventing my bot from doing its job on commons. It seems convinced that every file is a gallery when it isn't.

@Jidanni What needs to be added to resolve this? Just == {{int:filedesc}} == or is there more needed?

@Bstorm would it be something that could be easily attached or would that require the instance recreated? If the latter, we should probably do now. If it can be added seamlessly, then later is fine. Just wondering...if later, I’d just file a ticket here requesting or?

@Bstorm both. Re storage space: potentially yes, but only for the odd edge case if it runs across a gigapixel image (also why Pillow is forked to add handling for niche use case). Being space conscious, each worker only downloads/works with one file at a time and deletes the temp copy once done. I tend to run variable numbers of workers depending on the work load.

I agree with @MarcoAurelio that it would be useful to have implemented, @Legoktm .

@Legoktm since it is hard deprecated, shouldn't it be removed from tests/phpunit/MediaWikiIntegrationTestCase.php in core?

Since this is the wrong venue. Thanks @Aklapper for pointing this out and @cscott for filing over there.

Thanks @zhuyifei1999 ! Have you been able to work on this any?

@Aklapper @Krenair do either of you know if this could be done? I would still like to resize my instance as redoing everything would be a real pain/time consuming.

AND it looks like this task can finally be closed :)

Thanks for merging this @Aklapper !

@Yaron_Koren could you please take a look at this? Thanks!

Thanks again for the ping @Aklapper

@Clump could you please take a look at this? Thanks!

@Aklapper thanks for the ping. Committed :)

Thanks for the prod @Aklapper . For this one it appears like it has since been resolved by someone else.

It took several hours (didn't immediately report this), but it just decided to work. Bizarre.

There is currently an ongoing discussion very closely relating to this on Meta that appears to be garnering consensus in the other direction/against this ticket/proposal.

The "vanilla" requests code that would accomplish this (using the same format as API:Users) is:

I am using the latest sseclient. I assume that it isn't "real" as event streams shouldn't be 504ing constantly? That wouldn't make sense & would mean that recent changes should be broken etc? This script is connecting to it through pywikibot's. site_rc_listener().

This is happening only a few seconds after I start the script.

Going to resolve this then, given that it appears unavoidable. At least now that it throws an exception, it can be handled (ie skip to the next item in the queue).

I just discovered that rcwatcher.py crashed at some point within the past couple of days. Interesting.

Traceback (most recent call last):
  File "rcwatcher.py", line 65, in <module>
    main()
  File "rcwatcher.py", line 57, in main
    run_watcher()
  File "rcwatcher.py", line 41, in run_watcher
    for change in rc:
  File "/usr/local/lib/python3.8/site-packages/pywikibot/comms/eventstreams.py", line 291, in __iter__
    self.source = EventSource(**self.sse_kwargs)
  File "/home/thesanddoctor/sseclient/sseclient.py", line 48, in __init__
    self._connect()
  File "/home/thesanddoctor/sseclient/sseclient.py", line 63, in _connect
    self.resp.raise_for_status()
  File "/home/ccc/.local/lib/python3.8/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://stream.wikimedia.org/v2/stream/recentchange

In T181443#5927865, @zhuyifei1999 wrote:

Whichever is simplest.

In T181443#5927759, @zhuyifei1999 wrote:

I will be running the script with pdb + save all sseclient trace over the weekend.

@Dvorapa I saw that myself & just updated my above comment prior to seeing your response. It appears to have slipped through; I will have to add a catch for that. Others still relevant though.

Since updating to the latest master version of sseclient (post-fix merges) more of the workers crashed than usual (4 of the 5). 3 of the 4 crashes were due to the same issue.

@zhuyifei1999 requests has been updated & the workers/feeder all restarted. I have re-started test_rc.py and will post back here if anything crashes. If it is good in a few days/week or something like that I think we could consider this resolved. Thanks for your help so far!

@zhuyifei1999 Yes, first and third are two separate customers. The first and second are working with the same customer. The third (code) is just a plain/direct printing of the file name straight from recent changes listener and trying to turn it into a file page (until it fails).

@zhuyifei1999 the first and second traceback are from "production" worker instances and pop items off of the same redis queue (all fed by a single instance of rcwatcher.py), thus they wouldn't get the same image. So it isn't feasible that they would crash all at once. They basically get images first-come, first-serve from recent changes.

In T181443#5915796, @Dvorapa wrote:

Also, what is the output of echo $LC_CTYPE and echo $LANG? Weird is this happens only for some letters. I can not reproduce it on my machine. If you use Pywikibot packaged with scripts, could you share the output of python pwb.py version? And what is your version of requests and sseclient?

@Dvorapa python 3(.8)

@Dvorapa all encoding is set to UTF-8.

@zhuyifei1999 For both of these worth noting that I have not updated the the latest version with the change in behaviour that this task merged.

Thanks @Yaron_Koren !

@zhuyifei1999 Unknown at this point. Implemented and running alongside it now. If/when either it or the any of the 5 workers crash, will report back here.

@zhuyifei1999 The only log currently available is as follows (and linked above):

@Dvorapa Commons. The issue appears to happen at random. I have improved the ordering of my logs so next time it happens it should hopefully actually tell me the file name at issue (configured to log the file name before trying to make a FilePage object out of it, hopefully it will do that before crashing). Given that the files are only run from recent changes if they are new uploads, this isn't something easily repeatable and does appear to happen at random. I will update here when I have more logs. Thank you for your patch to make the exception catchable.

@Dvorapa grabs the file from recent changes using site_rc_listener (script, ImageObj) and then sends it to rcworker (linked above) using redis. rcworker then creates a pwb FilePage object out of the title from the recent changes log and processes the file. site_rc_listener is what must be giving it the invalid image titles? Something just doesn't add up here for me as it doesn't make sense why the script is being given invalid image titles by pwb's site_rc_listener.

@Dvorapa Just trying to create a file page reference given a name of a file from recent changes. Here is the error log.

Merged. Thanks @Tgr !

With @Tgr 's help, a new patch set has been uploaded that is functional. Just awaiting review.

@Dvorapa But what would cause it to return it then when looking at images? I am sort of confused here.

@Yaron_Koren could you please take a look?

@zhuyifei1999 Do you think that such a raise could be made? The problem that I see with both handlings though is that the titles are not "invalid" as they are the valid image titles on the wiki(?). I am also having this issue when it comes to my Commons Corruption Checking task.