@Dimpizzy: how did you create the family file?

@Xqt: I just copied the example from the manual and changed it to suit my wiki.

I was able to reproduce it. This issue is already solved with Pywikibot 8 which is the current master branch but still unser development. I could backport a relatedchange to stable version 7.7.4 or we can just wait for finishing Pywikibot 8 development (which will be in few weeks). You have to update your bot anyway. What would you prefer?

1. use the current master branch
2. wait for a stable release which solves this bug

In T326046#8492986, @Xqt wrote:

I was able to reproduce it. This issue is already solved with Pywikibot 8 which is the current master branch but still unser development. I could backport a relatedchange to stable version 7.7.4 or we can just wait for finishing Pywikibot 8 development (which will be in few weeks). You have to update your bot anyway. What would you prefer?

1. use the current master branch
2. wait for a stable release which solves this bug

#2, I can wait, thanks. Is Pywikibot 8 usable in its current state?

Edit: Oh, I thought #1 meant you backporting. If the current master branch works already, then I'd like to try that.

Just to satisfy my curiosity since I spent a while trying to find it, could you say what the issue was, if it's not super complex?

In T326046#8492990, @Dimpizzy wrote:

#2, I can wait, thanks. Is Pywikibot 8 usable in its current state?

Sure, Pywikibot 8 (master) is ready to use but it has "under development" state which means it could fail if new features are implemented whereas "stable" means it is ready for production systems. Anyway using master could help to find and fix those bugs which aren't detected yet during CI tests.

Sure, Pywikibot 8 (master) is ready to use but it has "under development" state which means it could fail if new features are implemented whereas "stable" means it is ready for production systems. Anyway using master could help to find and fix those bugs which aren't detected yet during CI tests.

Ok, great, thanks.

In T326046#8493004, @Dimpizzy wrote:

Just to satisfy my curiosity since I spent a while trying to find it, could you say what the issue was, if it's not super complex?

I have no idea at the moment but the LoginManager was rewritten and support of older wikis was given up.

In T326046#8493007, @Xqt wrote:

In T326046#8493004, @Dimpizzy wrote:

Just to satisfy my curiosity since I spent a while trying to find it, could you say what the issue was, if it's not super complex?

I have no idea at the moment but the LoginManager was rewritten and support of older wikis was given up.

Oh ok, no problem, thanks for helping with this!

I played a bit with this issue; this bug occurres randomly. It looks like the server is temporarily not available. This also means that Pywikibot 8 does not solve it.

In T326046#8493049, @Xqt wrote:

I played a bit with this issue; this bug occurres randomly. It looks like the server is temporarily not available. This also means that Pywikibot 8 does not solve it.

Oh ok, thanks for the heads up.

Change 874026 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@master] [IMPR] improve warning if a Non-JSON response was received from server

https://gerrit.wikimedia.org/r/874026

@Dimpizzy: With the patch given above I've adjusted the error warning a bit. This is all I can do in this case if the server is temporary down. The request will be retried several times.

In T326046#8493149, @Xqt wrote:

@Dimpizzy: With the patch given above I've adjusted the error warning a bit. This is all I can do in this case if the server is temporary down. The request will be retried several times.

You mean there's an issue with Miraheze? Everytime I run it, though, it does receive the api.php HTML page, and I can view the JSON formatted version in the browser everytime, so doesn't that mean the server is fine?

In T326046#8493150, @Dimpizzy wrote:

You mean there's an issue with Miraheze?

Yes.

Everytime I run it, though, it does receive the api.php HTML page, and I can view the JSON formatted version in the browser everytime, so doesn't that mean the server is fine?

I cannot confirm that this is every time when an api call was made. It happens sometimes. The simpliest way to test the connection might be (clear the cache previously):

> import pywikibot
> site = pywikibot.Site('comprehensibleinputwiki:en')
> site.version()
'1.38.4'

This works if the connection is ok and a json result is given. Otherwise that html page is shown.

@Xqt Ok, thanks, I'll create a bug report on Miraheze's phabricator then.

Edit: https://phabricator.miraheze.org/T10235

Is there a way to output and see the exact request sent to the server when it serves up HTML instead of JSON? This is what I spent a couple hours digging through the code for, to see if maybe the format=json was missing or something else was wrong, but I didn't have the skills to pinpoint where the request is made.

In T326046#8494108, @Dimpizzy wrote:

Is there a way to output and see the exact request sent to the server when it serves up HTML instead of JSON?

pywikibot/data/api/_requests.py#L715-L739 is where pywikibot outputs the warning for the non-JSON response. (You can also access the response for any request earliest after pywikibot/comms/http.py#L384-L386.) response.request is the PreparedRequest that was sent to the server. Look at its properties (headers/url).

Change 874026 merged by jenkins-bot:

[pywikibot/core@master] [IMPR] improve warning if a Non-JSON response was received from server

https://gerrit.wikimedia.org/r/874026

So Void over at Miraheze's Phabricator figured out that I needed to add this code to the family file, and now it logs in fine:

Hmm, I've been looking over your logs and the only suggestion I have at the moment is to force the family to use https. I'm wondering if perhaps the request data is being lost in a http -> https redirection? Really not sure at the moment. However, if I'm correct, all you need to do is add the following to the family:

def protocol(self, code):
    return 'HTTPS'

If I use the above code in the family file, then response.request.body equals:

action=query&meta=userinfo&uiprop=blockinfo%7Cgroups%7Chasmsg%7Cratelimits%7Crights&rawcontinue=&maxlag=5&format=json

If the https code isn't included, it equals None.

In T326046#8495095, @Dimpizzy wrote:

So Void over at Miraheze's Phabricator figured out that I needed to add this code to the family file, and now it logs in fine:

I've tested the generate-family-file script and indeed it added that method to the Family file. I think it is a better approach not to copy a family file example from mediawikiorg but either to use this script or use AutoFamily. This can be done in the user_config.py:

family_files['comprehensibleinputwiki'] = 'https://comprehensibleinput.miraheze.org'

Maybe we should have the 'https' scheme by default.

Anyway the request was redirected to the https url:

18:59:01.85 <<< Return value from request: <Response instance at 0x7f9c62f3f0a0 (repr suppressed)>
WARNING: Non-JSON response received from server comprehensibleinputwiki:en for url

https://comprehensibleinput.miraheze.org/w/api.php
The server may be down.
Status code: 200

The text message is:




MediaWiki API help - Comprehensible Input Wiki

In T326046#8495181, @Xqt wrote:

I've tested the generate-family-file script and indeed it added that method to the Family file. I think it is a better approach not to copy a family file example from mediawikiorg but either to use this script or use AutoFamily.

Ok, good to know, thanks.

Change 874903 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@master] [IMPR] return 'https' scheme with Family.protocol()

https://gerrit.wikimedia.org/r/874903

This is solved for me. I'm not sure if there's actually any bug to fix, so if not it can be closed.

Change 874903 merged by jenkins-bot:

[pywikibot/core@master] [IMPR] return 'https' scheme with Family.protocol()

https://gerrit.wikimedia.org/r/874903