The main problem is that the default is None in config2.py.

This task needs a little analysis to determine why the default is None, and what this currently does as the default for a user who doesnt modify this setting.

From memory, weblinkchecker.py needs user-agent faking to default to enabled, as many sites actively reject user-agents with bot-like parts, such as the string requests. (Ah, T71204: weblinkchecker.py uses a fake user-agent)

But we should not use a fake user-agent unless it is necessary.

From T71204, the second part wasnt done:

Faking the user-agent should be an option, default disabled, or only used for servers known to be problematic.

Maybe we need a list of servers that require a fake user-agent, so that it can be disabled by default in config2.py.

This task needs a little analysis to determine why the default is None

None behaves the same way as True (if set a value, UA becomes that value. if False, default UA returned). The default is Introduced since the patch was submitted and unchanged since then.

Potential actions:

• Set fake_user_agent value to false by default.
• Change reflinks and weblinkchecker to retry with fake UA if first request gets rejected (need to decide what errors/status codes constitute as a rejection).
• Maintain a list of problematic sites in the configuration file (or should this be hosted on WikiTech?)
• Split config entries to separate behavior when a site is/isn't in the problematic list.

I like these two combined:

• Change reflinks and weblinkchecker to retry with fake UA if first request gets rejected (need to decide what errors/status codes constitute as a rejection).
• Maintain a list of problematic sites in the configuration file (or should this be hosted on WikiTech?)

i.e. we would add a use_fake_user_agent parameter to methods in http.comms, which is False by default, True forces a fake user-agent, and None means automatic use of fallback logic (use fake if in the config list, otherwise try with real user-agent, and fallback to fake).

Then we can remove the fake_user_agent config option.

That could be an extra feature.

It would also be nice if the config list allowed literal strings: when connecting to this server, use this user-agent, rather than a random one.

In T152075#2837739, @jayvdb wrote:

i.e. we would add a use_fake_user_agent parameter to methods in http.comms, which is False by default, True forces a fake user-agent, and None means automatic use of fallback logic (use fake if in the config list, otherwise try with real user-agent, and fallback to fake).

http.comms seems to be used only to get the user_agent, but not for making the requests themselves. Shouldn't this parameter and logic be implemented in the scripts that need fake UAs (i.e. reflinks and weblinkchecker) instead?

It would also be nice if the config list allowed literal strings: when connecting to this server, use this user-agent, rather than a random one.

Yup. This list would also be better if it is dynamically updated, instead of hard-coded in config2.py, so maybe we should put this on a WikiTech page. This make user-agent specifications like this less messy as well.

In T152075#2837748, @Dargasea wrote:

In T152075#2837739, @jayvdb wrote:

i.e. we would add a use_fake_user_agent parameter to methods in http.comms, which is False by default, True forces a fake user-agent, and None means automatic use of fallback logic (use fake if in the config list, otherwise try with real user-agent, and fallback to fake).

http.comms seems to be used only to get the user_agent, but not for making the requests themselves. Shouldn't this parameter and logic be implemented in the scripts that need fake UAs (i.e. reflinks and weblinkchecker) instead?

See pywikibot.comms.http.request

It would also be nice if the config list allowed literal strings: when connecting to this server, use this user-agent, rather than a random one.

Yup. This list would also be better if it is dynamically updated, instead of hard-coded in config2.py, so maybe we should put this on a WikiTech page. This make user-agent specifications like this less messy as well.

Nah, it should be part of the config, but at the end of a bot run it should warn the user that they should add the extra sites to the config.
We have been developing "loading config from the wiki", using JSON. I'll look into how that is going. We can continue with that after this task.

https://codein.withgoogle.com/tasks/4719662784839680/ will be the GCI task for this, when it is publish - waiting for admin to press the button

Update:

Maintain a list of problematic sites in the configuration file

This has been worked on, but:

automatic use of fallback logic

I am debating if this should be implemented. Currently I'm thinking of HTTP status codes, but I thought if a server returns something like 403 Forbidden, we should honour it instead. Also in T113596 the website returned 200, but with invalid content.

Unless there is a better way to detect failures, I'm considering submitting a first patch first, implementing per-module toggles (on/off for reflinks and weblinkchecker) and default behaviour exceptions (list of problematic sites part).

Currently I'm thinking of HTTP status codes, but I thought if a server returns something like 403 Forbidden, we should honour it instead.

Yea, that is reasonable.

Also in T113596 the website returned 200, but with invalid content.

Right, the user needs to explicitly add an entry; automatic fallback wont help with that one.

Unless there is a better way to detect failures

The best way to detect these websites is to try with the normal user-agent, then try a generated user-agent. The time to do this extra check is when the connection to the site is already established and open.

I'm considering submitting a first patch first, implementing per-module toggles .. and default behaviour exceptions ...

No worries splitting this into chunks, and stopping after each chunk to reconsider about whether it is wise to add more voodoo.
The first part is sufficient for a GCI task approval. I'm always happy to add more tasks to allow extra improvements to be done.

Change 325241 had a related patch set uploaded (by Dargasia):
Improve fake user agent usage control

https://gerrit.wikimedia.org/r/325241

Change 325241 abandoned by Dargasia:
Improve fake user agent usage control

Reason:
John: mistakes were made - screwed up my branch. Allow me some time to fix it. I'll start another Change later to make things less messy. Sorry for the trouble.

https://gerrit.wikimedia.org/r/325241

Change 325241 restored by John Vandenberg:
Improve fake user agent usage control

Reason:
no worries.

The biggest benefit of Gerrit is it is possible to re-do your branch.

If you add "Change-Id: I28594f" to your new commit message, on any branch, even with a different branch name, anywhere, git review will figure it all out magically.

https://gerrit.wikimedia.org/r/325241

Change 325241 merged by jenkins-bot:
Improve fake user agent usage control

https://gerrit.wikimedia.org/r/325241