When `reflinks.py` works, sometimes web pages return information that "You use an outdated browser" as a title of the page.
e.g. http://base.consultant.ru/cons/cgi/online.cgi?req=doc;base=MOB;n=212237;fld=134;from=1-34;rnd=0.16269673267379403 redirects to http://base.consultant.ru/cons/static4012_00_88_184768/invalid_browser.htm?ext=1; when the user-agent is not recognised. (the title of that page is "Устаревший или неподдерживаемый веб-обозреватель" which in Russian means "outdated browser")
`reflinks.py` does not set a user agent for the bot. It uses `urllib2`, which has a default user-agent of "Python-urllib/2.6" (on Python 2.6) according to https://docs.python.org/2/library/urllib2.html.
The problem can be seen and tested with the following command, which uses https://en.wikipedia.org/wiki/User:John_Vandenberg/test_T113596 as a test page. Notice the script suggests adding a title "Устаревший или неподдерживаемый веб-обозреватель", which is incorrect.
```
$ python pwb.py reflinks -family:wikipedia -lang:en -page:User:John_Vandenberg/test_T113596
No handlers could be found for logger "pywiki"
Retrieving 1 pages from wikipedia:en.
>>> User:John Vandenberg/test T113596 <<<
@@ -3 +3 @@
- <ref>http://base.consultant.ru/cons/cgi/online.cgi?req=doc;base=MOB;n=212237;fld=134;from=1-34;rnd=0.16269673267379403</ref>
+ <ref>[http://base.consultant.ru/cons/cgi/online.cgi?req=doc;base=MOB;n=212237;fld=134;from=1-34;rnd=0.16269673267379403 Устаревший или неподдерживаемый веб-обозреватель<!-- Bot generated title -->]</ref>
Edit summary: Bot: Converting bare references, using ref names to avoid duplicates, see [[mw:Manual:Pywikibot/refLinks|FAQ]]
Do you want to accept these changes? ([y]es, [N]o, [a]ll, [q]uit): n
```
Regarding fixing the bug
==============
Pywikibot has a function `pywikibot.comms.http.user_agent` which produces a slightly better user agent, however it is also not accepted by http://base.consultant.ru
However http://base.consultant.ru does provide the correct resource when the user-agent is spoofed to be `Mozilla/5.0 (X11; U; Linux i686; de; rv:1.8) Gecko/20051128 SUSE/1.5-0.1 Firefox/1.5`
It is the `Firefox/1.5` which makes it work; e.g.
```
wget -O 'out.html' --user-agent="Firefox/1.5 Pywikibot/2.0rc4 (g5802) httplib2/0.9.1 Python/2.7.10.final.0" "http://base.consultant.ru/cons/cgi/online.cgi?req=doc;base=MOB;n=212237;fld=134;from=1-34;rnd=0.16269673267379403"
```
As a result, a custom/spoofed user agent is needed for `reflinks.py`, and possible also `weblinkchecker.py` (see T71204).
One design problem is that each website may have its own user-agent rejection rules, so to properly solve this problem, pywikibot needs to support a custom user-agent for each website.
This could be a configuration item in `pywikibot/config2.py`, however that would become out of date and could cause sniffers to reject the user-agent.
Another approach is to obtain the user-agent using [[https://pypi.python.org/pypi/fake-useragent | fake-useragent]] or [[https://pypi.python.org/pypi/browseragents | browseragents]], or similar.
An acceptable #easy solution is to have one configuration variable, which defaults to blank. When blank, **if** fake-useragent or browseragents are installed, prefill the configuration variable with a spoof user-agent from either package. If blank and neither is installed, use `pywikibot.comms.http.user_agent`.