When reflinks.py works, sometimes web pages return information that "You use an outdated browser" as a title of the page.
e.g. http://base.consultant.ru/cons/cgi/online.cgi?req=doc;base=MOB;n=212237;fld=134;from=1-34;rnd=0.16269673267379403 redirects to http://base.consultant.ru/cons/static4012_00_88_184768/invalid_browser.htm?ext=1; when the user-agent is not recognised. (the title of that page is "Устаревший или неподдерживаемый веб-обозреватель" which in Russian means "outdated browser")
reflinks.py does not set a user agent for the bot. It uses urllib2, which has a default user-agent of "Python-urllib/2.6" (on Python 2.6) according to https://docs.python.org/2/library/urllib2.html.
The problem can be seen and tested with the following command, which uses https://en.wikipedia.org/wiki/User:John_Vandenberg/test_T113596 as a test page. Notice the script suggests adding a title "Устаревший или неподдерживаемый веб-обозреватель", which is incorrect.
$ python pwb.py reflinks -family:wikipedia -lang:en -page:User:John_Vandenberg/test_T113596 No handlers could be found for logger "pywiki" Retrieving 1 pages from wikipedia:en. >>> User:John Vandenberg/test T113596 <<< @@ -3 +3 @@ - <ref>http://base.consultant.ru/cons/cgi/online.cgi?req=doc;base=MOB;n=212237;fld=134;from=1-34;rnd=0.16269673267379403</ref> + <ref>[http://base.consultant.ru/cons/cgi/online.cgi?req=doc;base=MOB;n=212237;fld=134;from=1-34;rnd=0.16269673267379403 Устаревший или неподдерживаемый веб-обозреватель<!-- Bot generated title -->]</ref> Edit summary: Bot: Converting bare references, using ref names to avoid duplicates, see [[mw:Manual:Pywikibot/refLinks|FAQ]] Do you want to accept these changes? ([y]es, [N]o, [a]ll, [q]uit): n
The title of http://base.consultant.ru/cons/cgi/online.cgi?req=doc;base=MOB;n=212237;fld=134;from=1-34;rnd=0.16269673267379403 should be "Постановление Губернатора МО от 03.07.2015 N 282-ПГ "Об объединении рабочего поселка Львовский Подольского района Московской области и города Подольска Московской области" - КонсультантПлюс:"
Regarding fixing the bug
Pywikibot has a function pywikibot.comms.http.user_agent which produces a slightly better user agent, however it is also not accepted by http://base.consultant.ru
However http://base.consultant.ru does provide the correct resource when the user-agent is spoofed to be Mozilla/5.0 (X11; U; Linux i686; de; rv:1.8) Gecko/20051128 SUSE/1.5-0.1 Firefox/1.5
It is the Firefox/1.5 which makes it work; e.g.
wget -O 'out.html' --user-agent="Firefox/1.5 Pywikibot/2.0rc4 (g5802) httplib2/0.9.1 Python/2.7.10.final.0" "http://base.consultant.ru/cons/cgi/online.cgi?req=doc;base=MOB;n=212237;fld=134;from=1-34;rnd=0.16269673267379403"
As a result, a custom/spoofed user agent is needed for reflinks.py, and possible also weblinkchecker.py (see T71204).
One design problem is that each website may have its own user-agent rejection rules, so to properly solve this problem, pywikibot needs to support a custom user-agent for each website.
This could be a configuration item in pywikibot/config2.py, however that would become out of date and could cause sniffers to reject the user-agent.
Another approach is to obtain the user-agent using fake-useragent or browseragents, or similar.
An acceptable good first task solution is to have one configuration variable, which defaults to blank. When blank, if fake-useragent or browseragents are installed, prefill the configuration variable with a spoof user-agent from either package. If blank and neither is installed, use pywikibot.comms.http.user_agent.