Page MenuHomePhabricator

MySQL page generator throws error on sock.close() on toolforge
Open, Stalled, Needs TriagePublic

Description

If I run MySQL page generator for longer time (here is the example of cca 1.5 day long run), it goes all the way through the retrieved list of pages and after bot.run() ends, bot throws the following error on toolforge (and therefore does not continue with the code following after bot.run()):

422844 pages read
0 pages written
Execution time: 1 days, 33969 seconds
Read operation time: 0 seconds
Script terminated by exception:

ERROR: AttributeError: 'NoneType' object has no attribute 'close'
Traceback (most recent call last):
  File "pwb.py", line 257, in <module>
    if not main():
  File "pwb.py", line 250, in main
    run_python_file(filename, [filename] + args, argvu, file_package)
  File "pwb.py", line 119, in run_python_file
    main_mod.__dict__)
  File "./nepodporovane-parametry-infoboxu.py", line 290, in <module>
    main()
  File "./nepodporovane-parametry-infoboxu.py", line 277, in main
    bot2.run()  # guess what it does
  File "/mnt/nfs/labstore-secondary-tools-project/pywikibot/public_html/core/pywikibot/bot.py", line 1479, in run
    for item in self.generator:
  File "/mnt/nfs/labstore-secondary-tools-project/pywikibot/public_html/core/pywikibot/pagegenerators.py", line 2777, in MySQLPageGenerator
    for row in row_gen:
  File "/mnt/nfs/labstore-secondary-tools-project/pywikibot/public_html/core/pywikibot/data/mysql.py", line 100, in mysql_query
    conn.close()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 734, in close
    sock.close()
AttributeError: 'NoneType' object has no attribute 'close'
CRITICAL: Exiting due to uncaught exception <class 'AttributeError'>

If I run it for a shorter time period, it works as expected, therefore it seems to me the connection to the database is lost? Or closed automatically after some amount of time? And it is closed by the database, pywikibot or pymysql?

The code looks simply like:

gen = pagegenerators.MySQLPageGenerator('select page_namespace, page_title from page where page_namespace like 0 and not page_is_redirect order by page_title')
bot = BasicBot(gen, **options)
bot.run()
page = pywikibot.Page(bot.site, 'Wikipedie:Údržbové seznamy/Nepodporované parametry infoboxů/seznam')
page.text += bot.list
page.save(summary='Robot: ' + bot.summary)
return True

Event Timeline

Dvorapa created this task.Feb 21 2019, 5:03 PM
Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptFeb 21 2019, 5:03 PM
Dvorapa updated the task description. (Show Details)Feb 21 2019, 5:05 PM
Dvorapa updated the task description. (Show Details)
Dvorapa updated the task description. (Show Details)
Dvorapa renamed this task from mysql.py throws error on sock.close() on toolforge to MySQL page generator throws error on sock.close() on toolforge.Feb 21 2019, 5:10 PM
Dvorapa updated the task description. (Show Details)
Dvorapa updated the task description. (Show Details)Feb 21 2019, 5:12 PM
Xqt added a subscriber: Mpaa.Feb 21 2019, 5:24 PM
Dvorapa updated the task description. (Show Details)Feb 21 2019, 5:35 PM

Which host (or via what means; bastion/grid/k8s; trusty/jessie/stretch) is the script executed on?

Dvorapa added a comment.EditedFeb 21 2019, 5:49 PM

stretch, using jsub in crontab (I think it runs on gridengine?), tool dvorapabot

Xqt added a subscriber: Xqt.Feb 21 2019, 5:54 PM

Are you sure that the pymysql is up-to-date (0.9.3)?
Can't find this line at https://github.com/PyMySQL/PyMySQL/blob/1ef6c587337bd6ff3272c2c4771948676fd2a9e6/pymysql/connections.py

Dvorapa added a comment.EditedFeb 21 2019, 5:54 PM

I tried the script multiple times and all the time the same result. If I limit the MySQL query (limit 100, limit 1000), it does not happen. Only if I run the whole (422 000 articles), it takes more than 30 hours to complete the task for every article on cswiki. I switched that easy query to site.allpages() there, but we should definitely try to fix this for more complicated queries.

tools.dvorapabot@tools-sgebastion-07:~$ python3
Python 3.5.3 (default, Sep 27 2018, 17:25:39) 
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pymysql
>>> pymysql.__version__
'0.7.10.None'
Xqt added a comment.Feb 21 2019, 6:19 PM

I can't see that the close statement is wrong on our side. Can't you try to update the package?

def close(self):
    """Send the quit message and close the socket"""
    if self._closed:
        raise err.Error("Already closed")
    self._closed = True
    if self._sock is None:
        return
    send_data = struct.pack('<iB', 1, COMMAND.COM_QUIT)
    try:
        self._write_bytes(send_data)
    except Exception:
        pass
    finally:
        sock = self._sock
        self._sock = None
        self._rfile = None
        sock.close()

sock.close() complains about sock being None, sock = self._sock, so self._sock must have been None, and self._sock was checked in if self._sock is None: so self._sock must not have been None at that time. So this status muct have been changed either in another thread (unlikely) or in self._write_bytes.

Can't you try to update the package?

No. You have to persuade debian package maintainers to update it. No other version is install-able under our current policy.

$ apt-cache policy python-pymysql
python-pymysql:
  Installed: 0.7.10-1
  Candidate: 0.7.10-1
  Version table:
 *** 0.7.10-1 500
        500 http://deb.debian.org/debian stretch/main amd64 Packages
        100 /var/lib/dpkg/status
def _write_bytes(self, data):
    self._sock.settimeout(self._write_timeout)
    try:
        self._sock.sendall(data)
    except IOError as e:
        self._force_close()
        raise err.OperationalError(
            CR.CR_SERVER_GONE_ERROR,
            "MySQL server has gone away (%r)" % (e,))
def _force_close(self):
    """Close connection without QUIT message"""
    if self._sock:
        try:
            self._sock.close()
        except:
            pass
    self._sock = None
    self._rfile = None

If close happened due to 'MySQL server has gone away', we should have received a OperationalError.

Dvorapa added a comment.EditedFeb 21 2019, 6:38 PM

Well, we can use venv and run the script again with updated pip package. But I never used venv on forge, don't know how it works and the forge tutorials are not really good written. The 0.7.10 is 2 years old, maybe the've fixed that already

If close happened due to 'MySQL server has gone away', we should have received a OperationalError.

This was nullified by except Exception: pass

v0.7.11? Soooo close!

Dvorapa changed the task status from Open to Stalled.Feb 21 2019, 6:53 PM

Okay, we can currently do nothing about, just wait until Debian developers will update the package.

bd808 added a subscriber: bd808.Feb 21 2019, 7:07 PM
bd808 added a comment.Feb 21 2019, 7:10 PM

Well, we can use venv and run the script again with updated pip package. But I never used venv on forge, don't know how it works and the forge tutorials are not really good written. The 0.7.10 is 2 years old, maybe the've fixed that already

Docs are something we can fix! Is there a specific set of actions that you would like to see a tutorial for? It sounds like you are interested in one about how to run a continuous job on the job grid using a Python virtualenv?

Dvorapa added a comment.EditedFeb 21 2019, 7:21 PM

@bd808 Well, this issue is from my point of view a little bit deep. To run Gerrit + Git review, there is nice tutorial with easy steps. On the other hand to gain acces to Toolforge, connect to it using ssh in terminal or file browser supporting ssh, run any type of command here, python commands specifically, create venv, maintain crontab, all these stuff I must've learn from my wiki colleague on wikimedia:cs workshop event as no tutorial on Wikitech was clear about what to do. I've already asked this somewhere (I can not find where).

But yes, for python venv I could only find https://wikitech.wikimedia.org/wiki/Help:Toolforge/FAQ#My_Tool_requires_a_package_that_is_not_currently_installed_in_Toolforge._How_can_I_add_it? and some bad tutorials for webservice (I don't want to run python on website)

bd808 added a comment.Feb 21 2019, 7:30 PM

@bd808 Well, this issue is from my point of view a little bit deep. To run Gerrit + Git review, there is nice tutorial with easy steps. On the other hand to gain acces to Toolforge, connect to it using ssh in terminal or file browser supporting ssh, run any type of command here, python commands specifically, create venv, maintain crontab, all these stuff I must've learn from my wiki colleague on wikimedia:cs workshop event as no tutorial on Wikitech was clear about what to do. I've already asked this somewhere (I can not find where).
But yes, for python venv I could only find https://wikitech.wikimedia.org/wiki/Help:Toolforge/FAQ#My_Tool_requires_a_package_that_is_not_currently_installed_in_Toolforge._How_can_I_add_it? and some bad tutorials for webservice (I don't want to run python on website)

Is https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Flask_OAuth_tool a bad tutorial in your opinion, or just not relevant to your particular problem? Which of the many git+gerrit tutorials on mediawiki.org helped you most? Improving documentation for getting started in Toolforge is something I care a lot about, but I need feedback from folks like you who are having trouble finding what they need to prioritize what to work on and how to present the material.

@bd808 Well, this issue is from my point of view a little bit deep. To run Gerrit + Git review, there is nice tutorial with easy steps. On the other hand to gain acces to Toolforge, connect to it using ssh in terminal or file browser supporting ssh, run any type of command here, python commands specifically, create venv, maintain crontab, all these stuff I must've learn from my wiki colleague on wikimedia:cs workshop event as no tutorial on Wikitech was clear about what to do. I've already asked this somewhere (I can not find where).
But yes, for python venv I could only find https://wikitech.wikimedia.org/wiki/Help:Toolforge/FAQ#My_Tool_requires_a_package_that_is_not_currently_installed_in_Toolforge._How_can_I_add_it? and some bad tutorials for webservice (I don't want to run python on website)

Is https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Flask_OAuth_tool a bad tutorial in your opinion, or just not relevant to your particular problem? Which of the many git+gerrit tutorials on mediawiki.org helped you most? Improving documentation for getting started in Toolforge is something I care a lot about, but I need feedback from folks like you who are having trouble finding what they need to prioritize what to work on and how to present the material.

As I was unable to run venv on many attempts and I learned how to use Toolforge from my experienced colleague, I definitively want to help and share more details about tutorials for Toolforge. But not here as it is off-topic to the task. I also wanted to write some Help:Toolforge/tl;dr for complete beginners page, but had no time to summarize all my experiences into a good manual. I referred to https://www.mediawiki.org/wiki/Gerrit/Tutorial manual btw

Is https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Flask_OAuth_tool a bad tutorial in your opinion, or just not relevant to your particular problem?

It focuses on python webservice on k8s. $ webservice --backend=kubernetes python shell would be misleading. It also does not contain information on how to load the venv from a cron job submitthing to grid.

It's a good tutorial for its use case, but it needs quite a bit digestion for one to setup a venv for a grid cron job,

Xqt added a comment.Feb 21 2019, 8:04 PM

We can catch the exception in our mysql.py like

with contextlib.suppress(AttributeError):

conn.close()

@bd808 https://wikitech.wikimedia.org/wiki/Help:Toolforge/Pywikibot is the best Toolforge manual from all Toolforge manuals. It just misses some stuff about how to create a tool and maybe a little more detail about how to run webservice (but it is not in the scope of the page)