MySQL page generator throws error on sock.close() on toolforge
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Dvorapa
	Feb 21 2019, 5:03 PM

Description

If I run MySQL page generator for longer time (here is the example of cca 1.5 day long run), it goes all the way through the retrieved list of pages and after bot.run() ends, bot throws the following error on toolforge (and therefore does not continue with the code following after bot.run()):

422844 pages read
0 pages written
Execution time: 1 days, 33969 seconds
Read operation time: 0 seconds
Script terminated by exception:

ERROR: AttributeError: 'NoneType' object has no attribute 'close'
Traceback (most recent call last):
  File "pwb.py", line 257, in <module>
    if not main():
  File "pwb.py", line 250, in main
    run_python_file(filename, [filename] + args, argvu, file_package)
  File "pwb.py", line 119, in run_python_file
    main_mod.__dict__)
  File "./nepodporovane-parametry-infoboxu.py", line 290, in <module>
    main()
  File "./nepodporovane-parametry-infoboxu.py", line 277, in main
    bot2.run()  # guess what it does
  File "/mnt/nfs/labstore-secondary-tools-project/pywikibot/public_html/core/pywikibot/bot.py", line 1479, in run
    for item in self.generator:
  File "/mnt/nfs/labstore-secondary-tools-project/pywikibot/public_html/core/pywikibot/pagegenerators.py", line 2777, in MySQLPageGenerator
    for row in row_gen:
  File "/mnt/nfs/labstore-secondary-tools-project/pywikibot/public_html/core/pywikibot/data/mysql.py", line 100, in mysql_query
    conn.close()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 734, in close
    sock.close()
AttributeError: 'NoneType' object has no attribute 'close'
CRITICAL: Exiting due to uncaught exception <class 'AttributeError'>

If I run it for a shorter time period, it works as expected, therefore it seems to me the connection to the database is lost? Or closed automatically after some amount of time? And it is closed by the database, pywikibot or pymysql?

The code looks simply like:

gen = pagegenerators.MySQLPageGenerator('select page_namespace, page_title from page where page_namespace like 0 and not page_is_redirect order by page_title')
bot = BasicBot(gen, **options)
bot.run()
page = pywikibot.Page(bot.site, 'Wikipedie:Údržbové seznamy/Nepodporované parametry infoboxů/seznam')
page.text += bot.list
page.save(summary='Robot: ' + bot.summary)
return True

Details

Subject	Repo	Branch	Lines +/-
[cleanup] Require 0.9.3 for pymysql	pywikibot/core	master	+3 -43
[dist] Show a deprecation warning when using PyMySQL < 0.7.11	pywikibot/core	master	+6 -0
[mysql] Revert deprecation warning when using PyMySQL < 0.7.11	pywikibot/core	master	+3 -12
[bugfix] Fix AttributeError on sock.close() on toolforge	pywikibot/core	master	+52 -8
[bugfix] require mysql >= 0.7.11	pywikibot/core	master	+2 -2

Customize query in gerrit

Event Timeline

Dvorapa created this task.Feb 21 2019, 5:03 PM

Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptFeb 21 2019, 5:03 PM

Dvorapa updated the task description. (Show Details)Feb 21 2019, 5:05 PM

Dvorapa updated the task description. (Show Details)

Dvorapa renamed this task from mysql.py throws error on sock.close() on toolforge to MySQL page generator throws error on sock.close() on toolforge.Feb 21 2019, 5:10 PM

Dvorapa updated the task description. (Show Details)

Dvorapa updated the task description. (Show Details)Feb 21 2019, 5:12 PM

Xqt added a subscriber: Mpaa.Feb 21 2019, 5:24 PM

Dvorapa updated the task description. (Show Details)Feb 21 2019, 5:35 PM

Which host (or via what means; bastion/grid/k8s; trusty/jessie/stretch) is the script executed on?

stretch, using jsub in crontab (I think it runs on gridengine?), tool dvorapabot

Are you sure that the pymysql is up-to-date (0.9.3)?
Can't find this line at https://github.com/PyMySQL/PyMySQL/blob/1ef6c587337bd6ff3272c2c4771948676fd2a9e6/pymysql/connections.py

I tried the script multiple times and all the time the same result. If I limit the MySQL query (limit 100, limit 1000), it does not happen. Only if I run the whole (422 000 articles), it takes more than 30 hours to complete the task for every article on cswiki. I switched that easy query to site.allpages() there, but we should definitely try to fix this for more complicated queries.

tools.dvorapabot@tools-sgebastion-07:~$ python3
Python 3.5.3 (default, Sep 27 2018, 17:25:39) 
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pymysql
>>> pymysql.__version__
'0.7.10.None'

I can't see that the close statement is wrong on our side. Can't you try to update the package?

def close(self):
    """Send the quit message and close the socket"""
    if self._closed:
        raise err.Error("Already closed")
    self._closed = True
    if self._sock is None:
        return
    send_data = struct.pack('<iB', 1, COMMAND.COM_QUIT)
    try:
        self._write_bytes(send_data)
    except Exception:
        pass
    finally:
        sock = self._sock
        self._sock = None
        self._rfile = None
        sock.close()

sock.close() complains about sock being None, sock = self._sock, so self._sock must have been None, and self._sock was checked in if self._sock is None: so self._sock must not have been None at that time. So this status muct have been changed either in another thread (unlikely) or in self._write_bytes.

In T216741#4972984, @Xqt wrote:

Can't you try to update the package?

No. You have to persuade debian package maintainers to update it. No other version is install-able under our current policy.

$ apt-cache policy python-pymysql
python-pymysql:
  Installed: 0.7.10-1
  Candidate: 0.7.10-1
  Version table:
 *** 0.7.10-1 500
        500 http://deb.debian.org/debian stretch/main amd64 Packages
        100 /var/lib/dpkg/status

def _write_bytes(self, data):
    self._sock.settimeout(self._write_timeout)
    try:
        self._sock.sendall(data)
    except IOError as e:
        self._force_close()
        raise err.OperationalError(
            CR.CR_SERVER_GONE_ERROR,
            "MySQL server has gone away (%r)" % (e,))

def _force_close(self):
    """Close connection without QUIT message"""
    if self._sock:
        try:
            self._sock.close()
        except:
            pass
    self._sock = None
    self._rfile = None

If close happened due to 'MySQL server has gone away', we should have received a OperationalError.

Well, we can use venv and run the script again with updated pip package. But I never used venv on forge, don't know how it works and the forge tutorials are not really good written. The 0.7.10 is 2 years old, maybe the've fixed that already

In T216741#4973067, @zhuyifei1999 wrote:

If close happened due to 'MySQL server has gone away', we should have received a OperationalError.

This was nullified by except Exception: pass

The fix was https://github.com/PyMySQL/PyMySQL/commit/0e01158fb8a204144c5adddde983bea2b3e4ff93, part of v0.7.11 release.

v0.7.11? Soooo close!

Okay, we can currently do nothing about, just wait until Debian developers will update the package.

bd808 subscribed.Feb 21 2019, 7:07 PM

In T216741#4973080, @Dvorapa wrote:

Well, we can use venv and run the script again with updated pip package. But I never used venv on forge, don't know how it works and the forge tutorials are not really good written. The 0.7.10 is 2 years old, maybe the've fixed that already

Docs are something we can fix! Is there a specific set of actions that you would like to see a tutorial for? It sounds like you are interested in one about how to run a continuous job on the job grid using a Python virtualenv?

@bd808 Well, this issue is from my point of view a little bit deep. To run Gerrit + Git review, there is nice tutorial with easy steps. On the other hand to gain acces to Toolforge, connect to it using ssh in terminal or file browser supporting ssh, run any type of command here, python commands specifically, create venv, maintain crontab, all these stuff I must've learn from my wiki colleague on wikimedia:cs workshop event as no tutorial on Wikitech was clear about what to do. I've already asked this somewhere (I can not find where).

But yes, for python venv I could only find https://wikitech.wikimedia.org/wiki/Help:Toolforge/FAQ#My_Tool_requires_a_package_that_is_not_currently_installed_in_Toolforge._How_can_I_add_it? and some bad tutorials for webservice (I don't want to run python on website)

In T216741#4973307, @Dvorapa wrote:

@bd808 Well, this issue is from my point of view a little bit deep. To run Gerrit + Git review, there is nice tutorial with easy steps. On the other hand to gain acces to Toolforge, connect to it using ssh in terminal or file browser supporting ssh, run any type of command here, python commands specifically, create venv, maintain crontab, all these stuff I must've learn from my wiki colleague on wikimedia:cs workshop event as no tutorial on Wikitech was clear about what to do. I've already asked this somewhere (I can not find where).

But yes, for python venv I could only find https://wikitech.wikimedia.org/wiki/Help:Toolforge/FAQ#My_Tool_requires_a_package_that_is_not_currently_installed_in_Toolforge._How_can_I_add_it? and some bad tutorials for webservice (I don't want to run python on website)

Is https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Flask_OAuth_tool a bad tutorial in your opinion, or just not relevant to your particular problem? Which of the many git+gerrit tutorials on mediawiki.org helped you most? Improving documentation for getting started in Toolforge is something I care a lot about, but I need feedback from folks like you who are having trouble finding what they need to prioritize what to work on and how to present the material.

In T216741#4973335, @bd808 wrote:

In T216741#4973307, @Dvorapa wrote:

@bd808 Well, this issue is from my point of view a little bit deep. To run Gerrit + Git review, there is nice tutorial with easy steps. On the other hand to gain acces to Toolforge, connect to it using ssh in terminal or file browser supporting ssh, run any type of command here, python commands specifically, create venv, maintain crontab, all these stuff I must've learn from my wiki colleague on wikimedia:cs workshop event as no tutorial on Wikitech was clear about what to do. I've already asked this somewhere (I can not find where).

But yes, for python venv I could only find https://wikitech.wikimedia.org/wiki/Help:Toolforge/FAQ#My_Tool_requires_a_package_that_is_not_currently_installed_in_Toolforge._How_can_I_add_it? and some bad tutorials for webservice (I don't want to run python on website)

Is https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Flask_OAuth_tool a bad tutorial in your opinion, or just not relevant to your particular problem? Which of the many git+gerrit tutorials on mediawiki.org helped you most? Improving documentation for getting started in Toolforge is something I care a lot about, but I need feedback from folks like you who are having trouble finding what they need to prioritize what to work on and how to present the material.

As I was unable to run venv on many attempts and I learned how to use Toolforge from my experienced colleague, I definitively want to help and share more details about tutorials for Toolforge. But not here as it is off-topic to the task. I also wanted to write some Help:Toolforge/tl;dr for complete beginners page, but had no time to summarize all my experiences into a good manual. I referred to https://www.mediawiki.org/wiki/Gerrit/Tutorial manual btw

In T216741#4973335, @bd808 wrote:

Is https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Flask_OAuth_tool a bad tutorial in your opinion, or just not relevant to your particular problem?

It focuses on python webservice on k8s. $ webservice --backend=kubernetes python shell would be misleading. It also does not contain information on how to load the venv from a cron job submitthing to grid.

It's a good tutorial for its use case, but it needs quite a bit digestion for one to setup a venv for a grid cron job,

We can catch the exception in our mysql.py like

with contextlib.suppress(AttributeError):

conn.close()

@bd808 https://wikitech.wikimedia.org/wiki/Help:Toolforge/Pywikibot is the best Toolforge manual from all Toolforge manuals. It just misses some stuff about how to create a tool and maybe a little more detail about how to run webservice (but it is not in the scope of the page)

• GTirloni added a project: cloud-services-team (Kanban).Mar 24 2019, 11:00 AM

bd808 edited projects, added Upstream; removed cloud-services-team (Kanban).Aug 8 2019, 4:15 AM

Xqt changed the task status from Stalled to Open.Nov 10 2021, 10:20 AM

Xqt claimed this task.

Change 737873 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@master] [bugfix] require mysql >= 0.7.11

https://gerrit.wikimedia.org/r/737873

gerritbot added a project: Patch-For-Review.Nov 10 2021, 10:24 AM

Xqt triaged this task as Medium priority.Nov 10 2021, 10:24 AM

Solved upstream already.

Xqt lowered the priority of this task from Medium to Lowest.Nov 27 2021, 9:21 AM

Change 737873 merged by jenkins-bot:

[pywikibot/core@master] [bugfix] require mysql >= 0.7.11

https://gerrit.wikimedia.org/r/737873

Maintenance_bot removed a project: Patch-For-Review.Nov 27 2021, 10:10 AM

Change 742198 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@master] [bugfix] Fix AttributeError on sock.close() on toolforge

https://gerrit.wikimedia.org/r/742198

gerritbot added a project: Patch-For-Review.Nov 27 2021, 1:46 PM

Xqt removed a project: Upstream.Nov 27 2021, 3:56 PM

Change 742198 merged by jenkins-bot:

[pywikibot/core@master] [bugfix] Fix AttributeError on sock.close() on toolforge

https://gerrit.wikimedia.org/r/742198

Maintenance_bot removed a project: Patch-For-Review.Dec 4 2021, 2:10 AM

Xqt closed this task as Resolved.Dec 4 2021, 7:41 AM

WARNING: /mnt/nfs/labstore-secondary-tools-project/dvorapabot/pywikibot/pywikibot/pagegenerators.py:2815: FutureWarning: pymysql package release 0.7.10 is deprecated since release 7.0.0; use pymysql >= 0.7.11 instead.

So the solution is to make sql requests on Toolforge fail since 7.0.0 because of unreachable dependency requirement? (does it make the whole Pagegenerators module fail btw?) Anyway I suggest a revert

In T216741#7531673, @Xqt wrote:

Solved upstream already.

We know:

In T216741#4973090, @zhuyifei1999 wrote:

The fix was https://github.com/PyMySQL/PyMySQL/commit/0e01158fb8a204144c5adddde983bea2b3e4ff93, part of v0.7.11 release.

In T216741#4973157, @Dvorapa wrote:

Okay, we can currently do nothing about, just wait until Debian developers will update the package.

Dvorapa reopened this task as Open.Dec 9 2021, 9:43 PM

We can either ignore dependency check in Toolforge for pymysql, or wait until Debian devs add 0.7.11 to Stretch (never gonna happen), or wait until Toolforge updates to Buster (hopefully soon), or go back and revert the change, or release 7.0.0 after Toolforge updates to Buster. No solution is ideal but every solution is better than the current state.

The current state is that the dependency is checked and a warning is thrown but the script uses the backported fix.

Change 750799 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@master] [mysql] Update requirement for mysql

https://gerrit.wikimedia.org/r/750799

gerritbot added a project: Patch-For-Review.Jan 2 2022, 6:38 PM

Xqt added a subscriber: JJMC89.Jan 3 2022, 9:10 AM

Xqt closed this task as Resolved.Jan 9 2022, 6:54 PM

Change 750799 merged by jenkins-bot:

[pywikibot/core@master] [mysql] Revert deprecation warning when using PyMySQL < 0.7.11

https://gerrit.wikimedia.org/r/750799

Maintenance_bot removed a project: Patch-For-Review.Jan 9 2022, 7:10 PM

Change 792993 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@master] [dist] Show a deprecation warning when using PyMySQL < 0.7.11"

https://gerrit.wikimedia.org/r/792993

gerritbot added a project: Patch-For-Review.May 18 2022, 9:12 AM

Change 792993 merged by jenkins-bot:

[pywikibot/core@master] [dist] Show a deprecation warning when using PyMySQL < 0.7.11

https://gerrit.wikimedia.org/r/792993

Change 835153 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@master] [cleanup] Require 0.9.3 for pymysql

https://gerrit.wikimedia.org/r/835153

Change 835153 merged by jenkins-bot: