Maniphest T206398

List of images without id causes SQL timeout
Open, Needs TriagePublic
Actions

Assigned To

None

Authored By

	Lokal_Profil
	Oct 6 2018, 3:47 PM

Description

From the log file

2018-10-06_14:36:30 Make a list of images without id...
Traceback (most recent call last):
  File "/data/project/heritage/heritage/erfgoedbot/images_of_monuments_without_id.py", line 419, in <module>
    main()
  File "/data/project/heritage/heritage/erfgoedbot/images_of_monuments_without_id.py", line 411, in main
    countryconfig, add_template, conn, cursor, conn2, cursor2))
  File "/data/project/heritage/heritage/erfgoedbot/images_of_monuments_without_id.py", line 65, in processCountry
    withTemplate = getMonumentsWithTemplate(countryconfig, conn2, cursor2)
  File "/data/project/heritage/heritage/erfgoedbot/images_of_monuments_without_id.py", line 262, in getMonumentsWithTemplate
    cursor.execute(query, (commonsTrackerCategory,))
  File "/mnt/nfs/labstore-secondary-tools-project/heritage/.venv/local/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
    self.errorhandler(self, exc, value)
  File "/mnt/nfs/labstore-secondary-tools-project/heritage/.venv/local/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
_mysql_exceptions.OperationalError: (2006, 'MySQL server has gone away')
CRITICAL: Closing network session.
<class '_mysql_exceptions.OperationalError'>
2018-10-06_14:58:55 Dump database...

Details

	Subject	Repo	Branch	Lines +/-
	[WIP]Make images_of_monuments_without_id ignore subcategories	labs/tools/heritage	master	+4 -6

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		JeanFred	T230769 Revive Erfgoedbot for WLM 2019
		Open		None	T206398 List of images without id causes SQL timeout

Event Timeline

Lokal_Profil created this task.Oct 6 2018, 3:47 PM

This is currently a query for all images in a category+subcategories with a sub-query of all images transcluding a certain template.

We could move the sub-query to a separate query and do the exclusion in python.
We could also drop the sub-categories bit if we consider the focus to rather be facilitating future categorisation.

It's unclear to me which would have the larger effect on the length of the SQL query.

Lokal_Profil mentioned this in T206034: Collect statistics for monuments_without_id reports.Oct 6 2018, 3:59 PM

@JeanFred any thoughts/suggestions? It seems like timing out SQL is constantly plaguing some part of this project.

Lokal_Profil added a parent task: T230769: Revive Erfgoedbot for WLM 2019.Aug 28 2019, 7:21 PM

I believe that the first country where the crash occurs is be-vlg. Looking at its base category it does seem to have a crazy amount of subcategories. Based on that unscientific analysis I suggest we only look at the base category itself.

Change 533105 had a related patch set uploaded (by Lokal Profil; owner: Lokal Profil):
[labs/tools/heritage@master] [WIP]Make images_of_monuments_without_id ignore subcategories

https://gerrit.wikimedia.org/r/533105

gerritbot added a project: Patch-For-Review.Aug 28 2019, 9:46 PM

In T206398#5447308, @gerritbot wrote:

Change 533105 had a related patch set uploaded (by Lokal Profil; owner: Lokal Profil):
[labs/tools/heritage@master] [WIP]Make images_of_monuments_without_id ignore subcategories

https://gerrit.wikimedia.org/r/533105

Making a stab at this to see if I can unblock part of T231484: Write instructions email about the monuments database and wikidata for the wlm-announce

SJu mentioned this in T242034: Process findings of Commons:Monuments database/Images without id.Jan 6 2020, 10:22 PM

Pppery edited projects, added Patch-Needs-Improvement; removed Patch-For-Review.Mar 27 2023, 2:09 AM

List of images without id causes SQL timeoutOpen, Needs TriagePublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

List of images without id causes SQL timeout
Open, Needs TriagePublic
Actions

Related Objects
Search...