Page MenuHomePhabricator

Revive Erfgoedbot for WLM 2019
Closed, ResolvedPublic

Description

So It looks like Erfgoedbot (some parts at least) hasn't run successfully for some time.

I've attached a cleaned up log below but the main issue seems to be calls to mysql-connect which doesn't exist in PHP 7.0. This is likely related to T216364: Upgrade heritage to Strech where we spotted similar things.

There also seems to be some issues with invalid CSFR tokens (might it have expired during the bot un) and of course the ever present MySQL server has gone away.

For categorization it looks like the mapping we use for ASI Monument is wrong since there is a trove of Errors trying to interpret the id (e.g. N-AP-27) as a DOUBLE and failing. The list where the example entry can be found is w:en:List_of_Monuments_of_National_Importance_in_Andhra_Pradesh.

12019-08-18_03:00:25 Starting full monument update.
22019-08-18_03:00:25 Load changes to monuments_config...
32019-08-18_03:00:53 Recreating the source tables...
42019-08-18_03:00:59 Full source database update...
5WARNING: /data/project/heritage/heritage/erfgoedbot/update_database.py:449: Warning: Data truncated for column 'site' at row 1
6 (countryconfig.get('lang'), page.title(True), params[0]))
7Sleeping for 5.0 seconds, 2019-08-18 04:17:49
8Sleeping for 5.0 seconds, 2019-08-18 07:03:12
9WARNING: API error badtoken: Invalid CSRF token.
10WARNING: API error badtoken: Invalid CSRF token.
11No job 'categorize_images' is currently queued or running
122019-08-18_13:16:41 Update monuments_all table...
132019-08-18_13:22:09 Make statistics...
14WARNING: /data/project/heritage/heritage/erfgoedbot/database_statistics.py:28: Warning: Truncated incorrect DOUBLE value: ''
15 cursor.execute(query)
162019-08-18_13:22:56 Make more detailed statistics...
17PHP Fatal error: Uncaught Error: Call to undefined function mysql_connect() in /mnt/nfs/labstore-secondary-tools-project/heritage/heritage/api/includes/Database.php:150
18Stack trace:
19#0 /mnt/nfs/labstore-secondary-tools-project/heritage/heritage/maintenance/_buildStats.php(25): Database::define('tools-db', 's51138__heritag...', 'XXXXXX', 'XXXXX')
20#1 {main}
21 thrown in /mnt/nfs/labstore-secondary-tools-project/heritage/heritage/api/includes/Database.php on line 150
22Fatal error: Uncaught Error: Call to undefined function mysql_connect() in /mnt/nfs/labstore-secondary-tools-project/heritage/heritage/api/includes/Database.php on line 150
23Error: Call to undefined function mysql_connect() in /mnt/nfs/labstore-secondary-tools-project/heritage/heritage/api/includes/Database.php on line 150
24Call Stack:
25 0.0073 393904 1. {main}() /mnt/nfs/labstore-secondary-tools-project/heritage/heritage/maintenance/_buildStats.php:0
26 0.0233 430128 2. Database::define() /mnt/nfs/labstore-secondary-tools-project/heritage/heritage/maintenance/_buildStats.php:25
272019-08-18_13:22:56 Update unused images list...
282019-08-18_13:55:03 Make a list of missing commonscat links...
292019-08-18_14:01:01 Make a list of images without id...
30Traceback (most recent call last):
31 File "/data/project/heritage/heritage/erfgoedbot/images_of_monuments_without_id.py", line 419, in <module>
32 main()
33 File "/data/project/heritage/heritage/erfgoedbot/images_of_monuments_without_id.py", line 411, in main
34 countryconfig, add_template, conn, cursor, conn2, cursor2))
35 File "/data/project/heritage/heritage/erfgoedbot/images_of_monuments_without_id.py", line 62, in processCountry
36 countryconfig.get('country'), countryconfig.get('lang'), conn, cursor)
37 File "/data/project/heritage/heritage/erfgoedbot/images_of_monuments_without_id.py", line 193, in getMonumentsWithPhoto
38 cursor.execute(query, (countrycode, lang))
39 File "/data/project/heritage/.venv/local/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
40 self.errorhandler(self, exc, value)
41 File "/data/project/heritage/.venv/local/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
42 raise errorclass, errorvalue
43_mysql_exceptions.OperationalError: (2006, 'MySQL server has gone away')
44CRITICAL: Closing network session.
45<class '_mysql_exceptions.OperationalError'>
462019-08-18_14:33:04 Dump database...
472019-08-18_14:38:21 Replace the sql file atomically...
482019-08-18_14:38:21 Refill prox_search table...
49PHP Fatal error: Uncaught Error: Call to undefined function mysql_connect() in /mnt/nfs/labstore-secondary-tools-project/heritage/heritage/api/includes/Database.php:150
50Stack trace:
51#0 /mnt/nfs/labstore-secondary-tools-project/heritage/heritage/api/common.php(76): Database::define('tools-db', 's51138__heritag...', 'XXXXX', 'XXXXX')
52#1 /mnt/nfs/labstore-secondary-tools-project/heritage/heritage/prox_search/fill_table_prox_search.php(7): require_once('/mnt/nfs/labsto...')
53#2 {main}
54 thrown in /mnt/nfs/labstore-secondary-tools-project/heritage/heritage/api/includes/Database.php on line 150
55<br />
56<font size='1'><table class='xdebug-error xe-uncaught-exception' dir='ltr' border='1' cellspacing='0' cellpadding='1'>
57<tr><th align='left' bgcolor='#f57900' colspan="5"><span style='background-color: #cc0000; color: #fce94f; font-size: x-large;'>( ! )</span> Fatal error: Uncaught Error: Call to undefined function mysql_connect() in /mnt/nfs/labstore-secondary-tools-project/heritage/heritage/api/includes/Database.php on line <i>150</i></th></tr>
58<tr><th align='left' bgcolor='#f57900' colspan="5"><span style='background-color: #cc0000; color: #fce94f; font-size: x-large;'>( ! )</span> Error: Call to undefined function mysql_connect() in /mnt/nfs/labstore-secondary-tools-project/heritage/heritage/api/includes/Database.php on line <i>150</i></th></tr>
59<tr><th align='left' bgcolor='#e9b96e' colspan='5'>Call Stack</th></tr>
60<tr><th align='center' bgcolor='#eeeeec'>#</th><th align='left' bgcolor='#eeeeec'>Time</th><th align='left' bgcolor='#eeeeec'>Memory</th><th align='left' bgcolor='#eeeeec'>Function</th><th align='left' bgcolor='#eeeeec'>Location</th></tr>
61<tr><td bgcolor='#eeeeec' align='center'>1</td><td bgcolor='#eeeeec' align='center'>0.0018</td><td bgcolor='#eeeeec' align='right'>400312</td><td bgcolor='#eeeeec'>{main}( )</td><td title='/mnt/nfs/labstore-secondary-tools-project/heritage/heritage/prox_search/fill_table_prox_search.php' bgcolor='#eeeeec'>.../fill_table_prox_search.php<b>:</b>0</td></tr>
62<tr><td bgcolor='#eeeeec' align='center'>2</td><td bgcolor='#eeeeec' align='center'>0.1182</td><td bgcolor='#eeeeec' align='right'>584728</td><td bgcolor='#eeeeec'>require_once( <font color='#00bb00'>'/mnt/nfs/labstore-secondary-tools-project/heritage/heritage/api/common.php'</font> )</td><td title='/mnt/nfs/labstore-secondary-tools-project/heritage/heritage/prox_search/fill_table_prox_search.php' bgcolor='#eeeeec'>.../fill_table_prox_search.php<b>:</b>7</td></tr>
63<tr><td bgcolor='#eeeeec' align='center'>3</td><td bgcolor='#eeeeec' align='center'>0.4825</td><td bgcolor='#eeeeec' align='right'>1256280</td><td bgcolor='#eeeeec'>Database::define( )</td><td title='/mnt/nfs/labstore-secondary-tools-project/heritage/heritage/api/common.php' bgcolor='#eeeeec'>.../common.php<b>:</b>76</td></tr>
64</table></font>
652019-08-18_14:38:22 Update admin structure tree...
66Deleting previous data...
67Rebuilding table...
68Querying data...
69Inserting data...
70Inserted 142732 admin zones in 117.81044912338 seconds.
712019-08-18_14:40:20 Done with the update!