Page MenuHomePhabricator

Monuments database dropped to 10% of its contents
Open, HighPublic

Description

It suddenly dropped from 1.5M to 119K.

Event Timeline

JeanFred created this task.Nov 17 2017, 6:56 PM

Mentioned in SAL (#wikimedia-cloud) [2017-11-17T19:30:26Z] <JeanFred> Started a new harvest to better investigate T180833

Amy hints from the logs?

ERROR: Unknown error occurred when processing country gb-sct in lang en
(2006, "MySQL server has gone away (error(32, 'Broken pipe'))")
ERROR: Unknown error occurred when processing country es-vc in lang ca
(0, '')
ERROR: Unknown error occurred when processing country th in lang th
(0, '')
ERROR: Unknown error occurred when processing country es-ct in lang ca
(0, '')
ERROR: Unknown error occurred when processing country es in lang es
(0, '')
…

Mentioned in SAL (#wikimedia-cloud) [2017-11-17T21:31:33Z] <JeanFred> Reverted to old database replicas (via git reset HEAD~1 && git stash) as part of T180833 investigation

Same thing

ERROR: Unknown error occurred when processing country gb-sct in lang en
(2006, "MySQL server has gone away (error(32, 'Broken pipe'))")
ERROR: Unknown error occurred when processing country es-vc in lang ca
(0, '')
ERROR: Unknown error occurred when processing country th in lang th
(0, '')
ERROR: Unknown error occurred when processing country es-ct in lang ca
(0, '')
ERROR: Unknown error occurred when processing country es in lang es
(0, '')

Might be related to the move to pymysql :-/

Latest run − with myslqdb and old replicas − did succeed. Monuments DB is backed to 1.5M. Now, is that just coincidence...

JeanFred triaged this task as High priority.Nov 18 2017, 3:07 PM

Which did you revert for it to work?

Lokal_Profil added a comment.EditedFeb 25 2018, 8:54 PM

So the reverted commit was https://gerrit.wikimedia.org/r/#/c/390895/.

@JeanFred Should we revert this in the repo as well (to get deployed code in sync with master again)

Lokal_Profil closed this task as Resolved.Aug 20 2018, 8:54 AM

The issue has been resolved (monumenta are back). Cleanup and figuring out the underlying issue is part of T200101: Resolve usage of pymysql vs. MySQLdb.

Atsirlin reopened this task as Open.Apr 12 2019, 6:19 AM

I am re-opening this task, because the same thing happened again. Many of the countries "lost" all their monuments.

I am re-opening this task, because the same thing happened again. Many of the countries "lost" all their monuments.

Oh geez, indeed: https://commons.wikimedia.org/wiki/Special:Diff/345626245

Thanks for flagging this @Atsirlin − will investigate.

JeanFred added a comment.EditedApr 12 2019, 10:00 AM
Retrieving 50 pages from wikipedia:de.
/mnt/nfs/labstore-secondary-tools-project/heritage/heritage/bin/update_monuments.sh: line 29: 31241 Killed                  $PYWIKIBOT_BIN $ERFGOED_PATH/update_database.py -fullupdate -log -skip_wd
/mnt/nfs/labstore-secondary-tools-project/heritage/heritage/bin/update_monuments.sh: line 32: jstop: command not found
2019-04-12_05:52:42 Update monuments_all table...
/mnt/nfs/labstore-secondary-tools-project/heritage/heritage/bin/update_monuments.sh: line 39: jsub: command not found
2019-04-12_05:53:42 Make statistics...
WARNING: /data/project/heritage/heritage/erfgoedbot/database_statistics.py:28: Warning: Truncated incorrect DOUBLE value: ''
  cursor.execute(query)

Page [[commons:Commons:Monuments database/Statistics]] saved

So update_database.py was killed somehow ; and then the Shell script went on to replace the database table anyways…

We should set a pipefail or smth so that the Python script being killed results in the entire Shell script being killed ; otherwise this is bound to happen.

(We likely will not be able to keep the shell script anyways, since it calls different runtime that need different containers anyways >_>)

Mentioned in SAL (#wikimedia-cloud) [2019-04-12T10:04:17Z] <JeanFred> Started a new harvest to better investigate T180833

Same problem again...

This time it failed for ru: only. Could anyone take a look, please?