Page MenuHomePhabricator

update_database crashing due to KeyboardInterrupt
Closed, ResolvedPublic

Description

Ever since we bumped pywikibot the harvest job has been crashing

Retrieving 50 pages from wikipedia:en.
Retrieving 50 pages from wikipedia:en.
WARNING: /mnt/nfs/labstore-secondary-tools-project/heritage/venv/local/lib/python2.7/site-packages/urllib3/util/ssl_.py:160: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning

Traceback (most recent call last):
  File "/data/project/heritage/heritage/erfgoedbot/update_database.py", line 728, in <module>
    main()
  File "/data/project/heritage/heritage/erfgoedbot/update_database.py", line 716, in main
    days_back))
  File "/data/project/heritage/heritage/erfgoedbot/update_database.py", line 465, in process_country
    countryconfig, conn, cursor, full_update, days_back)
  File "/data/project/heritage/heritage/erfgoedbot/update_database.py", line 504, in process_country_list
    conn, cursor, unknown_fields=unknown_fields)
  File "/data/project/heritage/heritage/erfgoedbot/update_database.py", line 425, in process_page
    templates = page.templatesWithParams()
  File "/data/project/heritage/pywikibot/pywikibot/tools/__init__.py", line 1746, in wrapper
    return obj(*__args, **__kw)
  File "/data/project/heritage/pywikibot/pywikibot/page.py", line 2305, in templatesWithParams
    titles = [t.title() for t in self.templates()]
  File "/data/project/heritage/pywikibot/pywikibot/tools/__init__.py", line 1746, in wrapper
    return obj(*__args, **__kw)
  File "/data/project/heritage/pywikibot/pywikibot/page.py", line 1561, in templates
    self._templates = list(self.itertemplates(content=content))
  File "/data/project/heritage/pywikibot/pywikibot/data/api.py", line 2921, in __iter__
    self.data = self.request.submit()
  File "/data/project/heritage/pywikibot/pywikibot/data/api.py", line 2173, in submit
    self.site.throttle(write=self.write)
  File "/data/project/heritage/pywikibot/pywikibot/throttle.py", line 275, in __call__
    self.wait(wait)
  File "/data/project/heritage/pywikibot/pywikibot/throttle.py", line 253, in wait
    time.sleep(seconds)
KeyboardInterrupt
CRITICAL: Closing network session.
<type 'exceptions.KeyboardInterrupt'>
/mnt/nfs/labstore-secondary-tools-project/heritage/heritage/bin/update_monuments.sh: line 32: jstop: command not found
2018-09-03_08:00:19 Update monuments_all table...

Event Timeline

I have no clue why the pywikibot/throttle.py:wait function raises a KeyboardInterrupt when run through cron but it's happened the last 4 times

If this is running on the toolforge grid, KeyboardInterrupt usually means you're running out of memory.

If this is running on the toolforge grid, KeyboardInterrupt usually means you're running out of memory.

Jupp running on grid. Thanks for the tip about it being the memory (which was not obvious from the error =) )

Mentioned in SAL (#wikimedia-cloud) [2018-09-04T06:35:30Z] <JeanFrd> Submit update_monuments manually with -mem 2000m (see T203417)

Mentioned in SAL (#wikimedia-cloud) [2018-09-04T06:35:58Z] <JeanFrd> Edit crontab to run update_monuments with -mem 2000m (see T203417)

Thanks @Legoktm for the pointer :)

Per https://wikitech.wikimedia.org/wiki/Help:Toolforge/Grid#Allocating_additional_memory I just bumped the memory allocated to that job from 1000m to 2000m. Not the ideal solution but it will have to do for now :)

We previously bumped the memory allowance for the populate_image_table job but that shouldn't be the cause here since the abort occurs before. Other than that, bar any changes to grid settings, the only culprit I can figure out is the pywikibot version bump.

Both changes coincide with the start of these crashes.

JeanFred claimed this task.

Tentatively closing this as resolved − grepped through the latest update logs, no mention of KeyboardInterrupt since September 2018