Page MenuHomePhabricator

Reportupdater queries jobs failing
Closed, ResolvedPublic

Description

There's no data for any of the reportupdater output tsv since 28 October 2018, and there are errors in the logs:

https://analytics.wikimedia.org/dashboards/browsers/

41 Traceback (most recent call last):
42   File "/srv/reportupdater/reportupdater/update_reports.py", line 39, in <module>
43     main()
44   File "/srv/reportupdater/reportupdater/update_reports.py", line 35, in main
45     reportupdater.run(**args)
46   File "/srv/reportupdater/reportupdater/reportupdater/reportupdater.py", line 36, in run
47     if only_instance_running(params):
48   File "/srv/reportupdater/reportupdater/reportupdater/reportupdater.py", line 92, in only_instance_running
49     pid = int(pid_file.read().strip())
50 ValueError: invalid literal for int() with base 10: ''

Event Timeline

turns out the .reportupdater.pid file was just an empty file written on Nov 6, 2019!!! Which... seems not possible. And was messing up all reportupdater-queries runs for the last few months. How we didn't notice this, I don't know. This seems like something that went wrong when we migrated from stat1005 to stat1007, maybe we migrated in the middle of the reportupdater job writing the pid. That would be pretty unlucky!

Update: executing fine now, will move this to done tentatively, and bring it back if anything else goes wrong.

Nuria subscribed.

Thanks to @Milimetric for the fast turnaround here

Change 485673 had a related patch set uploaded (by Milimetric; owner: Milimetric):
[analytics/reportupdater@master] Clarify pid file error message

https://gerrit.wikimedia.org/r/485673

Change 485673 merged by Milimetric:
[analytics/reportupdater@master] Clarify pid file error message

https://gerrit.wikimedia.org/r/485673