Page MenuHomePhabricator

RU reportupdater-ee-beta-features keeps logging a lot of daily errors to its logs
Closed, ResolvedPublic3 Estimated Story Points

Description

Today I checked icinga and an-launcher1002 had the root partition almost saturated (100G). The biggest problem was:

elukey@an-launcher1001:/srv/reportupdater/log/reportupdater-ee-beta-features$ du -hs *
48G syslog.log

I truncated the file to 1G and now we are in a good stage, but we should investigate why this is happening. A lot of errors are like:

ERROR - Report "rcfilters_beta" could not be executed because of error: pymysql can not execute query ((1146, "Table 'dewiki.user_properties' doesn't exist")).

Event Timeline

elukey triaged this task as High priority.Jun 24 2020, 6:58 AM
elukey created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 24 2020, 6:58 AM
fdans assigned this task to mforns.Jul 13 2020, 4:44 PM
fdans moved this task from Incoming to Ops Week on the Analytics board.

@mforns the /srv partition on an-launcher1002 was already filling up, we should prioritize this and see if we can remove these errors from the logs :(

mforns added a comment.EditedJul 31 2020, 3:29 PM

Just got notified from the Growth team, that this data set is no longer used.
I will remove the job from puppet. And clean the logs.

Change 618311 had a related patch set uploaded (by Mforns; owner: Mforns):
[operations/puppet@production] analytics::reportupdater::jobs: Absent ee-beta-features job

https://gerrit.wikimedia.org/r/618311

Change 618311 merged by Elukey:
[operations/puppet@production] analytics::reportupdater::jobs: Absent ee-beta-features job

https://gerrit.wikimedia.org/r/618311

mforns moved this task from Next Up to Done on the Analytics-Kanban board.
Nuria closed this task as Resolved.Aug 5 2020, 11:00 PM
Nuria set the point value for this task to 3.