Page MenuHomePhabricator

🐛 Matomo cronjob fails regularly
Closed, ResolvedPublic1 Estimated Story Points

Description

Problem:
Since 2nd of April, 2024 a Matomo-related cronjob fails regularly.
It fails almost every night around 00:00-02:00 in the morning.
Every time it fails, cron sends an e-mail with a subject like Cron <deploy@web03> umask 0002; php -d memory_limit=8G /local/sites/tracking.wikimedia.de/htdocs/console core:archive --url="https://tracking.wikimedia.de/", referring to tracking.wikimedia.de/logs/matomo.log
It indicates that it is hitting a memory limit.

Since April we increased the memory limit for the cronjob from 2GB to 8GB.
The cronjob still keeps sending error notifaction emails every 1-2 days).

Acceptance criteria:

  • find the source of the error
  • prevent the error from happening

Related Objects

Event Timeline

CorinnaHillebrand_WMDE renamed this task from Matomo cronjob fails regularly to 🐛 Matomo cronjob fails regularly.Apr 15 2024, 4:24 PM
CorinnaHillebrand_WMDE updated the task description. (Show Details)

Our current theory is that the cron job runs too long (15+ minutes) on a day change and multiple cron jobs (started every 5 minutes) affect each other.

I've changed the cron frequency to 20 minutes, let's wait a few days to see if the error messages go away

Part of the theory is that we have too many "segments" that are archived. We could regularly remove the archive flags of old segments.

current TODOs:
@kai.nissen:

  • The campaigns team needs to be asked whether they need the auto archive feature for segments

fun-tech:

  • remove auto archive flags in the database for the segments of 2022
gabriel-wmde set the point value for this task to 1.

I'ved updated the matomo database with the command update segment set auto_archive=0 where auto_archive=1; and set the cron job frequency back to 5 minutes. Let's wait for at least 3 days (until Monday, 2027-05-27) if the error occurs again (it shouldn't)