Page MenuHomePhabricator
Create Task
Maniphest T362554

🐛 Matomo cronjob fails regularly
Open, Needs TriagePublic1 Estimated Story Points
Actions

Description

Problem:
Since 2nd of April, 2024 a Matomo-related cronjob fails regularly.
It fails almost every night around 00:00-02:00 in the morning.
Every time it fails, cron sends an e-mail with a subject like Cron <deploy@web03> umask 0002; php -d memory_limit=8G /local/sites/tracking.wikimedia.de/htdocs/console core:archive --url="https://tracking.wikimedia.de/", referring to tracking.wikimedia.de/logs/matomo.log
It indicates that it is hitting a memory limit.

Since April we increased the memory limit for the cronjob from 2GB to 8GB.
The cronjob still keeps sending error notifaction emails every 1-2 days).

Acceptance criteria:

  • find the source of the error
  • prevent the error from happening

Related Objects

Duplicates Merged Here
T362517: Investigate Matomo issues

Event Timeline

CorinnaHillebrand_WMDE created this task.Mon, Apr 15, 4:21 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMon, Apr 15, 4:21 PM
CorinnaHillebrand_WMDE renamed this task from Matomo cronjob fails regularly to 🐛 Matomo cronjob fails regularly.Mon, Apr 15, 4:24 PM
CorinnaHillebrand_WMDE updated the task description. (Show Details)
gabriel-wmde subscribed.Mon, Apr 15, 5:26 PM

Our current theory is that the cron job runs too long (15+ minutes) on a day change and multiple cron jobs (started every 5 minutes) affect each other.

I've changed the cron frequency to 20 minutes, let's wait a few days to see if the error messages go away

Part of the theory is that we have too many "segments" that are archived. We could regularly remove the archive flags of old segments.

gabriel-wmde merged a task: T362517: Investigate Matomo issues.Mon, Apr 15, 5:27 PM
gabriel-wmde moved this task from Todo to Blocked externally on the WMDE-FUN-Sprint-2024-04-09 board.
CorinnaHillebrand_WMDE moved this task from Blocked externally to Todo on the WMDE-FUN-Sprint-2024-04-09 board.Mon, Apr 22, 8:10 AM
CorinnaHillebrand_WMDE moved this task from Todo to Blocked externally on the WMDE-FUN-Sprint-2024-04-09 board.
CorinnaHillebrand_WMDE added a subscriber: kai.nissen.Mon, Apr 22, 8:12 AM

current TODOs:
@kai.nissen:

  • The campaigns team needs to be asked whether they need the auto archive feature for segments

fun-tech:

  • remove auto archive flags in the database for the segments of 2022
gabriel-wmde added a project: WMDE-FUN-Sprint-2024-04-23.Tue, Apr 23, 10:03 AM
gabriel-wmde moved this task from Todo to Blocked externally on the WMDE-FUN-Sprint-2024-04-23 board.Tue, Apr 23, 10:03 AM
gabriel-wmde set the point value for this task to 1.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL