Page MenuHomePhabricator
Create Task
Maniphest T211250

Create a mediawiki::cronjob define
Closed, ResolvedPublic
Actions

Description

We need all mediawiki cron jobs to:

  • append in a log file under /var/log/mediawiki
  • have logrotation
  • run cronjobs only in the datacentre where mediawiki is active
    • Use a cli wrapper to get the active DC value from conftool locally on disk from the yaml file and uses that to decide whether a "periodic job" should run or not

Right now we have them writing log files, where we only keep logs from their last run.

current plan:

Details

SubjectRepoBranchLines +/-
profile::mediawiki: remove comment references to cron jobsoperations/puppetproduction+11 -11
maintenance: Migrate wikidata prune jobs to periodic_joboperations/puppetproduction+6 -12
maintenance: Migrate updatequerypages to periodic_joboperations/puppetproduction+90 -119
maintenance: Migrate startupregistrystats to periodic_joboperations/puppetproduction+20 -28
maintenance: Migrate initsitestats to periodic_joboperations/puppetproduction+7 -11
maintenance: Migrate readinglists to periodic_joboperations/puppetproduction+7 -12
maintenance: Migrate pageassessments to periodic_joboperations/puppetproduction+8 -12
maintenance: Migrate generatecaptcha to periodic_joboperations/puppetproduction+35 -76
maintenance: Migrate cirrussearch to periodic_joboperations/puppetproduction+20 -78
maintenance: Migrate db_lag_stats_reporter to periodic_joboperations/puppetproduction+12 -16
maintenance: Migrate purge_securepoll to periodic_joboperations/puppetproduction+7 -10
maintenance: Migrate purge_old_cx_drafts to periodic_joboperations/puppetproduction+9 -13
maintenance: Migrate purged_expired_userrights to periodic_joboperations/puppetproduction+7 -11
maintenance: Migrate purge_abusefilter to periodic_joboperations/puppetproduction+7 -10
maintenance: Migrate purge_checkuser to periodic_joboperations/puppetproduction+7 -10
maintenance: Migrate update_special_pages to periodic_joboperations/puppetproduction+8 -12
maintenance: Migrate update_flaggedrev_stats to periodic_joboperations/puppetproduction+8 -11
maintenance: Migrate refreshlinks to periodic_joboperations/puppetproduction+17 -31
maintenance: Migrate parsercachepurging to periodic_joboperations/puppetproduction+11 -16
maintenance: migrate cleanup_upload_stash to periodic_joboperations/puppetproduction+7 -11
maintenance: Migrate echo_mail_batch to periodic_joboperations/puppetproduction+8 -11
maintenance: Migrate updatetranslationstats to periodic_joboperations/puppetproduction+17 -22
maintenance: Migrate translationnotifications jobs to periodic_joboperations/puppetproduction+16 -51
profile::mediawiki::maintenance: Migrate pagetriage jobs to periodic_joboperations/puppetproduction+22 -34
profile::mediawiki::maintenance: migrate tor job to periodic_joboperations/puppetproduction+4 -10
profile::mediawiki::maintenance: systemd-timer based periodic jobsoperations/puppetproduction+64 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 5 2018, 10:16 PM
jijiki added a project: User-jijiki.Dec 5 2018, 10:17 PM
Paladox subscribed.Dec 5 2018, 10:17 PM
Joe added a comment.Dec 6 2018, 11:36 AM

I would add another requirement:

  • we want all mediawiki cronjobs to only run in the datacenter where mediawiki is active right now.
jijiki updated the task description. (Show Details)Dec 6 2018, 11:36 AM
In T211250#4803144, @Joe wrote:

I would add another requirement:

  • we want all mediawiki cronjobs to only run in the datacenter where mediawiki is active right now.

Added in description

fgiunchedi subscribed.Dec 10 2018, 8:46 AM

I recommend sending cronjobs output to logstash (as well as files?), when cronjobs are logging to syslog you can opt-in via ./modules/profile/files/rsyslog/lookup_table_output.json

elukey subscribed.Dec 10 2018, 8:52 AM

We (analytics) have been trying to move away from crons in favor of systemd timers, adding some automation in profile::analytics::systemd_timer. It shouldn't need too much work to be generalized and adapted to the mediawiki use case, I can help/work on it if you think it is good!

jijiki added a comment.Dec 10 2018, 10:22 AM

@elukey @fgiunchedi we should definitely take into account logging to logstash and using systemd timers, tx!

jijiki added a project: serviceops.Dec 14 2018, 1:24 PM
gerritbot subscribed.Jan 8 2019, 11:50 AM

Change 482792 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] profile::mediawiki::maintenance: systemd-timer based periodic jobs

https://gerrit.wikimedia.org/r/482792

gerritbot added a project: Patch-For-Review.Jan 8 2019, 11:50 AM

Change 482793 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] profile::mediawiki::maintenance: migrate tor job to periodic_job

https://gerrit.wikimedia.org/r/482793

Dzahn added a comment.EditedJan 8 2019, 10:40 PM

summarizing joe's work:

  1. introduce systemd::timer::job
  1. introduce profile::mediawiki::periodic_job

2.1) 2) uses 1) and a cli wrapper to get the active DC value from conftool locally on disk from the yaml file and uses that to decide whether a "periodic job" should run or not

  1. first example to convert a cronjob to the new code
  1. do the same for all other crons
Dzahn updated the task description. (Show Details)Jan 8 2019, 10:42 PM
elukey added a comment.Jan 9 2019, 8:25 AM
  1. is done :)
jijiki moved this task from Incoming 🐫 to Doing 😎 on the serviceops board.Jan 9 2019, 1:38 PM
jijiki moved this task from Inbox 🐅 to In Progress 🏋️‍♀️ on the User-jijiki board.Jan 9 2019, 3:15 PM
jijiki updated the task description. (Show Details)Jan 11 2019, 10:08 AM
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 10:49 PM
jijiki moved this task from Doing 😎 to Incoming 🐫 on the serviceops board.Feb 6 2019, 12:11 PM
jijiki moved this task from In Progress 🏋️‍♀️ to St on the User-jijiki board.Feb 8 2019, 8:27 AM
gerritbot added a comment.Mar 20 2019, 7:22 AM

Change 482792 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] profile::mediawiki::maintenance: systemd-timer based periodic jobs

https://gerrit.wikimedia.org/r/482792

gerritbot added a comment.Apr 29 2019, 11:56 AM

Change 482792 merged by Giuseppe Lavagetto:
[operations/puppet@production] profile::mediawiki::maintenance: systemd-timer based periodic jobs

https://gerrit.wikimedia.org/r/482792

gerritbot added a comment.Apr 29 2019, 3:08 PM

Change 482793 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] profile::mediawiki::maintenance: migrate tor job to periodic_job

https://gerrit.wikimedia.org/r/482793

gerritbot added a comment.Apr 29 2019, 3:20 PM

Change 482793 merged by Giuseppe Lavagetto:
[operations/puppet@production] profile::mediawiki::maintenance: migrate tor job to periodic_job

https://gerrit.wikimedia.org/r/482793

jijiki moved this task from St to In Progress 🏋️‍♀️ on the User-jijiki board.May 6 2019, 9:04 AM
jijiki moved this task from In Progress 🏋️‍♀️ to Inbox 🐅 on the User-jijiki board.Jun 21 2019, 11:55 AM
Joe updated the task description. (Show Details)Jan 17 2020, 11:34 AM
Joe added a comment.Jan 17 2020, 11:37 AM

We should complete this work before we perform the MediaWiki switchover.

Joe added a parent task: T243314: FY2020-2021 Q1 DC switchover and switchback.Jan 23 2020, 2:43 PM
gerritbot added a comment.Mar 23 2020, 10:24 PM

Change 582933 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] profile::mediawiki::maintenance: Migrate pagetriage jobs to periodic_job

https://gerrit.wikimedia.org/r/582933

gerritbot added a comment.Apr 2 2020, 8:00 PM

Change 582933 merged by RLazarus:
[operations/puppet@production] profile::mediawiki::maintenance: Migrate pagetriage jobs to periodic_job

https://gerrit.wikimedia.org/r/582933

gerritbot added a comment.Apr 2 2020, 8:19 PM

Change 585250 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate translationnotifications jobs to periodic_job

https://gerrit.wikimedia.org/r/585250

gerritbot added a comment.Apr 3 2020, 7:01 PM

Change 585795 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate updatetranslationstats to periodic_job

https://gerrit.wikimedia.org/r/585795

gerritbot added a comment.Apr 3 2020, 7:20 PM

Change 585796 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate echo_mail_batch to periodic_job

https://gerrit.wikimedia.org/r/585796

gerritbot added a comment.Apr 6 2020, 6:55 PM

Change 585250 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate translationnotifications jobs to periodic_job

https://gerrit.wikimedia.org/r/585250

gerritbot added a comment.Apr 7 2020, 7:31 PM

Change 585795 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate updatetranslationstats to periodic_job

https://gerrit.wikimedia.org/r/585795

gerritbot added a comment.Apr 7 2020, 7:36 PM

Change 585796 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate echo_mail_batch to periodic_job

https://gerrit.wikimedia.org/r/585796

gerritbot added a comment.Apr 7 2020, 8:10 PM

Change 587324 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate parsercachepurging to periodic_job

https://gerrit.wikimedia.org/r/587324

gerritbot added a comment.Apr 7 2020, 8:14 PM

Change 587327 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: migrate cleanup_upload_stash to periodic_job

https://gerrit.wikimedia.org/r/587327

gerritbot added a comment.Apr 7 2020, 8:26 PM

Change 587328 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate update_flaggedrev_stats to periodic_job

https://gerrit.wikimedia.org/r/587328

gerritbot added a comment.Apr 7 2020, 8:38 PM

Change 587331 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate refreshlinks to period_job

https://gerrit.wikimedia.org/r/587331

gerritbot added a comment.Apr 7 2020, 8:54 PM

Change 587334 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate update_special_pages to periodic_job

https://gerrit.wikimedia.org/r/587334

gerritbot added a comment.Apr 9 2020, 4:15 PM

Change 587327 merged by RLazarus:
[operations/puppet@production] maintenance: migrate cleanup_upload_stash to periodic_job

https://gerrit.wikimedia.org/r/587327

gerritbot added a comment.Apr 9 2020, 4:23 PM

Change 587324 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate parsercachepurging to periodic_job

https://gerrit.wikimedia.org/r/587324

gerritbot added a comment.Apr 15 2020, 8:12 PM

Change 587331 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate refreshlinks to periodic_job

https://gerrit.wikimedia.org/r/587331

gerritbot added a comment.Apr 16 2020, 3:05 PM

Change 587328 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate update_flaggedrev_stats to periodic_job

https://gerrit.wikimedia.org/r/587328

gerritbot added a comment.Apr 16 2020, 5:05 PM

Change 587334 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate update_special_pages to periodic_job

https://gerrit.wikimedia.org/r/587334

gerritbot added a comment.Apr 16 2020, 5:30 PM

Change 589369 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate purge_abusefilter to periodic_jobo

https://gerrit.wikimedia.org/r/589369

gerritbot added a comment.Apr 16 2020, 5:42 PM

Change 589377 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate purge_checkuser to periodic_job

https://gerrit.wikimedia.org/r/589377

gerritbot added a comment.Apr 16 2020, 5:50 PM

Change 589378 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate purged_expired_userrights to periodic_job

https://gerrit.wikimedia.org/r/589378

gerritbot added a comment.Apr 16 2020, 5:59 PM

Change 589379 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate purge_old_cx_drafts to periodic_job

https://gerrit.wikimedia.org/r/589379

gerritbot added a comment.Apr 16 2020, 6:09 PM

Change 589384 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate purge_securepoll to periodic_job

https://gerrit.wikimedia.org/r/589384

Stashbot added a comment.Apr 17 2020, 3:20 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-17T15:20:39Z] <rzl> remove cronjobs from mwmaint1002 previously updated to systemd timers and erroneously left in crontab -- diffs: https://phabricator.wikimedia.org/P11012 T211250

gerritbot added a comment.Apr 17 2020, 6:56 PM

Change 589672 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate db_lag_stats_reporter to periodic_job

https://gerrit.wikimedia.org/r/589672

gerritbot added a comment.Apr 17 2020, 7:26 PM

Change 589680 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate cirrussearch to periodic_job

https://gerrit.wikimedia.org/r/589680

gerritbot added a comment.Apr 17 2020, 7:50 PM

Change 589688 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate generatecaptcha to periodic_job

https://gerrit.wikimedia.org/r/589688

gerritbot added a comment.Apr 17 2020, 8:12 PM

Change 589695 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate pageassessments to periodic_job

https://gerrit.wikimedia.org/r/589695

gerritbot added a comment.Apr 17 2020, 8:56 PM

Change 589706 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate readinglists to periodic_job

https://gerrit.wikimedia.org/r/589706

gerritbot added a comment.Apr 30 2020, 2:41 PM

Change 589377 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate purge_checkuser to periodic_job

https://gerrit.wikimedia.org/r/589377

gerritbot added a comment.Apr 30 2020, 2:45 PM

Change 589369 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate purge_abusefilter to periodic_job

https://gerrit.wikimedia.org/r/589369

gerritbot added a comment.Apr 30 2020, 2:51 PM

Change 589378 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate purged_expired_userrights to periodic_job

https://gerrit.wikimedia.org/r/589378

gerritbot added a comment.Apr 30 2020, 3:13 PM

Change 589379 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate purge_old_cx_drafts to periodic_job

https://gerrit.wikimedia.org/r/589379

gerritbot added a comment.Apr 30 2020, 3:17 PM

Change 589384 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate purge_securepoll to periodic_job

https://gerrit.wikimedia.org/r/589384

gerritbot added a comment.Apr 30 2020, 6:52 PM

Change 589672 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate db_lag_stats_reporter to periodic_job

https://gerrit.wikimedia.org/r/589672

gerritbot added a comment.Apr 30 2020, 6:56 PM

Change 589680 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate cirrussearch to periodic_job

https://gerrit.wikimedia.org/r/589680

gerritbot added a comment.Apr 30 2020, 7:02 PM

Change 589688 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate generatecaptcha to periodic_job

https://gerrit.wikimedia.org/r/589688

gerritbot added a comment.Apr 30 2020, 7:06 PM

Change 589695 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate pageassessments to periodic_job

https://gerrit.wikimedia.org/r/589695

gerritbot added a comment.Apr 30 2020, 7:10 PM

Change 589706 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate readinglists to periodic_job

https://gerrit.wikimedia.org/r/589706

gerritbot added a comment.May 1 2020, 5:39 PM

Change 593772 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate initsitestats to periodic_job

https://gerrit.wikimedia.org/r/593772

gerritbot added a comment.May 1 2020, 5:43 PM

Change 593774 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate startupregistrystats to periodic_job

https://gerrit.wikimedia.org/r/593774

gerritbot added a comment.May 1 2020, 6:43 PM

Change 593797 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate updatequerypages to periodic_job

https://gerrit.wikimedia.org/r/593797

gerritbot added a comment.May 29 2020, 5:49 PM

Change 593772 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate initsitestats to periodic_job

https://gerrit.wikimedia.org/r/593772

gerritbot added a comment.May 29 2020, 5:55 PM

Change 593774 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate startupregistrystats to periodic_job

https://gerrit.wikimedia.org/r/593774

gerritbot added a comment.May 29 2020, 6:02 PM

Change 593797 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate updatequerypages to periodic_job

https://gerrit.wikimedia.org/r/593797

gerritbot added a comment.May 29 2020, 6:39 PM

Change 599956 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate wikidata prune jobs to periodic_job

https://gerrit.wikimedia.org/r/599956

gerritbot added a comment.Jun 16 2020, 7:54 PM

Change 599956 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate wikidata prune jobs to periodic_job

https://gerrit.wikimedia.org/r/599956

jijiki moved this task from Incoming 🐫 to 🔦Unused2 on the serviceops board.Aug 17 2020, 11:48 PM
jijiki removed a project: User-jijiki.Sep 8 2020, 10:21 AM
RLazarus added a subtask: T288175: Migrate wikibase-dispatch-changes crons to systemd timers.Aug 17 2021, 5:12 PM
RLazarus closed this task as Resolved.Aug 17 2021, 7:04 PM
RLazarus claimed this task.
RLazarus subscribed.

🎉 With the dispatcher jobs migrated to systemd timers today, this is done! There are no maintenance cronjobs left.

rzl@mwmaint2002:~$ sudo crontab -u www-data -l
# HEADER: This file was autogenerated at 2021-08-17 17:18:41 +0000 by puppet.
# HEADER: While it can still be managed manually, it is definitely not recommended.
# HEADER: Note particularly that the comments starting with 'Puppet Name' should
# HEADER: not be deleted, as doing so could cause duplicate cron jobs.

I'll file a separate task under Datacenter-Switchover to clean up the cron-specific stuff in the switchdc cookbooks.

RLazarus mentioned this in T289078: Clean up cron-specific elements of switchdc cookbooks.Aug 17 2021, 7:12 PM
Ladsgroup closed subtask T288175: Migrate wikibase-dispatch-changes crons to systemd timers as Resolved.Aug 24 2021, 7:26 AM
Dzahn updated the task description. (Show Details)Sep 22 2021, 7:05 PM
gerritbot added a comment.Sep 22 2021, 7:05 PM

Change 722946 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] profile::mediawiki: remove comment references to cron jobs

https://gerrit.wikimedia.org/r/722946

gerritbot added a comment.Sep 22 2021, 7:43 PM

Change 722946 merged by Dzahn:

[operations/puppet@production] profile::mediawiki: remove comment references to cron jobs

https://gerrit.wikimedia.org/r/722946

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL