Page MenuHomePhabricator

Create a mediawiki::cronjob define
Closed, ResolvedPublic

Description

We need all mediawiki cron jobs to:

  • append in a log file under /var/log/mediawiki
  • have logrotation
  • run cronjobs only in the datacentre where mediawiki is active
    • Use a cli wrapper to get the active DC value from conftool locally on disk from the yaml file and uses that to decide whether a "periodic job" should run or not

Right now we have them writing log files, where we only keep logs from their last run.


current plan:

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+11 -11
operations/puppetproduction+6 -12
operations/puppetproduction+90 -119
operations/puppetproduction+20 -28
operations/puppetproduction+7 -11
operations/puppetproduction+7 -12
operations/puppetproduction+8 -12
operations/puppetproduction+35 -76
operations/puppetproduction+20 -78
operations/puppetproduction+12 -16
operations/puppetproduction+7 -10
operations/puppetproduction+9 -13
operations/puppetproduction+7 -11
operations/puppetproduction+7 -10
operations/puppetproduction+7 -10
operations/puppetproduction+8 -12
operations/puppetproduction+8 -11
operations/puppetproduction+17 -31
operations/puppetproduction+11 -16
operations/puppetproduction+7 -11
operations/puppetproduction+8 -11
operations/puppetproduction+17 -22
operations/puppetproduction+16 -51
operations/puppetproduction+22 -34
operations/puppetproduction+4 -10
operations/puppetproduction+64 -0
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. ยท View Herald TranscriptDec 5 2018, 10:16 PM

I would add another requirement:

  • we want all mediawiki cronjobs to only run in the datacenter where mediawiki is active right now.

I would add another requirement:

  • we want all mediawiki cronjobs to only run in the datacenter where mediawiki is active right now.

Added in description

I recommend sending cronjobs output to logstash (as well as files?), when cronjobs are logging to syslog you can opt-in via ./modules/profile/files/rsyslog/lookup_table_output.json

We (analytics) have been trying to move away from crons in favor of systemd timers, adding some automation in profile::analytics::systemd_timer. It shouldn't need too much work to be generalized and adapted to the mediawiki use case, I can help/work on it if you think it is good!

@elukey @fgiunchedi we should definitely take into account logging to logstash and using systemd timers, tx!

Change 482792 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] profile::mediawiki::maintenance: systemd-timer based periodic jobs

https://gerrit.wikimedia.org/r/482792

Change 482793 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] profile::mediawiki::maintenance: migrate tor job to periodic_job

https://gerrit.wikimedia.org/r/482793

summarizing joe's work:

  1. introduce systemd::timer::job
  1. introduce profile::mediawiki::periodic_job

2.1) 2) uses 1) and a cli wrapper to get the active DC value from conftool locally on disk from the yaml file and uses that to decide whether a "periodic job" should run or not

  1. first example to convert a cronjob to the new code
  1. do the same for all other crons

Change 482792 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] profile::mediawiki::maintenance: systemd-timer based periodic jobs

https://gerrit.wikimedia.org/r/482792

Change 482792 merged by Giuseppe Lavagetto:
[operations/puppet@production] profile::mediawiki::maintenance: systemd-timer based periodic jobs

https://gerrit.wikimedia.org/r/482792

Change 482793 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] profile::mediawiki::maintenance: migrate tor job to periodic_job

https://gerrit.wikimedia.org/r/482793

Change 482793 merged by Giuseppe Lavagetto:
[operations/puppet@production] profile::mediawiki::maintenance: migrate tor job to periodic_job

https://gerrit.wikimedia.org/r/482793

We should complete this work before we perform the MediaWiki switchover.

Change 582933 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] profile::mediawiki::maintenance: Migrate pagetriage jobs to periodic_job

https://gerrit.wikimedia.org/r/582933

Change 582933 merged by RLazarus:
[operations/puppet@production] profile::mediawiki::maintenance: Migrate pagetriage jobs to periodic_job

https://gerrit.wikimedia.org/r/582933

Change 585250 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate translationnotifications jobs to periodic_job

https://gerrit.wikimedia.org/r/585250

Change 585795 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate updatetranslationstats to periodic_job

https://gerrit.wikimedia.org/r/585795

Change 585796 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate echo_mail_batch to periodic_job

https://gerrit.wikimedia.org/r/585796

Change 585250 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate translationnotifications jobs to periodic_job

https://gerrit.wikimedia.org/r/585250

Change 585795 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate updatetranslationstats to periodic_job

https://gerrit.wikimedia.org/r/585795

Change 585796 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate echo_mail_batch to periodic_job

https://gerrit.wikimedia.org/r/585796

Change 587324 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate parsercachepurging to periodic_job

https://gerrit.wikimedia.org/r/587324

Change 587327 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: migrate cleanup_upload_stash to periodic_job

https://gerrit.wikimedia.org/r/587327

Change 587328 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate update_flaggedrev_stats to periodic_job

https://gerrit.wikimedia.org/r/587328

Change 587331 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate refreshlinks to period_job

https://gerrit.wikimedia.org/r/587331

Change 587334 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate update_special_pages to periodic_job

https://gerrit.wikimedia.org/r/587334

Change 587327 merged by RLazarus:
[operations/puppet@production] maintenance: migrate cleanup_upload_stash to periodic_job

https://gerrit.wikimedia.org/r/587327

Change 587324 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate parsercachepurging to periodic_job

https://gerrit.wikimedia.org/r/587324

Change 587331 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate refreshlinks to periodic_job

https://gerrit.wikimedia.org/r/587331

Change 587328 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate update_flaggedrev_stats to periodic_job

https://gerrit.wikimedia.org/r/587328

Change 587334 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate update_special_pages to periodic_job

https://gerrit.wikimedia.org/r/587334

Change 589369 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate purge_abusefilter to periodic_jobo

https://gerrit.wikimedia.org/r/589369

Change 589377 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate purge_checkuser to periodic_job

https://gerrit.wikimedia.org/r/589377

Change 589378 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate purged_expired_userrights to periodic_job

https://gerrit.wikimedia.org/r/589378

Change 589379 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate purge_old_cx_drafts to periodic_job

https://gerrit.wikimedia.org/r/589379

Change 589384 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate purge_securepoll to periodic_job

https://gerrit.wikimedia.org/r/589384

Mentioned in SAL (#wikimedia-operations) [2020-04-17T15:20:39Z] <rzl> remove cronjobs from mwmaint1002 previously updated to systemd timers and erroneously left in crontab -- diffs: https://phabricator.wikimedia.org/P11012 T211250

Change 589672 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate db_lag_stats_reporter to periodic_job

https://gerrit.wikimedia.org/r/589672

Change 589680 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate cirrussearch to periodic_job

https://gerrit.wikimedia.org/r/589680

Change 589688 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate generatecaptcha to periodic_job

https://gerrit.wikimedia.org/r/589688

Change 589695 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate pageassessments to periodic_job

https://gerrit.wikimedia.org/r/589695

Change 589706 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate readinglists to periodic_job

https://gerrit.wikimedia.org/r/589706

Change 589377 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate purge_checkuser to periodic_job

https://gerrit.wikimedia.org/r/589377

Change 589369 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate purge_abusefilter to periodic_job

https://gerrit.wikimedia.org/r/589369

Change 589378 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate purged_expired_userrights to periodic_job

https://gerrit.wikimedia.org/r/589378

Change 589379 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate purge_old_cx_drafts to periodic_job

https://gerrit.wikimedia.org/r/589379

Change 589384 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate purge_securepoll to periodic_job

https://gerrit.wikimedia.org/r/589384

Change 589672 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate db_lag_stats_reporter to periodic_job

https://gerrit.wikimedia.org/r/589672

Change 589680 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate cirrussearch to periodic_job

https://gerrit.wikimedia.org/r/589680

Change 589688 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate generatecaptcha to periodic_job

https://gerrit.wikimedia.org/r/589688

Change 589695 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate pageassessments to periodic_job

https://gerrit.wikimedia.org/r/589695

Change 589706 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate readinglists to periodic_job

https://gerrit.wikimedia.org/r/589706

Change 593772 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate initsitestats to periodic_job

https://gerrit.wikimedia.org/r/593772

Change 593774 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate startupregistrystats to periodic_job

https://gerrit.wikimedia.org/r/593774

Change 593797 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate updatequerypages to periodic_job

https://gerrit.wikimedia.org/r/593797

Change 593772 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate initsitestats to periodic_job

https://gerrit.wikimedia.org/r/593772

Change 593774 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate startupregistrystats to periodic_job

https://gerrit.wikimedia.org/r/593774

Change 593797 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate updatequerypages to periodic_job

https://gerrit.wikimedia.org/r/593797

Change 599956 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] maintenance: Migrate wikidata prune jobs to periodic_job

https://gerrit.wikimedia.org/r/599956

Change 599956 merged by RLazarus:
[operations/puppet@production] maintenance: Migrate wikidata prune jobs to periodic_job

https://gerrit.wikimedia.org/r/599956

RLazarus claimed this task.
RLazarus added a subscriber: RLazarus.

๐ŸŽ‰ With the dispatcher jobs migrated to systemd timers today, this is done! There are no maintenance cronjobs left.

rzl@mwmaint2002:~$ sudo crontab -u www-data -l
# HEADER: This file was autogenerated at 2021-08-17 17:18:41 +0000 by puppet.
# HEADER: While it can still be managed manually, it is definitely not recommended.
# HEADER: Note particularly that the comments starting with 'Puppet Name' should
# HEADER: not be deleted, as doing so could cause duplicate cron jobs.

I'll file a separate task under Datacenter-Switchover to clean up the cron-specific stuff in the switchdc cookbooks.

Change 722946 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] profile::mediawiki: remove comment references to cron jobs

https://gerrit.wikimedia.org/r/722946

Change 722946 merged by Dzahn:

[operations/puppet@production] profile::mediawiki: remove comment references to cron jobs

https://gerrit.wikimedia.org/r/722946