Page MenuHomePhabricator
Create Task
Maniphest T218881

labweb - wikitech_run_jobs.service in failed state
Closed, ResolvedPublic
Actions

Description

● wikitech_run_jobs.service - Run Wikitech runJobs.php maintenance script
   Loaded: loaded (/lib/systemd/system/wikitech_run_jobs.service; static; vendor preset: enabled)
   Active: failed (Result: start-limit-hit) since Thu 2019-03-21 13:46:21 UTC; 752ms ago
  Process: 20367 ExecStart=/usr/local/bin/mwscript maintenance/runJobs.php --wiki=labswiki > /dev/null 2>&
 Main PID: 20367 (code=exited, status=0/SUCCESS)

Mar 21 13:46:21 labweb1001 systemd[1]: Started Run Wikitech runJobs.php maintenance script.
Mar 21 13:46:21 labweb1001 mwscript[20367]: Job queue is empty.
Mar 21 13:46:22 labweb1001 systemd[1]: wikitech_run_jobs.service: Start request repeated too quickly.
Mar 21 13:46:22 labweb1001 systemd[1]: Failed to start Run Wikitech runJobs.php maintenance script.
Mar 21 13:46:22 labweb1001 systemd[1]: wikitech_run_jobs.service: Unit entered failed state.
Mar 21 13:46:22 labweb1001 systemd[1]: wikitech_run_jobs.service: Failed with result 'start-limit-hit'.

Details

SubjectRepoBranchLines +/-
wikitech: Add missing zero in time specificationoperations/puppetproduction+1 -1
wikitech: Fix error in timer specification (every 1s -> 1min)operations/puppetproduction+1 -1
wikitech: Do not hide output from mwscript runjobsoperations/puppetproduction+1 -1
Customize query in gerrit

Event Timeline

• GTirloni triaged this task as Medium priority.Mar 21 2019, 1:48 PM
• GTirloni created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 21 2019, 1:48 PM
• GTirloni added a comment.Mar 21 2019, 2:04 PM

Same happening on labtestweb2001

• GTirloni added a comment.Mar 21 2019, 2:17 PM

Additionally, labtestweb2001 is paging for this error while labweb1001 is not. Pages are not expected because the timer is not defined as critical in Puppet.

To clarify, this is simply surfacing errors that were being ignored when this was a cronjob. Now that it's a systemd timer, we get these alerts.

Failure history:

Mar 21 14:15:58 labweb1001 systemd[1]: Started Run Wikitech runJobs.php maintenance script.
Mar 21 14:15:58 labweb1001 mwscript[36424]: Job queue is empty.
Mar 21 14:15:59 labweb1001 systemd[1]: Started Run Wikitech runJobs.php maintenance script.
Mar 21 14:15:59 labweb1001 mwscript[36446]: Job queue is empty.
Mar 21 14:16:07 labweb1001 systemd[1]: Started Run Wikitech runJobs.php maintenance script.
Mar 21 14:16:08 labweb1001 mwscript[36475]: Job queue is empty.
Mar 21 14:16:08 labweb1001 systemd[1]: Started Run Wikitech runJobs.php maintenance script.
Mar 21 14:16:08 labweb1001 mwscript[36487]: Job queue is empty.
Mar 21 14:16:09 labweb1001 systemd[1]: Started Run Wikitech runJobs.php maintenance script.
Mar 21 14:16:09 labweb1001 mwscript[36499]: Job queue is empty.
Mar 21 14:16:12 labweb1001 systemd[1]: Started Run Wikitech runJobs.php maintenance script.
Mar 21 14:16:12 labweb1001 mwscript[36534]: Job queue is empty.
Mar 21 14:16:13 labweb1001 systemd[1]: Started Run Wikitech runJobs.php maintenance script.
Mar 21 14:16:14 labweb1001 mwscript[36553]: Job queue is empty.
Mar 21 14:16:14 labweb1001 systemd[1]: wikitech_run_jobs.service: Start request repeated too quickly.
Mar 21 14:16:14 labweb1001 systemd[1]: Failed to start Run Wikitech runJobs.php maintenance script.
Mar 21 14:16:14 labweb1001 systemd[1]: wikitech_run_jobs.service: Unit entered failed state.
Mar 21 14:16:14 labweb1001 systemd[1]: wikitech_run_jobs.service: Failed with result 'start-limit-hit'.
Mar 21 14:16:15 labweb1001 systemd[1]: wikitech_run_jobs.service: Start request repeated too quickly.
Mar 21 14:16:15 labweb1001 systemd[1]: Failed to start Run Wikitech runJobs.php maintenance script.
Mar 21 14:16:15 labweb1001 systemd[1]: wikitech_run_jobs.service: Failed with result 'start-limit-hit'.
Mar 21 14:16:17 labweb1001 systemd[1]: Started Run Wikitech runJobs.php maintenance script.
Mar 21 14:16:18 labweb1001 mwscript[36606]: Job queue is empty.
Mar 21 14:16:18 labweb1001 systemd[1]: Started Run Wikitech runJobs.php maintenance script.
Mar 21 14:16:19 labweb1001 mwscript[36618]: Job queue is empty.
Mar 21 14:16:19 labweb1001 systemd[1]: Started Run Wikitech runJobs.php maintenance script.
Mar 21 14:16:19 labweb1001 mwscript[36645]: Job queue is empty.
Mar 21 14:16:22 labweb1001 systemd[1]: Started Run Wikitech runJobs.php maintenance script.
Mar 21 14:16:22 labweb1001 mwscript[36664]: Job queue is empty.
Mar 21 14:16:24 labweb1001 systemd[1]: Started Run Wikitech runJobs.php maintenance script.
Mar 21 14:16:25 labweb1001 mwscript[36692]: Job queue is empty.
Mar 21 14:16:25 labweb1001 systemd[1]: wikitech_run_jobs.service: Start request repeated too quickly.
Mar 21 14:16:25 labweb1001 systemd[1]: Failed to start Run Wikitech runJobs.php maintenance script.
Mar 21 14:16:25 labweb1001 systemd[1]: wikitech_run_jobs.service: Unit entered failed state.
Mar 21 14:16:25 labweb1001 systemd[1]: wikitech_run_jobs.service: Failed with result 'start-limit-hit'.
Mar 21 14:16:27 labweb1001 systemd[1]: wikitech_run_jobs.service: Start request repeated too quickly.
Mar 21 14:16:27 labweb1001 systemd[1]: Failed to start Run Wikitech runJobs.php maintenance script.
Mar 21 14:16:27 labweb1001 systemd[1]: wikitech_run_jobs.service: Failed with result 'start-limit-hit'.
Mar 21 14:16:28 labweb1001 systemd[1]: Started Run Wikitech runJobs.php maintenance script.
Mar 21 14:16:29 labweb1001 mwscript[36713]: Job queue is empty.
Mar 21 14:16:29 labweb1001 systemd[1]: Started Run Wikitech runJobs.php maintenance script.
Mar 21 14:16:30 labweb1001 mwscript[36726]: Job queue is empty.
Mar 21 14:16:30 labweb1001 systemd[1]: Started Run Wikitech runJobs.php maintenance script.
Mar 21 14:16:30 labweb1001 mwscript[36738]: Job queue is empty.
Stashbot added a comment.Mar 21 2019, 2:18 PM

Mentioned in SAL (#wikimedia-operations) [2019-03-21T14:18:04Z] <gtirloni> downtimed labtestweb2001 (T218881)

gerritbot added a comment.Mar 21 2019, 2:31 PM

Change 498099 had a related patch set uploaded (by GTirloni; owner: GTirloni):
[operations/puppet@production] wikitech: Do not hide output from mwscript runjobs

https://gerrit.wikimedia.org/r/498099

gerritbot added a project: Patch-For-Review.Mar 21 2019, 2:31 PM
gerritbot added a comment.Mar 21 2019, 2:34 PM

Change 498099 merged by GTirloni:
[operations/puppet@production] wikitech: Do not hide output from mwscript runjobs

https://gerrit.wikimedia.org/r/498099

gerritbot added a comment.Mar 21 2019, 2:43 PM

Change 498105 had a related patch set uploaded (by GTirloni; owner: GTirloni):
[operations/puppet@production] wikitech: Fix error in timer specification (every 1s -> 1min)

https://gerrit.wikimedia.org/r/498105

gerritbot added a comment.Mar 21 2019, 2:44 PM

Change 498105 merged by GTirloni:
[operations/puppet@production] wikitech: Fix error in timer specification (every 1s -> 1min)

https://gerrit.wikimedia.org/r/498105

gerritbot added a comment.Mar 21 2019, 2:49 PM

Change 498107 had a related patch set uploaded (by GTirloni; owner: GTirloni):
[operations/puppet@production] wikitech: Add missing zero in time specification

https://gerrit.wikimedia.org/r/498107

gerritbot added a comment.Mar 21 2019, 2:49 PM

Change 498107 merged by GTirloni:
[operations/puppet@production] wikitech: Add missing zero in time specification

https://gerrit.wikimedia.org/r/498107

• GTirloni added a comment.Mar 21 2019, 2:58 PM

Due to an error in the timer specification, mwscript was running every second instead of every minute. That was too much for systemd and the unit would be put in failed state ocassionally.

I've fixed the timer specification and also allowed the command to create logs (by removing "2>&1"). If this happens again, we should see why it failed.

This confirmed systemd timers have better monitoring than cronjobs, which was very nice to see in action.

• GTirloni closed this task as Resolved.Mar 21 2019, 2:58 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL