Page MenuHomePhabricator
Create Task
Maniphest T292619

Implement a watchdog mechanism on acme-chief
Closed, ResolvedPublic
Actions

Description

As reported on T273956 acme-chief sometimes hangs and it's unable to keep performing its duties. A watchdog mechanism would help detecting this quickly and on a reliable manner.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Vgutierrez created this task.Oct 6 2021, 9:29 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 6 2021, 9:29 AM
Vgutierrez triaged this task as Medium priority.Oct 6 2021, 9:29 AM
Vgutierrez moved this task from Triage to TLS on the Traffic board.
Maintenance_bot added a project: SRE.Oct 6 2021, 9:45 AM
gerritbot added a comment.Oct 8 2021, 11:32 AM

Change 728379 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/software/acme-chief@master] acme_chief: implement file and systemd based watchdogs

https://gerrit.wikimedia.org/r/728379

gerritbot added a project: Patch-For-Review.Oct 8 2021, 11:32 AM
BBlack moved this task from TLS to In Progress on the Traffic board.Oct 8 2021, 8:16 PM
gerritbot added a comment.Oct 11 2021, 3:34 PM

Change 730016 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] acme_chief: Enable file and systemd watchdogs

https://gerrit.wikimedia.org/r/730016

gerritbot added a comment.Oct 14 2021, 8:51 AM

Change 730703 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/software/acme-chief@master] Release 0.33

https://gerrit.wikimedia.org/r/730703

gerritbot added a comment.Oct 14 2021, 8:52 AM

Change 728379 merged by jenkins-bot:

[operations/software/acme-chief@master] acme_chief: Add systemd based watchdog support

https://gerrit.wikimedia.org/r/728379

Vgutierrez mentioned this in rOSACde8747cdd903: acme_chief: Add systemd based watchdog support.Oct 14 2021, 8:53 AM
gerritbot added a comment.Oct 14 2021, 9:07 AM

Change 730703 merged by jenkins-bot:

[operations/software/acme-chief@master] Release 0.33

https://gerrit.wikimedia.org/r/730703

Vgutierrez mentioned this in rOSAC985fe6e5a15e: Release 0.33.Oct 14 2021, 9:09 AM
gerritbot added a comment.Oct 14 2021, 9:14 AM

Change 730711 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/software/acme-chief@debian] acme_chief: Add systemd based watchdog support

https://gerrit.wikimedia.org/r/730711

gerritbot added a comment.Oct 14 2021, 9:14 AM

Change 730712 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/software/acme-chief@debian] Release 0.33

https://gerrit.wikimedia.org/r/730712

gerritbot added a comment.Oct 14 2021, 9:14 AM

Change 730713 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/software/acme-chief@debian] debian: Add release 0.33 to the changelog

https://gerrit.wikimedia.org/r/730713

gerritbot added a comment.Oct 14 2021, 9:27 AM

Change 730711 merged by jenkins-bot:

[operations/software/acme-chief@debian] acme_chief: Add systemd based watchdog support

https://gerrit.wikimedia.org/r/730711

gerritbot added a comment.Oct 14 2021, 9:27 AM

Change 730712 merged by jenkins-bot:

[operations/software/acme-chief@debian] Release 0.33

https://gerrit.wikimedia.org/r/730712

gerritbot added a comment.Oct 14 2021, 9:27 AM

Change 730713 merged by jenkins-bot:

[operations/software/acme-chief@debian] debian: Add release 0.33 to the changelog

https://gerrit.wikimedia.org/r/730713

Vgutierrez mentioned this in rOSACcbeca694a9f8: debian: Add release 0.33 to the changelog.Oct 14 2021, 9:29 AM
Vgutierrez mentioned this in rOSAC4272a7a51031: Release 0.33.
Vgutierrez mentioned this in rOSACb75324f47b37: acme_chief: Add systemd based watchdog support.
gerritbot added a comment.Oct 14 2021, 10:54 AM

Change 730749 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/software/acme-chief@master] acme_chief: auto-detect systemd watchdog

https://gerrit.wikimedia.org/r/730749

gerritbot added a comment.Oct 14 2021, 4:06 PM

Change 730749 merged by Vgutierrez:

[operations/software/acme-chief@master] acme_chief: auto-detect systemd watchdog

https://gerrit.wikimedia.org/r/730749

Vgutierrez mentioned this in rOSAC248b6f1ca275: acme_chief: auto-detect systemd watchdog.Oct 14 2021, 4:09 PM
gerritbot added a comment.Oct 15 2021, 9:15 AM

Change 730977 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/software/acme-chief@master] Release 0.34

https://gerrit.wikimedia.org/r/730977

gerritbot added a comment.Oct 15 2021, 9:24 AM

Change 730977 merged by jenkins-bot:

[operations/software/acme-chief@master] Release 0.34

https://gerrit.wikimedia.org/r/730977

Vgutierrez mentioned this in rOSAC3894009d6455: Release 0.34.Oct 15 2021, 9:26 AM
gerritbot added a comment.Oct 15 2021, 9:48 AM

Change 730979 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/software/acme-chief@debian] acme_chief: auto-detect systemd watchdog

https://gerrit.wikimedia.org/r/730979

gerritbot added a comment.Oct 15 2021, 9:48 AM

Change 730980 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/software/acme-chief@debian] Release 0.34

https://gerrit.wikimedia.org/r/730980

gerritbot added a comment.Oct 15 2021, 9:48 AM

Change 730981 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/software/acme-chief@debian] debian: Add release 0.34 to the changelog

https://gerrit.wikimedia.org/r/730981

gerritbot added a comment.Oct 15 2021, 9:57 AM

Change 730979 merged by jenkins-bot:

[operations/software/acme-chief@debian] acme_chief: auto-detect systemd watchdog

https://gerrit.wikimedia.org/r/730979

gerritbot added a comment.Oct 15 2021, 9:58 AM

Change 730980 merged by jenkins-bot:

[operations/software/acme-chief@debian] Release 0.34

https://gerrit.wikimedia.org/r/730980

gerritbot added a comment.Oct 15 2021, 9:58 AM

Change 730981 merged by jenkins-bot:

[operations/software/acme-chief@debian] debian: Add release 0.34 to the changelog

https://gerrit.wikimedia.org/r/730981

Vgutierrez mentioned this in rOSAC3dfb70bd6b8b: Release 0.34.Oct 15 2021, 9:59 AM
Vgutierrez mentioned this in rOSACdabc46932029: debian: Add release 0.34 to the changelog.
Vgutierrez mentioned this in rOSAC3d54c8bcaed2: acme_chief: auto-detect systemd watchdog.
gerritbot added a comment.Oct 15 2021, 11:13 AM

Change 731018 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] acme_chief: Enable watchdog on acmechief-test1001

https://gerrit.wikimedia.org/r/731018

Stashbot added a comment.Oct 15 2021, 1:14 PM

Mentioned in SAL (#wikimedia-operations) [2021-10-15T13:14:21Z] <vgutierrez> upload acme-chief 0.34 to apt.wikimedia.org (buster) - T292619

gerritbot added a comment.Oct 15 2021, 1:17 PM

Change 730016 merged by Vgutierrez:

[operations/puppet@production] acme_chief: Support systemd watchdog

https://gerrit.wikimedia.org/r/730016

gerritbot added a comment.Oct 15 2021, 1:18 PM

Change 731018 merged by Vgutierrez:

[operations/puppet@production] acme_chief: Enable watchdog on acmechief-test1001

https://gerrit.wikimedia.org/r/731018

Stashbot added a comment.Oct 15 2021, 1:21 PM

Mentioned in SAL (#wikimedia-operations) [2021-10-15T13:21:17Z] <vgutierrez> updating acme-chief to version 0.34 on acmechief-test instances - T292619

gerritbot added a comment.Oct 15 2021, 1:41 PM

Change 731101 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] systemd: Allow paging on a systemd::service failure

https://gerrit.wikimedia.org/r/731101

gerritbot added a comment.Oct 18 2021, 8:17 AM

Change 731101 merged by Vgutierrez:

[operations/puppet@production] systemd: Allow paging on a systemd::service failure

https://gerrit.wikimedia.org/r/731101

gerritbot added a comment.Oct 18 2021, 9:03 AM

Change 731335 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] acme_chief: Enable watchdog on production servers

https://gerrit.wikimedia.org/r/731335

Stashbot added a comment.Oct 18 2021, 9:39 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-18T09:39:09Z] <vgutierrez> updating acme-chief to version 0.34 on acmechief instances - T292619

gerritbot added a comment.Oct 18 2021, 9:39 AM

Change 731335 merged by Vgutierrez:

[operations/puppet@production] acme_chief: Enable watchdog on production servers

https://gerrit.wikimedia.org/r/731335

gerritbot added a comment.Oct 18 2021, 9:48 AM

Change 731343 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] acme_chief: Enable monitoring on systemd

https://gerrit.wikimedia.org/r/731343

gerritbot added a comment.Oct 18 2021, 10:52 AM

Change 731343 merged by Vgutierrez:

[operations/puppet@production] acme_chief: Enable monitoring on systemd

https://gerrit.wikimedia.org/r/731343

Vgutierrez closed this task as Resolved.Oct 18 2021, 3:39 PM
Vgutierrez claimed this task.
gerritbot added a comment.Oct 28 2021, 8:20 AM

Change 735297 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] acme_chief: Page on acme-chief unit failure

https://gerrit.wikimedia.org/r/735297

gerritbot added a comment.Nov 2 2021, 9:25 AM

Change 735297 merged by Vgutierrez:

[operations/puppet@production] acme_chief: Page on acme-chief unit failure

https://gerrit.wikimedia.org/r/735297

gerritbot added a comment.Feb 3 2022, 8:05 AM

Change 759439 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:acme_chief: set watchdog_sec default on cloud

https://gerrit.wikimedia.org/r/759439

gerritbot added a comment.Feb 4 2022, 1:49 PM

Change 759439 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] P:acme_chief: set watchdog_sec default on cloud

https://gerrit.wikimedia.org/r/759439

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL