Page MenuHomePhabricator
Create Task
Maniphest T236505

Monitor mailman outbound mail queue
Closed, ResolvedPublic
Actions

Description

We currently monitor "in", "bounces", and "virgin". We should also monitor "out".

Details

SubjectRepoBranchLines +/-
profile: add anchor to mailman monitoring sectionoperations/puppetproduction+7 -7
profile: check_prometheus expects intoperations/puppetproduction+2 -2
profile: add mailman outbound queue monitoringoperations/puppetproduction+14 -0
mtail: track new subscription requests in prometheusoperations/puppetproduction+147 -0
hiera: set mtail to run as group adm on listsoperations/puppetproduction+3 -0
prometheus: add lists server mtail scrape to mtail jobsoperations/puppetproduction+7 -0
profile: get exim metrics from listsoperations/puppetproduction+11 -4
mtail,profile: add smtp metrics collection with mtailoperations/puppetproduction+27 -1
prometheus: update file count script to use single metric instanceoperations/puppetproduction+14 -8
prometheus, profile: add file count feature and enable lists queue trackingoperations/puppetproduction+136 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

colewhite created this task.Oct 25 2019, 5:48 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 25 2019, 5:48 PM
colewhite claimed this task.Oct 25 2019, 5:48 PM
colewhite triaged this task as Medium priority.
gerritbot added a comment.Oct 25 2019, 6:01 PM

Change 546260 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] prometheus, profile: add file count feature and enable lists queue tracking

https://gerrit.wikimedia.org/r/546260

gerritbot added a project: Patch-For-Review.Oct 25 2019, 6:01 PM
colewhite added a comment.Oct 25 2019, 7:49 PM

Historically, out queue monitoring has been noisy. One idea to have less noisy outbound monitoring is to take the queue depth and estimate how long it will take to send that queue based on the average send time.

gerritbot added a comment.Oct 25 2019, 8:34 PM

Change 546290 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] mtail,profile: add smtp metrics collection with mtail

https://gerrit.wikimedia.org/r/546290

fgiunchedi moved this task from Inbox to In progress on the observability board.Oct 28 2019, 2:15 PM
herron subscribed.Oct 28 2019, 4:19 PM
gerritbot added a comment.Oct 29 2019, 5:59 PM

Change 546992 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: get exim metrics from lists

https://gerrit.wikimedia.org/r/546992

gerritbot added a comment.Oct 29 2019, 7:52 PM

Change 546260 merged by Cwhite:
[operations/puppet@production] prometheus, profile: add file count feature and enable lists queue tracking

https://gerrit.wikimedia.org/r/546260

gerritbot added a comment.Oct 29 2019, 8:20 PM

Change 547028 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] prometheus: update file count script to use single metric instance

https://gerrit.wikimedia.org/r/547028

gerritbot added a comment.Oct 29 2019, 8:25 PM

Change 547028 merged by Cwhite:
[operations/puppet@production] prometheus: update file count script to use single metric instance

https://gerrit.wikimedia.org/r/547028

gerritbot added a comment.Nov 6 2019, 6:58 PM

Change 546290 merged by Cwhite:
[operations/puppet@production] mtail,profile: add smtp metrics collection with mtail

https://gerrit.wikimedia.org/r/546290

gerritbot added a comment.Nov 6 2019, 7:15 PM

Change 549179 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] prometheus: add lists server mtail scrape to mtail jobs

https://gerrit.wikimedia.org/r/549179

gerritbot added a comment.Nov 6 2019, 11:19 PM

Change 546992 merged by Cwhite:
[operations/puppet@production] profile: get exim metrics from lists

https://gerrit.wikimedia.org/r/546992

gerritbot added a comment.Nov 11 2019, 8:31 AM

Change 549179 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: add lists server mtail scrape to mtail jobs

https://gerrit.wikimedia.org/r/549179

Maintenance_bot removed a project: Patch-For-Review.Nov 11 2019, 9:10 AM
colewhite added a parent task: T230030: Prometheus failing to ingest some mtail samples.Nov 25 2019, 4:08 PM
gerritbot added a comment.Nov 26 2019, 4:57 PM

Change 553147 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] hiera: set mtail to run as group adm on lists

https://gerrit.wikimedia.org/r/553147

gerritbot added a project: Patch-For-Review.Nov 26 2019, 4:57 PM
gerritbot added a comment.Nov 26 2019, 5:30 PM

Change 553147 merged by Cwhite:
[operations/puppet@production] hiera: set mtail to run as group adm on lists

https://gerrit.wikimedia.org/r/553147

Maintenance_bot removed a project: Patch-For-Review.Nov 26 2019, 6:10 PM
gerritbot added a comment.Jan 13 2020, 9:11 PM

Change 564129 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] mtail: track new subscription requests in prometheus

https://gerrit.wikimedia.org/r/564129

gerritbot added a project: Patch-For-Review.Jan 13 2020, 9:11 PM
gerritbot added a comment.Jan 15 2020, 4:29 PM

Change 564129 merged by Cwhite:
[operations/puppet@production] mtail: track new subscription requests in prometheus

https://gerrit.wikimedia.org/r/564129

Maintenance_bot removed a project: Patch-For-Review.Jan 15 2020, 5:11 PM
gerritbot added a comment.May 14 2020, 3:56 PM

Change 596471 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: add mailman outbound queue monitoring

https://gerrit.wikimedia.org/r/596471

gerritbot added a project: Patch-For-Review.May 14 2020, 3:56 PM
gerritbot added a comment.May 14 2020, 8:16 PM

Change 596471 merged by Cwhite:
[operations/puppet@production] profile: add mailman outbound queue monitoring

https://gerrit.wikimedia.org/r/596471

gerritbot added a comment.May 14 2020, 8:19 PM

Change 596507 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: check_prometheus expects int

https://gerrit.wikimedia.org/r/596507

gerritbot added a comment.May 14 2020, 8:22 PM

Change 596507 merged by Cwhite:
[operations/puppet@production] profile: check_prometheus expects int

https://gerrit.wikimedia.org/r/596507

Maintenance_bot removed a project: Patch-For-Review.May 14 2020, 9:10 PM
gerritbot added a comment.May 14 2020, 9:22 PM

Change 596517 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: add anchor to mailman monitoring section

https://gerrit.wikimedia.org/r/596517

gerritbot added a project: Patch-For-Review.May 14 2020, 9:22 PM
gerritbot added a comment.May 15 2020, 3:59 PM

Change 596517 merged by Cwhite:
[operations/puppet@production] profile: add anchor to mailman monitoring section

https://gerrit.wikimedia.org/r/596517

colewhite closed this task as Resolved.May 15 2020, 4:00 PM

Monitoring deployed and updated some docs as well.

Maintenance_bot removed a project: Patch-For-Review.May 15 2020, 4:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits