Page MenuHomePhabricator
Create Task
Maniphest T286135

Toolforge jobs framework: email maintainers on job failure
Closed, ResolvedPublicFeature
Actions

Description

Feature summary:

When a job fails, email tool maintainers.

Example available status information
status:
  conditions:
  - last_probe_time: null
    last_transition_time: 2021-07-03 07:07:12+00:00
    message: null
    reason: null
    status: 'True'
    type: Initialized
  - last_probe_time: null
    last_transition_time: 2021-07-03 12:03:30+00:00
    message: 'containers with unready status: [bsicons-replacer-old]'
    reason: ContainersNotReady
    status: 'False'
    type: Ready
  - last_probe_time: null
    last_transition_time: 2021-07-03 12:03:30+00:00
    message: 'containers with unready status: [bsicons-replacer-old]'
    reason: ContainersNotReady
    status: 'False'
    type: ContainersReady
  - last_probe_time: null
    last_transition_time: 2021-07-03 07:07:12+00:00
    message: null
    reason: null
    status: 'True'
    type: PodScheduled
  container_statuses:
  - container_id: docker://c7ce910349a993b98434ce9da21a8903b2b8ad82b14534d2d692a0ed1c670475
    image: docker-registry.tools.wmflabs.org/toolforge-python37-sssd-base:latest
    image_id: docker-pullable://docker-registry.tools.wmflabs.org/toolforge-python37-sssd-base@sha256:e42965a00ec91f52d051723277b81cce5de8339146d9010c3e735e0924fcc4a5
    last_state:
      running: null
      terminated: null
      waiting: null
    name: bsicons-replacer-old
    ready: false
    restart_count: 0
    state:
      running: null
      terminated:
        container_id: docker://c7ce910349a993b98434ce9da21a8903b2b8ad82b14534d2d692a0ed1c670475
        exit_code: 137
        finished_at: 2021-07-03 12:03:29+00:00
        message: null
        reason: OOMKilled
        signal: null
        started_at: 2021-07-03 07:07:14+00:00
      waiting: null
  host_ip: 172.16.1.183
  init_container_statuses: null
  message: null
  nominated_node_name: null
  phase: Failed
  pod_ip: 192.168.68.110
  qos_class: Burstable
  reason: null
  start_time: 2021-07-03 07:07:12+00:00

Use case(s):

I want to know when a job fails and what caused the failure.

Details

ProjectBranchLines +/-Subject
cloud/toolforge/jobs-framework-emailermain+6 -4emailer: config: increase task_compose_emails_loop_sleep value
cloud/toolforge/jobs-framework-emailermain+1 -1send: relax log message about not sending emails
cloud/toolforge/jobs-framework-emailermain+3 -1compose: include tool name in the body
cloud/toolforge/jobs-framework-emailermain+13 -3compose: make the email subject more dynamic
cloud/toolforge/jobs-framework-emailermain+2 -2configmap: update from address
cloud/toolforge/jobs-framework-emailermain+80 -20events: validate all labels are present
cloud/toolforge/jobs-framework-emailermain+42 -16emailer: refresh email handling
cloud/toolforge/jobs-framework-emailermain+8 -16events: flush cache all at once
cloud/toolforge/jobs-framework-emailermain+321 -187jobs-framework-emailer: introduce helm chart
cloud/toolforge/jobs-framework-emailermain+211 -185jobs-framework-emailer: introduce kustomize deployment
cloud/toolforge/jobs-framework-emailermain+341 -158emailer: introduce job event abstraction
cloud/toolforge/jobs-framework-apimain+283 -0jobs-framework-api: add emailer component
cloud/toolforge/jobs-framework-emailermain+7 -6emailer: don't use TLS to contact the smtp server
cloud/toolforge/jobs-framework-emailermain+1 -1deployment/toolsbeta: update smtp server port number
cloud/toolforge/jobs-framework-climaster+15 -8toolforge-jobs-framework-cli.1: refresh manpage
cloud/toolforge/jobs-framework-emailermain+19 -0emailer: handle containers in waiting state
cloud/toolforge/jobs-framework-emailermain+15 -14emailer: revisit email send routine
cloud/toolforge/jobs-framework-emailermain+67 -17emailer: filter events from label configuration
cloud/toolforge/jobs-framework-emailermain+66 -43deployment: refresh
cloud/toolforge/jobs-framework-climaster+20 -2jobs: add support for selecting email notifications
cloud/toolforge/jobs-framework-apimain+37 -2api: add new parameter to allow users to request email notifications
cloud/toolforge/jobs-framework-emailermain+1 K -0Initial code commit
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Sep 13 2021, 12:27 PM

Change 720745 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[cloud/toolforge/jobs-framework-emailer@main] emailer: filter events from label configuration

https://gerrit.wikimedia.org/r/720745

gerritbot added a comment.Sep 14 2021, 7:59 AM

Change 720745 merged by Arturo Borrero Gonzalez:

[cloud/toolforge/jobs-framework-emailer@main] emailer: filter events from label configuration

https://gerrit.wikimedia.org/r/720745

aborrero mentioned this in rCTJE893a2575d566: emailer: filter events from label configuration.Sep 14 2021, 7:59 AM
gerritbot added a comment.Sep 14 2021, 9:26 AM

Change 720933 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[cloud/toolforge/jobs-framework-emailer@main] emailer: revisit email send routine

https://gerrit.wikimedia.org/r/720933

gerritbot added a comment.Sep 14 2021, 9:30 AM

Change 720933 merged by Arturo Borrero Gonzalez:

[cloud/toolforge/jobs-framework-emailer@main] emailer: revisit email send routine

https://gerrit.wikimedia.org/r/720933

aborrero mentioned this in rCTJEc867574b1ae0: emailer: revisit email send routine.Sep 14 2021, 9:30 AM
gerritbot added a comment.Sep 14 2021, 10:09 AM

Change 720942 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[cloud/toolforge/jobs-framework-emailer@main] emailer: handle containers in waiting state

https://gerrit.wikimedia.org/r/720942

gerritbot added a comment.Sep 14 2021, 10:10 AM

Change 720942 merged by Arturo Borrero Gonzalez:

[cloud/toolforge/jobs-framework-emailer@main] emailer: handle containers in waiting state

https://gerrit.wikimedia.org/r/720942

aborrero mentioned this in rCTJE51032af302c3: emailer: handle containers in waiting state.Sep 14 2021, 10:10 AM
gerritbot added a comment.Sep 14 2021, 10:20 AM

Change 720946 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[cloud/toolforge/jobs-framework-cli@master] toolforge-jobs-framework-cli.1: refresh manpage

https://gerrit.wikimedia.org/r/720946

gerritbot added a comment.Sep 14 2021, 10:25 AM

Change 720946 merged by jenkins-bot:

[cloud/toolforge/jobs-framework-cli@master] toolforge-jobs-framework-cli.1: refresh manpage

https://gerrit.wikimedia.org/r/720946

aborrero mentioned this in rCTKF63cfb3fd6ca4: toolforge-jobs-framework-cli.1: refresh manpage.Sep 14 2021, 10:25 AM
Stashbot added a comment.Sep 14 2021, 10:39 AM

Mentioned in SAL (#wikimedia-cloud) [2021-09-14T10:39:27Z] <arturo> deploying jobs-framework-api 16fbf51 (T286135)

Stashbot added a comment.Sep 14 2021, 10:44 AM

Mentioned in SAL (#wikimedia-cloud) [2021-09-14T10:44:57Z] <arturo> deploying jobs-framework-emailer 51032af (T286135)

gerritbot added a comment.Sep 14 2021, 11:15 AM

Change 720954 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[cloud/toolforge/jobs-framework-emailer@main] deployment/toolsbeta: update smtp server port number

https://gerrit.wikimedia.org/r/720954

gerritbot added a comment.Sep 14 2021, 11:17 AM

Change 720954 merged by Arturo Borrero Gonzalez:

[cloud/toolforge/jobs-framework-emailer@main] deployment/toolsbeta: update smtp server port number

https://gerrit.wikimedia.org/r/720954

aborrero mentioned this in rCTJE7b248a8963fe: deployment/toolsbeta: update smtp server port number.Sep 14 2021, 11:18 AM
gerritbot added a comment.Sep 14 2021, 11:23 AM

Change 720956 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[cloud/toolforge/jobs-framework-emailer@main] emailer: don't use TLS to contact the smtp server

https://gerrit.wikimedia.org/r/720956

gerritbot added a comment.Sep 14 2021, 11:40 AM

Change 720956 merged by Arturo Borrero Gonzalez:

[cloud/toolforge/jobs-framework-emailer@main] emailer: don't use TLS to contact the smtp server

https://gerrit.wikimedia.org/r/720956

aborrero mentioned this in rCTJE3045601ca56a: emailer: don't use TLS to contact the smtp server.Sep 14 2021, 11:40 AM
Stashbot added a comment.Sep 14 2021, 11:42 AM

Mentioned in SAL (#wikimedia-cloud) [2021-09-14T11:42:33Z] <arturo> deploying jobs-framework-emailer 3045601 (T286135)

JJMC89 added a comment.Sep 14 2021, 8:41 PM

Is it supposed to be working?

I did toolforge-jobs run test-fail --command this-does-not-exist --image tf-python39 --emails onfailure --no-filelog, but didn't get an email.

For my other jobs that I added --emails onfailure (via load from yaml) all show Emails: Unknown.

Would you include the tool name in the email subject? I'd also prefer the job name in the subject if the email applies to a single job.

aborrero added a comment.Sep 15 2021, 9:24 AM
In T286135#7353351, @JJMC89 wrote:

Is it supposed to be working?

No, this wasn't deployed in toolforge yet, only in toolsbeta.

The new software piece emailer works pretty well apparently but I need to conduct a few more tests and adjusts a few things before deploying to Toolforge.

Unfortunately it is unlikely that I will have time for this until next month due to some priority changes in the team.

aborrero added a subscriber: taavi.Sep 15 2021, 11:06 AM

@Majavah asked me to share my TODO and here it is, from the top of my head:

  • the email composition phase needs to be a bit smarter, which is probably related to the event filtering stage.. As of yesterday, we could end up emailing duplicated events for a given job
  • try/catch more exceptions
  • conduct some careful tests to ensure the email flooding controls that were introduced in the system actually works (I only tested them partially with actual emails in toolsbeta)
  • halt the program entirely if one of the tasks fails. Otherwise, the daemon will keep looping but not doing its job. If we halt the program, at least k8s will restart the pod. One way to do this is to create *another task* that every 2 minutes checks if all other tasks are still scheduled, and if not, exit().
  • add comments to functions in the source code, update README and such
  • validate that our deployment manifest is right (kubernetes configuration to be loaded by kubectl -f), probably just need another pair of eyes.
  • automate deployment steps (I was planning to use our new spicerack setup for this). Steps are: build docker image, push docker image to the registry, deploy on k8s

The first 3 things are probably the real blockers for deploying this in toolforge now.

aborrero claimed this task.Dec 13 2021, 2:03 PM
gerritbot added a comment.Dec 14 2021, 1:08 PM

Change 747107 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[cloud/toolforge/jobs-framework-emailer@main] jobs-framework-emailer: introduce helm chart

https://gerrit.wikimedia.org/r/747107

aborrero changed the task status from Open to Stalled.Dec 15 2021, 10:18 AM
aborrero removed aborrero as the assignee of this task.
aborrero moved this task from Doing to Soon! on the cloud-services-team (Kanban) board.
gerritbot added a comment.Feb 21 2022, 8:15 AM

Change 719304 abandoned by Arturo Borrero Gonzalez:

[cloud/toolforge/jobs-framework-api@main] jobs-framework-api: add emailer component

Reason:

this lives now on its own repository

https://gerrit.wikimedia.org/r/719304

aborrero changed the task status from Stalled to In Progress.Feb 21 2022, 11:16 AM
aborrero claimed this task.
aborrero moved this task from Soon! to Doing on the cloud-services-team (Kanban) board.

I'm restarting work on this.

gerritbot added a comment.Feb 24 2022, 1:44 PM

Change 764432 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[cloud/toolforge/jobs-framework-emailer@main] emailer: introduce job event abstraction

https://gerrit.wikimedia.org/r/764432

diegodlh added a subscriber: diegodlh.EditedFeb 27 2022, 1:24 AM

Hi, all! Do you think something like this could help with T53434? I mean, would it make sense adding something like this to the webservice tool so an email is sent when the webservice container fails?

One would still one to know if a web service is working as expected from the outside, as discussed in T53434. But I feel that getting a notification when the service's container fails (even if it is automatically restarted immediately) may also help detect possible bugs.

I'm new to Toolforge and Kubernetes, so please apologize if my question is too naive.

On the other hand, I just gave a quick look at the source code for jobs-framework-api, jobs-framework-client, and jobs-framework-emailer. I couldn't find where the api is using the emailer code. I was expecting to find something in the line of lifecycle event hooks, as per the resource linked by @aborrero above. Is this still pending?

Edit: Well, I've been playing with this idea a bit. I've forked the webservice tool to have it create the container with a lifecycle preStop hook that runs a prestop.sh script in the tool's home directory. In my case this script in turn runs a node script that sends an email. This works when the container is terminated via webservice stop, for example. However, the preStop event doesn't seem to be sent when the container fails and is automatically restarted due to the restart policy. So, no luck with this, unfortunately :(

diegodlh mentioned this in T302689: Improve error monitoring.Feb 28 2022, 12:24 AM
aborrero added a comment.Mar 1 2022, 9:14 AM

hey @diegodlh thanks for your interest. Let me share a bit more information here.

Now, to the other question: it should be simple to extend this emailer component to support notifications for other pods (not only those jobs-related). After all, we evaluate if a pod event is relevant based on things like labels and such.

That being said, I think webservices need some metric-based monitoring and alerting system. In general, a webservice workload is a bit different than the ephemeral-in-nature job workloads, so monitoring them need to use different approaches. What I'm tryng to say is that toolforge webservices could definitely benefit the most from any development in T194333: [Epic] Provide logging/metrics/monitoring SaaS for Cloud VPS tenants.
Think of: a loop running curl/wget against all active webservices and alerting if they any are down. Discussing this with more details is out of scope for this ticket though.

gerritbot added a comment.Mar 10 2022, 11:42 AM

Change 764432 merged by Arturo Borrero Gonzalez:

[cloud/toolforge/jobs-framework-emailer@main] emailer: introduce job event abstraction

https://gerrit.wikimedia.org/r/764432

aborrero mentioned this in rCTJE060fb4451fc3: emailer: introduce job event abstraction.Mar 10 2022, 11:42 AM
gerritbot added a comment.Mar 10 2022, 1:19 PM

Change 769694 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[cloud/toolforge/jobs-framework-emailer@main] jobs-framework-emailer: introduce kustomize deployment

https://gerrit.wikimedia.org/r/769694

gerritbot added a comment.Mar 11 2022, 9:42 AM

Change 769694 merged by Arturo Borrero Gonzalez:

[cloud/toolforge/jobs-framework-emailer@main] jobs-framework-emailer: introduce kustomize deployment

https://gerrit.wikimedia.org/r/769694

gerritbot added a comment.Mar 11 2022, 9:43 AM

Change 747107 abandoned by Arturo Borrero Gonzalez:

[cloud/toolforge/jobs-framework-emailer@main] jobs-framework-emailer: introduce helm chart

Reason:

merged https://gerrit.wikimedia.org/r/c/cloud/toolforge/jobs-framework-emailer/+/769694 instead

https://gerrit.wikimedia.org/r/747107

aborrero mentioned this in rCTJE07e5d69afb0a: jobs-framework-emailer: introduce kustomize deployment.Mar 11 2022, 9:43 AM
Stashbot added a comment.Mar 11 2022, 11:59 AM

Mentioned in SAL (#wikimedia-cloud) [2022-03-11T11:59:40Z] <arturo> deploy jobs-framework-emailer d60ffd6 (T286135)

gerritbot added a comment.Mar 11 2022, 1:41 PM

Change 769970 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[cloud/toolforge/jobs-framework-emailer@main] events: flush cache all at once

https://gerrit.wikimedia.org/r/769970

gerritbot added a comment.Mar 11 2022, 1:41 PM

Change 769971 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[cloud/toolforge/jobs-framework-emailer@main] emailer: refresh email handling

https://gerrit.wikimedia.org/r/769971

gerritbot added a comment.Mar 11 2022, 2:33 PM

Change 769984 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[cloud/toolforge/jobs-framework-emailer@main] events: validate all labels are present

https://gerrit.wikimedia.org/r/769984

gerritbot added a comment.Mar 11 2022, 2:37 PM

Change 769970 merged by jenkins-bot:

[cloud/toolforge/jobs-framework-emailer@main] events: flush cache all at once

https://gerrit.wikimedia.org/r/769970

gerritbot added a comment.Mar 11 2022, 2:38 PM

Change 769971 merged by jenkins-bot:

[cloud/toolforge/jobs-framework-emailer@main] emailer: refresh email handling

https://gerrit.wikimedia.org/r/769971

gerritbot added a comment.Mar 11 2022, 2:38 PM

Change 769984 merged by jenkins-bot:

[cloud/toolforge/jobs-framework-emailer@main] events: validate all labels are present

https://gerrit.wikimedia.org/r/769984

aborrero mentioned this in rCTJE9b1eda4eaa09: events: validate all labels are present.Mar 11 2022, 2:38 PM
aborrero mentioned this in rCTJE6889c3acff17: emailer: refresh email handling.
aborrero mentioned this in rCTJE3d22a538f699: events: flush cache all at once.
Stashbot added a comment.Mar 11 2022, 3:02 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-11T15:02:35Z] <arturo> deploy jobs-framework-emailer 9470a5f (T286135)

Stashbot added a comment.Mar 14 2022, 11:44 AM

Mentioned in SAL (#wikimedia-cloud) [2022-03-14T11:44:38Z] <arturo> deploy jobs-framework-emailer 9470a5f339fd5a44c97c69ce97239aef30f5ee41 (T286135)

gerritbot added a comment.Mar 14 2022, 11:59 AM

Change 770477 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[cloud/toolforge/jobs-framework-emailer@main] configmap: update from address

https://gerrit.wikimedia.org/r/770477

gerritbot added a comment.Mar 14 2022, 12:14 PM

Change 770477 merged by Arturo Borrero Gonzalez:

[cloud/toolforge/jobs-framework-emailer@main] configmap: update from address

https://gerrit.wikimedia.org/r/770477

aborrero mentioned this in rCTJEd9d5b1194d6c: configmap: update from address.Mar 14 2022, 12:14 PM
aborrero added a comment.Mar 14 2022, 1:19 PM

hey @JJMC89 could you please give this a try?

When creating a new job, try the --emails parameter.

JJMC89 added a comment.Mar 14 2022, 5:13 PM
In T286135#7773876, @aborrero wrote:

hey @JJMC89 could you please give this a try?

When creating a new job, try the --emails parameter.

I tested with the below command and got two emails with the same content.

$ toolforge-jobs run test-fail --command this-does-not-exist --image tf-python39 --emails onfailure --no-filelog
[Toolforge] notification about 1 jobs
We wanted to notify you about the activity of some jobs in Toolforge.

* Job 'test-fail' (normal) (emails: onfailure) had 2 events:
  -- Pod 'test-fail-lnf5q'. Phase: 'failed'. Container state: 'terminated'. Start timestamp 2022-03-14T17:00:23Z. Finish timestamp 2022-03-14T17:00:23Z. Exit code was '127'. With reason 'Error'.
  -- Pod 'test-fail-9n4x6'. Phase: 'failed'. Container state: 'terminated'. Start timestamp 2022-03-14T17:00:26Z. Finish timestamp 2022-03-14T17:00:26Z. Exit code was '127'. With reason 'Error'.


If you requested 'filelog' for any of the jobs mentioned above, you may find additional information about what happened in the associated log files. Check them from Toolforge bastions as usual.

You are receiving this email because:
 1) when the job was created, it was requested to send email notfications.
 2) you are listed as tool maintainer for this tool.

Find help and more information in wikitech: https://wikitech.wikimedia.org/

Thanks for your contributions to the Wikimedia movement.

I've also reloaded my cronjobs with onfailure emails for ongoing testing.

In T286135#7353351, @JJMC89 wrote:

Would you include the tool name in the email subject? I'd also prefer the job name in the subject if the email applies to a single job.

JJMC89 added a comment.Mar 14 2022, 9:21 PM

I also get two emails when only one pod for a job fails.

aborrero added a comment.Mar 15 2022, 11:49 AM

A single email is sent each time:

2022-03-15 02:09:16 INFO: 1 new pending emails in the queue, new total queue size: 1
2022-03-15 02:09:43 INFO: Sending email FROM: noreply@toolforge.org TO: tools.jjmc89-bot@tools.wmflabs.org via mail.tools.wmflabs.org:25

This may suggest you have some kind of alias or duplicated inbox behind tools.jjmc89-bot@tools.wmflabs.org

gerritbot added a comment.Mar 15 2022, 11:58 AM

Change 770901 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[cloud/toolforge/jobs-framework-emailer@main] compose: make the email subject more dynamic

https://gerrit.wikimedia.org/r/770901

gerritbot added a comment.Mar 15 2022, 12:03 PM

Change 770901 merged by Arturo Borrero Gonzalez:

[cloud/toolforge/jobs-framework-emailer@main] compose: make the email subject more dynamic

https://gerrit.wikimedia.org/r/770901

aborrero mentioned this in rCTJE585139194f91: compose: make the email subject more dynamic.Mar 15 2022, 12:03 PM
gerritbot added a comment.Mar 15 2022, 12:11 PM

Change 770902 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[cloud/toolforge/jobs-framework-emailer@main] compose: include tool name in the body

https://gerrit.wikimedia.org/r/770902

gerritbot added a comment.Mar 15 2022, 12:55 PM

Change 770902 merged by Arturo Borrero Gonzalez:

[cloud/toolforge/jobs-framework-emailer@main] compose: include tool name in the body

https://gerrit.wikimedia.org/r/770902

aborrero mentioned this in rCTJE2cf9456202a7: compose: include tool name in the body.Mar 15 2022, 12:56 PM
gerritbot added a comment.Mar 15 2022, 1:09 PM

Change 770908 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[cloud/toolforge/jobs-framework-emailer@main] send: relax log message about not sending emails

https://gerrit.wikimedia.org/r/770908

gerritbot added a comment.Mar 15 2022, 1:09 PM

Change 770908 merged by Arturo Borrero Gonzalez:

[cloud/toolforge/jobs-framework-emailer@main] send: relax log message about not sending emails

https://gerrit.wikimedia.org/r/770908

aborrero mentioned this in rCTJE084ee51bf6d8: send: relax log message about not sending emails.Mar 15 2022, 1:10 PM
JJMC89 added a comment.Mar 16 2022, 4:40 AM
In T286135#7777382, @aborrero wrote:

A single email is sent each time:

2022-03-15 02:09:16 INFO: 1 new pending emails in the queue, new total queue size: 1
2022-03-15 02:09:43 INFO: Sending email FROM: noreply@toolforge.org TO: tools.jjmc89-bot@tools.wmflabs.org via mail.tools.wmflabs.org:25

This may suggest you have some kind of alias or duplicated inbox behind tools.jjmc89-bot@tools.wmflabs.org

I don't think that is the cause since I never received duplicate emails from the grid, which are sent to the same address.

Below are the Message-Ids and dates for the emails that I received. Emails 1 and 2 are the same, and 3 and 4 are the same. 5 is the one from your log, which I didn't receive two of.

  1. Message-Id: <E1nTo47-00067c-Ay@mail.tools.wmflabs.org>; Date: Mon, 14 Mar 2022 17:01:03 +0000
  2. Message-Id: <E1nTo74-0006Gu-L1@mail.tools.wmflabs.org>; Date: Mon, 14 Mar 2022 17:04:06 +0000
  3. Message-Id: <E1nTrwB-0002uZ-VK@mail.tools.wmflabs.org>; Date: Mon, 14 Mar 2022 21:09:07 +0000
  4. Message-Id: <E1nTrzF-0003Lf-8d@mail.tools.wmflabs.org>; Date: Mon, 14 Mar 2022 21:12:17 +0000
  5. Message-Id: <E1nTwd5-0002Uh-3c@mail.tools.wmflabs.org>; Date: Tue, 15 Mar 2022 02:09:43 +0000
aborrero added a comment.Mar 17 2022, 2:00 PM

I don't think I can explain why there are 2 emails. I couldn't reproduce this.

My working theory is the daemon being restarted and both the old/new daemon sending an email about the same events. But I don't think the emailer was restarted in those timestamps.

JJMC89 added a comment.Mar 18 2022, 10:36 PM

In case this helps, I received another pair of duplicates today.

  1. Message-Id: <E1nVIuM-0005sr-Nm@mail.tools.wmflabs.org>; Date: Fri, 18 Mar 2022 20:09:10 +0000
  2. Message-Id: <E1nVIxD-0006Hy-0A@mail.tools.wmflabs.org>; Date: Fri, 18 Mar 2022 20:12:07 +0000
aborrero added a comment.Mar 21 2022, 5:22 PM

Can you please paste here the full repeated emails, with the complete email source and headers?

JJMC89 added a comment.Mar 22 2022, 4:59 AM
  1. P22908, P22909
  2. P22910, P22911
  3. P22912, P22913
  4. P22914, P22915
aborrero added a comment.Mar 22 2022, 10:17 AM

thanks!

aborrero added a comment.Mar 22 2022, 10:45 AM

I think I have a theory of what's happening. The k8s API is really chatty about events going on for pods, which is good, but forces the emailer to do some filtering and caching to avoid flooding you with meaningless emails, which could be tricky.

  • certain pod events happen for a tool (creation, destroy, state change, restart, etc).
  • we filter out irrelevant events and cache the rest. The cache is to catch repeated events that may happen a few moments later. We cannot simply send an email per pod event.
  • This works, and at some point an email is sent to you with that event. We clear the event from the cache.
  • A few moments later, the pod (still running) generates a similar event (which I don't know yet how to uniquely identify), but we already sent an email and the cache is clean, so the code doesn't know it is duplicated.
  • The code understand is a new event, caches it, sends it a few moment later. Duplicated.
aborrero added a comment.Mar 22 2022, 10:52 AM

As a quick counter measure, will try increasing the time we cache events before we send an email. Hopefully this is enough to catch repeated events.

From your pastes:

  1. timestamp diff is ~3 minutes
  2. timestamp diff is ~3 minutes
  3. timestamp diff is ~3 minutes
  4. timestamp diff is ~3 minutes

So I will increase caching time by 4 minutes. This has the side effect of emails being delayed.

Other option would be to clearly identify what is the event k8s is reporting and effectively ignore it right away.

aborrero added a comment.Mar 22 2022, 10:55 AM

Done, note the 400:

2022-03-22 10:54:38 INFO: new configuration: {'task_compose_emails_loop_sleep': '400', 'task_send_emails_loop_sleep': '10', 'task_send_emails_max': '10', 'task_watch_pods_timeout': '60', 'task_read_configmap_sleep': '10', 'email_to_domain': 'tools.wmflabs.org', 'email_to_prefix': 'tools', 'email_from_addr': 'noreply@toolforge.org', 'smtp_server_fqdn': 'mail.tools.wmflabs.org', 'smtp_server_port': '25', 'send_emails_for_real': 'yes', 'debug': 'yes'}
gerritbot added a comment.Mar 22 2022, 10:59 AM

Change 772808 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[cloud/toolforge/jobs-framework-emailer@main] emailer: config: increase task_compose_emails_loop_sleep value

https://gerrit.wikimedia.org/r/772808

gerritbot added a comment.Mar 23 2022, 10:41 AM

Change 772808 merged by Arturo Borrero Gonzalez:

[cloud/toolforge/jobs-framework-emailer@main] emailer: config: increase task_compose_emails_loop_sleep value

https://gerrit.wikimedia.org/r/772808

aborrero mentioned this in rCTJE6e5b11d630d2: emailer: config: increase task_compose_emails_loop_sleep value.Mar 23 2022, 10:44 AM
aborrero closed this task as Resolved.Mar 29 2022, 1:47 PM

This is done for now.

Wbm1058 added a subscriber: Wbm1058.Nov 12 2022, 3:10 PM

If this task has been finished and closed as resolved, then why isn't --emails EMAILS listed by the Help command?

JJMC89 added a comment.Nov 12 2022, 5:56 PM

Each command has additional help text.

$ toolforge-jobs run -h
usage: toolforge-jobs run [-h] --command COMMAND --image IMAGE [--no-filelog]
                          [--mem MEM] [--cpu CPU]
                          [--emails {none,all,onfinish,onfailure}]
                          [--schedule SCHEDULE | --continuous | --wait]
                          name

positional arguments:
  name                  new job name

optional arguments:
  -h, --help            show this help message and exit
  --command COMMAND     full path of command to run in this job
  --image IMAGE         image shortname (check them with `images`)
  --no-filelog          don't store job stdout in `jobname`.out and stderr in
                        `jobname`.err files in the user home directory
  --mem MEM             specify additional memory limit required for this job
  --cpu CPU             specify additional CPU limit required for this job
  --emails {none,all,onfinish,onfailure}
                        specify if the system should email notifications about
                        this job. Defaults to 'none'.
  --schedule SCHEDULE   run a job with a cron-like schedule (example '1 * * *
                        *')
  --continuous          run a continuous job
  --wait                run a job and wait for completition. Timeout is 300
                        seconds.
Wbm1058 added a comment.Nov 12 2022, 8:59 PM

Oh, sorry, thanks. I tried that twice

$ toolforge-jobs -h run which just gave me the main help, ignoring the " run" at the end -- order "run" before "-h" is important there

then

$ tollforge-jobs run -h which was "command not found". I didn't notice my typo 'till later. Guess my mind was fried for the day by then

Thanks again

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL