Change Details

== Common information * **dashboard**: https://w.wiki/DocP * **description**: Use `kube-env mw-cron eqiad; kubectl get jobs -l team=notifications-echo,cronjob=echo-mail-batch --field-selector status.successful=0` to see failures * **runbook**: https://wikitech.wikimedia.org/wiki/Periodic_jobs#Troubleshooting * **summary**: MediaWiki periodic job echo-mail-batch failed * **alertname**: MediaWikiCronJobFailed * **label_cronjob**: echo-mail-batch * **label_team**: notifications-echo * **prometheus**: k8s * **severity**: task * **site**: eqiad * **source**: prometheus * **team**: notifications-echo == Firing alerts --- * **dashboard**: https://w.wiki/DocP * **description**: Use `kube-env mw-cron eqiad; kubectl get jobs -l team=notifications-echo,cronjob=echo-mail-batch --field-selector status.successful=0` to see failures * **runbook**: https://wikitech.wikimedia.org/wiki/Periodic_jobs#Troubleshooting * **summary**: MediaWiki periodic job echo-mail-batch failed * **alertname**: MediaWikiCronJobFailed * **label_cronjob**: echo-mail-batch * **label_team**: notifications-echo * **prometheus**: k8s * **severity**: task * **site**: eqiad * **source**: prometheus * **team**: notifications-echo * [Source](https://prometheus-eqiad.wikimedia.org/k8s/graph?g0.expr=sum+by+%28label_cronjob%2C+label_team%29+%28kube_job_status_failed%7Bjob_name%21%3D%22wikidata-updatequeryservicelag%22%2Cnamespace%3D%22mw-cron%22%7D+%2A+on+%28namespace%2C+job_name%29+group_left+%28label_cronjob%2C+label_team%29+kube_job_labels%7Blabel_cronjob%21%3D%22wikidata-updatequeryservicelag%22%2Cnamespace%3D%22mw-cron%22%7D%29+%3E+0&g0.tab=1)

== Common information * **dashboard**: https://logstash.wikimedia.org/app/dashboards#/view/d51552d0-e309-11ef-87d0-9371e01d3c68?_a=(filters:!((query:(match_phrase:(kubernetes.labels.cronjob.keyword:echo-mail-batch)))))&_g=(time:(from:now-24h,to:now)) * **description**: Use `kube-env mw-cron codfw; kubectl get jobs -l team=notifications-echo,cronjob=echo-mail-batch --field-selector status.successful=0; kubectl logs -l cronjob=echo-mail-batch --timestamps=true --prefix=true -c mediawiki-main-app` to see failures and logs * **runbook**: https://wikitech.wikimedia.org/wiki/Periodic_jobs#Troubleshooting * **summary**: MediaWiki periodic job echo-mail-batch failed * **alertname**: MediaWikiCronJobFailed * **label_cronjob**: echo-mail-batch * **label_team**: notifications-echo * **prometheus**: k8s * **severity**: task * **site**: codfw * **source**: prometheus * **team**: notifications-echo == Firing alerts --- * **dashboard**: https://logstash.wikimedia.org/app/dashboards#/view/d51552d0-e309-11ef-87d0-9371e01d3c68?_a=(filters:!((query:(match_phrase:(kubernetes.labels.cronjob.keyword:echo-mail-batch)))))&_g=(time:(from:now-24h,to:now)) * **description**: Use `kube-env mw-cron codfw; kubectl get jobs -l team=notifications-echo,cronjob=echo-mail-batch --field-selector status.successful=0; kubectl logs -l cronjob=echo-mail-batch --timestamps=true --prefix=true -c mediawiki-main-app` to see failures and logs * **runbook**: https://wikitech.wikimedia.org/wiki/Periodic_jobs#Troubleshooting * **summary**: MediaWiki periodic job echo-mail-batch failed * **alertname**: MediaWikiCronJobFailed * **label_cronjob**: echo-mail-batch * **label_team**: notifications-echo * **prometheus**: k8s * **severity**: task * **site**: codfw * **source**: prometheus * **team**: notifications-echo * [Source](https://prometheus-codfw.wikimedia.org/k8s/graph?g0.expr=sum+by+%28label_cronjob%2C+label_team%29+%28kube_job_status_failed%7Bjob_name%21%3D%22wikidata-updatequeryservicelag%22%2Cnamespace%3D%22mw-cron%22%7D+%2A+on+%28namespace%2C+job_name%29+group_left+%28label_cronjob%2C+label_team%29+kube_job_labels%7Blabel_cronjob%21%3D%22wikidata-updatequeryservicelag%22%2Cnamespace%3D%22mw-cron%22%7D%29+%3E+0&g0.tab=1)

== Common information * **dashboard**: https://w.wiki/DocPlogstash.wikimedia.org/app/dashboards#/view/d51552d0-e309-11ef-87d0-9371e01d3c68?_a=(filters:!((query:(match_phrase:(kubernetes.labels.cronjob.keyword:echo-mail-batch)))))&_g=(time:(from:now-24h,to:now)) * **description**: Use `kube-env mw-cron eqiad;codfw; kubectl get jobs -l team=notifications-echo,cronjob=echo-mail-batch --field-selector status.successful=0; kubectl logs -l cronjob=echo-mail-batch --timestamps=true --prefix=true -c mediawiki-main-app` to see failuress and logs * **runbook**: https://wikitech.wikimedia.org/wiki/Periodic_jobs#Troubleshooting * **summary**: MediaWiki periodic job echo-mail-batch failed * **alertname**: MediaWikiCronJobFailed * **label_cronjob**: echo-mail-batch * **label_team**: notifications-echo * **prometheus**: k8s * **severity**: task * **site**: eqiadcodfw * **source**: prometheus * **team**: notifications-echo == Firing alerts --- * **dashboard**: https://w.wiki/DocPlogstash.wikimedia.org/app/dashboards#/view/d51552d0-e309-11ef-87d0-9371e01d3c68?_a=(filters:!((query:(match_phrase:(kubernetes.labels.cronjob.keyword:echo-mail-batch)))))&_g=(time:(from:now-24h,to:now)) * **description**: Use `kube-env mw-cron eqiad;codfw; kubectl get jobs -l team=notifications-echo,cronjob=echo-mail-batch --field-selector status.successful=0; kubectl logs -l cronjob=echo-mail-batch --timestamps=true --prefix=true -c mediawiki-main-app` to see failuress and logs * **runbook**: https://wikitech.wikimedia.org/wiki/Periodic_jobs#Troubleshooting * **summary**: MediaWiki periodic job echo-mail-batch failed * **alertname**: MediaWikiCronJobFailed * **label_cronjob**: echo-mail-batch * **label_team**: notifications-echo * **prometheus**: k8s * **severity**: task * **site**: eqiadcodfw * **source**: prometheus * **team**: notifications-echo * [Source](https://prometheus-eqiadcodfw.wikimedia.org/k8s/graph?g0.expr=sum+by+%28label_cronjob%2C+label_team%29+%28kube_job_status_failed%7Bjob_name%21%3D%22wikidata-updatequeryservicelag%22%2Cnamespace%3D%22mw-cron%22%7D+%2A+on+%28namespace%2C+job_name%29+group_left+%28label_cronjob%2C+label_team%29+kube_job_labels%7Blabel_cronjob%21%3D%22wikidata-updatequeryservicelag%22%2Cnamespace%3D%22mw-cron%22%7D%29+%3E+0&g0.tab=1)