Page MenuHomePhabricator

ES Curator cron jobs are not cleaned up when output no longer exists
Closed, ResolvedPublic

Description

The curator configuration is removed

Notice: /Stage[main]/Elasticsearch::Curator/File[/etc/curator/cleanup_w3creportingapi-1.0.0-1.yaml]/ensure: removed
Notice: /Stage[main]/Elasticsearch::Curator/File[/etc/curator/config-w3creportingapi-1.0.0-1.yaml]/ensure: removed

But there is no corresponding cron entry cleanup

Notice: /Stage[main]/Profile::Logstash::Collector7/Logstash::Output::Elasticsearch[w3creportingapi-1.0.0-2]/Cron[logstash_cleanup_indices_w3creportingapi-1.0.0-2]/ensure: created

sudo crontab -l output:

(...)
# Puppet Name: logstash_cleanup_indices_w3creportingapi-1.0.0-1
42 0 * * * /usr/bin/curator --config /etc/curator/config-w3creportingapi-1.0.0-1.yaml /etc/curator/cleanup_w3creportingapi-1.0.0-1.yaml > /dev/null
# Puppet Name: logstash_cleanup_indices_w3creportingapi-1.0.0-2
42 0 * * * /usr/bin/curator --config /etc/curator/config-w3creportingapi-1.0.0-2.yaml /etc/curator/cleanup_w3creportingapi-1.0.0-2.yaml > /dev/null

Event Timeline

colewhite triaged this task as Medium priority.Mar 22 2021, 3:29 PM

Change 676631 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: use curator cluster config when possible

https://gerrit.wikimedia.org/r/676631

Change 676685 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: set cluster name for elasticsearch outputs

https://gerrit.wikimedia.org/r/676685

Change 676687 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: use logstash output to manage ecs-test indexes

https://gerrit.wikimedia.org/r/676687

Change 676690 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: add curator config to manage w3creportingapi revision 1 indexes

https://gerrit.wikimedia.org/r/676690

Change 677593 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: refactor how curator jobs are defined and deployed

https://gerrit.wikimedia.org/r/677593

Mentioned in SAL (#wikimedia-operations) [2021-04-14T15:00:44Z] <shdubsh> run new curator actions on codfw - T274394

Change 677593 merged by Cwhite:

[operations/puppet@production] logstash: refactor how curator jobs are defined and deployed

https://gerrit.wikimedia.org/r/677593

Change 679524 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: provision per-datacenter apifeatureusage jobs

https://gerrit.wikimedia.org/r/679524

Change 679525 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: clean up apifeatureusage curator job

https://gerrit.wikimedia.org/r/679525

Change 679553 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] elasticsearch: curator remove stdout redirect

https://gerrit.wikimedia.org/r/679553

Change 679553 merged by Cwhite:

[operations/puppet@production] elasticsearch: curator remove stdout redirect

https://gerrit.wikimedia.org/r/679553

Change 679525 merged by Cwhite:

[operations/puppet@production] logstash: clean up apifeatureusage curator job

https://gerrit.wikimedia.org/r/679525

Change 679524 merged by Cwhite:

[operations/puppet@production] logstash: provision per-target apifeatureusage jobs

https://gerrit.wikimedia.org/r/679524

Change 680399 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: limit apifeatureusage curator job to jobs_host

https://gerrit.wikimedia.org/r/680399

Change 676631 abandoned by Cwhite:

[operations/puppet@production] logstash: use curator cluster config when possible

Reason:

superseded by Ia392fb108bd96f3106ed02615a03e7cc3e43993f

https://gerrit.wikimedia.org/r/676631

Change 676685 abandoned by Cwhite:

[operations/puppet@production] logstash: set cluster name for elasticsearch outputs

Reason:

superseded by Ia392fb108bd96f3106ed02615a03e7cc3e43993f

https://gerrit.wikimedia.org/r/676685

Change 676687 abandoned by Cwhite:

[operations/puppet@production] logstash: use logstash output to manage ecs-test indexes

Reason:

superseded by Ia392fb108bd96f3106ed02615a03e7cc3e43993f

https://gerrit.wikimedia.org/r/676687

Change 676690 abandoned by Cwhite:

[operations/puppet@production] logstash: add curator config to manage w3creportingapi revision 1 indexes

Reason:

superseded by Ia392fb108bd96f3106ed02615a03e7cc3e43993f

https://gerrit.wikimedia.org/r/676690

Change 680399 merged by Cwhite:

[operations/puppet@production] logstash: limit apifeatureusage curator job to jobs_host

https://gerrit.wikimedia.org/r/680399

Refactor complete!

Change 690539 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: clean up deprecated curator entries

https://gerrit.wikimedia.org/r/690539

Change 690539 merged by Cwhite:

[operations/puppet@production] logstash: clean up deprecated curator entries

https://gerrit.wikimedia.org/r/690539