Will the be only the druid-analytics cluster that is affected, or should we apply the same configuration to the druid-public cluster?
I take it that we can test the settings in the druid-test cluster, too?

We very much can test the setting onto the test-cluster.
I think there is small benefit in having the same parameter for the druid-public cluster: there is no reindexation, so no unused segments that should be deleted (we delete entire datasources in there).

Is there some overlap between this ticket and T296207: Review druid deep-storage making sure that old segments having been reindexed are deleted?

In T376118#10193763, @BTullis wrote:

Is there some overlap between this ticket and T296207: Review druid deep-storage making sure that old segments having been reindexed are deleted?

Good call! They are indeed pretty much the same.
I have made the test of deleting unused segments trough kill-tasks, so I'll merge T296207 into this one.
Thanks for keeping us clean @BTullis :)

Change #1077653 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Update druid test config to drop unused segments automatically

https://gerrit.wikimedia.org/r/1077653

I have the beginnings of a patch to enable the dropping of unused segments in the test Druid cluster. https://gerrit.wikimedia.org/r/1077653

I'm wondering whether what we should set for the other parameters that you mentioned:
druid.coordinator.kill.period is described as:

The frequency of sending kill tasks to the indexing service.

It defaults to the same value as druid.coordinator.period.indexingPeriod

We don't set druid.coordinator.period.indexingPeriod so I believe that it taskes the default value of PT1800S which is every 30 minutes.
This seems OK to me, but what do you think?

For druid.coordinator.kill.ignoreDurationToRetain I'd like to be sure that I understand the implications, regarding the an-druid cluster. It says:

A way to override druid.coordinator.kill.durationToRetain and tell the coordinator that you do not care about the end date of unused segment intervals when it comes to killing them.
If true, the coordinator considers all unused segments as eligible to be killed.

The default value for druid.coordinator.kill.durationToRetain is 90D and the default for druid.coordinator.kill.ignoreDurationToRetain is false.

So my understanding is that we would always retain the last 90 days' worth of segments by default.
Should I set this value to true so that we delete all unused segments, or should we reduce the durationToRetain value to something lower than 90D?

Should I set this value to true so that we delete all unused segments, or should we reduce the durationToRetain value to something lower than 90D?

I think we can keep the default values. Keeping a safe zone of data not being fully deleted is good, and while it grows the data we keep, it's not that long and should be ok.
Thank you for making this happen @BTullis :)

Change #1077653 merged by Btullis:

[operations/puppet@production] Update druid test config to drop unused segments automatically

https://gerrit.wikimedia.org/r/1077653

@JAllemandou - I've applied the changes to an-test-druid1001, but the coordinator won't start.

2024-11-13T11:25:45,353 ERROR org.apache.druid.cli.CliCoordinator: Error when starting up.  Failing.
com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) Error injecting constructor, java.lang.IllegalArgumentException: coordinator kill retainDuration must be >= 0

I've manually reverted for now and I will rework the patch.

Change #1090828 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Add missing params to enable druid test to delete unused segments

https://gerrit.wikimedia.org/r/1090828

Change #1090828 merged by Btullis:

[operations/puppet@production] Add missing params to enable druid test to delete unused segments

https://gerrit.wikimedia.org/r/1090828

Change #1090842 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Enable deletion of unused segments on the druid-analytics cluster

https://gerrit.wikimedia.org/r/1090842

@JAllemandou - I've now successfully applied the settings to the test cluster and here is the patch to apply the same settings the druid-analytics cluster: https://gerrit.wikimedia.org/r/1090842
If you'd like to take a look and check that it's working correctly, that would be great.

It looks like a couple of those default values aren't compiled into our slightly older version of Druid, after all.

Change #1090842 merged by Btullis:

[operations/puppet@production] Enable deletion of unused segments on the druid-analytics cluster

https://gerrit.wikimedia.org/r/1090842