Indexes created with monthly indexes %{+YYYY.MM} share the same pattern as weekly indexes %{+xxxx.ww}. This could lead to problems when curating weekly and monthly indexes if they matching the same curator pattern.
One possible solution is to make weekly buckets underscore-separated and calendar year buckets dot-separated.
@herron @fgiunchedi what do you think about that convention?
Sounds like that'd work, although I wonder if there are any alternatives that may be more obvious at a glance? Could we get away with doing something like including the unit as a keyword before the stamp? e.g.
ecs-1.7.0-5-alerts-bucket-daily-2022.01.01
ecs-1.7.0-5-alerts-bucket-weekly-2022.01
ecs-1.7.0-5-alerts-bucket-monthly-2022.01
Agreed, something explicit in the index name seems preferable to me too and less prone to errors.
I want to explore this idea further. It would remove the need to key off of outputs by default we included bucket in the index pattern. This would look like:
This matches ^(?<output>[a-z0-9]+)-((?<partition>[a-z0-9]+)-)?((?<version>[0-9\.]+)-(?<revision>[0-9]+)-)?(?<bucket>daily|weekly|monthly|yearly)-(?<created>[0-9.]+)$.
I do not think this will simplify our curator config much. We'll be trading output-defined actions for a set of implicit (bucket-defined) actions and adding exceptions. This would look like:
This introduces a problem where if we introduce a new monthly index with a 9-month retention (e.g. ecs-foo-*-monthly-2022.04), the monthly action (delete older than 3 months) would have to exclude the pattern in addition to setting up the 9-month action.
I see a link between the creation format and retention. If we were to extend the definition of "bucket" to combine format and retention ("policy", perhaps?), we could reuse curator actions more effectively. An example strawman approach:
^(?<output>[a-z0-9]+)-((?<partition>[a-z0-9]+)-)?((?<version>[0-9\.]+)-(?<revision>[0-9]+)-)?(?<policy>[a-z0-9]+)-(?<created>[0-9.]+)$
daily: format: %Y.%m.%d retention: 90 days example: logstash-mediawiki-daily-2022.04.04 weekly: format: %Y.%W retention: 14 weeks example: w3creportingapi-1.0.0-2-weekly-2022.14 monthly: format: %Y.%m retention: 3 months example: ecs-foo-1.7.0-5-monthly-2022.04 3years: format: %Y retention: 3 years example: ecs-alerts-1.7.0-5-3years-2022 2weeks: format: %Y.%W retention: 2 weeks example: ecs-test-1.7.0-5-2weeks-2022.14 2days: format: %Y.%m.%d retention: 2 days example: dlq-1.0.0-1-2days-2022.04.04
What do you think?
If I'm understanding correctly the idea is to have a set of generic curator rules that would automatically set retention based on patterns like "2weeks" or "2days" in the index name?
One challenge that comes to mind is transitioning retention time on a given index. Say increasing from 2 weeks to 4 weeks. IIUC we would be on the hook to handle renaming/reindexing of the affected existing indices from 2weeks to 4weeks or weekly shortly after the next index rollover to avoid losing them.
Yes, the pattern combines retention period and bucket. "2days", etc. is an example to illustrate the concept. They could just as easily be more kind-of-data-oriented approach (ecs-alerts-1.7.0-5-alerts-2022.14) or a policy revision approach (ecs-1.7.0-5-alerts-0-2022.14).
One challenge that comes to mind is transitioning retention time on a given index. Say increasing from 2 weeks to 4 weeks. IIUC we would be on the hook to handle renaming/reindexing of the affected existing indices from 2weeks to 4weeks or weekly shortly after the next index rollover to avoid losing them.
Good point. Policies named with such specific terms makes them less flexible. Perhaps kind-of-data-oriented policies is preferable here. To evaluate the procedure for adjustment:
It occurs to me we could also get there by leveraging the output-partition pair to achieve the same goal. For example:
ecs-1.7.0-5-alerts-2022.14: format: %Y.%W retention: 105 weeks ecs-1.7.0-5-alerts2-2022: format: %Y retention: 2 years
It also occurs to me a combination of output-partition-policy revision would get us there. For example:
ecs-1.7.0-5-alerts-0-2022.14 format: %Y.%W retention: 105 weeks ecs-1.7.0-5-alerts-1-2022 format: %Y retention: 2 years
The more we explore, the more I see the benefits of rolling out a modified index pattern to all indexes like so: ^(?<output>[a-z0-9]+)-(?<partition>[a-z0-9]+)-(?<policy_revision>[0-9]+)-(?<template_version>[0-9\.]+)-(?<template_revision>[0-9]+)-(?<created>[0-9.]+)$
This would:
The indexes would then be formatted like so:
This could homogenize the curator filters into a predictable format and replace all regex filters with prefix filters. Unfortunately, we could not construct any generic "catch-all" rules, but we do not have that even now and is not a hard requirement AFAIK.
Per discussion today in the Observability meeting, we agreed to evaluate the proposed index name refactor to:
Change 775375 had a related patch set uploaded (by Cwhite; author: Cwhite):
[operations/puppet@production] logstash: populate target index format and add pipeline diagnostics
Change 777874 had a related patch set uploaded (by Cwhite; author: Cwhite):
[operations/puppet@production] logstash: replace all instances of @metadata.partition
Change 777880 had a related patch set uploaded (by Cwhite; author: Cwhite):
[operations/puppet@production] logstash: set partition on legacy indexes
Change 777882 had a related patch set uploaded (by Cwhite; author: Cwhite):
[operations/puppet@production] logstash: transform human-friendly values to bucket date format
Change 777891 had a related patch set uploaded (by Cwhite; author: Cwhite):
[operations/puppet@production] logstash: add target index validation step
Change 777874 merged by Cwhite:
[operations/puppet@production] logstash: replace all instances of @metadata.partition
Change 775375 merged by Cwhite:
[operations/puppet@production] logstash: populate target index format and add pipeline diagnostics
Change 777880 merged by Cwhite:
[operations/puppet@production] logstash: set partition on legacy indexes
Change 777882 merged by Cwhite:
[operations/puppet@production] logstash: transform rotation frequency values to datestamp format
Change 777891 merged by Cwhite:
[operations/puppet@production] logstash: add target index validation step
Change 798974 had a related patch set uploaded (by Cwhite; author: Cwhite):
[operations/puppet@production] beta-logs: enable pipeline-managed index patterns
Change 798974 merged by Cwhite:
[operations/puppet@production] beta-logs: enable pipeline-managed index patterns
Change 798982 had a related patch set uploaded (by Cwhite; author: Cwhite):
[operations/puppet@production] logstash: curator support new and legacy index patterns
Change 799001 had a related patch set uploaded (by Cwhite; author: Cwhite):
[operations/puppet@production] logstash: enable pipeline-managed index patterns
Change 798982 merged by Cwhite:
[operations/puppet@production] logstash: curator support new and legacy index patterns
Change 802873 had a related patch set uploaded (by Cwhite; author: Cwhite):
[operations/software/ecs@master] add new index pattern format
Change 802873 merged by jenkins-bot:
[operations/software/ecs@master] add new index pattern format
Change 803345 had a related patch set uploaded (by Cwhite; author: Cwhite):
[operations/software/ecs@master] templates: replace all version instances
Change 803345 merged by jenkins-bot:
[operations/software/ecs@master] templates: replace all version instances
Change 803350 had a related patch set uploaded (by Cwhite; author: Cwhite):
[operations/puppet@production] add new index pattern to ecs templates
Change 803350 merged by Cwhite:
[operations/puppet@production] logstash: add new index pattern to ecs templates
Change 799001 merged by Cwhite:
[operations/puppet@production] logstash: enable pipeline-managed index patterns
Deployed! Will be watching it this week for any issues.
Last thing to do is migrate w3creportingapi-* and dlq-*.
Change 815799 had a related patch set uploaded (by Cwhite; author: Cwhite):
[operations/puppet@production] logstash: add missing closing curly brace
Change 815799 merged by Cwhite:
[operations/puppet@production] logstash: add missing closing curly brace
Change 822444 had a related patch set uploaded (by Cwhite; author: Cwhite):
[operations/puppet@production] logstash: replace legacy routing filters
Change 822450 had a related patch set uploaded (by Cwhite; author: Cwhite):
[operations/puppet@production] logstash: use logstash routing for w3creportingapi stream
Change 822452 had a related patch set uploaded (by Cwhite; author: Cwhite):
[operations/puppet@production] logstash: update production w3creportingapi guard condition
Change 822450 merged by Cwhite:
[operations/puppet@production] logstash: use logstash routing for w3creportingapi stream
Change 822452 merged by Cwhite:
[operations/puppet@production] logstash: update production w3creportingapi guard condition
Change 824751 had a related patch set uploaded (by Cwhite; author: Cwhite):
[operations/puppet@production] beta-logs: dlq use logstash-managed index pattern
Change 824752 had a related patch set uploaded (by Cwhite; author: Cwhite):
[operations/puppet@production] logstash: dlq use logstash-managed index pattern
Change 824753 had a related patch set uploaded (by Cwhite; author: Cwhite):
[operations/puppet@production] beta-logs: w3creportingapi to use logstash-managed index pattern
Change 824754 had a related patch set uploaded (by Cwhite; author: Cwhite):
[operations/puppet@production] logstash: w3creportingapi to use logstash-managed index pattern
Change 824751 merged by Cwhite:
[operations/puppet@production] beta-logs: dlq use logstash-managed index pattern
Change 824753 merged by Cwhite:
[operations/puppet@production] beta-logs: w3creportingapi to use logstash-managed index pattern
Change 824752 merged by Cwhite:
[operations/puppet@production] logstash: dlq use logstash-managed index pattern
Change 826372 had a related patch set uploaded (by Cwhite; author: Cwhite):
[operations/puppet@production] logstash: add dlq revision to index pattern
Change 826372 merged by Cwhite:
[operations/puppet@production] logstash: use puppet dlq version and revision for index pattern
Change 824754 merged by Cwhite:
[operations/puppet@production] logstash: w3creportingapi to use logstash-managed index pattern