@dcausse IIRC we said that "something in the areas of hours" would be considered a "short maintenance" and thus would not need any additional actions to be carried out, right?
As part of T251305 we will re-create the helm release of flink in both datacenters (one after the other ofc.) and that would mean flink will be down for a couple of minutes. If my memory and understanding is still intact, the checkpoint/tombstone metadata is not part of the helm release itself (it's in those flink managed configmaps). So it should survive purging and recreating the helm release.
@Jelto has alredy done that for the staging flink release. If you have the chance it would be nice if you could double check that is still working as expected.

Besides that I tried to understand what would be needed to do for a "longer downtime" of k8s and it's not exactly clear to me. Could we have a dedicated section for that on whe wikitech page? IIRC that also needed a change to WQDS itself.

In T293063#7491903, @JMeybohm wrote:

@dcausse IIRC we said that "something in the areas of hours" would be considered a "short maintenance" and thus would not need any additional actions to be carried out, right?

We are targeting a SLO with an update lag below 10minutes for 99% of the time, we are still learning what is the operational cost of this and are happy to discuss/re-adjust all this depending on your constraints.

As part of T251305 we will re-create the helm release of flink in both datacenters (one after the other ofc.) and that would mean flink will be down for a couple of minutes. If my memory and understanding is still intact, the checkpoint/tombstone metadata is not part of the helm release itself (it's in those flink managed configmaps). So it should survive purging and recreating the helm release.

Yes if the configmaps are kept flink will just autorestart on its own, regarding lag I'm not worried as flink already restarts on its own from time to time without affecting the 10min lag SLO.

@Jelto has alredy done that for the staging flink release. If you have the chance it would be nice if you could double check that is still working as expected.

Checking the logs I see 2 restarts in the last 7 days and both restarts properly restored the job:

Nov 3, 2021 @ 15:44:33.739	syslog	kubestage1002	Restoring job 095b671d83457ebf4c59166fda7a7055 from Checkpoint 106609 @ 1635954210959 for 095b671d83457ebf4c59166fda7a7055 located at swift://rdf-streaming-updater-staging.thanos-swift/wikidata/checkpoints/095b671d83457ebf4c59166fda7a7055/chk-106609.

Nov 4, 2021 @ 13:36:35.097	syslog	kubestage1002	Restoring job 095b671d83457ebf4c59166fda7a7055 from Checkpoint 109216 @ 1636032918483 for 095b671d83457ebf4c59166fda7a7055 located at swift://rdf-streaming-updater-staging.thanos-swift/wikidata/checkpoints/095b671d83457ebf4c59166fda7a7055/chk-109216.

So, if one of these restarts corresponds to the helm 3 upgrade then I can confirm that it will work properly the production clusters.

Besides that I tried to understand what would be needed to do for a "longer downtime" of k8s and it's not exactly clear to me. Could we have a dedicated section for that on whe wikitech page? IIRC that also needed a change to WQDS itself.

Certainly, this task is all about clarifying all this.

@JMeybohm thanks for the write-up! I added few more notes.

Hey @dcausse, I'm reading this again because of the upcoming k8s 1.23 upgrade and was wondering:
In "To restore:" section of "Alternate actions (not fully untested):" - do we need to start the job somehow as well, specifying which jar file to use? Or is that information part of the configmaps/safepoint and the job can start automatically without submitting a jar?

In T293063#8582548, @JMeybohm wrote:

Hey @dcausse, I'm reading this again because of the upcoming k8s 1.23 upgrade and was wondering:
In "To restore:" section of "Alternate actions (not fully untested):" - do we need to start the job somehow as well, specifying which jar file to use? Or is that information part of the configmaps/safepoint and the job can start automatically without submitting a jar?

Hey, clarified this a bit, renamed it to "Hard depool/re-pool", yes in this method the jobs should start right after the helm deploy, the jar is stored in swift so no need to deploy it manually.

In T293063#8582600, @dcausse wrote:

Hey, clarified this a bit, renamed it to "Hard depool/re-pool", yes in this method the jobs should start right after the helm deploy, the jar is stored in swift so no need to deploy it manually.

Cool, thanks. That would make it hands-off for anybody but sre/serviceops which ofc would be nice.

Anyhow. AIUI this process will be more or less the same for flink deployments managed by the flink operator. It would be nice if you could verify this during your tests with the operator (I'm happy to help/pair ofc.) or if there maybe even is a better option in flink-operator world.

In T293063#8582625, @JMeybohm wrote:

Anyhow. AIUI this process will be more or less the same for flink deployments managed by the flink operator. It would be nice if you could verify this during your tests with the operator (I'm happy to help/pair ofc.) or if there maybe even is a better option in flink-operator world.

Yes definitely! (I also have some hope that the k8s-operator might be able to discover the latest valid checkpoint without having to save the config maps but we'll see...). Thanks for the help!