Change Details

Hi! We would like to use an S3 backend for our flink-based [[ https://phabricator.wikimedia.org/T317045 | search update pipeline ]]. This is needed for persisting checkpoints, aka state of stateful operations inside the application, which in turn allows the application to seamlessly pick up where a predecessor left of (when being stopped for whatever reason). In analogy to [[ https://phabricator.wikimedia.org/T330693 | T330693 (s3 for flink-based enrichment application) ]] we would need an account to store flink checkpoints and savepoints. This account would have access to three containers: - cirrussearch-update-pipeline-eqiad - cirrussearch-update-pipeline-codfw - cirrussearch-update-pipeline-staging The storage needs for each container (excluding staging) depends on the various options that we are still debating in T341625 but should be: - 200Gb for the option 0 - 21Gb for the option 1 - 4Gb for the option 2 The storage needs for the staging container should be minimal as it will be used for staging deploys for which we'll probably cover only one test wiki so in total the account needs a storage quota twice as big as the values stated above. This space is primarily used for operating flink, the loss of this state should not be terrible and can be solved by restarting the job from earlier kafka offsets (~5min). See for the detailed estimation [[ https://docs.google.com/spreadsheets/d/1Fp44MdLxUVlxi03MBD_64m0zQErny-9jUD5C6RGf_bU/edit#gid=2037151092 | checkpoint storage estimation ]].

Hi! We would like to use an S3 backend for our flink-based [[ https://phabricator.wikimedia.org/T317045 | search update pipeline ]]. This is needed for persisting checkpoints, aka state of stateful operations inside the application, which in turn allows the application to seamlessly pick up where a predecessor left of (when being stopped for whatever reason). In analogy to [[ https://phabricator.wikimedia.org/T330693 | T330693 (s3 for flink-based enrichment application) ]] we would need an account to store flink checkpoints and savepoints. This account would have access to three containers: - cirrussearch-update-pipeline-eqiad - cirrussearch-update-pipeline-codfw - cirrussearch-update-pipeline-staging The storage needs for each container (excluding staging) would be 21G. The storage needs for the staging container should be minimal as it will be used for staging deploys for which we'll probably cover only one test wiki so in total the account needs a storage quota twice as big as the values stated above. This space is primarily used for operating flink, the loss of this state should not be terrible and can be solved by restarting the job from earlier kafka offsets (~5min). See for the detailed estimation [[ https://docs.google.com/spreadsheets/d/1Fp44MdLxUVlxi03MBD_64m0zQErny-9jUD5C6RGf_bU/edit#gid=2037151092 | checkpoint storage estimation ]]. Thanks for looking and please let us know if you need more info.

Hi! We would like to use an S3 backend for our flink-based [[ https://phabricator.wikimedia.org/T317045 | search update pipeline ]]. This is needed for persisting checkpoints, aka state of stateful operations inside the application, which in turn allows the application to seamlessly pick up where a predecessor left of (when being stopped for whatever reason). In analogy to [[ https://phabricator.wikimedia.org/T330693 | T330693 (s3 for flink-based enrichment application) ]] we would need an account to store flink checkpoints and savepoints. This account would have access to three containers: - cirrussearch-update-pipeline-eqiad - cirrussearch-update-pipeline-codfw - cirrussearch-update-pipeline-staging The storage needs for each container (excluding staging) depends on the various options that we are still debating in T341625 but should be: - 200Gb for the option 0 - 21Gb for the option 1 - 4Gb for the option 2would be 21G. The storage needs for the staging container should be minimal as it will be used for staging deploys for which we'll probably cover only one test wiki so in total the account needs a storage quota twice as big as the values stated above. This space is primarily used for operating flink, the loss of this state should not be terrible and can be solved by restarting the job from earlier kafka offsets (~5min). See for the detailed estimation [[ https://docs.google.com/spreadsheets/d/1Fp44MdLxUVlxi03MBD_64m0zQErny-9jUD5C6RGf_bU/edit#gid=2037151092 | checkpoint storage estimation ]]. Thanks for looking and please let us know if you need more info.