Change Details

At today's Wednesday meeting, @EBernhardson, @dcausse and the rest of us discussed the next steps for the Search Update Pipeline. We believe a backfill test is needed, so we have a better idea of how the Search Update Pipeline application, Flink, and our infrastructure (Kafka) behave during a backfill. Creating this ticket to: - Devise a backfill test - Run the test - Collect the results

At today's Wednesday meeting, @EBernhardson, @dcausse and the rest of us discussed the next steps for the Search Update Pipeline. We believe a backfill test is needed, so we have a better idea of how the Search Update Pipeline application, Flink, and our infrastructure (Kafka) behave during a backfill. Things that needs to be addressed by the test: * capacity on k8s and on all backends (swift, mw-api, kafka, elasticsearch) * functional correctness Potential issues: * We don't have a target elasticsearch cluster * This will require enabling re-render for wikis under test (with load impact) Creating this ticket to: - Devise a backfill test - Run the test - Collect the results - Address blocking issues

At today's Wednesday meeting, @EBernhardson, @dcausse and the rest of us discussed the next steps for the Search Update Pipeline. We believe a backfill test is needed, so we have a better idea of how the Search Update Pipeline application, Flink, and our infrastructure (Kafka) behave during a backfill. Things that needs to be addressed by the test: * capacity on k8s and on all backends (swift, mw-api, kafka, elasticsearch) * functional correctness Potential issues: * We don't have a target elasticsearch cluster * This will require enabling re-render for wikis under test (with load impact) Creating this ticket to: - Devise a backfill test - Run the test - Collect the results - Address blocking issues