Page MenuHomePhabricator
Create Task
Maniphest T350826

Test backfilling for cirrus-streaming-updater
Closed, ResolvedPublic5 Estimated Story Points
Actions

Description

At today's Wednesday meeting, @EBernhardson, @dcausse and the rest of us discussed the next steps for the Search Update Pipeline.

We believe a backfill test is needed, so we have a better idea of how the Search Update Pipeline application, Flink, and our infrastructure (Kafka) behave during a backfill.

Things that needs to be addressed by the test:

  • capacity on k8s and on all backends (swift, mw-api, kafka, elasticsearch)
  • functional correctness

Potential issues:

  • We don't have a target elasticsearch cluster
  • This will require enabling re-render for wikis under test (with load impact)

Creating this ticket to:

  • Devise a backfill test
  • Run the test
  • Collect the results
  • Address blocking issues

Related Objects
Search...

Event Timeline

bking created this task.Nov 8 2023, 7:48 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 8 2023, 7:48 PM
bking added a comment.Nov 8 2023, 7:52 PM

My rough notes around this subject are here . I'm still learning Flink and Kafka, so will need some help creating the backfill test.

bking added a parent task: T340548: [EPIC] Deployment of the Search Update Pipeline on Flink / k8s.Nov 8 2023, 7:55 PM
bking mentioned this in T350185: Estimate cirrus streaming updater's usage of MWAPI.Nov 8 2023, 8:28 PM
Gehel moved this task from needs triage to Current work on the Discovery-Search board.Nov 13 2023, 4:16 PM
Gehel edited projects, added Discovery-Search (Current work); removed Discovery-Search.
Gehel updated the task description. (Show Details)Nov 13 2023, 4:34 PM
Gehel set the point value for this task to 5.Nov 13 2023, 4:43 PM
Gehel moved this task from Incoming to Ready for Dev -- SWE on the Discovery-Search (Current work) board.
Gehel triaged this task as High priority.Nov 15 2023, 9:36 AM
Gehel moved this task from Incoming to Misc on the Data-Platform-SRE board.
Gehel moved this task from Misc to Quarterly Goals on the Data-Platform-SRE board.
bking mentioned this in T342538: Q1:rack/setup/install cloudelastic10[07-10].wikimedia.org.Nov 15 2023, 6:05 PM
ops-monitoring-bot added a comment.Nov 15 2023, 6:12 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cloudelastic1008.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Nov 15 2023, 6:15 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cloudelastic1009.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Nov 15 2023, 6:16 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cloudelastic1010.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Nov 15 2023, 7:34 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cloudelastic1008.wikimedia.org with OS bullseye executed with errors:

  • cloudelastic1008 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Nov 15 2023, 7:36 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cloudelastic1009.wikimedia.org with OS bullseye executed with errors:

  • cloudelastic1009 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Nov 15 2023, 7:37 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cloudelastic1010.wikimedia.org with OS bullseye executed with errors:

  • cloudelastic1010 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
bking added a comment.Nov 15 2023, 8:43 PM

Apologies for the reimage spam, it's from an unrelated operation.

ops-monitoring-bot added a comment.Nov 16 2023, 3:01 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cloudelastic1010.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Nov 16 2023, 3:01 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cloudelastic1009.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Nov 16 2023, 4:21 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cloudelastic1010.wikimedia.org with OS bullseye executed with errors:

  • cloudelastic1010 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Nov 16 2023, 4:21 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cloudelastic1009.wikimedia.org with OS bullseye executed with errors:

  • cloudelastic1009 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Nov 16 2023, 4:58 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cloudelastic1009.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Nov 16 2023, 5:16 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cloudelastic1010.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Nov 16 2023, 5:18 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cloudelastic1010.wikimedia.org with OS bullseye executed with errors:

  • cloudelastic1010 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Nov 16 2023, 5:19 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cloudelastic1010.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Nov 16 2023, 5:20 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cloudelastic1010.wikimedia.org with OS bullseye executed with errors:

  • cloudelastic1010 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Nov 16 2023, 5:23 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cloudelastic1010.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Nov 16 2023, 5:28 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cloudelastic1009.wikimedia.org with OS bullseye completed:

  • cloudelastic1009 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311161713_bking_2708107_cloudelastic1009.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Nov 16 2023, 6:44 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cloudelastic1010.wikimedia.org with OS bullseye executed with errors:

  • cloudelastic1010 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
Gehel moved this task from Ready for Dev -- SWE to In Progress on the Discovery-Search (Current work) board.Nov 17 2023, 9:32 AM
dcausse mentioned this in T351503: Enable mediawiki.cirrussearch.page_rerender.v1 on all public wikis.Nov 17 2023, 10:03 AM
Gehel assigned this task to bking.Dec 4 2023, 4:19 PM
Gehel subscribed.

Once we have a working deployment on Cloudelastic (T352335), we can just re-run a backfill operation there.

EBernhardson added a comment.Dec 7 2023, 5:34 PM

Another option we came up with was to backfill to a null flink sink, this would allow measuring capacity of the flink pipeline by itself, separate from the ability of the chosen elasticsearch cluster to consumer those updates.

Gehel added a comment.Dec 18 2023, 4:14 PM

We probably want to run the backfill test once we have enabled all wikis, so that the load generated is maximized.

Gehel added a comment.Jan 8 2024, 4:14 PM

Still needs to be done:

  • HTTP timeout
  • Documentation
  • Validate elasticsearch ingestion throughput once we're on Cloudelastic
pfischer subscribed.Jan 8 2024, 4:52 PM

One of the key findings of the backfill tests was a lower-than-expected throughput, see T353460. That is mainly caused by a bug inside the flink consumer but is still a unexpectedly high rate of timeouts (from enovys perspective) see T354289.

Gehel reassigned this task from bking to dcausse.Jan 22 2024, 4:12 PM
Gehel moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board.Feb 5 2024, 4:14 PM
Gehel closed this task as Resolved.Feb 9 2024, 9:21 AM
Gehel edited projects, added Data-Platform-SRE (2024.01.22 - 2024.02.11); removed Data-Platform-SRE.Jan 10 2025, 4:04 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits