Page MenuHomePhabricator
Create Task
Maniphest T341705

eqiad: 3 VMs requested for Zookeeper
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

Site/Location: eqiad
Number of systems: 3
Service: zookeeper - on behalf of the Search Platform Team
Networking Requirements: internal IP - needs to connect to WDQS, kafka-main, and Elasticsearch . I see VLANS 1017-1019 so I assume any of those will do.

Processor Requirements: 4 vCPUs
Memory: 8 GB of RAM
Disks: 50 GB
Other Requirements: None

Thanks for looking! Please let me know if you have any questions.

AC:

  • 3 VMs are create with the above specs, and the in_setup puppet role

Details

SubjectRepoBranchLines +/-
flink-zk: Add hostnames for CODFW clusteroperations/puppetproduction+5 -1
netboot.cfg: explicitly define partitioning schemeoperations/puppetproduction+1 -0
flink-zk: Provision hostnames for new ZK clusteroperations/puppetproduction+5 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

bking created this task.Jul 12 2023, 3:14 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 12 2023, 3:14 PM
jbond added a project: vm-requests.Jul 12 2023, 3:16 PM
jbond subscribed.

approved

bking added a parent task: T317045: [Epic] Re-architect the Search Update Pipeline.Jul 12 2023, 3:18 PM
bking added a parent task: T331283: [Event Platform] [NEEDS GROOMING] Store Flink HA metadata in Zookeeper.Jul 12 2023, 8:35 PM
Gehel moved this task from needs triage to Current work on the Discovery-Search board.Jul 17 2023, 3:29 PM
Gehel edited projects, added Discovery-Search (Current work); removed Discovery-Search.
Gehel moved this task from Incoming to Ready for Work on the Data-Platform-SRE board.
Gehel moved this task from Incoming to Blocked/Waiting on the Discovery-Search (Current work) board.Jul 17 2023, 3:43 PM
Gehel moved this task from Blocked/Waiting to Ready for Dev -- SWE on the Discovery-Search (Current work) board.
Gehel moved this task from Ready for Dev -- SWE to Incoming on the Discovery-Search (Current work) board.
Gehel updated the task description. (Show Details)Jul 17 2023, 3:48 PM
Gehel set the point value for this task to 3.
Gehel moved this task from Incoming to Ready for Dev -- SRE/Ops on the Discovery-Search (Current work) board.
gerritbot added a comment.Jul 17 2023, 4:35 PM

Change 938890 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] search-zk: Provision hostnames for new ZK cluster

https://gerrit.wikimedia.org/r/938890

gerritbot added a project: Patch-For-Review.Jul 17 2023, 4:35 PM
gerritbot added a comment.Jul 17 2023, 5:34 PM

Change 938890 merged by Bking:

[operations/puppet@production] flink-zk: Provision hostnames for new ZK cluster

https://gerrit.wikimedia.org/r/938890

Maintenance_bot removed a project: Patch-For-Review.Jul 17 2023, 6:10 PM
ops-monitoring-bot added a comment.Jul 17 2023, 9:01 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host flink-zk1001.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 17 2023, 9:52 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host flink-zk1001.eqiad.wmnet with OS bookworm executed with errors:

  • flink-zk1001 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jul 18 2023, 5:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host flink-zk1002.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 18 2023, 6:16 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host flink-zk1002.eqiad.wmnet with OS bookworm executed with errors:

  • flink-zk1002 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • The reimage failed, see the cookbook logs for the details
andrea.denisse subscribed.Jul 18 2023, 8:28 PM

Hi! I see this task in the SRE Clinic Duty Triage, feel free to let me know if you would like me to help with creating the VMs. :)

ops-monitoring-bot added a comment.Jul 18 2023, 9:15 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host flink-zk1003.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 18 2023, 10:06 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host flink-zk1003.eqiad.wmnet with OS bookworm executed with errors:

  • flink-zk1003 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • The reimage failed, see the cookbook logs for the details
bking added a parent task: T341792: Provision Zookeeper Cluster for storing Flink HA data.Jul 19 2023, 1:45 PM
ops-monitoring-bot added a comment.Jul 19 2023, 7:36 PM

cookbooks.sre.hosts.decommission executed by bking@cumin1001 for hosts: flink-zk1003.eqiad.wmnet

  • flink-zk1003.eqiad.wmnet (WARN)
    • Host not found on Icinga, unable to downtime it
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
ops-monitoring-bot added a comment.Jul 19 2023, 7:42 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host flink-zk1003.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 19 2023, 8:33 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host flink-zk1003.eqiad.wmnet with OS bookworm executed with errors:

  • flink-zk1003 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jul 19 2023, 8:55 PM

cookbooks.sre.hosts.decommission executed by bking@cumin1001 for hosts: flink-zk1003.eqiad.wmnet

  • flink-zk1003.eqiad.wmnet (WARN)
    • Host not found on Icinga, unable to downtime it
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
ops-monitoring-bot added a comment.Jul 19 2023, 9:41 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host flink-zk1003.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 19 2023, 10:36 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host flink-zk1003.eqiad.wmnet with OS bookworm executed with errors:

  • flink-zk1003 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jul 20 2023, 1:57 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host flink-zk1003.eqiad.wmnet with OS bookworm

gerritbot added a comment.Jul 20 2023, 2:21 PM

Change 940160 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] netboot.cfg: explicitly define partitioning scheme

https://gerrit.wikimedia.org/r/940160

gerritbot added a project: Patch-For-Review.Jul 20 2023, 2:21 PM
gerritbot added a comment.Jul 20 2023, 2:27 PM

Change 940160 merged by Bking:

[operations/puppet@production] netboot.cfg: explicitly define partitioning scheme

https://gerrit.wikimedia.org/r/940160

Maintenance_bot removed a project: Patch-For-Review.Jul 20 2023, 2:30 PM
ops-monitoring-bot added a comment.Jul 20 2023, 2:32 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host flink-zk1003.eqiad.wmnet with OS bookworm executed with errors:

  • flink-zk1003 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jul 20 2023, 2:37 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host flink-zk1003.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 20 2023, 2:38 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host flink-zk1003.eqiad.wmnet with OS bookworm executed with errors:

  • flink-zk1003 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jul 20 2023, 2:38 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host flink-zk1003.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 20 2023, 3:16 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host flink-zk1003.eqiad.wmnet with OS bookworm completed:

  • flink-zk1003 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202307201455_bking_2580988_flink-zk1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jul 20 2023, 4:42 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host flink-zk1001.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 20 2023, 5:22 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host flink-zk1001.eqiad.wmnet with OS bookworm completed:

  • flink-zk1001 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202307201700_bking_2613387_flink-zk1001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jul 20 2023, 5:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host flink-zk1002.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 20 2023, 6:00 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host flink-zk1002.eqiad.wmnet with OS bookworm completed:

  • flink-zk1002 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202307201739_bking_2628408_flink-zk1002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
bking moved this task from Ready for Dev -- SRE/Ops to Needs review on the Discovery-Search (Current work) board.Jul 21 2023, 1:45 PM

This is complete; moving to 'needs review'.

andrea.denisse unsubscribed.Jul 24 2023, 5:50 AM
Gehel assigned this task to bking.Jul 24 2023, 3:14 PM
Gehel moved this task from Ready for Work to Done on the Data-Platform-SRE board.
Gehel moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.
Gehel closed this task as Resolved.Jul 28 2023, 9:44 AM
gerritbot added a comment.Aug 7 2023, 1:23 PM

Change 946547 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] flink-zk: Add hostnames for CODFW cluster

https://gerrit.wikimedia.org/r/946547

gerritbot added a project: Patch-For-Review.Aug 7 2023, 1:23 PM
gerritbot added a comment.Aug 7 2023, 1:58 PM

Change 946547 merged by Bking:

[operations/puppet@production] flink-zk: Add hostnames for CODFW cluster

https://gerrit.wikimedia.org/r/946547

Maintenance_bot removed a project: Patch-For-Review.Aug 7 2023, 2:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL