Page MenuHomePhabricator
Create Task
Maniphest T341464

eqiad: 1 VM requested for karapace in support of datahub in staging
Closed, ResolvedPublic
Actions

Description

Site/Location: eqiad
Number of systems: 1
Service: datahub in staging
Networking Requirements: internal IP
Processor Requirements: 2 VCPUs
Memory: 2 GB of RAM
Disks: 20 GB disk
Other Requirements:

We have discovered that the production and staging deployments of datahub share an instance of karapace, which is proving problematic.
Having this additional machine will help to separate the deployments.

Details

SubjectRepoBranchLines +/-
Add a second karapace VMoperations/puppetproduction+2 -2
Customize query in gerrit

Related Objects

Event Timeline

BTullis triaged this task as High priority.Jul 10 2023, 12:19 PM
BTullis created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 10 2023, 12:19 PM
MoritzMuehlenhoff subscribed.Jul 10 2023, 12:55 PM

Looks good

BTullis added a project: Data-Platform-SRE.Jul 10 2023, 1:11 PM
BTullis moved this task from Incoming to In Progress on the Data-Platform-SRE board.
BTullis mentioned this in T329514: Upgrade Datahub to v0.10.4.Jul 10 2023, 1:19 PM
ops-monitoring-bot added a comment.Jul 10 2023, 1:27 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1001 for host karapace1002.eqiad.wmnet with OS bullseye

gerritbot added a comment.Jul 10 2023, 1:32 PM

Change 936706 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Add a second karapace VM

https://gerrit.wikimedia.org/r/936706

gerritbot added a project: Patch-For-Review.Jul 10 2023, 1:32 PM

Change 936706 merged by Btullis:

[operations/puppet@production] Add a second karapace VM

https://gerrit.wikimedia.org/r/936706

ops-monitoring-bot added a comment.Jul 10 2023, 1:52 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1001 for host karapace1002.eqiad.wmnet with OS bullseye completed:

  • karapace1002 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202307101339_btullis_4053396_karapace1002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
BTullis closed this task as Resolved.Jul 10 2023, 1:53 PM
BTullis moved this task from In Progress to Needs Reporting on the Data-Platform-SRE board.
Maintenance_bot removed a project: Patch-For-Review.Jul 10 2023, 2:10 PM
Gehel moved this task from Needs Reporting to Done on the Data-Platform-SRE board.Jul 19 2023, 8:51 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL