Page MenuHomePhabricator
Create Task
Maniphest T355617

Migrate cloudelastic from public to private IPs
Closed, ResolvedPublic
Actions

Description

All cloudelastic hosts currently use public IPs.

Per parent ticket, cloudelastic is behind a load balancer, so there is no reason for these hosts to use public IPs.

Creating this ticket to:

  • Create a migration plan from public to private IPs
  • Execute the plan
  • Verify operation

Details

SubjectRepoBranchLines +/-
cloudelastic: config changes for migration canaryoperations/puppetproduction+5 -1
cloudelastic: remove unneeded hostnames from cert alt namesoperations/puppetproduction+0 -9
cloudelastic: Complete cloudelastic1005's migrationoperations/puppetproduction+2 -5
cloudelastic: Begin private IP migration for cloudelastic1005operations/puppetproduction+6 -3
cloudelastic: Complete cloudelastic1006's migrationoperations/puppetproduction+2 -5
cloudelastic: Begin private IP migration for cloudelastic1006operations/puppetproduction+6 -3
cloudelastic: remove unneeded master eligiblesoperations/puppetproduction+0 -3
cloudelastic: Complete cloudelastic1007's migrationoperations/puppetproduction+2 -5
cloudelastic: Begin private IP migration for cloudelastic1007operations/puppetproduction+6 -3
cloudelastic: Add already-migrated hosts as master-eligiblesoperations/puppetproduction+2 -0
elasticsearch: avoid systemd timeouts when large clusters start upoperations/puppetproduction+3 -0
cloudelastic: Complete cloudelastic1008's migrationoperations/puppetproduction+2 -7
cloudelastic: Begin private IP migration for cloudelastic1008operations/puppetproduction+7 -8
cloudelastic: Complete cloudelastic1009's migrationoperations/puppetproduction+2 -1
cloudelastic: Begin private IP migration for cloudelastic1009operations/puppetproduction+9 -5
cloudelastic: Add private IP canary back to load balancer pooloperations/puppetproduction+1 -1
cloudelastic: stop issuing certs for soon-to-be defunct FQDNsoperations/puppetproduction+8 -3
cloudelastic: stop issuing certs for soon-to-be defunct FQDNsoperations/puppetproduction+5 -2
cloudelastic: stop issuing certs for soon-to-be defunct FQDNsoperations/puppetproduction+1 -1
cloudelastic: allow wmnet hosts to request certs from acme-chiefoperations/puppetproduction+1 -0
cloudelastic: use acme-chief/letsencrypt with canaryoperations/puppetproduction+0 -2
cloudelastic: add CNAME for migration canaryoperations/dnsmaster+2 -0
cloudelastic: enable DNS discovery/VIP for test serviceoperations/puppetproduction+44 -1
cloudelastic: bind to all interfacesoperations/puppetproduction+1 -3
cloudelastic: listen on "site" scoped IPs as welloperations/puppetproduction+1 -0
cloudelastic: Add migration canary to cloudelastic clusteroperations/puppetproduction+5 -2
cloudelastic: use CFSSL for TLS on canaryoperations/puppetproduction+1 -0
cloudelastic: apply cloudelastic role to canaryoperations/puppetproduction+3 -3
cloudelastic: remove references to cloudelastic1010operations/puppetproduction+0 -3
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Jan 31 2024, 5:56 PM

Change 994763 merged by Bking:

[operations/puppet@production] cloudelastic: allow wmnet hosts to request certs from acme-chief

https://gerrit.wikimedia.org/r/994763

Stashbot added a comment.Jan 31 2024, 6:04 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-31T18:04:20Z] <bking@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on cloudelastic1010.eqiad.wmnet with reason: T355617

Stashbot added a comment.Jan 31 2024, 6:04 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-31T18:04:36Z] <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cloudelastic1010.eqiad.wmnet with reason: T355617

gerritbot added a comment.Jan 31 2024, 6:21 PM

Change 994800 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: Don't validate certs against FQDN

https://gerrit.wikimedia.org/r/994800

gerritbot added a comment.Jan 31 2024, 9:19 PM

Change 994800 merged by Bking:

[operations/puppet@production] cloudelastic: stop issuing certs for soon-to-be defunct FQDNs

https://gerrit.wikimedia.org/r/994800

gerritbot added a comment.Jan 31 2024, 9:34 PM

Change 994838 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: stop issuing certs for soon-to-be defunct FQDNs

https://gerrit.wikimedia.org/r/994838

gerritbot added a comment.Jan 31 2024, 10:39 PM

Change 994838 merged by Bking:

[operations/puppet@production] cloudelastic: stop issuing certs for soon-to-be defunct FQDNs

https://gerrit.wikimedia.org/r/994838

gerritbot added a comment.Feb 1 2024, 2:17 PM

Change 995041 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: stop issuing certs for soon-to-be defunct FQDNs

https://gerrit.wikimedia.org/r/995041

gerritbot added a comment.Feb 1 2024, 9:46 PM

Change 995041 merged by Bking:

[operations/puppet@production] cloudelastic: stop issuing certs for soon-to-be defunct FQDNs

https://gerrit.wikimedia.org/r/995041

gerritbot added a comment.Feb 1 2024, 10:11 PM

Change 995107 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: remove soon-to-be-defunct hostnames from SNI

https://gerrit.wikimedia.org/r/995107

gerritbot added a comment.Feb 1 2024, 10:25 PM

Change 995110 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: Add private IP canary back to load balancer pool

https://gerrit.wikimedia.org/r/995110

gerritbot added a comment.Feb 1 2024, 10:43 PM

Change 995110 merged by Bking:

[operations/puppet@production] cloudelastic: Add private IP canary back to load balancer pool

https://gerrit.wikimedia.org/r/995110

bking mentioned this in T355720: Change TLS/load balancer configuration for cloudelastic.Feb 1 2024, 11:18 PM
bking closed subtask T355720: Change TLS/load balancer configuration for cloudelastic as Resolved.
gerritbot added a comment.Feb 2 2024, 3:48 PM

Change 995223 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: Begin private IP migration for cloudelastic1009

https://gerrit.wikimedia.org/r/995223

bking added a comment.Feb 5 2024, 2:57 PM
In T355617#9493745, @cmooney wrote:

@bking I see that cloudelastic1010 seems to be happy on it's new IP/hostname? Glad that it seems to have gone well. I am mostly on leave this week so may not be on irc, but I'll be checking in here and there so let me know on the task if there are any glitches.

Thanks @cmooney ! cloudelastic1010 is indeed working with its new IP/hostname and we're ready to migrate the next host. I CC'd you on the CR above.

gerritbot added a comment.Feb 5 2024, 10:21 PM

Change 995223 merged by Bking:

[operations/puppet@production] cloudelastic: Begin private IP migration for cloudelastic1009

https://gerrit.wikimedia.org/r/995223

Stashbot added a comment.Feb 5 2024, 10:25 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-05T22:25:15Z] <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Banning hosts: cloudelastic1009.wikimedia.org for migrate cloudelastic1009 to private IP - bking@cumin2002 - T355617

Stashbot added a comment.Feb 5 2024, 10:25 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-05T22:25:20Z] <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: cloudelastic1009.wikimedia.org for migrate cloudelastic1009 to private IP - bking@cumin2002 - T355617

Stashbot added a comment.Feb 5 2024, 10:28 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-05T22:28:50Z] <bking@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on cloudelastic1009.wikimedia.org with reason: T355617

Stashbot added a comment.Feb 5 2024, 10:29 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-05T22:29:06Z] <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cloudelastic1009.wikimedia.org with reason: T355617

Stashbot added a comment.Feb 5 2024, 10:30 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-05T22:30:15Z] <bking@cumin2002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply cluster settings before private IP migration - bking@cumin2002 - T355617

Stashbot added a comment.Feb 5 2024, 11:39 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-05T23:39:13Z] <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply cluster settings before private IP migration - bking@cumin2002 - T355617

ops-monitoring-bot added a comment.Feb 6 2024, 2:49 PM

cookbooks.sre.hosts.decommission executed by bking@cumin2002 for hosts: cloudelastic1009.wikimedia.org

  • cloudelastic1009.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
ops-monitoring-bot added a comment.Feb 6 2024, 3:06 PM

Icinga downtime and Alertmanager silence (ID=8b38d8fa-15e7-4772-81bd-035e55b1e01f) set by cmooney@cumin1002 for 0:30:00 on 1 host(s) and their services with reason: Restoring DB from backup on netboxdb1002

netbox1002.eqiad.wmnet
ops-monitoring-bot added a comment.Feb 6 2024, 3:53 PM

cookbooks.sre.hosts.decommission executed by bking@cumin2002 for hosts: cloudelastic1009.wikimedia.org

  • cloudelastic1009.wikimedia.org (FAIL)
    • Missing DNSName in Nebox for cloudelastic1009, unable to verify it.
    • Missing DNS record for cloudelastic1009.wikimedia.org, the steps requiring DNS will fail.
    • Unable to find/resolve the mgmt DNS record, using the IP instead: 10.65.3.166
    • Host not found on Icinga, unable to downtime it
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Unable to connect to the host, wipe of swraid, partition-table and filesystem signatures will not be performed: Cumin execution failed (exit_code=2)
    • Host is already powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Host steps raised exception: No non-mgmt connected interfaces found for cloudelastic1009. Please check Netbox.

ERROR: some step on some host failed, check the bolded items above

ops-monitoring-bot added a comment.Feb 6 2024, 4:35 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cloudelastic1009.eqiad.wmnet with OS bullseye

gerritbot added a comment.Feb 6 2024, 5:56 PM

Change 997933 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: Complete cloudelastic1009's migration

https://gerrit.wikimedia.org/r/997933

ops-monitoring-bot added a comment.Feb 6 2024, 7:21 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cloudelastic1009.eqiad.wmnet with OS bullseye completed:

  • cloudelastic1009 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202402061654_bking_587800_cloudelastic1009.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Failed to run the sre.puppet.sync-netbox-hiera cookbook, run it manually
Stashbot added a comment.Feb 6 2024, 7:22 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-06T19:22:21Z] <bking@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "finish cloudelastic1009 private IP migration - bking@cumin2002 - T355617"

Stashbot added a comment.Feb 6 2024, 7:23 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-06T19:23:13Z] <bking@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "finish cloudelastic1009 private IP migration - bking@cumin2002 - T355617"

gerritbot added a comment.Feb 6 2024, 7:44 PM

Change 997933 merged by Bking:

[operations/puppet@production] cloudelastic: Complete cloudelastic1009's migration

https://gerrit.wikimedia.org/r/997933

gerritbot added a comment.Feb 7 2024, 6:28 PM

Change 998494 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: Begin private IP migration for cloudelastic1008

https://gerrit.wikimedia.org/r/998494

gerritbot added a comment.Feb 7 2024, 6:43 PM

Change 998498 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: Complete cloudelastic1008's migration

https://gerrit.wikimedia.org/r/998498

gerritbot added a comment.Feb 7 2024, 7:33 PM

Change 998494 merged by Bking:

[operations/puppet@production] cloudelastic: Begin private IP migration for cloudelastic1008

https://gerrit.wikimedia.org/r/998494

ops-monitoring-bot added a comment.Feb 7 2024, 7:56 PM

cookbooks.sre.hosts.decommission executed by bking@cumin2002 for hosts: cloudelastic1008.wikimedia.org

  • cloudelastic1008.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
bking changed the status of subtask T356941: Stale data/failed queries on wikidatawiki index from Open to In Progress.Feb 8 2024, 4:32 PM
gerritbot added a comment.Feb 8 2024, 9:47 PM

Change 999088 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: Begin private IP migration for cloudelastic1007

https://gerrit.wikimedia.org/r/999088

gerritbot added a comment.Feb 8 2024, 10:04 PM

Change 999091 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: Complete cloudelastic1007's migration

https://gerrit.wikimedia.org/r/999091

Stashbot added a comment.Feb 8 2024, 10:38 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-08T22:38:38Z] <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Banning hosts: cloudelastic1005*,cloudelastic1006*,cloudelastic1007*,cloudelastic1008* for IP migration - bking@cumin2002 - T355617

Stashbot added a comment.Feb 8 2024, 10:38 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-08T22:38:42Z] <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: cloudelastic1005*,cloudelastic1006*,cloudelastic1007*,cloudelastic1008* for IP migration - bking@cumin2002 - T355617

Gehel edited projects, added Data-Platform-SRE (2024.02.12 - 2024.03.03); removed Data-Platform-SRE (2024.01.22 - 2024.02.11).Feb 9 2024, 10:46 AM
Gehel moved this task from Backlog to In Progress on the Data-Platform-SRE (2024.02.12 - 2024.03.03) board.
Gehel triaged this task as Medium priority.Feb 9 2024, 1:36 PM
Stashbot added a comment.Feb 9 2024, 3:05 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-09T15:05:47Z] <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Banning hosts: cloudelastic1005*,cloudelastic1006*,cloudelastic1007*,cloudelastic1008* for IP migration - bking@cumin2002 - T355617

Stashbot added a comment.Feb 9 2024, 3:06 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-09T15:06:02Z] <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: cloudelastic1005*,cloudelastic1006*,cloudelastic1007*,cloudelastic1008* for IP migration - bking@cumin2002 - T355617

bking closed subtask T356941: Stale data/failed queries on wikidatawiki index as Resolved.Feb 9 2024, 4:24 PM
VRiley-WMF closed subtask T356919: Comm Error: Backplane 0 on cloudelastic1008 as Resolved.Feb 9 2024, 6:15 PM
gerritbot added a comment.Feb 9 2024, 7:30 PM

Change 998498 merged by Bking:

[operations/puppet@production] cloudelastic: Complete cloudelastic1008's migration

https://gerrit.wikimedia.org/r/998498

gerritbot added a comment.Feb 9 2024, 8:22 PM

Change 1000018 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] elasticsearch: avoid systemd timeouts when large clusters start up

https://gerrit.wikimedia.org/r/1000018

gerritbot added a comment.Feb 9 2024, 8:29 PM

Change 1000018 merged by Bking:

[operations/puppet@production] elasticsearch: avoid systemd timeouts when large clusters start up

https://gerrit.wikimedia.org/r/1000018

Stashbot added a comment.Feb 9 2024, 8:46 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-09T20:46:07Z] <bking@cumin2002> START - Cookbook sre.elasticsearch.rolling-operation Operation.UPGRADE (1 nodes at a time) for ElasticSearch cluster relforge: apply new systemd settings - bking@cumin2002 - T355617

Stashbot added a comment.Feb 9 2024, 8:55 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-09T20:55:36Z] <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.UPGRADE (1 nodes at a time) for ElasticSearch cluster relforge: apply new systemd settings - bking@cumin2002 - T355617

Stashbot added a comment.Feb 9 2024, 9:06 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-09T21:06:53Z] <bking@cumin2002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply new systemd settings - bking@cumin2002 - T355617

Stashbot added a comment.Feb 9 2024, 9:09 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-09T21:09:42Z] <bking@cumin2002> END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply new systemd settings - bking@cumin2002 - T355617

gerritbot added a comment.Feb 13 2024, 8:57 PM

Change 1003100 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: Add already-migrated hosts as master-eligibles

https://gerrit.wikimedia.org/r/1003100

gerritbot added a comment.Feb 14 2024, 3:51 PM

Change 1003100 merged by Bking:

[operations/puppet@production] cloudelastic: Add already-migrated hosts as master-eligibles

https://gerrit.wikimedia.org/r/1003100

Stashbot added a comment.Feb 14 2024, 3:53 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T15:53:05Z] <bking@cumin2002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply new master settings - bking@cumin2002 - T355617

Stashbot added a comment.Feb 14 2024, 3:55 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T15:55:06Z] <bking@cumin2002> END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply new master settings - bking@cumin2002 - T355617

Stashbot added a comment.Feb 14 2024, 3:59 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T15:59:07Z] <bking@cumin2002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply new master settings - bking@cumin2002 - T355617

Stashbot added a comment.Feb 14 2024, 4:04 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T16:04:58Z] <bking@cumin2002> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply new master settings - bking@cumin2002 - T355617

Stashbot added a comment.Feb 14 2024, 5:44 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T17:44:14Z] <bking@cumin2002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply new master settings - bking@cumin2002 - T355617

Stashbot added a comment.Feb 14 2024, 6:01 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T18:01:38Z] <bking@cumin2002> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply new master settings - bking@cumin2002 - T355617

Stashbot added a comment.Feb 14 2024, 6:03 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T18:03:37Z] <bking@cumin2002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply new master settings - bking@cumin2002 - T355617

Stashbot added a comment.Feb 14 2024, 6:37 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T18:37:07Z] <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply new master settings - bking@cumin2002 - T355617

gerritbot added a comment.Feb 14 2024, 7:47 PM

Change 999088 merged by Bking:

[operations/puppet@production] cloudelastic: Begin private IP migration for cloudelastic1007

https://gerrit.wikimedia.org/r/999088

ops-monitoring-bot added a comment.Feb 14 2024, 8:04 PM

cookbooks.sre.hosts.decommission executed by bking@cumin2002 for hosts: cloudelastic1007.wikimedia.org

  • cloudelastic1007.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
Stashbot added a comment.Feb 14 2024, 8:51 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T20:51:09Z] <inflatador> bking@puppetmaster1001 manually updating facts data for PCC T355617

gerritbot added a comment.Feb 14 2024, 9:09 PM

Change 1003535 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: remove unneeded master eligibles

https://gerrit.wikimedia.org/r/1003535

gerritbot added a comment.Feb 14 2024, 10:04 PM

Change 999091 merged by Bking:

[operations/puppet@production] cloudelastic: Complete cloudelastic1007's migration

https://gerrit.wikimedia.org/r/999091

Stashbot added a comment.Feb 14 2024, 10:20 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T22:20:11Z] <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Banning hosts: cloudelastic1005*,cloudelastic1006* for IP migration - bking@cumin2002 - T355617

Stashbot added a comment.Feb 14 2024, 10:20 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T22:20:15Z] <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: cloudelastic1005*,cloudelastic1006* for IP migration - bking@cumin2002 - T355617

gerritbot added a comment.Feb 14 2024, 10:27 PM

Change 1003535 merged by Bking:

[operations/puppet@production] cloudelastic: remove unneeded master eligibles

https://gerrit.wikimedia.org/r/1003535

Stashbot added a comment.Feb 14 2024, 10:33 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T22:33:58Z] <bking@cumin2002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply new master settings - bking@cumin2002 - T355617

gerritbot added a comment.Feb 14 2024, 10:40 PM

Change 1003557 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: Begin private IP migration for cloudelastic1006

https://gerrit.wikimedia.org/r/1003557

gerritbot added a comment.Feb 14 2024, 10:47 PM

Change 1003558 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: Complete cloudelastic1006's migration

https://gerrit.wikimedia.org/r/1003558

Stashbot added a comment.Feb 14 2024, 10:48 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T22:48:03Z] <bking@cumin2002> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply new master settings - bking@cumin2002 - T355617

Stashbot added a comment.Feb 14 2024, 10:51 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T22:51:09Z] <bking@cumin2002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply new master settings - bking@cumin2002 - T355617

gerritbot added a comment.Feb 14 2024, 10:54 PM

Change 1003561 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: Begin private IP migration for cloudelastic1005

https://gerrit.wikimedia.org/r/1003561

gerritbot added a comment.Feb 14 2024, 10:57 PM

Change 1003563 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: Complete cloudelastic1005's migration

https://gerrit.wikimedia.org/r/1003563

Stashbot added a comment.Feb 14 2024, 11:14 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T23:14:03Z] <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply new master settings - bking@cumin2002 - T355617

gerritbot added a comment.Feb 15 2024, 5:59 PM

Change 1003557 merged by Bking:

[operations/puppet@production] cloudelastic: Begin private IP migration for cloudelastic1006

https://gerrit.wikimedia.org/r/1003557

ops-monitoring-bot added a comment.Feb 15 2024, 6:12 PM

cookbooks.sre.hosts.decommission executed by bking@cumin2002 for hosts: cloudelastic1006.wikimedia.org

  • cloudelastic1006.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
brennen mentioned this in T357713: RuntimeException: Received cirrusSearchElasticaWrite job for an unwritable cluster cloudelastic..Feb 15 2024, 8:33 PM
gerritbot added a comment.Feb 15 2024, 9:36 PM

Change 1003558 merged by Bking:

[operations/puppet@production] cloudelastic: Complete cloudelastic1006's migration

https://gerrit.wikimedia.org/r/1003558

Stashbot added a comment.Feb 15 2024, 9:52 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-15T21:52:25Z] <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Banning hosts: cloudelastic1005* for IP migration - bking@cumin2002 - T355617

Stashbot added a comment.Feb 15 2024, 9:52 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-15T21:52:30Z] <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: cloudelastic1005* for IP migration - bking@cumin2002 - T355617

gerritbot added a comment.Feb 15 2024, 10:02 PM

Change 1003561 merged by Bking:

[operations/puppet@production] cloudelastic: Begin private IP migration for cloudelastic1005

https://gerrit.wikimedia.org/r/1003561

ops-monitoring-bot added a comment.Feb 15 2024, 10:19 PM

cookbooks.sre.hosts.decommission executed by bking@cumin2002 for hosts: cloudelastic1005.wikimedia.org

  • cloudelastic1005.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
gerritbot added a comment.Feb 15 2024, 11:56 PM

Change 1003563 merged by Bking:

[operations/puppet@production] cloudelastic: Complete cloudelastic1005's migration

https://gerrit.wikimedia.org/r/1003563

bking closed this task as Resolved.Feb 16 2024, 2:29 PM
bking moved this task from In Progress to Done on the Data-Platform-SRE (2024.02.12 - 2024.03.03) board.

The public-to-private IP migration is complete!

Note that we still have 4 hosts using public IPs, but they will be decommissioned soon. Closing this ticket...

gerritbot added a comment.Feb 16 2024, 5:07 PM

Change 995107 merged by Bking:

[operations/puppet@production] cloudelastic: remove unneeded hostnames from cert alt names

https://gerrit.wikimedia.org/r/995107

Stashbot added a comment.Feb 28 2024, 12:06 AM

Mentioned in SAL (#wikimedia-operations) [2024-02-28T00:06:24Z] <bking@cumin2002> START - Cookbook sre.hosts.downtime for 15:00:00 on wdqs1011.eqiad.wmnet with reason: T355617

Stashbot added a comment.Feb 28 2024, 12:06 AM

Mentioned in SAL (#wikimedia-operations) [2024-02-28T00:06:39Z] <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 15:00:00 on wdqs1011.eqiad.wmnet with reason: T355617

Stashbot added a comment.Feb 28 2024, 2:27 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-28T14:27:04Z] <bking@cumin2002> START - Cookbook sre.hosts.downtime for 6:00:00 on wdqs2008.codfw.wmnet with reason: T355617

Stashbot added a comment.Feb 28 2024, 2:27 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-28T14:27:20Z] <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on wdqs2008.codfw.wmnet with reason: T355617

Maintenance_bot removed a project: Patch-For-Review.Feb 28 2024, 2:32 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL