Page MenuHomePhabricator
Create Task
Maniphest T338779

cloudservices2005-dev: reimage into new network setup
Closed, ResolvedPublic
Actions

Description

The cloudcontrol2005-dev server got a new network setup.

We should:

  • drop wikimedia.org domain in favor of .codfw.wmnet.
  • drop connection to asw
  • keep private.codfw.wikimedia.cloud address

Following procedure at https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Rename_while_reimaging

Details

Other Assignee
Papaul
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedaborreroT296411 cloud: decide on general idea for having cloud-dedicated hardware provide service in the cloud realm & the internet
 ResolvedaborreroT297596 have cloud hardware servers in the cloud realm using a dedicated LB layer
 ResolvedaborreroT324992 cloudlb: create PoC on codfw
 ResolvedaborreroT327908 rename cloudgw2001-dev into cloudlb2001-dev
 ResolvedayounsiT327930 Is Vlan 2122 cloud-support1-b-codfw required?
 ResolvedcmooneyT327919 Configure cloudsw1-b1-codfw and migrate cloud hosts in codfw B1 to it
 ResolvedPapaulT331470 Run 2x1G links from asw-b1-codfw to cloudsw1-b1-codfw
Restricted Task
 ResolvedcmooneyT333316 Homer unable to commit config to cloudsw1-b1-codfw (QFX5120 21.4R3.16)
 ResolvedcmooneyT336368 cloudgw: review security policy for edge network
 ResolvedaborreroT332153 cloudlb PoC: prepare backends
 ResolvedaborreroT336963 cloudcontrol2001-dev can't reach cloud-vps public IPs
 ResolvedaborreroT335759 cloud-private subnet: introduce new domain
 Resolved• taaviT336566 cloud: failures resolving some `wikimedia.cloud` domains
 OpencmooneyT346428 Netbox: Add support for our complex host network setups in provision script
 ResolvedaborreroT346410 need private IPs for cloudvirt200[4-6]-dev.codfw.wmnet
 OpencmooneyT347375 Netbox device location information not available on the first Puppet run of a device
 ResolvedaborreroT335760 Modify Bird module to allow source IP to be passed to template
 ResolvedaborreroT336071 cloudlb: figure out routing
 ResolvedaborreroT337758 puppet: interface::route resource does not persists settings
 ResolvedaborreroT338936 cloudlb: figure out plans for eqiad1
 ResolvedaborreroT338937 cloudlb: review swift/radosgw status
ResolvedAndrewT341380 Open swift port (28080) to the public internet
 ResolvedAndrewT341484 Horizon object store UI doesn't allow uploading of files
 ResolvedAndrewT341640 Move Horizon deploy to a Docker container
 ResolvedAndrewT341509 radosgw+keystone chokes on projects with '-' in their id
 ResolvedAndrewT343158 Support OpenStack projects where project name != project id
 OpenNoneT366679 openstack-browser support for projects where id != name
 ResolvedAndrewT347927 Better swift error messages for projects with invalid chars
 ResolvedAndrewT366301 Document and enforce restrictions on openstack project names
 ResolvedaborreroT340536 cloud-private: figure out, implement and test cross-DC traffic
 OpenNoneT317177 [tracking] Don't keep on the public vlans hosts that don't require it
 OpenNoneT340446 Cloud VPS Designate setup improvements
 ResolvedaborreroT307357 Move cloud vps ns-recursor IPs to host/row-independent addressing
 ResolvedcmooneyT336587 cloudservices[2004/2005]-dev & cloudweb2002-dev: connect them to cloudsw so they can have cloud-private vlan
 ResolvedaborreroT338779 cloudservices2005-dev: reimage into new network setup
 ResolvedaborreroT340047 LDAP problems following cloudservices2005-dev reimage
 OpenNoneT340127 systemctl restart keystone doesn't actually restart keystone sometimes
 ResolvedaborreroT340488 codfw1dev: OpenStack services can only sort of talk to memacached on cloudcontrols

Event Timeline

aborrero triaged this task as Medium priority.Jun 12 2023, 11:11 AM
aborrero created this task.
aborrero moved this task from Backlog to Next on the User-aborrero board.Jun 12 2023, 2:40 PM
aborrero updated the task description. (Show Details)Jun 12 2023, 3:33 PM
gerritbot added a comment.Jun 13 2023, 12:04 PM

Change 929687 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: codfw1dev: services: disable slapd mirror mode

https://gerrit.wikimedia.org/r/929687

gerritbot added a project: Patch-For-Review.Jun 13 2023, 12:04 PM
gerritbot added a comment.Jun 13 2023, 3:06 PM

Change 929687 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: codfw1dev: services: disable slapd mirror mode

https://gerrit.wikimedia.org/r/929687

Maintenance_bot removed a project: Patch-For-Review.Jun 13 2023, 3:11 PM
gerritbot added a comment.Jun 19 2023, 3:27 PM

Change 931291 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: codfw1dev: refresh ldap hosts

https://gerrit.wikimedia.org/r/931291

gerritbot added a project: Patch-For-Review.Jun 19 2023, 3:27 PM
gerritbot added a comment.Jun 20 2023, 11:59 AM

Change 931589 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/dns@master] wikimediacloud.org: refresh A for ns1.openstack.codfw1dev.wikimediacloud.org

https://gerrit.wikimedia.org/r/931589

gerritbot added a comment.Jun 20 2023, 4:25 PM

Change 931287 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloud: codfw1dev: fix labsldapconfig to use newer server

https://gerrit.wikimedia.org/r/931287

aborrero moved this task from Next to Doing on the User-aborrero board.Jun 20 2023, 4:28 PM
gerritbot added a comment.Jun 21 2023, 9:09 AM

Change 931589 merged by Arturo Borrero Gonzalez:

[operations/dns@master] wikimediacloud.org: refresh A for ns1.openstack.codfw1dev.wikimediacloud.org

https://gerrit.wikimedia.org/r/931589

gerritbot added a comment.Jun 21 2023, 11:52 AM

Change 931900 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloud: drop references to cloudservices2005-dev.wikimedia.org

https://gerrit.wikimedia.org/r/931900

gerritbot added a comment.Jun 21 2023, 12:00 PM

Change 931900 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloud: drop references to cloudservices2005-dev.wikimedia.org

https://gerrit.wikimedia.org/r/931900

ops-monitoring-bot added a comment.Jun 21 2023, 12:47 PM

cookbooks.sre.hosts.decommission executed by aborrero@cumin2002 for hosts: cloudservices2005-dev

  • cloudservices2005-dev (FAIL)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Management interface not found on Icinga, unable to downtime it
    • Unable to connect to the host, wipe of swraid, partition-table and filesystem signatures will not be performed: Cumin execution failed (exit_code=2)
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • COMMON_STEPS (FAIL)
    • Failed to run the sre.dns.netbox cookbook, run it manually

ERROR: some step on some host failed, check the bolded items above

aborrero updated the task description. (Show Details)Jun 21 2023, 12:56 PM
aborrero changed the task status from Open to In Progress.Jun 21 2023, 2:48 PM
aborrero updated Other Assignee, added: Papaul.
aborrero added a project: ops-codfw.
aborrero added subscribers: Jhancock.wm, Papaul.

Hey @Papaul (or @Jhancock.wm ) we would need this server connected in a similar fashion as cloudservices2004-dev:

  • primary interface connected to cloudsw (whatever port) -- relocate connection from server's eno2
  • no connection to asw
Jhancock.wm moved this task from Backlog to Racking Tasks on the ops-codfw board.Jun 21 2023, 2:56 PM
Jhancock.wm added a comment.Jun 21 2023, 3:13 PM

@aborrero I've physically removed the connection from asw. the cloudsw connection is now on port ge-0/0/12 in <-> eno1.

aborrero added a comment.Jun 21 2023, 3:21 PM
In T338779#8953114, @Jhancock.wm wrote:

@aborrero I've physically removed the connection from asw. the cloudsw connection is now on port ge-0/0/12 in <-> eno1.

Thanks!

Could you please make the changes in netbox so it shows this new reality ? https://netbox.wikimedia.org/dcim/devices/4144/interfaces/

Jhancock.wm updated the task description. (Show Details)Jun 21 2023, 3:28 PM

updated!

Maintenance_bot added a project: SRE.Jun 21 2023, 4:29 PM
aborrero changed the status of subtask T340047: LDAP problems following cloudservices2005-dev reimage from Open to In Progress.Jun 21 2023, 5:07 PM
gerritbot added a comment.Jun 26 2023, 8:49 AM

Change 932800 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudservices2005-dev: give it proper role and name.

https://gerrit.wikimedia.org/r/932800

aborrero added a comment.Jun 26 2023, 9:03 AM

image.png (579×1 px, 80 KB)

ops-monitoring-bot added a comment.Jun 26 2023, 9:11 AM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin2002 for host cloudservices2005-dev.codfw.wmnet with OS bullseye

gerritbot added a comment.Jun 26 2023, 9:14 AM

Change 932800 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudservices2005-dev: give it proper role and name

https://gerrit.wikimedia.org/r/932800

ops-monitoring-bot added a comment.Jun 26 2023, 9:17 AM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin2002 for host cloudservices2005-dev.codfw.wmnet with OS bullseye executed with errors:

  • cloudservices2005-dev (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jun 26 2023, 9:18 AM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin2002 for host cloudservices2005-dev.codfw.wmnet with OS bullseye

aborrero updated the task description. (Show Details)Jun 26 2023, 9:53 AM
aborrero closed subtask T340047: LDAP problems following cloudservices2005-dev reimage as Resolved.
gerritbot added a comment.Jun 26 2023, 10:03 AM

Change 932832 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudservices2005-dev: drop cloud-private base interface override

https://gerrit.wikimedia.org/r/932832

gerritbot added a comment.Jun 26 2023, 10:04 AM

Change 932833 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] acme_chief: allow cloudservices2005-dev to access ldap-codfw1dev cert

https://gerrit.wikimedia.org/r/932833

gerritbot added a comment.Jun 26 2023, 10:05 AM

Change 932832 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudservices2005-dev: drop cloud-private base interface override

https://gerrit.wikimedia.org/r/932832

gerritbot added a comment.Jun 26 2023, 10:05 AM

Change 932833 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] acme_chief: allow cloudservices2005-dev to access ldap-codfw1dev cert

https://gerrit.wikimedia.org/r/932833

gerritbot added a comment.Jun 26 2023, 10:07 AM

Change 932834 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] acme_chief: extend ldap-codfw1dev with cloudservices2005-dev SNI

https://gerrit.wikimedia.org/r/932834

gerritbot added a comment.Jun 26 2023, 10:08 AM

Change 932834 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] acme_chief: extend ldap-codfw1dev with cloudservices2005-dev SNI

https://gerrit.wikimedia.org/r/932834

ops-monitoring-bot added a comment.Jun 26 2023, 10:25 AM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin2002 for host cloudservices2005-dev.codfw.wmnet with OS bullseye completed:

  • cloudservices2005-dev (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202306260940_aborrero_2574582_cloudservices2005-dev.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Failed to run the sre.puppet.sync-netbox-hiera cookbook, run it manually
aborrero mentioned this in T339894: cloudservices: codfw1dev: fix backups.Jun 26 2023, 2:51 PM
aborrero closed this task as Resolved.Jun 26 2023, 3:34 PM
Andrew reopened this task as Open.Jun 26 2023, 7:46 PM
aborrero closed this task as Resolved.Jun 29 2023, 8:21 AM
aborrero updated the task description. (Show Details)

I'm closing this ticket as I believe the reimage is completed and the server is mostly working, barring any last minute firewalling changes which we are tracking in other tickets.

Andrew closed subtask T340488: codfw1dev: OpenStack services can only sort of talk to memacached on cloudcontrols as Resolved.Jun 29 2023, 4:55 PM
gerritbot added a comment.Jun 29 2023, 7:16 PM

Change 934404 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] designate: distinguish between designate IPs and pdns IPs in pools.yaml

https://gerrit.wikimedia.org/r/934404

gerritbot added a comment.Jun 29 2023, 7:17 PM

Change 934405 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloud-vps pdns: include designate hosts as allowed api clients

https://gerrit.wikimedia.org/r/934405

gerritbot added a comment.Jun 29 2023, 7:22 PM

Change 934404 merged by Andrew Bogott:

[operations/puppet@production] designate: distinguish between designate IPs and pdns IPs in pools.yaml

https://gerrit.wikimedia.org/r/934404

gerritbot added a comment.Jun 29 2023, 7:22 PM

Change 934405 merged by Andrew Bogott:

[operations/puppet@production] cloud-vps pdns: include designate hosts as allowed api clients

https://gerrit.wikimedia.org/r/934405

gerritbot added a comment.Jun 30 2023, 1:28 AM

Change 934427 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Designate/pdns: allow designate hosts to access the pdns rest api

https://gerrit.wikimedia.org/r/934427

gerritbot added a comment.Jun 30 2023, 1:30 AM

Change 934427 merged by Andrew Bogott:

[operations/puppet@production] Designate/pdns: allow designate hosts to access the pdns rest api

https://gerrit.wikimedia.org/r/934427

gerritbot added a comment.Jun 30 2023, 2:29 AM

Change 934430 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Designate/pdns: allow designate and pdns hosts to access mdns for axfr

https://gerrit.wikimedia.org/r/934430

gerritbot added a comment.Jun 30 2023, 2:31 AM

Change 934430 merged by Andrew Bogott:

[operations/puppet@production] Designate/pdns: allow designate and pdns hosts to access mdns for axfr

https://gerrit.wikimedia.org/r/934430

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits