Page MenuHomePhabricator
Create Task
Maniphest T372817

reimage gerrit1004.wikimedia.org as phab1005.eqiad.wmnet
Closed, ResolvedPublic
Actions

Description

The machine WMF11584 (netbox 5353) that was procured in T368918 and has been racked as gerrit1004.wikimedia.org in T369671 should be renamed.

Please:

  • move from external IP to internal IP (wikimedia.org -> eqiad.wmnet)
  • reimage it as phab1005.eqiad.wmnet, OS still bookworm

Budget-wise there should be no effects because this has been budgeted as "Phabricator / Gerrit / Contint spare host" and it is still one of those.

We are just changing which we use for Gerrit and which for Phabricator.

Thanks, and sorry for the change after racking.

Rename & VLAN change checklist

  • make note of server's network port and cable ID
  • run the decom script to remove all instances of host. (check if this remocves mgmt ip cuz we don't want that)
  • update netbox with new hostname
  • add the network data back for the host, check that mgmt updated ot new hostname
  • run dns update script
  • reimage host with new hostname
  • create sub-task to this task with ops-eqiad tag for onsites to change the hostname tags on the host.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
site: rename gerrit1004 to phab1005operations/puppetproduction+5 -5
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedFeatureAklapperT367500 Explore displaying additional info in comment field for open Outreach related tasks
 ResolvedFeatureAklapperT369982 Display info for new technical contributors in task comment field for new accounts
 ResolvedFeatureAklapperT354769 Display warning about commenting on a task already closed as duplicate
Restricted Task
 OpenNoneT84 Make sure anti-vandalism features are up to snuff
Restricted Task
 Resolvedvalerio.bozzolanT356384 Phorge (Phabricator) Code Sprint 2024
 ResolvedBUG REPORTvalerio.bozzolanT217984 Listing tickets "Closed By: Current Viewer" while logged out causes PhabricatorTypeaheadInvalidTokenException: This datasource ("PhabricatorPeopleUserFunctionDatasource") can not evaluate the function "viewer(...)".
 ResolvedAklapperT91428 tooltip mentions non-existent "Wishlist priority" in Open Tasks by Project and Priority report
 ResolvedAklapperT310543 Unable to "manage user" on mobile
 ResolvedAklapperT266079 Error when searching Maniphest task IDs with improper format could be less cryptic: "Expected a numeric scalar or null for %Ld conversion"
Restricted Task
 ResolvedAklapperT132335 After "Log In to Comment", go back to previous page instead of Phab home page
 ResolvedFeatureAklapperT208434 Allow contracting (collapsing and expanding) of columns on project workboard in mobile view
 ResolvedAklapperT338306 When I delete a comment, I don't want to be added as a subscriber
 ResolvedFeatureAklapperT288117 Add Open Graph support to Phabricator Maniphest Tasks to have link preview on Telegram, Slack, and other messaging apps
 ResolvedBUG REPORTAklapperT233541 Unhandled Exception when clicking "Award" without choosing a badge; No badge type-ahead proposals if no permission to award badges
 ResolvedbrennenT370266 Update to Phorge upstream 2024.35 release
 ResolvedJclark-ctrT372817 reimage gerrit1004.wikimedia.org as phab1005.eqiad.wmnet
 OpenNoneT377889 install a service on phab1005
 ResolvedDzahnT392386 PuppetFailure - phab1005
 ResolvedMarosteguiT390034 Prepare a database test for m3
 ResolvedABran-WMFT397886 PuppetDisabled - phab1005:9100
 ResolvedDzahnT399480 Can't update scap on phab1005.eqiad.wmnet

Event Timeline

Dzahn created this task.Aug 19 2024, 6:52 PM
Restricted Application added a project: DC-Ops. · View Herald TranscriptAug 19 2024, 6:52 PM
Dzahn updated the task description. (Show Details)Aug 19 2024, 6:52 PM
Dzahn updated the task description. (Show Details)
Dzahn updated the task description. (Show Details)
Dzahn renamed this task from reimage gerrit1004 as phab1003 to reimage gerrit1004 as phab1005.Aug 19 2024, 6:55 PM
Dzahn renamed this task from reimage gerrit1004 as phab1005 to reimage gerrit1004.wikimedia.org as phab1005.eqiad.wmnet.
Dzahn updated the task description. (Show Details)
Dzahn mentioned this in T369670: Q1:rack/setup/install gerrit2003.Aug 19 2024, 6:58 PM
gerritbot added a comment.Aug 19 2024, 7:05 PM

Change #1063870 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site: rename gerrit1004 to phab1005

https://gerrit.wikimedia.org/r/1063870

gerritbot added a project: Patch-For-Review.Aug 19 2024, 7:05 PM
RobH updated the task description. (Show Details)Aug 19 2024, 7:26 PM
ops-monitoring-bot added a comment.Sep 3 2024, 8:44 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host phab1005.eqiad.wmnet with OS bookworm

Jclark-ctr claimed this task.Sep 3 2024, 9:06 PM

@Dzahn please update preseed.yaml file for sw raid for this server. Reimage fails without this

Jclark-ctr updated the task description. (Show Details)Sep 3 2024, 9:07 PM
Dzahn added a comment.Sep 3 2024, 9:09 PM

@Jclark-ctr It also fails because coordination was needed to change the role:

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1063870

Maintenance_bot added a project: SRE.Sep 3 2024, 9:29 PM
gerritbot added a comment.Sep 3 2024, 9:32 PM

Change #1063870 merged by Dzahn:

[operations/puppet@production] site: rename gerrit1004 to phab1005

https://gerrit.wikimedia.org/r/1063870

Dzahn added a comment.Sep 3 2024, 9:35 PM

@Jclark-ctr I don't think it's preseed.yaml. Both existing gerrit1004 and phab* are set to standard/raid1-2dev.cfg.

I think it was that the host wasn't in site.pp with a role. But now it is.

ops-monitoring-bot added a comment.Sep 3 2024, 10:04 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host phab1005.eqiad.wmnet with OS bookworm executed with errors:

  • phab1005 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details,You can also try typing "sudo install-console phab1005.eqiad.wmnet" to get a root shellbut depending on the failure this may not work.
Maintenance_bot removed a project: Patch-For-Review.Sep 3 2024, 10:30 PM
LSobanski moved this task from Incoming to Backlog on the collaboration-services board.Sep 4 2024, 9:05 AM
ops-monitoring-bot added a comment.Sep 4 2024, 1:33 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host phab1005.eqiad.wmnet with OS bookworm

Jclark-ctr added a comment.Sep 4 2024, 2:19 PM

@Dzahn phab1005 is still continuing to fail imaging not picking up ip address for pxe booting would you be able to assist?

Jclark-ctr added a comment.Sep 4 2024, 2:33 PM

@Dzahn disregard i figured out issue

ops-monitoring-bot added a comment.Sep 4 2024, 3:12 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host phab1005.eqiad.wmnet with OS bookworm executed with errors:

  • phab1005 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details,You can also try typing "sudo install-console phab1005.eqiad.wmnet" to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 4 2024, 3:12 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host phab1005.eqiad.wmnet with OS bookworm

Dzahn added a comment.Sep 4 2024, 3:43 PM

Was this a bug in the cookbook?

ops-monitoring-bot added a comment.Sep 4 2024, 7:07 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host phab1005.eqiad.wmnet with OS bookworm executed with errors:

  • phab1005 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details,You can also try typing "sudo install-console phab1005.eqiad.wmnet" to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 4 2024, 7:15 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host phab1005.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 4 2024, 7:56 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host phab1005.eqiad.wmnet with OS bookworm completed:

  • phab1005 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409041937_jclark_2394602_phab1005.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr closed this task as Resolved.Sep 4 2024, 7:56 PM
Jclark-ctr updated the task description. (Show Details)
MoritzMuehlenhoff subscribed.Sep 9 2024, 9:17 AM

@Dzahn gerrit1004 is still in puppetdb: https://puppetboard.wikimedia.org/catalog/gerrit1004.wikimedia.org

ops-monitoring-bot added a comment.Sep 9 2024, 9:18 AM

Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: gerrit1004.wikimedia.org

Dzahn added a comment.Sep 13 2024, 3:57 PM

I ran the decom cookbook (without and with --force) but it errors out with

spicerack.netbox.NetboxHostNotFoundError: gerrit1004

I guess it's just the cert.. In the past this would have been fixable with a manual "puppet cert clean" on the puppetmaster.

Looking...

Dzahn added a comment.Sep 13 2024, 4:45 PM
[puppetserver1001:~] $ sudo puppet node clean gerrit1004.wikimedia.org
Notice: Certificate for gerrit1004.wikimedia.org has been revoked
Notice: Cleaned files related to gerrit1004.wikimedia.org
Dzahn added a comment.Sep 16 2024, 5:51 PM
In T372817#10129107, @MoritzMuehlenhoff wrote:

@Dzahn gerrit1004 is still in puppetdb: https://puppetboard.wikimedia.org/catalog/gerrit1004.wikimedia.org

This is now fixed for real after also running a "puppet node deactivate" on puppetmaster and puppetserver. Next puppet run on Icinga removed it from monitoring.

Wasn't aware this was also fixed in the cookbook a few days after it was used here (https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1071588).

Dzahn mentioned this in T354688: Request a spare host for Collab services (Phorge, Gerrit, Contint).Oct 22 2024, 6:22 PM
Dzahn mentioned this in T377889: install a service on phab1005.Oct 22 2024, 7:20 PM
Dzahn changed the status of subtask T377889: install a service on phab1005 from Open to Stalled.Apr 21 2025, 5:31 PM
Dzahn changed the status of subtask T377889: install a service on phab1005 from Stalled to Open.May 28 2025, 10:47 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits