Page MenuHomePhabricator

rack/setup/install bast3004
Closed, ResolvedPublic

Description

This task will track the racking/setup/installation of bast3004 ordered on task T230620.

Racking Proposal: Setup via the google sheet rack layout.

bast3004:

  • - receive in system on procurement task T230620
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - disable embedded NIC
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

Details

Event Timeline

RobH triaged this task as Normal priority.Thu, Oct 24, 3:22 PM
RobH created this task.
Restricted Application added a project: Operations. · View Herald TranscriptThu, Oct 24, 3:22 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
RobH added a parent task: Unknown Object (Task).Thu, Oct 24, 3:22 PM

Change 545880 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/dns@master] esams: mgmt dns for rack 16

https://gerrit.wikimedia.org/r/545880

Change 545880 merged by BBlack:
[operations/dns@master] esams: mgmt dns for rack 16

https://gerrit.wikimedia.org/r/545880

RobH removed a subscriber: RobH.Thu, Oct 24, 3:58 PM

Change 545893 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] Add dhcp macaddrs for esams rack 16 hosts

https://gerrit.wikimedia.org/r/545893

Change 545893 merged by BBlack:
[operations/puppet@production] Add dhcp macaddrs for esams rack 16 hosts

https://gerrit.wikimedia.org/r/545893

Dzahn added a subscriber: Dzahn.Thu, Oct 24, 5:35 PM

merging in duplicate ticket T236409 where i started OS install

Dzahn updated the task description. (Show Details)

confirmed mgmt and production DNS exists, mgtm password is set, IPMI over LAN working..

started OS install

Dzahn added a comment.Thu, Oct 24, 5:58 PM
[bast3004:~] $ gen_fingerprints
 +---------+---------+-----------------------------------------------------+
 | Cipher  | Algo    | Fingerprint                                         |
 +---------+---------+-----------------------------------------------------+
 | RSA     | SHA-256 | SHA256:/0S1drRdtp6jpPBDq0bb45228GwTdouVTOGNKYDGZ8Y  |
 +---------+---------+-----------------------------------------------------+
 | ECDSA   | SHA-256 | SHA256:vp48DKQHJ0x/cjPZ7MsXe+ChoMZdkVLS0m0xwg832dw  |
 +---------+---------+-----------------------------------------------------+
 | ED25519 | SHA-256 | SHA256:O7bKwLBunrULntnBP7G/z2Stcvk4mjJ3EgNNxqCb8+E  |
 +---------+---------+-----------------------------------------------------+

 +---[RSA 2048]----+ +---[ECDSA 256]---+ +--[ED25519 256]--+
 |       . o       | |         +..o= . | |      .o         |
 |        + E    . | |    .   o *.*.o E| |     .  +        |
 |       . + .  oo*| |   o .   B * .   | |    .  +         |
 |            ..o**| |    + = B = .    | |     o. .        |
 |        S   ..=o+| |     * +S= .     | |  . + ..S        |
 |         o o +.*.| |    . o.. o +    | |   = +..o. .     |
 |        ..=.+.=+.| |     o =.+ + =   | |  o * Eo+oo..    |
 |         ooBB=o..| |      +.+o+ + .  | | o.B *=.+B=o     |
 |        ..o+*Bo  | |     . .=. . .   | | oB +.+B*O=o.    |
 +----[SHA256]-----+ +----[SHA256]-----+ +----[SHA256]-----+
Dzahn updated the task description. (Show Details)Thu, Oct 24, 6:00 PM
Dzahn updated the task description. (Show Details)Thu, Oct 24, 6:55 PM

Change 545921 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] smokeping: replace bast3002 with bast3004 as target

https://gerrit.wikimedia.org/r/545921

Change 545922 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] hieradata/common: add bast3004 to bastion hosts for firewalls

https://gerrit.wikimedia.org/r/545922

Change 545922 merged by Dzahn:
[operations/puppet@production] hieradata/common: add bast3004 to bastion hosts for firewalls

https://gerrit.wikimedia.org/r/545922

Dzahn added a comment.Thu, Oct 24, 8:11 PM

^ made active bastion host with the global firewall change above

created wikitech pages

https://wikitech.wikimedia.org/wiki/Bast3004
https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/bast3004.wikimedia.org

Can be used as a jump host now.

Setting to "Active" in netbox.

Change 545921 merged by Dzahn:
[operations/puppet@production] smokeping: replace bast3002 with bast3004 as target

https://gerrit.wikimedia.org/r/545921

Papaul updated the task description. (Show Details)Thu, Oct 24, 8:22 PM
Dzahn claimed this task.Thu, Oct 24, 8:24 PM
Dzahn added a subscriber: Papaul.

service "bastion host" is ready but service "tftp" still needs to be migrated. taking it.

Change 545935 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] Revert "delete unused bastionhost::migration class"

https://gerrit.wikimedia.org/r/545935

Change 545935 merged by Dzahn:
[operations/puppet@production] bastionhost: recreate migration class, convert to profile

https://gerrit.wikimedia.org/r/545935

Change 545971 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] bastionhost: fix migration class fqdn comparison, rename vars

https://gerrit.wikimedia.org/r/545971

Change 545973 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] DHCP: replace bast3002 with bast3004 as next-server

https://gerrit.wikimedia.org/r/545973

Change 545971 merged by Dzahn:
[operations/puppet@production] bastionhost: fix migration class fqdn comparison, rename vars

https://gerrit.wikimedia.org/r/545971

Mentioned in SAL (#wikimedia-operations) [2019-10-24T23:46:41Z] <mutante> bast3002 - rsyncing /home, /srv/tfptboot and /srv/prometheus to /srv/bast3002/ on bast3004 (T236394 T236329)

  • home dirs copied to individual $user_bast1002.tar.gz files in each user home (where the user exists on both old and new server) so users have their old files if they need them and can just delete the file if they don't
  • prometheus data saved in /srv/bast3002/prometheus/ (36G) and local prometheus data /srv/prometheus is 1.2G and unchanged
  • tftpdata (installers) rsynced into the actual /srv/tftpboot so both servers are identical there

Change 546001 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] switch esams prometheus node from bast3002 to bast3004

https://gerrit.wikimedia.org/r/546001

Change 546003 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] install_server: set 'next-server' for bast3004

https://gerrit.wikimedia.org/r/546003

Change 546003 merged by Dzahn:
[operations/puppet@production] install_server: set 'next-server' for bast3004

https://gerrit.wikimedia.org/r/546003

Change 545973 merged by Dzahn:
[operations/puppet@production] DHCP: switch esams DHCP server from bast3002 to bast3004

https://gerrit.wikimedia.org/r/545973

Change 546001 merged by Filippo Giunchedi:
[operations/puppet@production] add bast3004 to esams prometheus nodes

https://gerrit.wikimedia.org/r/546001

Prometheus data sync'd again from bast3002 and copied in place, DNS flipped, Prometheus is live on this host now and not active on bast3002 anymore

Dzahn changed the status of subtask T236329: decommission bast3002 from Stalled to Open.Fri, Oct 25, 3:14 PM
Dzahn closed this task as Resolved.Fri, Oct 25, 3:20 PM
Dzahn updated the task description. (Show Details)

Thank you, @fgiunchedi!

It's set to active in Netbox and i tested an install to confirm DHCP/tftpboot is working after that was switched too.

Resolving.