Page MenuHomePhabricator
Create Task
Maniphest T179050

setup bast4002/WMF7218
Closed, ResolvedPublic
Actions

Description

This task will track the setup of bast4002/WMF7218. This system is 1 of 3 misc systems ordered on T160936 to replace ALL non-cp systems in ulsfo. @RobH will handle setup.

  • - refactor the role/profile(s) of bastion server, as it now applies multiple roles and fails our style check
  • - apply refactored role for bastion server to bast4002 (this will also have to ensure it works on other bastions)
  • - replace bast4001

virtualization note

Some of these systems will potentially be running ganeti. However, due to not knowing which will and will not, virtualization has been disabled on the CPU for ALL of the new ulsfo misc hosts orders on T160936. This can easily be changed back to enabled via the following instructions:

  • system will need to reboot, please ensure that this won't interrupt any vital services
  • ssh root@bast4002.mgmt.ulsfo.wmnet
  • run the following on the mgmt console:
racadm set iDRAC.ServerBoot.FirstBootDevice BIOS
console com2
  • in another ssh session, login to bast4002.wikimedia.org and reboot it. The racadm command set it to go into bios the next boot.
  • swap back to your mgmt console session and watch it launch system bios.
  • in bios, go to System Bios > Processor Settings > Virtualization Technology & Enable it. Then hit ESC and back out, confirming Yes when it asks to save and exit.
  • boot system back up and it will have the virtualization enabled on the processors.

Details

SubjectRepoBranchLines +/-
remove bast4001 from prometheus firewall exceptionsoperations/puppetproduction+0 -1
changing prometheus.svc.ulsfo.wmnet entry to bast4002operations/dnsmaster+1 -1
bast4002: switch over prometheusoperations/puppetproduction+1 -0
splitting bast4002 to its own entryoperations/puppetproduction+11 -1
setting bast4001 paramsoperations/puppetproduction+12 -2
set bast4002.wikimedia.org dns entriesoperations/dnsmaster+6 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedBBlackT104442 Investigate better DNS cache/lookup solutions
 ResolvedRobHT164327 replace ulsfo aging servers
Unknown Object (Task)
 ResolvedfgiunchediT179050 setup bast4002/WMF7218
 ResolvedRobHT178592 decommission/replace bast4001.wikimedia.org

Event Timeline

RobH created this task.Oct 26 2017, 12:46 AM
RobH added a subtask: T178592: decommission/replace bast4001.wikimedia.org.
RobH updated the task description. (Show Details)Oct 26 2017, 12:49 AM
gerritbot subscribed.Oct 26 2017, 4:30 PM

Change 386643 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] set bast4002.wikimedia.org production dns entries

https://gerrit.wikimedia.org/r/386643

gerritbot added a project: Patch-For-Review.Oct 26 2017, 4:30 PM
gerritbot added a comment.Oct 26 2017, 4:32 PM

Change 386643 merged by RobH:
[operations/dns@master] set bast4002.wikimedia.org dns entries

https://gerrit.wikimedia.org/r/386643

RobH updated the task description. (Show Details)Oct 26 2017, 4:34 PM
gerritbot added a comment.Oct 26 2017, 4:50 PM

Change 386652 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] setting bast4001 params

https://gerrit.wikimedia.org/r/386652

gerritbot added a comment.Oct 26 2017, 5:08 PM

Change 386652 merged by RobH:
[operations/puppet@production] setting bast4001 params

https://gerrit.wikimedia.org/r/386652

gerritbot added a comment.Oct 26 2017, 6:19 PM

Change 386672 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] splitting bast4002 to its own entry

https://gerrit.wikimedia.org/r/386672

gerritbot added a comment.Oct 26 2017, 6:25 PM

Change 386672 merged by RobH:
[operations/puppet@production] splitting bast4002 to its own entry

https://gerrit.wikimedia.org/r/386672

RobH removed a project: Patch-For-Review.Oct 26 2017, 7:01 PM
RobH updated the task description. (Show Details)
RobH mentioned this in T179204: setup/deploy dns400[12]/wmf721[56].Oct 27 2017, 7:11 PM
• MoritzMuehlenhoff subscribed.Nov 1 2017, 8:24 AM

This is currently installed with jessie, but if we setup a new box, let's use stretch from the start?

BBlack added a comment.Nov 1 2017, 1:49 PM
In T179050#3725651, @MoritzMuehlenhoff wrote:

This is currently installed with jessie, but if we setup a new box, let's use stretch from the start?

+1 We may as well move to stretch here. For the bastion/installserver role it should be pretty simple?

• MoritzMuehlenhoff added a comment.Nov 1 2017, 1:55 PM
In T179050#3726257, @BBlack wrote:

+1 We may as well move to stretch here. For the bastion/installserver role it should be pretty simple?

I wouldn't expect any problems. It also has role::prometheus::ops, but the prometheus version we're using on prometheus* is a backport of the prometheus version in stretch, so that should be fine as well.

fgiunchedi subscribed.Nov 2 2017, 4:19 PM
In T179050#3726279, @MoritzMuehlenhoff wrote:
In T179050#3726257, @BBlack wrote:

+1 We may as well move to stretch here. For the bastion/installserver role it should be pretty simple?

I wouldn't expect any problems. It also has role::prometheus::ops, but the prometheus version we're using on prometheus* is a backport of the prometheus version in stretch, so that should be fine as well.

Indeed I don't expect any problems, we don't have any stretch + prometheus machine yet but should be fairly easy to test in labs if need be. While we're at it we should be also copying (rsync is fine) the prometheus data from bast4002, I can assist with that.

RobH added a comment.Nov 2 2017, 4:28 PM

So this now reads 'bast4001' in the subject, but it is bast4002, just making sure no one changed that intentionally? (I setup the task as bast4002, so checking.)

Dzahn subscribed.Nov 2 2017, 4:33 PM

+1 to 4002 and stretch !

RobH added a comment.Nov 2 2017, 5:07 PM

Ok, I'll reimage, I'm also doing the conversion to bastion profile. (Unless Brandon tells me otherwise, I'm also making all references on this new server be bast4002, since bast4001 will remain online until this system is fully deployed.)

RobH renamed this task from setup bast4001/WMF7218 to setup bast4002/WMF7218.Nov 2 2017, 5:07 PM
• ema moved this task from Backlog to Radar/Not for service by Traffic on the Traffic board.Nov 9 2017, 7:20 AM
fgiunchedi added a comment.EditedDec 11 2017, 3:04 PM

More details on the Prometheus part:

  • Allow bast4002 as an additional Prometheus host (https://gerrit.wikimedia.org/r/#/c/393943/)
  • rsync Prometheus data from bast4001 to bast4002
    • Initial rsync can run while Prometheus is running on bast4001. Plus an additional rsync with Prometheus stopped on bast4001 to make sure all data is consistent and flushed
  • Start and verify Prometheus works on bast4002
  • Flip CNAME for prometheus.svc to bast4002
Dzahn added a comment.Mar 28 2018, 12:42 AM

Was just working on the Bastion related Wikitech pages due to Bast1001 being replaced and i noticed we have 2 bastions in ULSFO, 4001 and 4002. stalled?

BBlack added a comment.Mar 28 2018, 12:29 PM

Was stalled on my lack of time dealing with the prometheus switchover and then switching peoples' SSH configs, otherwise it's ready for switchover.

fgiunchedi added a comment.Mar 29 2018, 10:02 AM

FWIW I'm happy to assist with the Prometheus part, e.g. next week

RobH added a parent task: Unknown Object (Task).Aug 21 2018, 6:08 PM
gerritbot added a comment.Oct 3 2018, 5:24 PM

Change 464369 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] changing prometheus.svc.ulsfo.wmnet entry to bast4002

https://gerrit.wikimedia.org/r/464369

gerritbot added a project: Patch-For-Review.Oct 3 2018, 5:24 PM

Change 393943 had a related patch set uploaded (by RobH; owner: BBlack):
[operations/puppet@production] bast4002: switch over prometheus

https://gerrit.wikimedia.org/r/393943

gerritbot added a comment.Oct 3 2018, 5:28 PM

Change 393943 merged by RobH:
[operations/puppet@production] bast4002: switch over prometheus

https://gerrit.wikimedia.org/r/393943

gerritbot added a comment.Oct 3 2018, 5:33 PM

Change 464371 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] remove bast4001 from prometheus firewall exceptions

https://gerrit.wikimedia.org/r/464371

RobH reassigned this task from RobH to fgiunchedi.EditedOct 3 2018, 5:34 PM

Ok, updates from IRC sync up and followup actions:

RobH moved this task from Backlog to Racking Tasks on the ops-ulsfo board.Oct 3 2018, 5:50 PM
Stashbot subscribed.Oct 5 2018, 7:20 AM

Mentioned in SAL (#wikimedia-operations) [2018-10-05T07:20:29Z] <godog> temporarily stop prometheus on bast4001 to finalize data transfer - T179050

gerritbot added a comment.Oct 5 2018, 7:47 AM

Change 464369 merged by Filippo Giunchedi:
[operations/dns@master] changing prometheus.svc.ulsfo.wmnet entry to bast4002

https://gerrit.wikimedia.org/r/464369

fgiunchedi added a comment.Oct 5 2018, 8:01 AM

Data transfer and CNAME flip completed. I've documented the data transfer itself at https://wikitech.wikimedia.org/wiki/Prometheus#Sync_data_from_an_existing_Prometheus_host. Once TTL expires bast4001 should no longer receive queries, this can be verified by looking at /var/log/apache2/other_vhosts_access.log.

fgiunchedi added a comment.Oct 5 2018, 12:33 PM
In T179050#4644666, @fgiunchedi wrote:

Data transfer and CNAME flip completed. I've documented the data transfer itself at https://wikitech.wikimedia.org/wiki/Prometheus#Sync_data_from_an_existing_Prometheus_host. Once TTL expires bast4001 should no longer receive queries, this can be verified by looking at /var/log/apache2/other_vhosts_access.log.

This has happened, we can proceed with decom bast4001 (e.g. in puppet)

faidon added a comment.Nov 9 2018, 8:06 PM

Can this task be resolved, given we have T178592 to track the bast4001 decom?

RobH closed this task as Resolved.Dec 10 2018, 9:28 PM
RobH closed subtask T178592: decommission/replace bast4001.wikimedia.org as Resolved.Jun 25 2019, 4:58 PM
gerritbot added a comment.Sep 17 2020, 3:34 PM

Change 464371 abandoned by RobH:
[operations/puppet@production] remove bast4001 from prometheus firewall exceptions

Reason:
old neglected patchset, no longer needed.

https://gerrit.wikimedia.org/r/464371

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits