Page MenuHomePhabricator
Create Task
Maniphest T274206

(Need By: 2021-03-31) rack/setup/install ms-backup100[12]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of ms-backup100[12]

Hostname / Racking / Installation Details

Hostnames: ms-backup1001, ms-backup1002
Racking Proposal: Redundant to each other, preferentially different rows, it not, different racks, as to have power and network redundancy.
Networking/Subnet/VLAN/IP: 10G, single production link, single mgmt link.
Partitioning/Raid: Software RAID1 between disks is enough.
OS Distro: Buster

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

ms-backup1001:

  • - receive in system on procurement task T272018 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
    • end on-site specific steps
  • - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

ms-backup1002:

  • - receive in system on procurement task T272018 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - system incorrectly plugged into port 42 on switch, but reads 41 on this task and netbox, needs to be moved by onsite.
    • end on-site specific steps
  • - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

Once the system(s) above have had all checkbox steps completed, this task can be resolved.

Details

Other Assignee
Cmjohnson
ProjectBranchLines +/-Subject
operations/puppetproduction+2 -2fixing mac for ms-backup100[12]
operations/puppetproduction+15 -1ms-backup100[12] updates
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT262668 WMF media storage must be adequately backed up
Unknown Object (Task)
Unknown Object (Task)
 ResolvedjcrespoT274206 (Need By: 2021-03-31) rack/setup/install ms-backup100[12]

Event Timeline

RobH renamed this task from (Need By: TBD) rack/setup/install ms-backup100[12] to (Need By: 2021-03-31) rack/setup/install ms-backup100[12].Feb 8 2021, 10:13 PM
RobH created this task.
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.
RobH added a parent task: Unknown Object (Task).
RobH mentioned this in Unknown Object (Task).
RobH assigned this task to jcrespo.Feb 8 2021, 10:16 PM
RobH added subscribers: Jclark-ctr, jcrespo.

@jcrespo: Conversation between you and Arzhel on T272018 seems to indicate some kind of discussion is still pending for these to determine where they can be racked? They just need to be in different 10G racks than one another as far as I can tell, please advise on this task and assign to @Jclark-ctr once you have done so, as this is now pending order.

RobH mentioned this in T274202: (Need By: 2021-03-31) rack/setup/install ms-backup200[12].Feb 8 2021, 10:18 PM
RobH removed a subscriber: RobH.Feb 8 2021, 10:21 PM
Maintenance_bot added a project: SRE.Feb 8 2021, 10:45 PM
jcrespo reassigned this task from jcrespo to Jclark-ctr.Feb 9 2021, 9:31 AM
jcrespo added subscribers: ayounsi, RobH.

@RobH it got resolved. It will go in the regular production internal vlan (same as ms-fe hosts). In the future there is a chance it will move to the public vlan to get a public ip, but that is not the current plan, and @ayounsi mentioned if that is the case, it will not require physical location changes, just a network config change.

RobH updated the task description. (Show Details)Feb 9 2021, 4:01 PM
RobH removed a subscriber: RobH.
Jclark-ctr reassigned this task from Jclark-ctr to Cmjohnson.Feb 24 2021, 11:42 PM
Jclark-ctr updated the task description. (Show Details)

Racked & cabled host handing over to Chris for configuration
Ms-backup1001 A4 u4 p3 id5322.
Ms-backup1001. C2 U41 P41 ID5587

Cmjohnson reassigned this task from Cmjohnson to RobH.Mar 5 2021, 3:23 PM
Cmjohnson updated the task description. (Show Details)
Cmjohnson added subscribers: RobH, Cmjohnson.

on-site work completed, moving to @RobH to finish installs

RobH added a comment.Mar 5 2021, 7:25 PM

Please note that ms-backup1002 shows port 41 in netbox and on task, but shows its plugged into port 42 on the idrac interface.

@Jclark-ctr: It seems like perhaps you plugged this into port 42 by mistake, can you please move it to port 41?

RobH updated the task description. (Show Details)Mar 5 2021, 7:25 PM
RobH reassigned this task from RobH to Jclark-ctr.Mar 5 2021, 7:28 PM

Please move the connection for ms-backup1002 fromn port 42 (shows 42 on idrac) to the port 41 listed on this task via T274206#6859101. Once that is done please assign back to me for followup and bulk imaging of this task.

RobH reassigned this task from Jclark-ctr to Cmjohnson.Mar 5 2021, 7:56 PM

IRC update, Chris will be onsite next and can move this cable.

Summary: ms-backup1002 in C2:U41 has DAC cable 5887 plugged into switch port 42 when it should be plugged into switch port 41. Please move to port 41 and reassign this task back to me for imaging, thanks!

Cmjohnson reassigned this task from Cmjohnson to RobH.Mar 9 2021, 5:06 PM

the ports were reversed, kafka-logging1002 was in 42 and ms-backup1002 was in 41. I swapped them and all should be good now.

ops-monitoring-bot added a comment.Mar 9 2021, 11:08 PM

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

['ms-backup1001.eqiad.wmnet', 'ms-backup1002.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202103092308_robh_24019.log.

gerritbot added a comment.Mar 9 2021, 11:14 PM

Change 670318 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] ms-backup100[12] updates

https://gerrit.wikimedia.org/r/670318

gerritbot added a project: Patch-For-Review.Mar 9 2021, 11:14 PM
Jclark-ctr added a comment.Mar 9 2021, 11:14 PM
This comment was removed by Jclark-ctr.
gerritbot added a comment.Mar 9 2021, 11:16 PM

Change 670318 merged by RobH:
[operations/puppet@production] ms-backup100[12] updates

https://gerrit.wikimedia.org/r/670318

RobH removed a project: Patch-For-Review.Mar 9 2021, 11:24 PM
RobH updated the task description. (Show Details)
RobH removed subscribers: Cmjohnson, Jclark-ctr.
ops-monitoring-bot added a comment.Mar 9 2021, 11:25 PM

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

['ms-backup1001.eqiad.wmnet', 'ms-backup1002.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202103092325_robh_26266.log.

gerritbot added a comment.Mar 9 2021, 11:41 PM

Change 670323 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] fixing mac for ms-backup100[12]

https://gerrit.wikimedia.org/r/670323

gerritbot added a project: Patch-For-Review.Mar 9 2021, 11:41 PM

Change 670323 merged by RobH:
[operations/puppet@production] fixing mac for ms-backup100[12]

https://gerrit.wikimedia.org/r/670323

ops-monitoring-bot added a comment.Mar 9 2021, 11:44 PM

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

['ms-backup1001.eqiad.wmnet', 'ms-backup1002.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202103092344_robh_29801.log.

ops-monitoring-bot added a comment.Mar 10 2021, 12:09 AM

Completed auto-reimage of hosts:

['ms-backup1001.eqiad.wmnet', 'ms-backup1002.eqiad.wmnet']

and were ALL successful.

Maintenance_bot removed a project: Patch-For-Review.Mar 10 2021, 12:10 AM
RobH closed this task as Resolved.Mar 10 2021, 12:16 AM
RobH updated the task description. (Show Details)
jcrespo reopened this task as Open.Dec 23 2021, 12:37 PM

I just realized with jbond that the 2 servers (ms-backup1001.eqiad.wmnet and ms-backup1002.eqiad.wmnet) do not have ipv6 AAAA dns records.

While this is not a production-breaking issue, it is now causing slowdowns (IPv6 timeouts), as I cannot open through DNS a firewall of another host for these client, slowing down a potential media recovery.

Before doing any change to configuration on my own, I would like to double check with dcops if the IP6 assignment/network configuration/inventory on your records is correct (IP6 stack on both servers seems to work ok!), and it was just a case of just a pending step/script failure, etc. and not something else, like lost records or something weirder:

https://netbox.wikimedia.org/ipam/ip-addresses/7916/
https://netbox.wikimedia.org/ipam/ip-addresses/7919/

The DNS records for ms-backup200[12] on codfw seem to be ok.

RobH added a subscriber: Cmjohnson.Dec 23 2021, 3:04 PM

@Cmjohnson sets up dns so perhaps he knows?

RobH reassigned this task from RobH to Cmjohnson.Dec 23 2021, 3:04 PM
RobH removed a subscriber: RobH.
Cmjohnson added a comment.Dec 23 2021, 3:12 PM

@jcrespo I believe this would've been a script failure/issue. I would only select the skip IPv6 button if the task called for it.

jcrespo claimed this task.Dec 23 2021, 3:41 PM

Thank you for the feedback! As we all think this was not disabled for some reason (I've seen the setup scripts fail on some steps sometimes, I will do: https://wikitech.wikimedia.org/wiki/DNS/Netbox#How_can_I_add_the_IPv6_AAAA/PTR_records_to_a_host_that_doesn't_have_it? as documented, and that should take care of it.

It is likely super-easy and fast, but I am not going to change DNS minutes away from my Christmas vacation, so will take care of this after I return in 2022

Claiming the task to fix in a few weeks.

jcrespo closed this task as Resolved.Jan 10 2022, 4:54 PM
jcrespo updated Other Assignee, added: Cmjohnson.

Deployed the change as this:

commit baeb288d4d9713814ac88e9537bbcf0ece5bb9e4                                                
Author: generate-dns-snippets <noc@wikimedia.org>                                              
Date:   Mon Jan 10 16:17:22 2022 +0000                                                         
                                                                                               
    root@cumin1001: Add IPv6 dns records for ms-backup1001/2                                   
                                                                                               
diff --git a/1.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa b/1.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa                                                                                              
index 6e27ad2..cabd95e 100644                                                                  
--- a/1.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa                                                 
+++ b/1.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa                                                 
@@ -49,6 +49,7 @@                                                                              
 7.0.1.0.0.0.0.0.4.6.0.0.0.1.0.0 1H IN PTR aqs1004.eqiad.wmnet.                                
 8.0.1.0.0.0.0.0.4.6.0.0.0.1.0.0 1H IN PTR netboxdb1001.eqiad.wmnet.                           
 0.1.1.0.0.0.0.0.4.6.0.0.0.1.0.0 1H IN PTR kafkamon1002.eqiad.wmnet.                           
+1.1.1.0.0.0.0.0.4.6.0.0.0.1.0.0 1H IN PTR ms-backup1001.eqiad.wmnet.                          
 2.1.1.0.0.0.0.0.4.6.0.0.0.1.0.0 1H IN PTR krb1001.eqiad.wmnet.                                
 7.1.1.0.0.0.0.0.4.6.0.0.0.1.0.0 1H IN PTR kubemaster1001.eqiad.wmnet.                         
 9.1.1.0.0.0.0.0.4.6.0.0.0.1.0.0 1H IN PTR grafana1002.eqiad.wmnet.                            
diff --git a/3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa b/3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa                                                                                              
index 711b906..bf82f76 100644                                                                  
--- a/3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa                                                 
+++ b/3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa                                                 
@@ -40,6 +40,7 @@                                                                              
 7.3.1.0.2.3.0.0.4.6.0.0.0.1.0.0 1H IN PTR matomo1002.eqiad.wmnet.                             
 8.3.1.0.2.3.0.0.4.6.0.0.0.1.0.0 1H IN PTR aqs1005.eqiad.wmnet.                                
 3.4.1.0.2.3.0.0.4.6.0.0.0.1.0.0 1H IN PTR registry1004.eqiad.wmnet.                           
+4.4.1.0.2.3.0.0.4.6.0.0.0.1.0.0 1H IN PTR ms-backup1002.eqiad.wmnet.                          
 8.4.1.0.2.3.0.0.4.6.0.0.0.1.0.0 1H IN PTR mc1046.eqiad.wmnet.                                 
 9.4.1.0.2.3.0.0.4.6.0.0.0.1.0.0 1H IN PTR thumbor1006.eqiad.wmnet.                            
 1.5.1.0.2.3.0.0.4.6.0.0.0.1.0.0 1H IN PTR mc1047.eqiad.wmnet.                                 
diff --git a/eqiad.wmnet b/eqiad.wmnet                                                         
index c35be41..cc14904 100644                                                                  
--- a/eqiad.wmnet                                                                              
+++ b/eqiad.wmnet                                                                              
@@ -1007,7 +1007,9 @@ moss-fe1001                              1H IN AAAA 2620:0:861:101:10:64:0:45                                                                                           
 moss-fe1002                              1H IN A 10.64.48.76                                  
 moss-fe1002                              1H IN AAAA 2620:0:861:107:10:64:48:76                
 ms-backup1001                            1H IN A 10.64.0.111                                  
+ms-backup1001                            1H IN AAAA 2620:0:861:101:10:64:0:111                
 ms-backup1002                            1H IN A 10.64.32.144                                 
+ms-backup1002                            1H IN AAAA 2620:0:861:103:10:64:32:144               
 ms-be1028                                1H IN A 10.64.0.21                                   
 ms-be1029                                1H IN A 10.64.0.25                                   
 ms-be1030                                1H IN A 10.64.0.26                                   
METADATA: {"path": "/tmp/dns-c25pcHBldHM-frtl99bc", "sha1": "baeb288d4d9713814ac88e9537bbcf0ece5bb9e4", "insertions": 4, "deletions": 0, "lines": 4, "files": 3}

dns was a bit unreliable just after the deploy, but it seems fine now, and it solved my timeout issue.

jcrespo mentioned this in T270101: Grants not working with DB hosts with to ipv6.Jan 19 2022, 9:49 AM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL