Page MenuHomePhabricator
Create Task
Maniphest T224747

Move maps and scratch on cloudstore1008/9 to a DRBD failover similar to labstore1004/5
Open, MediumPublic
Actions

Description

On reviewing the setup obsessively, I realized that a locking issue will prevent smooth failover for maps in the event of an NFS server failover. As mentioned in T203469, NFS failover is not very good without shared cluster filesystems, so this isn't the only place where that is true. Currently I will be documenting the failover as suggesting that the maps servers be rebooted to clean up after.

Because the failover of the primary cluster now works, this task is now to go to that model. The steps are:

  1. Evaluate and fix the issue of having a DRBD network interface/IP (significant work here)
  2. Get all the monitoring and scripts in place for DRBD
  3. Set the standby server to be a DRBD primary system as an rsync target
  4. Move scratch to use the cluster IP if not already there
  5. Fail over to the DRBD primary node
  6. Stop/remove the rsync process
  7. Set up the DRBD secondary (destructive step) and connect the cluster

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+0 -8cloud nfs: clean up some of the secondary cluster materials
Customize query in gerrit

Related Objects
Search...

Event Timeline

Bstorm triaged this task as Medium priority.May 31 2019, 6:46 PM
Bstorm created this task.
Bstorm added a comment.Jun 5 2019, 6:38 PM

Active/active drbd requires detached storage. Overall, failover with NFS attached is never going to look very good. A clever solution may be possible once ceph is deployed.

bd808 moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.Jul 11 2019, 4:28 PM
Bstorm changed the task status from Open to Stalled.Aug 6 2019, 1:57 PM

Waiting on this in hopes that maybe one of the two volumes could be replaced with something else, like a cephfs.

Bstorm moved this task from Doing to Soon! on the cloud-services-team (Kanban) board.Sep 25 2019, 3:27 PM
Bstorm removed Bstorm as the assignee of this task.Sep 25 2019, 3:54 PM

Removing myself because cookie-licking is bad when I'm not working on it.

Bstorm changed the task status from Stalled to Open.Jun 11 2020, 11:17 PM

I think we know two things about this right now:

  1. This isn't going on cephfs directly.
  2. We should convert this cluster to using DRBD like the primary NFS system. It will work so much better.
Bstorm mentioned this in T224510: Document the new NFS setup on cloudstore1008/9.Jun 11 2020, 11:19 PM
Bstorm added a project: Data-Services.
Bstorm moved this task from Backlog to Shared Storage on the Data-Services board.
Bstorm claimed this task.Jun 22 2020, 10:59 PM
Bstorm moved this task from Soon! to Doing on the cloud-services-team (Kanban) board.
gerritbot added a comment.Jun 22 2020, 11:01 PM

Change 607142 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] cloud nfs: clean up some of the secondary cluster materials

https://gerrit.wikimedia.org/r/607142

gerritbot added a project: Patch-For-Review.Jun 22 2020, 11:01 PM
Bstorm renamed this task from Improve the failover mechanism for maps on cloudstore1008/9 to Move maps and scratch on cloudstore1008/9 to a DRBD failover similar to labstore1004/5.Jun 22 2020, 11:08 PM
Bstorm updated the task description. (Show Details)
gerritbot added a comment.Jun 23 2020, 3:59 PM

Change 607142 merged by Bstorm:
[operations/puppet@production] cloud nfs: clean up some of the secondary cluster materials

https://gerrit.wikimedia.org/r/607142

Maintenance_bot removed a project: Patch-For-Review.Jun 23 2020, 4:12 PM
Bstorm moved this task from Doing to Inbox on the cloud-services-team (Kanban) board.Jul 2 2020, 8:43 PM
bd808 moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.Jul 17 2020, 9:59 PM
Bstorm added a subscriber: ayounsi.Aug 10 2020, 4:47 PM

@ayounsi I was wondering if you could help me with a mystery on this task. Are the second port on labstore1004 and labstore1005 on a non-routed network connection to each other or are they directly linked via crossover cable?

ayounsi added a comment.Aug 10 2020, 4:53 PM
In T224747#6373732, @Bstorm wrote:

@ayounsi I was wondering if you could help me with a mystery on this task. Are the second port on labstore1004 and labstore1005 on a non-routed network connection to each other or are they directly linked via crossover cable?

Looks directly connected.

ayounsi@labstore1004:~$ sudo lldpcli 
[lldpcli] # show neighbors
-------------------------------------------------------------------------------
LLDP neighbors:
-------------------------------------------------------------------------------
[...]
-------------------------------------------------------------------------------
Interface:    eth1, via: LLDP, RID: 3, Time: 12 days, 11:11:56
  Chassis:     
    ChassisID:    mac 14:18:77:5d:2d:d2
    SysName:      labstore1005.eqiad.wmnet
    SysDescr:     Debian GNU/Linux 9 (stretch) Linux 4.9.0-0.bpo.12-amd64 #1 SMP Debian 4.9.210-1~deb8u1 (2020-02-21) x86_64
    TTL:          120
    MgmtIP:       10.64.37.20
    MgmtIP:       2620:0:861:119:10:64:37:20
    Capability:   Bridge, off
    Capability:   Router, off
    Capability:   Wlan, off
    Capability:   Station, on
  Port:        
    PortID:       mac 14:18:77:5d:2d:d3
    PortDescr:    eth1
-------------------------------------------------------------------------------
Cmjohnson closed subtask T266192: Connect cloudstore1008 and cloudstore1009 directly via second 10G interface similar to labstore1004/5 as Resolved.Feb 11 2021, 4:59 PM
Bstorm added a comment.Mar 8 2021, 11:05 PM

Ok, there should be a cable now. This is unblocked and should proceed.

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL