Page MenuHomePhabricator

Move labstore1002 and labstore1002-array1 and labstore1002-array2 to different rack (currently in C3)
Closed, ResolvedPublic

Description

labstore1001 and labstore1002 are currently located in the same rack C3, and we'd like to move to the following shelve arrangement:

  • Labstore1001 - attached to labstore1001-array1 and labstore1001-array2
  • Labstore1002 - attached to labstore1002-array1 and labstore1002-array2 (Moved to a different rack)
  • labstore1001-array3 - spare for now

Currently all 5 disk arrays are connected via H800 to labstore1001 and we are hoping to split that up.

Next steps would be:

  • Setup disks in BIOS for new shelve arrangement (RAID10)
  • Reinstall OS on both servers (needs a partman recipe applied)
  • Direct cable eth1 between labstore1001 and labstore1002

All of the data in the shelves, and the servers are unused and can be destroyed.

Event Timeline

Note: to do the DRBD direct link they may need to be close, even adjoining racks. I'm not sure. Kind of a call for @Cmjohnson to make.

I can make a cable to run anywhere in the data center so proximity is not an issue. I need to find a space that has available u for the server and 2 disk shelves. Row C is not going to be an option, Most likely going to row B5 where the Netapp used to reside but will need to check for available eth ports on the switch first.

@Cmjohnson Moving to row B is fine, but we'd have to move both servers to row B then. We need both servers to be in the same row for the NFS failover to work. As broadcast domains don't extend beyond a row, and we are using gratuitous ARP to fail a single IP between the nodes, and have to use IPs (not service urls) due to limitations with the NFS client.

@Cmjohnson Bumping this, we'd like to move forward with this soon if possible :) Thanks!

@madhuvishy . Sorry I was away on vacation. When do you want to do this next week. I would prefer if we could do it around 1000 EST any day that is convenient. Still going to move to B5.

@Cmjohnson Yup no problem! :) Just to confirm, can we move both servers to row B then? My last comment on NFS failover explains why we'd need to do this.

@madhuvishy You want both moved to row B but separate racks? If they need to be connected to each other that would be difficult to do across racks

@Cmjohnson Yes I think separate racks for reliability. We currently have labstore1004 and labstore1005 set up in the same way, in racks C2 and C5, which eth1 being a direct cable link. This is what I assumed we'll do for these two (labstore1001/2). We need them to be in the same row for NFS failover reasons, and the direct link across racks for DRBD replication. Let me know if I'm missing something, or can clarify anything better!

It's similar to what we are doing with labstore1004/1005, or same model. I believe those are in neighboring racks.

To clarify
labstore1001 and 2 arrays are going to rack B5
labstore1002 and all arrays are going to B6

Each has 1 production cable connected and I am using a crossover cable to connect eth1
I will setup raid as it's described previously.

Please give me the go ahead on this

@Cmjohnson - One edit

labstore1002 and all arrays are going to B6

I assume that's really all other* arrays, since two are already in B5. Just clarifying :)

Otherwise looks good +1.

@madhuvishy that is correct ... labstore1001 and 2 arrays in B5 and labstore1002 and 2 arrays in B6

labstore1001 and 2 arrays are in B5 connected to ge-5/0/4
labstore1002 and 2 arrays are in B7 connected to ge-7/0/0

I set up raid 10 for 3 VD's on each for a total of 36 disks
Connected both via eth1 using a cross-over cable
updated switch descriptions but vlans are not set
updated racktables

Needs DNS once vlans are confirmed.

I mentioned this to Chase in PM on IRC...there is not a labs-support vlan in row B currently.

Hi @Cmjohnson, apologies for the delay here, we were working through the possibilities of what the next steps should be, given that row B doesn't have the labs-support vlan. We had (mistakenly) assumed that row B had the vlan, but in fact it looks row A and C are the ones that have it enabled in eqiad. The labs-support vlan is a mechanism we are looking to phase out in the near future, and hence setting it up in row B may not be a useful venture. Given that, we wonder if we could just move the servers once again (at great pain to you, apologies :( ) to row A. I've looked at the rack utilization, and it looks like row A has space, but I am not aware of all the parameters involved in making those decisions. Let us know if that'd be possible, and your thoughts. Thanks, and sorry for the trouble!

I really do not know where else they can go...each one of these requires 6u of space and has to be in (preferably) adjacent racks. Even without the adjacent rack requirement I do not really have to many places I can move them in row A or row D. The closest thing is D1 and D2 but I would need to move ocg1002 and ocg1003.

@Cmjohnson and I spoke a bit on IRC, and we talked about moving back to row C, and possibly to C2 and C3, provided we can move db1056 2 units down in the same rack C2. This would be ideal for us, and we can go ahead if the db server move is alright :) Thank you!

No problem with me on this, it just takes a few ( minutes to depool/shutdown mysql/shutdown, etc. - if you schedule a time, I can have it ready for you in advance so I do not make you lose too much time waiting, @Cmjohnson.).

@madhuvishy @chasemp Moved the labstores back to row C...labstore1001 is in c2 and labstore1002 in c3...all are cabled, vlans are set, switch ports enabled, cross connect in place and I verified the disk shelves are accessible from the hosts. racktables updated.

**Outstanding Item is DNS