Page MenuHomePhabricator

Connect cloudstore1008 and cloudstore1009 directly via second 10G interface similar to labstore1004/5
Open, MediumPublic

Description

In order to provide a good failover service, this cluster needs a 10Gb crossover cable connecting them both like labstore1004 and labstore1005 have.

This will unblock setting up DRBD replication. I cannot find the appropriate task describing how this was done for the older pair of systems.

Work Checklist

cloudstore1008 - rack D2:U35 - https://netbox.wikimedia.org/dcim/devices/1864/
cloudstore1009 - rack D7:U30 - https://netbox.wikimedia.org/dcim/devices/1863/

Since cross-server connections are not common, I HIGHLY recommend that the cable connecting them be registered in netbox. (By clicking the 'connect' option for eno2 at the bottom of either of the netbox server entries, and connecting a cable between them with a new cable ID. Please run the cable report in netbox after completoin, to ensure the new cable ID was not one already in use.

Cable ID check: Load up https://netbox.wikimedia.org/dcim/cables/?q=&site=eqiad&type=&status=&color=&sort=label (which is for eqiad sorted by label) and just ensure what you plan to use on the cable isn't in there already as an eqiad label.

  • - run cat5 cable from cloudstore1008:eno2 to cloudstore1009:eno2
  • - update the cable entry in netbox under the server (see above directions)
  • - when this connection is complete, resolve this task so cloud-services-team know they can proceed with the setup of the eno2 interfaces on those hosts.

Event Timeline

Bstorm created this task.Oct 21 2020, 9:55 PM
Restricted Application added a project: SRE. · View Herald TranscriptOct 21 2020, 9:55 PM
Bstorm moved this task from Backlog to Shared Storage on the Data-Services board.Oct 21 2020, 9:56 PM
Bstorm reassigned this task from Bstorm to RobH.Oct 27 2020, 6:03 PM
Bstorm added a subscriber: RobH.

Assigning to @RobH to start discussion of how to actually make this happen.

RobH reassigned this task from RobH to Cmjohnson.Oct 27 2020, 6:16 PM
RobH moved this task from Backlog to Cloud Tasks on the ops-eqiad board.
RobH added subscribers: Jclark-ctr, wiki_willy, Cmjohnson.

So this appears like it is basically requiring someone in eqiad to perform a cross connection between the two hosts. I'm not actually the person to schedule this, either @wiki_willy (or one of the onsites in eqiad directly @Cmjohnson or @Jclark-ctr, but John is out due to a broken hand, so I'll assign this directly to Chris.

I cannot give any ETA on time (That is purely the domain of Willy and Chris), all I can do is suggest how I'd handle this below (and with my update of the task description.)

I've updated the task description with the following:

Checklist

cloudstore1008 - rack D2:U35 - https://netbox.wikimedia.org/dcim/devices/1864/
cloudstore1009 - rack D7:U30 - https://netbox.wikimedia.org/dcim/devices/1863/

Since cross-server connections are not common, I HIGHLY recommend that the cable connecting them be registered in netbox. (By clicking the 'connect' option for eno2 at the bottom of either of the netbox server entries, and connecting a cable between them with a new cable ID. Please run the cable report in netbox after completoin, to ensure the new cable ID was not one already in use.

Cable ID check: Load up https://netbox.wikimedia.org/dcim/cables/?q=&site=eqiad&type=&status=&color=&sort=label (which is for eqiad sorted by label) and just ensure what you plan to use on the cable isn't in there already as an eqiad label.

  • - run cat5 cable from cloudstore1008:eno2 to cloudstore1009:eno2
  • - update the cable entry in netbox under the server (see above directions)
  • - when this connection is complete, resolve this task so cloud-services-team know they can proceed with the setup of the eno2 interfaces on those hosts.
RobH triaged this task as Medium priority.Oct 27 2020, 6:16 PM
RobH updated the task description. (Show Details)
RobH removed a subscriber: RobH.

Thank you for helping me sort this out.

Bstorm updated the task description. (Show Details)Nov 4 2020, 8:03 PM
wiki_willy reassigned this task from Cmjohnson to Jclark-ctr.Nov 4 2020, 8:04 PM

Hey @Jclark-ctr - can you see if we have a cross-over cable long enough to make this connection?

Thanks,
Willy

@wiki_willy the crossover cable needs to be made. We have cat5 cable on-site and can be cut to the length needed. If you rather purchase a cross over cable then we need a blue 10M cable.

RobH mentioned this in Unknown Object (Task).Nov 5 2020, 7:58 PM
RobH added a subtask: Unknown Object (Task).
Bstorm added a comment.Dec 9 2020, 3:47 PM

T267366#6667864 suggests the cables should have arrived on-site.

Jclark-ctr closed subtask Unknown Object (Task) as Resolved.Dec 12 2020, 11:01 PM
Peachey88 updated the task description. (Show Details)Dec 12 2020, 11:43 PM

I was just looking into this. If the cross-over cables ordered will work, that's great (presuming the 10G card is RJ45, which I've been taking it on faith that it is based on this ticket). If it is not and is sfp+, any cable for that should work https://serverfault.com/questions/426869/10gbe-sfp-cross-over-cable-required-is-there-such-a-thing
@wiki_willy and @Cmjohnson

This is also the final step for the 10G upgrade to T266198.