- create new vlan, cloud-storage1-codfw-dev - https://netbox.wikimedia.org/ipam/vlans/147/
- reconfigure network so that each host has their primary nic ("eth0") on cloud-hosts1-b-codfw and secondary nic ("eth1") on cloud-storage1-codfw-dev
- rename cloudceph200x-dev hosts to cloudcephosd200x-dev https://gerrit.wikimedia.org/r/c/operations/puppet/+/642408
Description
Description
Details
Details
Related Changes in Gerrit:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • aborrero | T217891 CloudVPS: rework codfw deployments | |||
| Resolved | None | T229441 CloudVPS: codfw1dev: missing bits | |||
| Resolved | Andrew | T265965 Ceph dev cluster in codfw1dev | |||
| Resolved | Andrew | T266261 rearrange networking for cloudceph200[1-3]-dev and rename |
Event Timeline
Comment Actions
@Papaul, are these things you can help with or do we need Arzhel? (I can do most of the host renaming of course).
Comment Actions
The requested config seems good to me, I just compared it to the eqiad1 in https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Ceph#Network
Comment Actions
Change 642408 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Rename cloudceph200x-dev to cloudcephosd200x-dev
Comment Actions
@Papaul, if you want to make the netbox/network changes, I can do the actual re-imaging. There's nothing happening on these boxes currently so you can break them whenever :)
Comment Actions
As for the Netbox side of things I think that with this procedure we should be able to reach the desired status (experimental, never tried):
- From https://netbox.wikimedia.org/dcim/devices/2637/ delete the IPv4 and IPv6 currently assigned
- From https://netbox.wikimedia.org/ipam/prefixes/147/ip-addresses/ add an IP Address and set the DNS Name
- From https://netbox.wikimedia.org/ipam/prefixes/226/ip-addresses/ add an IP Address with the mapped version of the IPv4 created in step (2) and set the DNS Name (unless the service doesn't support v6)
- From https://netbox.wikimedia.org/dcim/interfaces/9637/ rename eno1 to ##PRIMARY##
- From https://netbox.wikimedia.org/dcim/devices/2637/ click on the green plus button on the ##PRIMARY## line and attach the IPs created in step (2) and (3)
- From https://netbox.wikimedia.org/dcim/devices/2637/ mark the two IPs as primary
As for the secondary IP that's on a VLAN whose prefixes are not managed by Netbox so should not appear at all
Comment Actions
- edit the switch ports to match the new access vlans (eg. https://netbox.wikimedia.org/dcim/interfaces/16057/edit/ -> Untagged VLAN -> cloud-hosts-b-codfw) then https://netbox.wikimedia.org/dcim/interfaces/17740/ cloud-storage
Comment Actions
If you do 2 before 5 it will not work. you get error " interface exist already" . so you don't have to to 2 . The final steps are
- From https://netbox.wikimedia.org/dcim/devices/2637/ delete the IPv4 and IPv6 currently assigned
- From https://netbox.wikimedia.org/dcim/interfaces/9637/ rename eno1 to PRIMARY
- From https://netbox.wikimedia.org/dcim/devices/2637/ click on the green plus button on the PRIMARY line and attach an IPs
- From https://netbox.wikimedia.org/dcim/devices/2637/ mark the two IPs as primary
- Edit the switch ports to match the new access vlans (eg. https://netbox.wikimedia.org/dcim/interfaces/16057/edit/ -> Untagged VLAN -> cloud-hosts-b-codfw) then https://netbox.wikimedia.org/dcim/interfaces/17740/ cloud-storage
Comment Actions
@ayounsi there is no interface range for VLAN cloud-storage-b-codfw in row b. Do you want me to create it, or just assign the second interface of those servers directly to the VLAN?
Comment Actions
As soon as you update Netbox with the proper vlan on the switch interface ("https://netbox.wikimedia.org/dcim/interfaces/17740/ cloud-storage"), homer will configure it automatically.
Comment Actions
I realized now that we should have decomm'ed them because of the rename, also to clear puppetdb/debmonitor/icinga. But if I'll do that now they will clear the already assigned IPs, so I will instead manually remove them from puppetdb/debmonitor that will in turn remove them from icinga too.
Comment Actions
[edit interfaces]
interface-range vlan-cloud-hosts1-b-codfw { ... }
+ interface-range vlan-cloud-storage1-b-codfw {
+ member ge-1/0/5;
+ mtu 9192;
+ unit 0 {
+ family ethernet-switching {
+ interface-mode access;
+ vlan {
+ members cloud-storage1-b-codfw;
+ }
+ }
+ }
+ }
interface-range vlan-private1-a-codfw { ... }
[edit interfaces interface-range vlan-private1-b-codfw]
- member ge-1/0/5;[edit interfaces interface-range vlan-cloud-storage1-b-codfw]
member ge-1/0/5 { ... }
+ member ge-5/0/10;
+ member ge-8/0/11;
[edit interfaces interface-range vlan-private1-b-codfw]
- member ge-5/0/10;
- member ge-8/0/11;Comment Actions
Change 642408 merged by Andrew Bogott:
[operations/puppet@production] Rename cloudceph200x-dev to cloudcephosd200x-dev
Comment Actions
Change 643275 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Move cloudcephosd2xxx-dev from .wikimedia.org to .codfw.wmnet
Comment Actions
Change 643275 merged by Andrew Bogott:
[operations/puppet@production] Move cloudcephosd2xxx-dev from .wikimedia.org to .codfw.wmnet
Comment Actions
I'm a bit lost in the backscroll :) Did the second nics get attached and assigned for all these? @dcaro points out that they kernel thinks they are disconnected:`
root@cloudcephosd2001-dev:~# grep "" /sys/class/net/eno*/carrier
/sys/class/net/eno1/carrier:1
grep: /sys/class/net/eno2/carrier: Invalid argument
Comment Actions
papaul@asw-b-codfw> show interfaces descriptions | match cloudcephosd
ge-1/0/4 up up cloudcephosd2001-dev:eno1 {#}
ge-1/0/5 up up cloudcephosd2001-dev:eno2 {#}
ge-5/0/8 up up cloudcephosd2002-dev:##PRIMARY## {#}
ge-5/0/10 up up cloudcephosd2002-dev:eno2 {#}
ge-8/0/10 up up cloudcephosd2003-dev:eno1 {#}
ge-8/0/11 up up cloudcephosd2003-dev:eno2 {#}papaul@asw-b-codfw> show ethernet-switching interface ge-1/0/5.0
Routing Instance Name : default-switch
Logical Interface flags (DL - disable learning, AD - packet action drop,
LH - MAC limit hit, DN - interface down,
SCTL - shutdown by Storm-control,
MMAS - Mac-move action shutdown, AS - Autostate-exclude enabled)
Logical Vlan TAG MAC STP Logical Tagging
interface members limit state interface flags
ge-1/0/5.0 294912 untagged
cloud-storage1-b-codfw 2106 294912 Forwarding untagged