starting maintenance do not expect any outages will be disconnecting pdu`s in about 1 hour

Hello John,

received raid controller

received memory placed in storage room

Received memory. placed in storage room

@elukey will these host be part of the fail over?

Racked and cabled host

located samarium removed from racks corrected netbox error

https://phabricator.wikimedia.org/T197630. corrected netbox removed host from rack

removed from rack updated netbox Resolving ticket previous comments list ports already removed

@elukey. ram has arrived how long will it take to drain host? and how long can they be down?

name rack position asset_tag switchport
an-worker1102 A4 39 WMF5406 38
an-worker1103 A7 39 WMF5407 25
an-worker1104 C2 34 WMF5408 34
an-worker1105 C4 18 WMF5409 43
an-worker1106 C4 22 WMF5410 41
an-worker1107 C4 24 WMF5411 42
an-worker1108 C4 36 WMF5412 40
an-worker1109 C7 4 WMF5413 29
an-worker1110 C7 17 WMF5414 39
an-worker1111 C8 12 WMF5415 12
an-worker1112 D2 39 WMF5416 19
an-worker1113 D5 18 WMF5417 16
an-worker1114 D5 20 WMF5418 17
an-worker1115 D7 15 WMF5419 24
an-worker1116 D7 39 WMF5420 23
an-worker1117 B4 38 WMF5421 10

@Marostegui removed bbu. host is booting up now

@Marostegui is host down now? i can remove in about 1 hour

Connected console port to scs-c1-eqiad updated netbox with connection

@CDanis connected console to port 41 on scs-c1-eqiad

@hnowlan
Replaced failed drive

@jcrespo maintenance is completed

@hnowlan replacement drive arrived today can you confirm drive can just be replaced. will take care of tomorrow.

@elukey can you let me know your availability for scheduling this project?

@Marostegui Replacement Dimm has arrived please reach out to me for scheduling down time i am available for the next 2 hours but will be on site tomorrow 9:30am est

finished with bbu swap

@fgiunchedi if you would like bbu swapped i am available tomorrow morning eastern time

Confirmed: Service Request 1030121866 was successfully submitted.

name rack position asset_tag switchport
an-worker1096 A2 39 WMF4839 27
an-worker1097 B4 36 WMF4840 47
an-worker1098 B7 35 WMF4841 39
an-worker1099 C2 32 WMF4842 31
an-worker1100 C4 28 WMF4843 39
an-worker1101 C7 37 WMF4844 37

@wiki_willy @fgiunchedi we do have 3 bbu remaining on site

Thanks willy

updated netbox

@wiki_willy checked storage we do not have any 3tb drives

@akosiaris Finished with upgrade on ganeti1008

@Marostegui Yes all items finished sorry for not commenting. Dell did not come till very late yesterday

starting on pdu upgrade now

@Cmjohnson correct racking
an-test-master1001 A3 25 WMF4836
an-test-master1002 C3 33 WMF4837
an-test-coord1001 D3 18 WMF4838

@Cmjohnson sorry spaced where flipped in netbox, netbox is correct now.
an-test-worker1001 A5 30 WMF4833
an-test-worker1002 C5 34 WMF4834
an-test-worker1003 D6 32 WMF4835

In T255520#6234640, @elukey wrote:

@RobH we ordered 6 nodes IIRC, 3 with more storage space and 3 called "lightweight", which batch is this?

I have in mind the following naming:

• an-test-worker100X for the hosts with more storage
• an-test-master100X for two of the "lightweight" nodes
• an-test-coord1001 for the remaining "lightweight" node

Already updated https://wikitech.wikimedia.org/wiki/Infrastructure_naming_conventions#Servers

replaced sfp on host Finisar model FTLX1471D3BCL S/N AQR0HM6

finished with memory upgrade on ganeti1007

@Marostegui Tsr report showed a few more errors and dell would like to address. what day works best to schedule downtime?

@akosiaris I will be on site tomorrow also if host is available to do 1 day earlier

updated netbox with cable id`s

Relabled host and resolved ticket

@Dzahn sorry that was my mistake. Thanks!

@elukey notebook1004 to an-scheduler1001 i do not see this in netbox. either names.

Confirm
Confirmed: Service Request 1029100504 was successfully submitted.

@Marostegui BBU replaced host is powering up now

@Marostegui yes believe we still have some. I will be on site in a few hours if we wanted to change it today

Server Asset tag Rack Switch port
icinga1001 WMF5405 c6 36

@akosiaris. finished upgrade host is powering up right now

host labeled resolving tast

host rack. switch port asset tag
cloudcephosd1004 C8 22 WMF5103
cloudcephosd1005 C8 23 WMF5104
cloudcephosd1006 C8 24 WMF4831
cloudcephosd1007 C8 25 WMF4830
cloudcephosd1008 C8 26 WMF4829
cloudcephosd1009 C8 27 WMF4828
cloudcephosd1010 D5 6 WMF4827
cloudcephosd1011 D5 7 WMF4826
cloudcephosd1012 D5 8 WMF4825
cloudcephosd1013 D5 9 WMF4824
cloudcephosd1014 D5 10 WMF4823
cloudcephosd1015 D5 11 WMF4822

host rack. Switchport. asset tag
cloudvirt1031 C8 7 WMF4817
cloudvirt1032 C8 8 WMF4816
cloudvirt1033 C8 9 WMF4815
cloudvirt1034 C8 10 WMF4814
cloudvirt1035 C8 11 WMF4813
cloudvirt1036 D5 12 WMF4812
cloudvirt1037 D5 13 WMF4811
cloudvirt1038 D5 14 WMF4810
cloudvirt1039 D5 15 WMF4832

@ayounsi switches are cabled and powered waiting on configuration

added to netbox

added to netbox