Page MenuHomePhabricator

rack and set up analyics1058-1069
Closed, ResolvedPublic

Description

Rack and do initial setup. They will be evenly spreadout across all 4 rows. Current plan is
1 in A1 2 in A3
3 in B8
3 in C2
3 in D2

@Ottomata Please let me know the raid config you need

Details

Related Gerrit Patches:

Event Timeline

Restricted Application added a project: Operations. · View Herald TranscriptApr 5 2017, 12:25 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

RAID config should be identical to other nodes, e.g. analytics1057.

I think /dev/sda is Hardware RAID 1 on the 2 2.5" flex bay drives. The rest 12 drives are JBOD, so you can leave them unpartitioned for the install.

Luca has done some great work to make the partman recipe for these nodes work decently well. See analytics-flex.cfg.

Change 347020 had a related patch set uploaded (by Cmjohnson):
[operations/puppet@production] Adding dhcpd entries for analytics 1058-1068, 1068 mac address is not defined yet. Updating netboot.cfg file for installs T162216

https://gerrit.wikimedia.org/r/347020

Change 347020 merged by Cmjohnson:
[operations/puppet@production] Adding dhcpd entries for analytics 1058-1068, 1068 mac address is not defined yet. Updating netboot.cfg file for installs T162216

https://gerrit.wikimedia.org/r/347020

Change 347042 had a related patch set uploaded (by Cmjohnson):
[operations/dns@master] Adding production dns for analytics1058-68 T162216

https://gerrit.wikimedia.org/r/347042

Change 347042 merged by Cmjohnson:
[operations/dns@master] Adding production dns for analytics1058-68 T162216

https://gerrit.wikimedia.org/r/347042

Change 347398 had a related patch set uploaded (by Cmjohnson):
[operations/puppet@production] Adding mac address that was missing for analytics1068 T162216

https://gerrit.wikimedia.org/r/347398

Change 347398 abandoned by Cmjohnson:
Adding mac address that was missing for analytics1068 T162216

Reason:
Don't know what happened there but I wa about to blow out the entire file. Abandoning this craziness

https://gerrit.wikimedia.org/r/347398

Ok! All but 2 of the nodes are up and running as Hadoop worker nodes.

analytics1064 doesn't seem to be able to contact puppetmaster1001:

puppet agent -t
Error: Could not request certificate: getaddrinfo: Name or service not known
Exiting; failed to retrieve certificate and waitforcert is disabled

analytics1068 got stuck on partitioning during the installer. I was able to manually partition in the installer menus, but I didn't get it done in the exact way that the analytics-flex partman recipe does (no boot partition?). We should probably figure out why partman didn't work, and get it to work and then reinstall this node so that it matches the others.

Ottomata added a parent task: Unknown Object (Task).Apr 10 2017, 8:04 PM
elukey added a subscriber: elukey.Apr 11 2017, 6:54 AM

Change 347577 had a related patch set uploaded (by Elukey):
[operations/dns@master] Correct some typos for analytics10[64,68]

https://gerrit.wikimedia.org/r/347577

Change 347577 merged by Elukey:
[operations/dns@master] Correct some typos for analytics10[64,68]

https://gerrit.wikimedia.org/r/347577

analytics1064 and 1068 should be up and running now!

Cmjohnson changed the task status from Open to Stalled.Apr 20 2017, 1:45 PM

Stalling this until the new servers arrives

Cmjohnson triaged this task as Lowest priority.Apr 20 2017, 1:45 PM
Cmjohnson moved this task from Backlog to Blocked on the ops-eqiad board.Apr 27 2017, 8:27 PM

@Cmjohnson any status update on the 1069 replacement?

Cmjohnson added a subscriber: RobH.May 16 2017, 1:36 PM

@Ottomata I am not sure where we are with status of the replacement server. @RobH may have a better idea

RobH added a comment.May 16 2017, 2:34 PM

The server that dropped? We are still working on resolving the existing one, and getting it gone, and then ordering a new one. The new order will result in its own #procurment task.

Change 357836 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding productin dns for analytics1069 T162216

https://gerrit.wikimedia.org/r/357836

Change 357836 merged by Cmjohnson:
[operations/dns@master] Adding productin dns for analytics1069 T162216

https://gerrit.wikimedia.org/r/357836

Change 357860 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Adding mac addresses to dhcpd file for several systems, wtp1025-1046, stat1005-1006, ganeti1005-1008, labvirt1015-1018, dumpsdata1001-1002, kubestage1001-1002, analytics1069 task #'s T165173 T165366 T166264 T165531 T165368 T165520 T162216 T166076

https://gerrit.wikimedia.org/r/357860

Change 357860 merged by Cmjohnson:
[operations/puppet@production] Adding mac addresses to dhcpd file for several systems, wtp1025-1046, stat1005-1006, ganeti1005-1008, labvirt1015-1018, dumpsdata1001-1002, kubestage1001-1002, analytics1069 task #'s T165173 T165366 T166264 T165531 T165368 T165520 T162216 T166076

https://gerrit.wikimedia.org/r/357860

Change 357870 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding production dns for several new servers, wtp1025-48, ganeti1005-1008, kubestage1001/1002, dumpsdata1001/2, labvirt1015-18 T165173 T166264 T165531 T165520 T162216 T166076

https://gerrit.wikimedia.org/r/357870

Change 357870 merged by Cmjohnson:
[operations/dns@master] Adding production dns for several new servers, wtp1025-48, ganeti1005-1008, kubestage1001/1002, dumpsdata1001/2, labvirt1015-18 and stat1005/6 T165366 T165368 T165173 T166264 T165531 T165520 T162216 T166076

https://gerrit.wikimedia.org/r/357870

Change 357879 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] Revert "Adding mac addresses to dhcpd file for several systems, wtp1025-1046, stat1005-1006, ganeti1005-1008, labvirt1015-1018, dumpsdata1001-1002, kubestage1001-1002, analytics1069 task #'s T165173 T165366 T166264 T165531 T165368 T165520 T162216 T166076"

https://gerrit.wikimedia.org/r/357879

Change 357879 abandoned by RobH:
Revert "Adding mac addresses to dhcpd file for several systems, wtp1025-1046, stat1005-1006, ganeti1005-1008, labvirt1015-1018, dumpsdata1001-1002, kubestage1001-1002, analytics1069 task #'s T165173 T165366 T166264 T165531 T165368 T165520 T162216 T166076"

https://gerrit.wikimedia.org/r/357879

Change 358777 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Adding mac address for analytics1069 T162216

https://gerrit.wikimedia.org/r/358777

Change 358777 merged by Cmjohnson:
[operations/puppet@production] Adding mac address for analytics1069 T162216

https://gerrit.wikimedia.org/r/358777

@elukey @Ottomata analytics1069 is installed, I stopped short of getting puppet running. I wasn't sure if you already had a config for this and did not want to chance breaking anything.

Thanks @Cmjohnson, we'll take it from here then! Appreciated!

@Cmjohnson, I was about to do this, but I need to know which row and rack it is in. I can see its in Row D, but which rack?

Ottomata closed this task as Resolved.Jun 14 2017, 6:40 PM