Page MenuHomePhabricator
Create Task
Maniphest T201341

rack/setup/install cloudservices1004.wikimedia.org
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and installation of labservices1004.wikimedia.org. These host was purchased as 1 'spare' system on T195418 and then allocated on T199727.

Racking Proposal: This host will use public IP address/vlan, so it can be placed in any row. It should be in a different row than wmf7225 (which is in B8).

labservices1004:

  • - receive in system on procurement task T195418
  • - rack system with proposed racking plan (see above) & update racktables (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run
  • - handoff for service implementation

Details

SubjectRepoBranchLines +/-
Fix ipv6 ptr record for cloudservices1004.wikimedia.orgoperations/dnsmaster+2 -1
eqiad1: add entries for cloudservices1004operations/puppetproduction+3 -3
Add DNS entries for labs-ns3 and labs-recursor3operations/dnsmaster+3 -0
cloudservices1004 mac address updateoperations/puppetproduction+7 -0
cloudservices1004 production dns entriesoperations/dnsmaster+5 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH triaged this task as Medium priority.Aug 6 2018, 6:23 PM
RobH created this task.
RobH added a parent task: Unknown Object (Task).
RobH mentioned this in Unknown Object (Task).
• Cmjohnson moved this task from Backlog to Racking Tasks on the ops-eqiad board.Aug 7 2018, 2:02 PM
Dzahn unsubscribed.Aug 7 2018, 3:22 PM
Andrew renamed this task from rack/setup/install labservices100[34].wikimedia.org to rack/setup/install cloudservices1004.wikimedia.org.Aug 7 2018, 5:21 PM
Andrew subscribed.

Thanks to T201439 I've just reduced the ask on this ticket from 2 servers to 1: cloudservices1004.

RobH updated the task description. (Show Details)Aug 7 2018, 5:26 PM
• Cmjohnson updated the task description. (Show Details)Aug 7 2018, 5:38 PM
• Cmjohnson updated the task description. (Show Details)Aug 15 2018, 3:29 PM
Andrew added a comment.Aug 24 2018, 2:28 PM

*bump* -- I'm happy to do the OS install &c. if that helps move this along. Thanks!

RobH added a comment.Aug 29 2018, 4:53 PM

So, this system is plugged into asw2-a-eqiad, which had deployment issues. @Cmjohnson will need to move this to asw-a-eqiad and update this task with the port. (It shows as allocated on ge-3/0/32 down down allocated to cloudservices1004 asw2)

So once this network port is migrated from asw2-a-eqiad to asw-a-eqiad, I can take it over from Chris.

gerritbot subscribed.Aug 29 2018, 5:09 PM

Change 456179 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] cloudservices1004 production dns entries

https://gerrit.wikimedia.org/r/456179

gerritbot added a project: Patch-For-Review.Aug 29 2018, 5:09 PM

Change 456179 merged by RobH:
[operations/dns@master] cloudservices1004 production dns entries

https://gerrit.wikimedia.org/r/456179

RobH claimed this task.Aug 29 2018, 5:11 PM
RobH removed a project: Patch-For-Review.
RobH updated the task description. (Show Details)
gerritbot added a comment.Aug 29 2018, 5:32 PM

Change 456183 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] cloudservices1004 mac address update

https://gerrit.wikimedia.org/r/456183

gerritbot added a project: Patch-For-Review.Aug 29 2018, 5:32 PM

Change 456183 merged by RobH:
[operations/puppet@production] cloudservices1004 mac address update

https://gerrit.wikimedia.org/r/456183

RobH added a comment.Aug 29 2018, 6:27 PM

Ok, firmware updated on the bios and the network card, as they were outdated. Other firmware versions are up to date (according to support.dell.com for the system service tag.)

OS installed and puppet's initial run is in progress. Once it is done, I'll assign this over to @Andrew for followup/implementation.

RobH removed a project: Patch-For-Review.Aug 29 2018, 6:28 PM
RobH updated the task description. (Show Details)
RobH reassigned this task from RobH to Andrew.Aug 29 2018, 6:31 PM
RobH removed a project: ops-eqiad.

Ok, the first puppet run fails due to:

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, 208.80.154.11 is not valid for profile::openstack::eqiad1::pdns::auth::service at /etc/puppet/modules/profile/manifests/openstack/eqiad1/pdns/auth/service.pp:33:5 on node cloudservices1004.wikimedia.org

I've already synced with @Andrew via irc (he is going to eat lunch, and then tackle this.) He is aware if we cannot get this fixed within an hour or so we'll need to power it off, or disable switch port, or reimage with role spare, so its just not a non puppet server sitting out there.

Dzahn mentioned this in T202100: Intermittent git-fat failure during deploy.Aug 29 2018, 6:39 PM
gerritbot added a comment.Aug 29 2018, 7:29 PM

Change 456207 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/dns@master] Add DNS entries for labs-ns2 and labs-recursor2

https://gerrit.wikimedia.org/r/456207

gerritbot added a project: Patch-For-Review.Aug 29 2018, 7:29 PM

Change 456208 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] eqiad1: add entries for cloudservices1004

https://gerrit.wikimedia.org/r/456208

gerritbot added a comment.Aug 29 2018, 7:33 PM

Change 456207 merged by Andrew Bogott:
[operations/dns@master] Add DNS entries for labs-ns3 and labs-recursor3

https://gerrit.wikimedia.org/r/456207

gerritbot added a comment.Aug 29 2018, 7:34 PM

Change 456208 merged by Andrew Bogott:
[operations/puppet@production] eqiad1: add entries for cloudservices1004

https://gerrit.wikimedia.org/r/456208

Andrew closed this task as Resolved.Aug 29 2018, 7:59 PM

puppet is running now. Thank you!

gerritbot added a comment.Aug 29 2018, 10:09 PM

Change 456293 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/dns@master] Fix ipv6 ptr record for cloudservices1004.wikimedia.org

https://gerrit.wikimedia.org/r/456293

gerritbot added a comment.Aug 29 2018, 10:09 PM

Change 456293 merged by Andrew Bogott:
[operations/dns@master] Fix ipv6 ptr record for cloudservices1004.wikimedia.org

https://gerrit.wikimedia.org/r/456293

RobH unsubscribed.Aug 30 2018, 4:28 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL