Page MenuHomePhabricator
Create Task
Maniphest T196691

rack/setup/install dns100[12].wikimedia.org
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and installation of 2 new single cpu misc class hosts for dns recursor role. These will replace the use of chromium and hydrogen.

Racking Proposal: Ensure dns1001 and dns1002 do not share the same rack or row.

dns1001:

  • - receive in system on procurement task T195417
  • - rack system with proposed racking plan (see above) & update racktables (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run
  • - handoff for service implementation

dns1002:

  • - receive in system on procurement task T195417
  • - rack system with proposed racking plan (see above) & update racktables (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run
  • - handoff for service implementation

Details

SubjectRepoBranchLines +/-
conftool-data: Remove chromium & hydrogen from pdns_recursor serviceoperations/puppetproduction+0 -2
standard: Remove chromium and hydrogen from ntp peer listoperations/puppetproduction+0 -2
hieradata: Get rid of hydrogen and chromium referencesoperations/puppetproduction+2 -2
lvs: use the new dns100[12] recursive DNS serversoperations/puppetproduction+3 -3
smokeping: Replace hydrogen & chromium with dns1001 and dns1002operations/puppetproduction+8 -8
conftool-data: Add dns100[12] to pdns_recursor serviceoperations/puppetproduction+2 -0
standard: add dns100[12] to eqiad ntp peer listoperations/puppetproduction+2 -0
setting dns100[12] install paramsoperations/puppetproduction+17 -1
Adding dns1001/1002 mgmt/production dnsoperations/dnsmaster+46 -32
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
Unknown Object (Task)
 ResolvedVgutierrezT196691 rack/setup/install dns100[12].wikimedia.org
 ResolvedayounsiT201414 Use dns100[12] as ntp servers in eqiad networking equipment

Event Timeline

RobH triaged this task as Medium priority.Jun 7 2018, 8:32 PM
RobH created this task.
Restricted Application added a project: Traffic. · View Herald TranscriptJun 7 2018, 8:32 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
• Cmjohnson moved this task from Backlog to Up next on the ops-eqiad board.Jun 11 2018, 3:38 PM
BBlack moved this task from Backlog to Hardware on the Traffic board.Jun 11 2018, 5:35 PM
• Cmjohnson moved this task from Up next to Racking Tasks on the ops-eqiad board.Jun 26 2018, 3:57 PM
• Vvjjkkii renamed this task from rack/setup/install dns100[12].wikimedia.org to ofbaaaaaaa.Jul 1 2018, 1:05 AM
• Vvjjkkii removed • Cmjohnson as the assignee of this task.
• Vvjjkkii raised the priority of this task from Medium to High.
• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).
• Vvjjkkii updated the task description. (Show Details)
• Vvjjkkii removed a subscriber: Aklapper.
• ema renamed this task from ofbaaaaaaa to rack/setup/install dns100[12].wikimedia.org.Jul 2 2018, 9:33 AM
• ema assigned this task to • Cmjohnson.
• ema updated the task description. (Show Details)
• ema removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.Jul 2 2018, 9:36 AM
• ema lowered the priority of this task from High to Medium.Jul 2 2018, 9:58 AM
• Cmjohnson updated the task description. (Show Details)Jul 3 2018, 1:24 PM
gerritbot subscribed.Aug 1 2018, 1:10 PM

Change 449715 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding dns1001/1002 mgmt/production dns

https://gerrit.wikimedia.org/r/449715

gerritbot added a project: Patch-For-Review.Aug 1 2018, 1:10 PM
gerritbot added a comment.Aug 1 2018, 1:32 PM

Change 449715 merged by Cmjohnson:
[operations/dns@master] Adding dns1001/1002 mgmt/production dns

https://gerrit.wikimedia.org/r/449715

• Cmjohnson updated the task description. (Show Details)Aug 1 2018, 1:33 PM
• Cmjohnson reassigned this task from • Cmjohnson to RobH.Aug 2 2018, 7:56 PM
• Cmjohnson updated the task description. (Show Details)
• Cmjohnson moved this task from Racking Tasks to Blocked on the ops-eqiad board.

assigning to @RobH to help complete the install

gerritbot added a comment.Aug 3 2018, 10:08 PM

Change 450286 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] setting dns100[12] install params

https://gerrit.wikimedia.org/r/450286

gerritbot added a comment.Aug 3 2018, 10:09 PM

Change 450286 merged by RobH:
[operations/puppet@production] setting dns100[12] install params

https://gerrit.wikimedia.org/r/450286

RobH removed a project: Patch-For-Review.Aug 3 2018, 10:16 PM
RobH updated the task description. (Show Details)
RobH reassigned this task from RobH to BBlack.Aug 3 2018, 10:39 PM

So these two systems fail their puppet runs, but fail for the following:

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Resource Statement, Duplicate declaration: Package[ntp] is already declared in file /etc/puppet/modules/standard/manifests/ntp/timesyncd.pp:8; cannot redeclare at /etc/puppet/modules/ntp/manifests/daemon.pp:3 at /etc/puppet/modules/ntp/manifests/daemon.pp:3:5 at /etc/puppet/modules/profile/manifests/ntp.pp:80 on node dns1001.wikimedia.org

RobH removed a project: ops-eqiad.Aug 3 2018, 10:39 PM
Vgutierrez subscribed.Aug 6 2018, 4:50 PM

I'll handle this from here @RobH, thanks :)

Vgutierrez claimed this task.Aug 6 2018, 4:51 PM
Vgutierrez added a subscriber: BBlack.
gerritbot added a comment.Aug 7 2018, 8:30 AM

Change 450922 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] standard: add dns100[12] to eqiad ntp peer list

https://gerrit.wikimedia.org/r/450922

gerritbot added a project: Patch-For-Review.Aug 7 2018, 8:30 AM
gerritbot added a comment.Aug 7 2018, 9:35 AM

Change 450922 merged by Vgutierrez:
[operations/puppet@production] standard: add dns100[12] to eqiad ntp peer list

https://gerrit.wikimedia.org/r/450922

ops-monitoring-bot subscribed.Aug 7 2018, 9:38 AM

Script wmf-auto-reimage was launched by vgutierrez on neodymium.eqiad.wmnet for hosts:

dns1001.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201808070938_vgutierrez_8140_dns1001_wikimedia_org.log.

ops-monitoring-bot added a comment.Aug 7 2018, 9:38 AM

Completed auto-reimage of hosts:

['dns1001.wikimedia.org']

Of which those FAILED:

['dns1001.wikimedia.org']
ops-monitoring-bot added a comment.Aug 7 2018, 9:38 AM

Script wmf-auto-reimage was launched by vgutierrez on neodymium.eqiad.wmnet for hosts:

dns1001.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201808070938_vgutierrez_8288_dns1001_wikimedia_org.log.

ops-monitoring-bot added a comment.Aug 7 2018, 9:40 AM

Script wmf-auto-reimage was launched by vgutierrez on neodymium.eqiad.wmnet for hosts:

dns1002.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201808070939_vgutierrez_8501_dns1002_wikimedia_org.log.

ops-monitoring-bot added a comment.Aug 7 2018, 10:08 AM

Completed auto-reimage of hosts:

['dns1001.wikimedia.org']

and were ALL successful.

ops-monitoring-bot added a comment.Aug 7 2018, 10:09 AM

Completed auto-reimage of hosts:

['dns1002.wikimedia.org']

and were ALL successful.

gerritbot added a comment.Aug 7 2018, 10:37 AM

Change 450940 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] conftool-data: Add dns100[12] to pdns_recursor service

https://gerrit.wikimedia.org/r/450940

Vgutierrez updated the task description. (Show Details)Aug 7 2018, 10:38 AM
gerritbot added a comment.Aug 7 2018, 12:38 PM

Change 450940 merged by Vgutierrez:
[operations/puppet@production] conftool-data: Add dns100[12] to pdns_recursor service

https://gerrit.wikimedia.org/r/450940

gerritbot added a comment.Aug 7 2018, 1:11 PM

Change 450977 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] smokeping: Replace hydrogen & chromium with dns1001 and dns1002

https://gerrit.wikimedia.org/r/450977

gerritbot added a comment.Aug 7 2018, 1:32 PM

Change 450982 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] lvs: use the new dns100[12] recursive DNS servers

https://gerrit.wikimedia.org/r/450982

gerritbot added a comment.Aug 7 2018, 1:33 PM

Change 450977 merged by Vgutierrez:
[operations/puppet@production] smokeping: Replace hydrogen & chromium with dns1001 and dns1002

https://gerrit.wikimedia.org/r/450977

gerritbot added a comment.Aug 7 2018, 1:36 PM

Change 450982 merged by Vgutierrez:
[operations/puppet@production] lvs: use the new dns100[12] recursive DNS servers

https://gerrit.wikimedia.org/r/450982

gerritbot added a comment.Aug 7 2018, 1:42 PM

Change 450984 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] hieradata: Get rid of hydrogen and chromium references

https://gerrit.wikimedia.org/r/450984

gerritbot added a comment.Aug 7 2018, 1:50 PM

Change 450986 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] standard: Remove chromium and hydrogen from ntp peer list

https://gerrit.wikimedia.org/r/450986

gerritbot added a comment.Aug 7 2018, 1:51 PM

Change 450984 merged by Vgutierrez:
[operations/puppet@production] hieradata: Get rid of hydrogen and chromium references

https://gerrit.wikimedia.org/r/450984

gerritbot added a comment.Aug 7 2018, 2:54 PM

Change 450986 merged by Vgutierrez:
[operations/puppet@production] standard: Remove chromium and hydrogen from ntp peer list

https://gerrit.wikimedia.org/r/450986

gerritbot added a comment.Aug 7 2018, 3:01 PM

Change 451002 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] conftool-data: Remove chromium & hydrogen from pdns_recursor service

https://gerrit.wikimedia.org/r/451002

gerritbot added a comment.Aug 7 2018, 3:06 PM

Change 451002 merged by Vgutierrez:
[operations/puppet@production] conftool-data: Remove chromium & hydrogen from pdns_recursor service

https://gerrit.wikimedia.org/r/451002

Vgutierrez closed subtask T201414: Use dns100[12] as ntp servers in eqiad networking equipment as Resolved.Aug 8 2018, 3:20 PM
Vgutierrez mentioned this in T201522: Decommission chromium and hydrogen.Aug 8 2018, 3:40 PM
Vgutierrez closed this task as Resolved.Sep 27 2018, 3:03 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL