Page MenuHomePhabricator
Create Task
Maniphest T187090

Migrate dns caches to stretch
Closed, ResolvedPublic
Actions

Description

Various stretch-specific puppetization and operational issues for role::recursor, which encapsulates recursive DNS + NTP server functionalities have been resolved during the process of installing the new recdns boxes in eqsin and ulsfo (dns[45]00[12]). The new configuration is considerably better and it's time to upgrade the others as we're able

  • eqiad - chromium
  • eqiad - hydrogen
  • codfw - acamar
  • codfw - achernar
  • esams - maerlant
  • esams - nescio

Some of these are also due for hardware refresh soon, so in some cases it may make more sense to do this as part of that refresh (with the new replacement hosts taking on the standard naming scheme dnsS00N).

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
ntp: Cleanup jessie only codeVgutierrezoperations/puppetproduction+2 -50
Revert "Remove chromium from eqiad LVS name server config"Vgutierrezoperations/puppetproduction+1 -0
Remove chromium from eqiad LVS name server configVgutierrezoperations/puppetproduction+0 -1
install_server: Reimage chromium as stretchVgutierrezoperations/puppetproduction+0 -2
Revert "Remove hydrogen from eqiad LVS name server config"Vgutierrezoperations/puppetproduction+1 -0
Remove hydrogen from eqiad LVS name server configVgutierrezoperations/puppetproduction+0 -1
install_server: Reimage hydrogen as stretchVgutierrezoperations/puppetproduction+0 -2
install_server: Reimage acamar as stretchVgutierrezoperations/puppetproduction+0 -2
install_server: Reimage achernar as stretchVgutierrezoperations/puppetproduction+0 -2
install_server: Reimage maerlant.wikimedia.org as stretchVgutierrezoperations/puppetproduction+0 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

BBlack created this task.Feb 12 2018, 4:26 PM
BBlack triaged this task as Medium priority.
Restricted Application added a project: SRE. · View Herald TranscriptFeb 12 2018, 4:26 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
MoritzMuehlenhoff subscribed.Feb 12 2018, 4:44 PM
BBlack moved this task from Backlog to Some old column on the Traffic board.Feb 13 2018, 1:20 PM
• ema subscribed.Apr 13 2018, 7:56 AM
ops-monitoring-bot subscribed.Apr 13 2018, 8:58 AM

Script wmf-auto-reimage was launched by vgutierrez on neodymium.eqiad.wmnet for hosts:

nescio.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201804130858_vgutierrez_27111_nescio_wikimedia_org.log.

ops-monitoring-bot added a comment.Apr 13 2018, 9:29 AM

Completed auto-reimage of hosts:

['nescio.wikimedia.org']

Of which those FAILED:

['nescio.wikimedia.org']
Vgutierrez updated the task description. (Show Details)Apr 13 2018, 10:19 AM
Vgutierrez updated the task description. (Show Details)
gerritbot subscribed.Apr 13 2018, 10:21 AM

Change 425994 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] install_server: Reimage maerlant.wikimedia.org as stretch

https://gerrit.wikimedia.org/r/425994

gerritbot added a project: Patch-For-Review.Apr 13 2018, 10:21 AM
gerritbot added a comment.Apr 13 2018, 10:34 AM

Change 425994 merged by Vgutierrez:
[operations/puppet@production] install_server: Reimage maerlant.wikimedia.org as stretch

https://gerrit.wikimedia.org/r/425994

ops-monitoring-bot added a comment.Apr 13 2018, 10:43 AM

Script wmf-auto-reimage was launched by vgutierrez on neodymium.eqiad.wmnet for hosts:

maerlant.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201804131043_vgutierrez_18025_maerlant_wikimedia_org.log.

ops-monitoring-bot added a comment.Apr 13 2018, 11:14 AM

Completed auto-reimage of hosts:

['maerlant.wikimedia.org']

Of which those FAILED:

['maerlant.wikimedia.org']
Vgutierrez updated the task description. (Show Details)Apr 13 2018, 12:29 PM
gerritbot added a comment.Apr 16 2018, 7:35 AM

Change 426851 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] install_server: Reimage achernar as stretch

https://gerrit.wikimedia.org/r/426851

gerritbot added a comment.Apr 16 2018, 7:37 AM

Change 426851 merged by Vgutierrez:
[operations/puppet@production] install_server: Reimage achernar as stretch

https://gerrit.wikimedia.org/r/426851

Stashbot subscribed.Apr 16 2018, 7:39 AM

Mentioned in SAL (#wikimedia-operations) [2018-04-16T07:39:23Z] <vgutierrez> Depool and reimage achernar.wikimedia.org - T187090

ops-monitoring-bot added a comment.Apr 16 2018, 7:46 AM

Script wmf-auto-reimage was launched by vgutierrez on neodymium.eqiad.wmnet for hosts:

achernar.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201804160746_vgutierrez_21034_achernar_wikimedia_org.log.

ops-monitoring-bot added a comment.Apr 16 2018, 8:24 AM

Completed auto-reimage of hosts:

['achernar.wikimedia.org']

Of which those FAILED:

['achernar.wikimedia.org']
Vgutierrez updated the task description. (Show Details)Apr 16 2018, 9:26 AM
gerritbot added a comment.Apr 16 2018, 9:27 AM

Change 426876 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] install_server: Reimage acamar as stretch

https://gerrit.wikimedia.org/r/426876

gerritbot added a comment.Apr 16 2018, 9:29 AM

Change 426876 merged by Vgutierrez:
[operations/puppet@production] install_server: Reimage acamar as stretch

https://gerrit.wikimedia.org/r/426876

Stashbot added a comment.Apr 16 2018, 9:49 AM

Mentioned in SAL (#wikimedia-operations) [2018-04-16T09:49:54Z] <vgutierrez> Depool and reimage acamar as stretch - T187090

ops-monitoring-bot added a comment.Apr 16 2018, 9:58 AM

Script wmf-auto-reimage was launched by vgutierrez on neodymium.eqiad.wmnet for hosts:

acamar.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201804160957_vgutierrez_17117_acamar_wikimedia_org.log.

ops-monitoring-bot added a comment.Apr 16 2018, 10:36 AM

Completed auto-reimage of hosts:

['acamar.wikimedia.org']

Of which those FAILED:

['acamar.wikimedia.org']
Vgutierrez updated the task description. (Show Details)Apr 16 2018, 11:55 AM
gerritbot added a comment.Apr 16 2018, 12:00 PM

Change 426894 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] install_server: Reimage hydrogen as stretch

https://gerrit.wikimedia.org/r/426894

gerritbot added a comment.Apr 16 2018, 12:00 PM

Change 426895 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] Remove hydrogen from eqiad LVS name server config

https://gerrit.wikimedia.org/r/426895

gerritbot added a comment.Apr 16 2018, 12:02 PM

Change 426894 merged by Vgutierrez:
[operations/puppet@production] install_server: Reimage hydrogen as stretch

https://gerrit.wikimedia.org/r/426894

gerritbot added a comment.Apr 16 2018, 12:02 PM

Change 426895 merged by Vgutierrez:
[operations/puppet@production] Remove hydrogen from eqiad LVS name server config

https://gerrit.wikimedia.org/r/426895

Stashbot added a comment.Apr 16 2018, 12:11 PM

Mentioned in SAL (#wikimedia-operations) [2018-04-16T12:11:39Z] <vgutierrez> Depool and reimage hydrogen as stretch - T187090

ops-monitoring-bot added a comment.Apr 16 2018, 12:17 PM

Script wmf-auto-reimage was launched by vgutierrez on neodymium.eqiad.wmnet for hosts:

hydrogen.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201804161217_vgutierrez_15753_hydrogen_wikimedia_org.log.

ops-monitoring-bot added a comment.Apr 16 2018, 12:47 PM

Completed auto-reimage of hosts:

['hydrogen.wikimedia.org']

Of which those FAILED:

['hydrogen.wikimedia.org']
gerritbot added a comment.Apr 16 2018, 2:59 PM

Change 426940 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] Revert "Remove hydrogen from eqiad LVS name server config"

https://gerrit.wikimedia.org/r/426940

gerritbot added a comment.Apr 16 2018, 3:03 PM

Change 426940 merged by Vgutierrez:
[operations/puppet@production] Revert "Remove hydrogen from eqiad LVS name server config"

https://gerrit.wikimedia.org/r/426940

Vgutierrez updated the task description. (Show Details)Apr 16 2018, 3:04 PM
gerritbot added a comment.Apr 17 2018, 6:35 AM

Change 427061 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] install_server: Reimage chromium as stretch

https://gerrit.wikimedia.org/r/427061

gerritbot added a comment.Apr 17 2018, 6:35 AM

Change 427062 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] Remove chromium from eqiad LVS name server config

https://gerrit.wikimedia.org/r/427062

gerritbot added a comment.Apr 17 2018, 6:40 AM

Change 427061 merged by Vgutierrez:
[operations/puppet@production] install_server: Reimage chromium as stretch

https://gerrit.wikimedia.org/r/427061

gerritbot added a comment.Apr 17 2018, 6:41 AM

Change 427062 merged by Vgutierrez:
[operations/puppet@production] Remove chromium from eqiad LVS name server config

https://gerrit.wikimedia.org/r/427062

Stashbot added a comment.Apr 17 2018, 6:47 AM

Mentioned in SAL (#wikimedia-operations) [2018-04-17T06:47:45Z] <vgutierrez> Depool and reimage chromium as stretch - T187090

ops-monitoring-bot added a comment.Apr 17 2018, 6:54 AM

Script wmf-auto-reimage was launched by vgutierrez on neodymium.eqiad.wmnet for hosts:

chromium.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201804170653_vgutierrez_26909_chromium_wikimedia_org.log.

ops-monitoring-bot added a comment.Apr 17 2018, 7:25 AM

Completed auto-reimage of hosts:

['chromium.wikimedia.org']

Of which those FAILED:

['chromium.wikimedia.org']
gerritbot added a comment.Apr 17 2018, 8:32 AM

Change 427076 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] Revert "Remove chromium from eqiad LVS name server config" chromium has been reimaged successfully.

https://gerrit.wikimedia.org/r/427076

gerritbot added a comment.Apr 17 2018, 8:47 AM

Change 427076 merged by Vgutierrez:
[operations/puppet@production] Revert "Remove chromium from eqiad LVS name server config"

https://gerrit.wikimedia.org/r/427076

Vgutierrez closed this task as Resolved.Apr 17 2018, 8:51 AM
Vgutierrez claimed this task.
Vgutierrez updated the task description. (Show Details)
gerritbot added a comment.Apr 17 2018, 12:39 PM

Change 427101 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ntp: Cleanup jessie only code

https://gerrit.wikimedia.org/r/427101

gerritbot added a comment.Apr 19 2018, 10:15 PM

Change 427101 merged by BBlack:
[operations/puppet@production] ntp: Cleanup jessie only code

https://gerrit.wikimedia.org/r/427101

• Cmjohnson closed subtask T192280: sda failure in hydrogen.wikimedia.org as Resolved.Jan 30 2019, 10:18 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits