Page MenuHomePhabricator

labtestnet2003.codfw.wmnet: rename to cloudnet2003-dev.codfw.wmnet and reimage to stretch
Closed, ResolvedPublic

Description

The server was running Trusty. Reimage to stretch and rename to the new naming scheme.

  • disable puppet in labtestnet2003
  • merge puppet patch to rename and get the new debian installer working
  • merge dns patch to add the new FQDNs (partial, the old mgmt names still remains)
  • run the wmf-auto-reimage-host script
  • merge DNS cleanup patch
  • netbox update https://netbox.wikimedia.org/dcim/devices/1686/
  • get the physical relabeling done, switch port labels, etc (T219861)
  • done

Event Timeline

aborrero updated the task description. (Show Details)Apr 1 2019, 11:29 AM

Change 500423 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] labtestnet2003: rename to cloudnet2003-dev and reimage to stretch

https://gerrit.wikimedia.org/r/500423

Change 500423 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] labtestnet2003: rename to cloudnet2003-dev and reimage to stretch

https://gerrit.wikimedia.org/r/500423

aborrero updated the task description. (Show Details)Apr 1 2019, 11:39 AM

Change 500424 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/dns@master] wmnet: introduce cloudnet2003-dev.codfw.wmnet FQDNs

https://gerrit.wikimedia.org/r/500424

Change 500424 merged by Arturo Borrero Gonzalez:
[operations/dns@master] wmnet: introduce cloudnet2003-dev.codfw.wmnet FQDNs

https://gerrit.wikimedia.org/r/500424

aborrero updated the task description. (Show Details)Apr 1 2019, 11:47 AM

Change 500430 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/dns@master] wmnet: introduce cloudnet2003-dev.codfw.wmnet FQDNs

https://gerrit.wikimedia.org/r/500430

Change 500430 merged by Arturo Borrero Gonzalez:
[operations/dns@master] wmnet: introduce cloudnet2003-dev.codfw.wmnet FQDNs

https://gerrit.wikimedia.org/r/500430

Script wmf-auto-reimage was launched by aborrero on cumin1001.eqiad.wmnet for hosts:

labtesnet2003.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201904011556_aborrero_203001_labtesnet2003_codfw_wmnet.log.

Completed auto-reimage of hosts:

['labtesnet2003.codfw.wmnet']

Of which those FAILED:

['labtesnet2003.codfw.wmnet']

Script wmf-auto-reimage was launched by aborrero on cumin1001.eqiad.wmnet for hosts:

cloudnet2003-dev.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201904011556_aborrero_203186_cloudnet2003-dev_codfw_wmnet.log.

Completed auto-reimage of hosts:

['cloudnet2003-dev.codfw.wmnet']

Of which those FAILED:

['cloudnet2003-dev.codfw.wmnet']

Script wmf-auto-reimage was launched by aborrero on cumin1001.eqiad.wmnet for hosts:

labtesnet2003.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201904011557_aborrero_203277_labtesnet2003_codfw_wmnet.log.

Completed auto-reimage of hosts:

['labtesnet2003.codfw.wmnet']

Of which those FAILED:

['labtesnet2003.codfw.wmnet']

Script wmf-auto-reimage was launched by aborrero on cumin1001.eqiad.wmnet for hosts:

cloudnet2003-dev.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201904011557_aborrero_203487_cloudnet2003-dev_codfw_wmnet.log.

Mentioned in SAL (#wikimedia-operations) [2019-04-01T16:15:45Z] <arturo> T219776 reimaging + renaming labtestnet2003 into cloudnet2003-dev

Change 500483 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/dns@master] labtestnet2003: cleanup old FQDNs

https://gerrit.wikimedia.org/r/500483

Completed auto-reimage of hosts:

['cloudnet2003-dev.codfw.wmnet']

Of which those FAILED:

['cloudnet2003-dev.codfw.wmnet']

Script wmf-auto-reimage was launched by aborrero on cumin1001.eqiad.wmnet for hosts:

cloudnet2003-dev.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201904011708_aborrero_218195_cloudnet2003-dev_codfw_wmnet.log.

Completed auto-reimage of hosts:

['cloudnet2003-dev.codfw.wmnet']

Of which those FAILED:

['cloudnet2003-dev.codfw.wmnet']

Change 500483 merged by Arturo Borrero Gonzalez:
[operations/dns@master] labtestnet2003: cleanup old FQDNs

https://gerrit.wikimedia.org/r/500483

aborrero updated the task description. (Show Details)Apr 1 2019, 5:30 PM
aborrero added projects: ops-codfw, DC-Ops.
Restricted Application added a project: Operations. · View Herald TranscriptApr 1 2019, 5:31 PM

Script wmf-auto-reimage was launched by aborrero on cumin1001.eqiad.wmnet for hosts:

cloudnet2003-dev.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201904011732_aborrero_222393_cloudnet2003-dev_codfw_wmnet.log.

Change 500500 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/dns@master] wmnet: fix typo in cloudnet2003-dev FQDN

https://gerrit.wikimedia.org/r/500500

Change 500500 merged by Arturo Borrero Gonzalez:
[operations/dns@master] wmnet: fix typo in cloudnet2003-dev FQDN

https://gerrit.wikimedia.org/r/500500

Completed auto-reimage of hosts:

['cloudnet2003-dev.codfw.wmnet']

Of which those FAILED:

['cloudnet2003-dev.codfw.wmnet']

Script wmf-auto-reimage was launched by aborrero on cumin1001.eqiad.wmnet for hosts:

cloudnet2003-dev.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201904011743_aborrero_225254_cloudnet2003-dev_codfw_wmnet.log.

Completed auto-reimage of hosts:

['cloudnet2003-dev.codfw.wmnet']

Of which those FAILED:

['cloudnet2003-dev.codfw.wmnet']

Script wmf-auto-reimage was launched by aborrero on cumin1001.eqiad.wmnet for hosts:

cloudnet2003-dev.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201904020910_aborrero_17135_cloudnet2003-dev_codfw_wmnet.log.

Mentioned in SAL (#wikimedia-operations) [2019-04-02T09:14:36Z] <arturo> T219776 finally reimaging cloudnet2003-dev.codfw.wmnet (was labtestnet2003)

Change 500679 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] install_server: fix typo in partman recipe selector for cloudnet2003-dev

https://gerrit.wikimedia.org/r/500679

Change 500679 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] install_server: fix typo in partman recipe selector for cloudnet2003-dev

https://gerrit.wikimedia.org/r/500679

Completed auto-reimage of hosts:

['cloudnet2003-dev.codfw.wmnet']

Of which those FAILED:

['cloudnet2003-dev.codfw.wmnet']

Script wmf-auto-reimage was launched by aborrero on cumin1001.eqiad.wmnet for hosts:

cloudnet2003-dev.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201904020923_aborrero_39144_cloudnet2003-dev_codfw_wmnet.log.

aborrero updated the task description. (Show Details)Apr 2 2019, 9:46 AM
aborrero updated the task description. (Show Details)

Completed auto-reimage of hosts:

['cloudnet2003-dev.codfw.wmnet']

and were ALL successful.

aborrero closed this task as Resolved.Apr 2 2019, 12:49 PM
aborrero triaged this task as Normal priority.
aborrero updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2019-04-03T09:45:36Z] <moritzm> removed labtestnet2003.codfw.wmnet from debmonitor (T219776)

MoritzMuehlenhoff reopened this task as Open.Apr 8 2019, 7:33 AM

labtestnet2003 is still in puppetdb:

jmm@cumin2001:~$ sudo cumin labtestnet2003*
1 hosts will be targeted:
labtestnet2003.codfw.wmnet
DRY-RUN mode enabled, aborting

Mentioned in SAL (#wikimedia-operations) [2019-04-08T09:30:38Z] <arturo> T219776 puppet node clean labtestnet2003.codfw.wmnet

Not sure if I should drop the concrete registers in the puppetdb, or if cleaning the node is enough.

Dzahn added a comment.Apr 8 2019, 10:27 AM

I think you need both "node clean" and "node deactivate".

Mentioned in SAL (#wikimedia-operations) [2019-04-08T12:03:04Z] <arturo> T219776 puppet node deactivate labtestnet2003.codfw.wmnet

aborrero closed this task as Resolved.Apr 8 2019, 1:02 PM
aborrero@cumin1001:~ $ sudo cumin labtestnet2003*
No hosts found that matches the query

Created some docs about this: https://wikitech.wikimedia.org/wiki/Puppet#node_cleanup