Page MenuHomePhabricator

Reimage wezen to Buster (and rename to centrallog2001)
Closed, ResolvedPublic

Description

wezen is currently running Jessie and must be reimaged to Stretch or Buster

Steps for implementation

  • setup rsync::quickdatacopy between centrallog hosts in eqiad/codfw
  • drain traffic from wezen (i.e. remove from remote_syslog_tls list in puppet)
  • reimage and rename to centrallog2001
  • validate log receiving works
  • add centrallog2001 back to remote_syslog_tls in puppet
  • transfer missing logs from centrallog1001 onto centrallog2001
  • remove rsync::quickdatacopy once done

Event Timeline

ArielGlenn triaged this task as Medium priority.Jun 11 2019, 7:57 AM
fgiunchedi renamed this task from Reimage wezen to Stretch (and rename to centrallog2001) to Reimage wezen to Stretch or Buster (and rename to centrallog2001).Oct 17 2019, 10:10 AM
fgiunchedi added a project: observability.
fgiunchedi added a project: User-fgiunchedi.
fgiunchedi updated the task description. (Show Details)
fgiunchedi moved this task from Backlog to Up next on the User-fgiunchedi board.Oct 17 2019, 3:09 PM
fgiunchedi moved this task from Inbox to Up next on the observability board.Oct 28 2019, 2:17 PM

Change 547241 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/dns@master] Rename wezen to centrallog2001

https://gerrit.wikimedia.org/r/547241

Change 547245 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] rsyslog: setup temporary rsync for logs transfer

https://gerrit.wikimedia.org/r/547245

Change 547246 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: remove wezen from service

https://gerrit.wikimedia.org/r/547246

Change 547247 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] Rename wezen to centrallog2001

https://gerrit.wikimedia.org/r/547247

Change 547248 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: pool centrallog2001 in eqsin

https://gerrit.wikimedia.org/r/547248

Change 547246 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: remove wezen from service

https://gerrit.wikimedia.org/r/547246

Change 547247 merged by Filippo Giunchedi:
[operations/puppet@production] Rename wezen to centrallog2001

https://gerrit.wikimedia.org/r/547247

Change 547241 merged by Filippo Giunchedi:
[operations/dns@master] Rename wezen to centrallog2001

https://gerrit.wikimedia.org/r/547241

Change 551518 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/dns@master] wmnet: rename wezen mgmt to centrallog2001

https://gerrit.wikimedia.org/r/551518

Change 551518 merged by Filippo Giunchedi:
[operations/dns@master] wmnet: rename wezen mgmt to centrallog2001

https://gerrit.wikimedia.org/r/551518

Script wmf-auto-reimage was launched by filippo on cumin1001.eqiad.wmnet for hosts:

wezen.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201911181201_filippo_12902_wezen_codfw_wmnet.log.

Completed auto-reimage of hosts:

['wezen.codfw.wmnet']

Of which those FAILED:

['wezen.codfw.wmnet']

Change 551522 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: cleanup wezen

https://gerrit.wikimedia.org/r/551522

Change 551522 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: cleanup wezen

https://gerrit.wikimedia.org/r/551522

Change 551547 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] install_server: add raid1+lvm standard recipe for GPT

https://gerrit.wikimedia.org/r/551547

Change 551548 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] install_server: use raid1-gpt-lvm-ext4-srv for centrallog

https://gerrit.wikimedia.org/r/551548

Change 551547 merged by Filippo Giunchedi:
[operations/puppet@production] install_server: add raid1+lvm standard recipe for GPT

https://gerrit.wikimedia.org/r/551547

Change 551548 merged by Filippo Giunchedi:
[operations/puppet@production] install_server: use raid1-gpt-lvm-ext4-srv for centrallog

https://gerrit.wikimedia.org/r/551548

fgiunchedi updated the task description. (Show Details)Nov 19 2019, 9:09 AM

Change 547248 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: pool centrallog2001 in eqsin

https://gerrit.wikimedia.org/r/547248

Change 551779 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: pool centrallog2001

https://gerrit.wikimedia.org/r/551779

Change 551779 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: pool centrallog2001

https://gerrit.wikimedia.org/r/551779

fgiunchedi updated the task description. (Show Details)Nov 19 2019, 10:16 AM
fgiunchedi moved this task from Up next to Doing on the User-fgiunchedi board.Nov 20 2019, 12:33 PM

Change 547245 merged by Filippo Giunchedi:
[operations/puppet@production] rsyslog: setup temporary secure rsync for logs transfer

https://gerrit.wikimedia.org/r/547245

fgiunchedi updated the task description. (Show Details)Nov 20 2019, 1:08 PM

Change 552208 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] centrallog: remove quickdatacopy after sync

https://gerrit.wikimedia.org/r/552208

Change 552208 merged by Filippo Giunchedi:
[operations/puppet@production] centrallog: remove quickdatacopy after sync

https://gerrit.wikimedia.org/r/552208

fgiunchedi closed this task as Resolved.Nov 21 2019, 9:13 AM
fgiunchedi updated the task description. (Show Details)

This is complete!

Volans added a subscriber: Volans.Dec 2 2019, 11:20 AM

I've updated the mgmt DNS name record in Netbox that was still reporting wezen. I've also a patch to cleanup the wezen record from DNS, will push it later today.

Change 554083 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] codfw: remove old wezen record

https://gerrit.wikimedia.org/r/554083

Change 554083 merged by Filippo Giunchedi:
[operations/dns@master] codfw: remove old wezen record

https://gerrit.wikimedia.org/r/554083

MoritzMuehlenhoff renamed this task from Reimage wezen to Stretch or Buster (and rename to centrallog2001) to Reimage wezen to Buster (and rename to centrallog2001).Dec 3 2019, 10:12 AM