Page MenuHomePhabricator
Create Task
Maniphest T224564

Reimage wezen to Buster (and rename to centrallog2001)
Closed, ResolvedPublic
Actions

Description

wezen is currently running Jessie and must be reimaged to Stretch or Buster

Steps for implementation

  • setup rsync::quickdatacopy between centrallog hosts in eqiad/codfw
  • drain traffic from wezen (i.e. remove from remote_syslog_tls list in puppet)
  • reimage and rename to centrallog2001
  • validate log receiving works
  • add centrallog2001 back to remote_syslog_tls in puppet
  • transfer missing logs from centrallog1001 onto centrallog2001
  • remove rsync::quickdatacopy once done

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

MoritzMuehlenhoff created this task.May 29 2019, 11:43 AM
MoritzMuehlenhoff mentioned this in T224549: Track remaining jessie systems in production.
ArielGlenn triaged this task as Medium priority.Jun 11 2019, 7:57 AM
fgiunchedi mentioned this in T225604: log spam from mtail 3.0.0~rc19 on wezen.Jun 12 2019, 11:04 AM
fgiunchedi renamed this task from Reimage wezen to Stretch (and rename to centrallog2001) to Reimage wezen to Stretch or Buster (and rename to centrallog2001).Oct 17 2019, 10:10 AM
fgiunchedi added a project: observability.
fgiunchedi added a project: User-fgiunchedi.
fgiunchedi claimed this task.Oct 17 2019, 10:16 AM
fgiunchedi updated the task description. (Show Details)
fgiunchedi moved this task from Backlog to Up next on the User-fgiunchedi board.Oct 17 2019, 3:09 PM
fgiunchedi moved this task from Inbox to Up next on the observability board.Oct 28 2019, 2:17 PM
gerritbot added a comment.Oct 30 2019, 3:30 PM

Change 547241 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/dns@master] Rename wezen to centrallog2001

https://gerrit.wikimedia.org/r/547241

gerritbot added a project: Patch-For-Review.Oct 30 2019, 3:30 PM
gerritbot added a comment.Oct 30 2019, 3:33 PM

Change 547245 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] rsyslog: setup temporary rsync for logs transfer

https://gerrit.wikimedia.org/r/547245

gerritbot added a comment.Oct 30 2019, 3:33 PM

Change 547246 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: remove wezen from service

https://gerrit.wikimedia.org/r/547246

gerritbot added a comment.Oct 30 2019, 3:33 PM

Change 547247 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] Rename wezen to centrallog2001

https://gerrit.wikimedia.org/r/547247

gerritbot added a comment.Oct 30 2019, 3:33 PM

Change 547248 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: pool centrallog2001 in eqsin

https://gerrit.wikimedia.org/r/547248

gerritbot added a comment.Nov 18 2019, 9:32 AM

Change 547246 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: remove wezen from service

https://gerrit.wikimedia.org/r/547246

gerritbot added a comment.Nov 18 2019, 11:39 AM

Change 547247 merged by Filippo Giunchedi:
[operations/puppet@production] Rename wezen to centrallog2001

https://gerrit.wikimedia.org/r/547247

gerritbot added a comment.Nov 18 2019, 11:48 AM

Change 547241 merged by Filippo Giunchedi:
[operations/dns@master] Rename wezen to centrallog2001

https://gerrit.wikimedia.org/r/547241

gerritbot added a comment.Nov 18 2019, 11:52 AM

Change 551518 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/dns@master] wmnet: rename wezen mgmt to centrallog2001

https://gerrit.wikimedia.org/r/551518

gerritbot added a comment.Nov 18 2019, 11:52 AM

Change 551518 merged by Filippo Giunchedi:
[operations/dns@master] wmnet: rename wezen mgmt to centrallog2001

https://gerrit.wikimedia.org/r/551518

ops-monitoring-bot added a comment.Nov 18 2019, 12:02 PM

Script wmf-auto-reimage was launched by filippo on cumin1001.eqiad.wmnet for hosts:

wezen.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201911181201_filippo_12902_wezen_codfw_wmnet.log.

ops-monitoring-bot added a comment.Nov 18 2019, 12:02 PM

Completed auto-reimage of hosts:

['wezen.codfw.wmnet']

Of which those FAILED:

['wezen.codfw.wmnet']
gerritbot added a comment.Nov 18 2019, 12:18 PM

Change 551522 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: cleanup wezen

https://gerrit.wikimedia.org/r/551522

gerritbot added a comment.Nov 18 2019, 12:22 PM

Change 551522 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: cleanup wezen

https://gerrit.wikimedia.org/r/551522

gerritbot added a comment.Nov 18 2019, 2:39 PM

Change 551547 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] install_server: add raid1+lvm standard recipe for GPT

https://gerrit.wikimedia.org/r/551547

gerritbot added a comment.Nov 18 2019, 2:39 PM

Change 551548 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] install_server: use raid1-gpt-lvm-ext4-srv for centrallog

https://gerrit.wikimedia.org/r/551548

gerritbot added a comment.Nov 18 2019, 3:21 PM

Change 551547 merged by Filippo Giunchedi:
[operations/puppet@production] install_server: add raid1+lvm standard recipe for GPT

https://gerrit.wikimedia.org/r/551547

gerritbot added a comment.Nov 18 2019, 3:22 PM

Change 551548 merged by Filippo Giunchedi:
[operations/puppet@production] install_server: use raid1-gpt-lvm-ext4-srv for centrallog

https://gerrit.wikimedia.org/r/551548

fgiunchedi updated the task description. (Show Details)Nov 19 2019, 9:09 AM
gerritbot added a comment.Nov 19 2019, 9:15 AM

Change 547248 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: pool centrallog2001 in eqsin

https://gerrit.wikimedia.org/r/547248

gerritbot added a comment.Nov 19 2019, 9:40 AM

Change 551779 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: pool centrallog2001

https://gerrit.wikimedia.org/r/551779

gerritbot added a comment.Nov 19 2019, 10:02 AM

Change 551779 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: pool centrallog2001

https://gerrit.wikimedia.org/r/551779

fgiunchedi updated the task description. (Show Details)Nov 19 2019, 10:16 AM
fgiunchedi moved this task from Up next to Doing on the User-fgiunchedi board.Nov 20 2019, 12:33 PM
gerritbot added a comment.Nov 20 2019, 12:54 PM

Change 547245 merged by Filippo Giunchedi:
[operations/puppet@production] rsyslog: setup temporary secure rsync for logs transfer

https://gerrit.wikimedia.org/r/547245

fgiunchedi updated the task description. (Show Details)Nov 20 2019, 1:08 PM
Maintenance_bot removed a project: Patch-For-Review.Nov 20 2019, 1:10 PM
Papaul closed subtask T238642: Update label and switch port for wezen -> centrallog2001 as Resolved.Nov 20 2019, 3:47 PM
gerritbot added a comment.Nov 21 2019, 8:51 AM

Change 552208 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] centrallog: remove quickdatacopy after sync

https://gerrit.wikimedia.org/r/552208

gerritbot added a project: Patch-For-Review.Nov 21 2019, 8:51 AM
gerritbot added a comment.Nov 21 2019, 8:56 AM

Change 552208 merged by Filippo Giunchedi:
[operations/puppet@production] centrallog: remove quickdatacopy after sync

https://gerrit.wikimedia.org/r/552208

Maintenance_bot removed a project: Patch-For-Review.Nov 21 2019, 9:10 AM
fgiunchedi closed this task as Resolved.Nov 21 2019, 9:13 AM
fgiunchedi updated the task description. (Show Details)

This is complete!

Volans subscribed.Dec 2 2019, 11:20 AM

I've updated the mgmt DNS name record in Netbox that was still reporting wezen. I've also a patch to cleanup the wezen record from DNS, will push it later today.

gerritbot added a comment.Dec 2 2019, 2:41 PM

Change 554083 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] codfw: remove old wezen record

https://gerrit.wikimedia.org/r/554083

gerritbot added a project: Patch-For-Review.Dec 2 2019, 2:41 PM
gerritbot added a comment.Dec 3 2019, 9:22 AM

Change 554083 merged by Filippo Giunchedi:
[operations/dns@master] codfw: remove old wezen record

https://gerrit.wikimedia.org/r/554083

Maintenance_bot removed a project: Patch-For-Review.Dec 3 2019, 10:10 AM
MoritzMuehlenhoff renamed this task from Reimage wezen to Stretch or Buster (and rename to centrallog2001) to Reimage wezen to Buster (and rename to centrallog2001).Dec 3 2019, 10:12 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL