Page MenuHomePhabricator

Put centrallog2002 in service
Closed, ResolvedPublic

Description

Followup from T289624: Q1: (Need By: TBD) rack/setup/install centrallog2002.codfw.wmnet, the host needs to be put in service

Event Timeline

herron changed the task status from Open to In Progress.Oct 7 2021, 5:15 PM
herron triaged this task as Medium priority.
herron moved this task from Inbox to In progress on the SRE Observability (FY2021/2022-Q2) board.

Change 730843 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] centrallog2002: apply role::syslog::centralserver

https://gerrit.wikimedia.org/r/730843

Change 730843 merged by Herron:

[operations/puppet@production] centrallog2002: apply role::syslog::centralserver

https://gerrit.wikimedia.org/r/730843

Mentioned in SAL (#wikimedia-operations) [2021-10-18T14:54:19Z] <herron> rebuilt and uploaded kafkatee for bullseye T292196

Mentioned in SAL (#wikimedia-operations) [2021-10-18T15:16:49Z] <herron> reprepro copied anycast-healthchecker, python3-json-logger and python3-anycast-healthchecker from buster-wikimedia to bullseye-wikimedia T292196

role::syslog::centralserver is now applied to centrallog2002, and the puppet run is clean after rebuilding/copying the above packages

Next steps will be to enable anycast on the routers and copy over the log history from centrallog2001.

Change 731828 had a related patch set uploaded (by Herron; author: Herron):

[operations/homer/public@master] add centrallog2002 to codfw anycast_neighbors and syslog fw allows

https://gerrit.wikimedia.org/r/731828

Change 732711 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] kafka-jumbo: permit centrallog2002 via ferm

https://gerrit.wikimedia.org/r/732711

Change 734401 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] rsyslog: centralize remote_server_tls lookups into single location in hiera

https://gerrit.wikimedia.org/r/734401

Change 734405 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] rsyslog: switch codfw TLS remote syslog destination to centrallog2002

https://gerrit.wikimedia.org/r/734405

Change 734401 merged by Herron:

[operations/puppet@production] rsyslog: centralize remote_syslog_tls lookups into single location in hiera

https://gerrit.wikimedia.org/r/734401

Change 731828 merged by Herron:

[operations/homer/public@master] add centrallog2002 to codfw anycast_neighbors and syslog fw allows

https://gerrit.wikimedia.org/r/731828

Change 732711 merged by Herron:

[operations/puppet@production] kafka-jumbo: permit centrallog2002 via ferm

https://gerrit.wikimedia.org/r/732711

Change 734405 merged by Herron:

[operations/puppet@production] rsyslog: switch codfw TLS remote syslog destination to centrallog2002

https://gerrit.wikimedia.org/r/734405

Centrallog2002 is now the codfw syslog TLS destination. Logs archives from centrallog2001 have been copied over, and the partial logs for today when the switchover happened are now being copied into /srv/syslog/00-partial_logs.

I see logs still arriving to centrallog2001 6514/TCP from cloudbackup1001-dev, and restbase-dev hosts. These hosts have puppet disabled and will switch over on the next puppet run.

Resolving as centrallog2002 is in service, tracking centrallog2001 decom with T298994