Page MenuHomePhabricator
Create Task
Maniphest T200706

rack/setup/install centrallog1001.eqiad.wmnet
Closed, ResolvedPublic
Actions

Description

This task will track the racking and setup of the new central syslogging system for eqiad. This replaces the outdated lithium.eqiad.wmnet, which will be decommisioned once this is fully online.

Hostname Proposal: Change from using elements (which are generic when a system has varied services/roles) to a cluster name, since this host is replicated in codfw. Currently proposed is centrallog1001, but could also do something else. Other options were log1001, and syslog1001.

Racking Proposal: Any 1G capable rack. This host will be in the private vlan for whatever row its assigned.

centrallog1001:

  • - receive in system on procurement task T195416
  • - rack system with proposed racking plan (see above) & update racktables (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

Details

SubjectRepoBranchLines +/-
wmnet: flip syslog.eqiad.wmnet to centrallog1001operations/dnsmaster+3 -1
Remove lithium from serviceoperations/puppetproduction+8 -8
Revert "syslog: add temp rsync to copy data"operations/puppetproduction+0 -8
syslog: add temp rsync to copy dataoperations/puppetproduction+8 -0
hieradata: enable centrallog1001 in eqiadoperations/puppetproduction+1 -1
hieradata: enable centrallog1001 in codfwoperations/puppetproduction+1 -1
Enable centrallog1001 on all popsoperations/puppetproduction+2 -2
eqsin: send logs to centrallog1001 toooperations/puppetproduction+1 -1
install_server: use buster for centrallog1001operations/puppetproduction+1 -0
Adding mgmt/production dns for centrallog1001operations/dnsmaster+27 -22
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH triaged this task as Medium priority.Jul 30 2018, 4:56 PM
RobH created this task.
RobH renamed this task from rack/setup/install syslog1001.eqiad.wmnet to rack/setup/install centrallog1001.eqiad.wmnet.Jul 30 2018, 5:04 PM
RobH updated the task description. (Show Details)
• Cmjohnson moved this task from Backlog to Racking Tasks on the ops-eqiad board.Aug 1 2018, 2:32 PM
gerritbot subscribed.Aug 2 2018, 8:09 PM

Change 450094 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding mgmt/production dns for centrallog1001

https://gerrit.wikimedia.org/r/450094

gerritbot added a project: Patch-For-Review.Aug 2 2018, 8:09 PM

Change 450094 merged by Cmjohnson:
[operations/dns@master] Adding mgmt/production dns for centrallog1001

https://gerrit.wikimedia.org/r/450094

• Cmjohnson reassigned this task from • Cmjohnson to RobH.Aug 2 2018, 8:11 PM
• Cmjohnson updated the task description. (Show Details)
• Cmjohnson moved this task from Racking Tasks to Blocked on the ops-eqiad board.

assigning to @RobH for help with the final installation

RobH removed a project: Patch-For-Review.Aug 3 2018, 8:11 PM
RobH updated the task description. (Show Details)
RobH reassigned this task from RobH to fgiunchedi.Aug 3 2018, 9:54 PM
RobH removed a project: ops-eqiad.
RobH updated the task description. (Show Details)

@fgiunchedi: You were the SRE team member to provide feedback regarding the disk capacity, so I'm assuming you would be the service owner. If this isn't correct, please comment/assign back to me/assign to service owner as needed.

This system is ready to have services transferred to it, and replace lithium (which is 5+ years old.)

fgiunchedi added a project: User-fgiunchedi.Aug 6 2018, 12:36 PM

Thanks @RobH !

fgiunchedi moved this task from Backlog to Up next on the User-fgiunchedi board.Aug 21 2018, 12:56 AM
fgiunchedi added a comment.Aug 27 2018, 7:22 AM

Steps for service implementation:

  • Include centrallog1001 in router ACLs
  • Add centrallog1001 to remote_syslog(_tls) destinations so logs start flowing to that host too
  • Let logs accumulate for more than one day so rotation kicks in, rsync the rest of the logs from lithium

Decommissioning lithium from service:

  • Remove lithium from remote_syslog(_tls) destinations and router ACLs
fgiunchedi added a project: Wikimedia-Logstash.Oct 15 2018, 12:19 PM

Not really "logstash" but using Wikimedia-Logstash for logging-related tasks

fgiunchedi moved this task from Backlog to Up next on the Wikimedia-Logstash board.Oct 15 2018, 2:29 PM
herron added a project: User-herron.Oct 16 2018, 6:43 PM
herron subscribed.
fgiunchedi removed fgiunchedi as the assignee of this task.Dec 17 2018, 11:49 AM
fgiunchedi moved this task from Up next to Radar on the User-fgiunchedi board.
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 10:13 PM
MoritzMuehlenhoff mentioned this in T224549: Track remaining jessie systems in production.May 29 2019, 11:41 AM
fgiunchedi mentioned this in T225604: log spam from mtail 3.0.0~rc19 on wezen.Jun 12 2019, 11:04 AM
Volans moved this task from Up next to In Dev/Progress on the Wikimedia-Logstash board.Jun 24 2019, 3:19 PM
gerritbot added a comment.Jun 27 2019, 3:05 PM

Change 519420 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] install_server: use buster for centrallog1001

https://gerrit.wikimedia.org/r/519420

gerritbot added a project: Patch-For-Review.Jun 27 2019, 3:05 PM
gerritbot added a comment.Jun 27 2019, 3:17 PM

Change 519420 merged by Filippo Giunchedi:
[operations/puppet@production] install_server: use buster for centrallog1001

https://gerrit.wikimedia.org/r/519420

Maintenance_bot removed a project: Patch-For-Review.Jun 27 2019, 4:12 PM
fgiunchedi moved this task from Radar to Doing on the User-fgiunchedi board.Jul 1 2019, 12:50 PM
ayounsi closed subtask T226813: Add centrallog1001 to syslog servers in network ACLs as Resolved.Jul 2 2019, 3:30 PM
gerritbot added a comment.Jul 4 2019, 9:43 AM

Change 520713 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] eqsin: send logs to centrallog1001 too

https://gerrit.wikimedia.org/r/520713

gerritbot added a project: Patch-For-Review.Jul 4 2019, 9:43 AM
gerritbot added a comment.Jul 4 2019, 12:35 PM

Change 520713 merged by Filippo Giunchedi:
[operations/puppet@production] eqsin: send logs to centrallog1001 too

https://gerrit.wikimedia.org/r/520713

Maintenance_bot removed a project: Patch-For-Review.Jul 4 2019, 1:11 PM
gerritbot added a comment.Jul 4 2019, 2:14 PM

Change 520761 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] Enable centrallog1001 on all pops

https://gerrit.wikimedia.org/r/520761

gerritbot added a project: Patch-For-Review.Jul 4 2019, 2:14 PM
gerritbot added a comment.Jul 4 2019, 2:32 PM

Change 520761 merged by Filippo Giunchedi:
[operations/puppet@production] Enable centrallog1001 on all pops

https://gerrit.wikimedia.org/r/520761

Maintenance_bot removed a project: Patch-For-Review.Jul 4 2019, 3:10 PM
gerritbot added a comment.Jul 8 2019, 10:29 AM

Change 521245 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: enable centrallog1001 in codfw

https://gerrit.wikimedia.org/r/521245

gerritbot added a project: Patch-For-Review.Jul 8 2019, 10:29 AM
gerritbot added a comment.Jul 8 2019, 2:01 PM

Change 521245 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: enable centrallog1001 in codfw

https://gerrit.wikimedia.org/r/521245

Maintenance_bot removed a project: Patch-For-Review.Jul 8 2019, 2:10 PM
gerritbot added a comment.Jul 9 2019, 8:33 AM

Change 521428 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: enable centrallog1001 in eqiad

https://gerrit.wikimedia.org/r/521428

gerritbot added a project: Patch-For-Review.Jul 9 2019, 8:33 AM
gerritbot added a comment.Jul 9 2019, 8:51 AM

Change 521428 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: enable centrallog1001 in eqiad

https://gerrit.wikimedia.org/r/521428

Maintenance_bot removed a project: Patch-For-Review.Jul 9 2019, 9:11 AM
gerritbot added a comment.Jul 15 2019, 8:57 AM

Change 523102 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] syslog: add temp rsync to copy data

https://gerrit.wikimedia.org/r/523102

gerritbot added a project: Patch-For-Review.Jul 15 2019, 8:57 AM
gerritbot added a comment.Jul 15 2019, 9:04 AM

Change 523102 merged by Filippo Giunchedi:
[operations/puppet@production] syslog: add temp rsync to copy data

https://gerrit.wikimedia.org/r/523102

Maintenance_bot removed a project: Patch-For-Review.Jul 15 2019, 9:10 AM
gerritbot added a comment.Jul 16 2019, 9:59 AM

Change 523669 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] Revert "syslog: add temp rsync to copy data"

https://gerrit.wikimedia.org/r/523669

gerritbot added a project: Patch-For-Review.Jul 16 2019, 9:59 AM

Change 523670 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] Remove lithium from service

https://gerrit.wikimedia.org/r/523670

gerritbot added a comment.Jul 16 2019, 10:04 AM

Change 523669 merged by Filippo Giunchedi:
[operations/puppet@production] Revert "syslog: add temp rsync to copy data"

https://gerrit.wikimedia.org/r/523669

gerritbot added a comment.Jul 17 2019, 8:01 AM

Change 523670 merged by Filippo Giunchedi:
[operations/puppet@production] Remove lithium from service

https://gerrit.wikimedia.org/r/523670

Maintenance_bot removed a project: Patch-For-Review.Jul 17 2019, 8:10 AM
gerritbot added a comment.Jul 17 2019, 4:00 PM

Change 523957 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/dns@master] wmnet: flip syslog.eqiad.wmnet to centrallog1001

https://gerrit.wikimedia.org/r/523957

gerritbot added a project: Patch-For-Review.Jul 17 2019, 4:00 PM
gerritbot added a comment.Jul 18 2019, 8:44 AM

Change 523957 merged by Filippo Giunchedi:
[operations/dns@master] wmnet: flip syslog.eqiad.wmnet to centrallog1001

https://gerrit.wikimedia.org/r/523957

Maintenance_bot removed a project: Patch-For-Review.Jul 18 2019, 9:11 AM
fgiunchedi closed subtask T228277: Use centrallog1001 instead of lithium for PDU syslog as Invalid.Jul 18 2019, 10:46 AM
fgiunchedi closed subtask T228275: Use centrallog1001 for network devices syslog as Invalid.
fgiunchedi reopened subtask T228275: Use centrallog1001 for network devices syslog as Open.Jul 18 2019, 12:44 PM
fgiunchedi closed subtask T228275: Use centrallog1001 for network devices syslog as Resolved.Jul 31 2019, 4:56 PM
fgiunchedi closed this task as Resolved.Aug 1 2019, 8:48 AM
fgiunchedi claimed this task.

This is done, decom for lithium is at T229557: decommission lithium

fgiunchedi added a project: observability.Aug 19 2019, 2:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL