Page MenuHomePhabricator
Create Task
Maniphest T318778

Put centrallog1002 in service
Closed, ResolvedPublic
Actions

Description

Host was installed/racked in T313858: Q1:rack/setup/install centrallog1002

  • Apply partman standard software raid recipe #882718
    • Reimage centrallog1002
  • Apply role::syslog::centralserver on centrallog1002 #881939
  • Add Add centrallog1002 to the kafka-jumbo allow list #882747
  • Add centrallog1002 to eqiad anycast_neighbors #882724
  • Sync centrallog1001 to centrallog1002 #882760
    • Show transfer progress when using quickdatacopy #889239
  • Add centrallog1002 as eqiad TLS rsyslog destination #882761
  • Add centrallog1002 as logsource for logstash tests #882762

Issues found:

  • The sync took many hours to complete 11:48:23

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
centrallog: Show transfer progress when using quickdatacopyoperations/puppetproduction+1 -0
centrallog: Enable auto_ferm_ipv6 to quickdatacopyoperations/puppetproduction+1 -0
rsyslog: Add centrallog1002 as eqiad TLS rsyslog destinationoperations/puppetproduction+1 -0
centrallog: Sync centrallog1001 to centrallog1002operations/puppetproduction+3 -4
centrallog: Add centrallog1002 to the kafka-jumbo allow listoperations/puppetproduction+1 -0
centrallog1002: Add to eqiad anycast_neighborsoperations/homer/publicmaster+1 -0
centrallog: apply role::syslog::centralserver on centrallog instancesoperations/puppetproduction+2 -6
logstash: Add centrallog1002 as logsource for logstash testsoperations/puppetproduction+2 -2
centrallog: Apply partman standard software raid recipeoperations/puppetproduction+1 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

fgiunchedi created this task.Sep 28 2022, 7:38 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 28 2022, 7:38 AM
andrea.denisse claimed this task.Oct 19 2022, 2:26 PM
• lmata edited projects, added SRE Observability (FY2022/2023-Q2); removed SRE Observability (FY2022/2023-Q3).Oct 24 2022, 12:52 PM
• lmata edited projects, added SRE Observability (FY2022/2023-Q3); removed SRE Observability (FY2022/2023-Q2).Jan 12 2023, 6:49 PM
gerritbot added a comment.Jan 20 2023, 11:15 PM

Change 881939 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] centrallog: apply role::syslog::centralserver on centrallog instances

https://gerrit.wikimedia.org/r/881939

gerritbot added a project: Patch-For-Review.Jan 20 2023, 11:15 PM
fgiunchedi added a comment.Jan 23 2023, 9:20 AM

It looks like the host hasn't been provisioned with the correct partman recipe (some drives are not partitioned)

centrallog1002:~$ sudo vgs
  VG  #PV #LV #SN Attr   VSize   VFree  
  vg0   1   3   0 wz--n- 893.84g 178.77g
centrallog1002:~$ cat /proc/partitions 
major minor  #blocks  name

   8        0  937692504 sda
   8        1     291840 sda1
   8        2  937399296 sda2
   8       32  937692504 sdc
   8       48  937692504 sdd
   8       16  937692504 sdb
   8       17     291840 sdb1
   8       18  937399296 sdb2
   9        0  937267200 md0
 253        0     999424 dm-0
 253        1   78123008 dm-1
 253        2  670683136 dm-2
andrea.denisse added a comment.Jan 23 2023, 8:28 PM

Thanks @fgiunchedi , I've sent patch #882718 to apply the partman standard software raid recipe.

gerritbot added a comment.Jan 23 2023, 9:09 PM

Change 882724 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/homer/public@master] centrallog1002: Add to eqiad anycast_neighbors

https://gerrit.wikimedia.org/r/882724

gerritbot added a comment.Jan 23 2023, 9:31 PM

Change 882747 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] centrallog: Add centrallog1002 as Kafka broker

https://gerrit.wikimedia.org/r/882747

andrea.denisse changed the task status from Open to In Progress.Jan 23 2023, 9:49 PM
andrea.denisse updated the task description. (Show Details)Jan 24 2023, 1:03 AM
gerritbot added a comment.Jan 24 2023, 1:18 AM

Change 882760 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] centrallog: Sync centrallog1001 to centrallog1002

https://gerrit.wikimedia.org/r/882760

andrea.denisse updated the task description. (Show Details)Jan 24 2023, 1:20 AM
gerritbot added a comment.Jan 24 2023, 1:31 AM

Change 882761 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] rsyslog: Add centrallog1002 as eqiad TLS rsyslog destination

https://gerrit.wikimedia.org/r/882761

andrea.denisse updated the task description. (Show Details)Jan 24 2023, 1:32 AM
gerritbot added a comment.Jan 24 2023, 1:52 AM

Change 882762 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] logstash: Add centrallog1002 as logsource for logstash tests

https://gerrit.wikimedia.org/r/882762

andrea.denisse updated the task description. (Show Details)Jan 24 2023, 1:54 AM
gerritbot added a comment.Jan 24 2023, 5:29 PM

Change 882718 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] centrallog: Apply partman standard software raid recipe

https://gerrit.wikimedia.org/r/882718

gerritbot added a comment.Jan 24 2023, 6:14 PM

Change 882718 merged by Andrea Denisse:

[operations/puppet@production] centrallog: Apply partman standard software raid recipe

https://gerrit.wikimedia.org/r/882718

andrea.denisse updated the task description. (Show Details)Jan 24 2023, 6:28 PM
ops-monitoring-bot added a comment.Jan 25 2023, 12:10 AM

Cookbook cookbooks.sre.hosts.reimage was started by denisse@cumin1001 for host centrallog1002.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 25 2023, 12:10 AM

Cookbook cookbooks.sre.hosts.reimage started by denisse@cumin1001 for host centrallog1002.eqiad.wmnet with OS bullseye executed with errors:

  • centrallog1002 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • The reimage failed, see the cookbook logs for the details
gerritbot added a comment.Jan 25 2023, 12:33 AM

Change 882762 merged by Cwhite:

[operations/puppet@production] logstash: Add centrallog1002 as logsource for logstash tests

https://gerrit.wikimedia.org/r/882762

ops-monitoring-bot added a comment.Jan 25 2023, 7:00 PM

Cookbook cookbooks.sre.hosts.reimage was started by denisse@cumin1001 for host centrallog1002.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 25 2023, 7:00 PM

Cookbook cookbooks.sre.hosts.reimage started by denisse@cumin1001 for host centrallog1002.eqiad.wmnet with OS bullseye executed with errors:

  • centrallog1002 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jan 25 2023, 7:21 PM

Cookbook cookbooks.sre.hosts.reimage was started by denisse@cumin1001 for host centrallog1002.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 25 2023, 7:52 PM

Cookbook cookbooks.sre.hosts.reimage started by denisse@cumin1001 for host centrallog1002.eqiad.wmnet with OS bullseye completed:

  • centrallog1002 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202301251920_denisse_3238543_centrallog1002.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Jan 25 2023, 8:11 PM

Change 881939 merged by Andrea Denisse:

[operations/puppet@production] centrallog: apply role::syslog::centralserver on centrallog instances

https://gerrit.wikimedia.org/r/881939

andrea.denisse updated the task description. (Show Details)Jan 25 2023, 8:17 PM
gerritbot added a comment.Jan 25 2023, 8:23 PM

Change 882724 merged by Andrea Denisse:

[operations/homer/public@master] centrallog1002: Add to eqiad anycast_neighbors

https://gerrit.wikimedia.org/r/882724

andrea.denisse updated the task description. (Show Details)Jan 25 2023, 8:24 PM
gerritbot added a comment.Jan 25 2023, 10:05 PM

Change 882747 merged by Andrea Denisse:

[operations/puppet@production] centrallog: Add centrallog1002 to the kafka-jumbo allow list

https://gerrit.wikimedia.org/r/882747

andrea.denisse updated the task description. (Show Details)Jan 25 2023, 10:06 PM
andrea.denisse updated the task description. (Show Details)
gerritbot added a comment.Jan 25 2023, 10:19 PM

Change 882760 merged by Andrea Denisse:

[operations/puppet@production] centrallog: Sync centrallog1001 to centrallog1002

https://gerrit.wikimedia.org/r/882760

gerritbot added a comment.Feb 2 2023, 7:02 PM

Change 882761 merged by Andrea Denisse:

[operations/puppet@production] rsyslog: Add centrallog1002 as eqiad TLS rsyslog destination

https://gerrit.wikimedia.org/r/882761

Maintenance_bot removed a project: Patch-For-Review.Feb 2 2023, 7:30 PM
andrea.denisse added a subtask: T328803: Decommission centrallog1001.Feb 3 2023, 7:08 PM
andrea.denisse updated the task description. (Show Details)Feb 8 2023, 3:04 PM
gerritbot added a comment.Feb 8 2023, 5:42 PM

Change 887812 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] centrallog: Add centrallog1001 to quickdatacopy allow hosts

https://gerrit.wikimedia.org/r/887812

gerritbot added a project: Patch-For-Review.Feb 8 2023, 5:42 PM
gerritbot added a comment.Feb 9 2023, 5:33 PM

Change 887812 abandoned by Andrea Denisse:

[operations/puppet@production] centrallog: Enable auto_ferm_ipv6 to quickdatacopy

Reason:

Closing in favor of achieving the same result using Hiera lookups

https://gerrit.wikimedia.org/r/887812

Maintenance_bot removed a project: Patch-For-Review.Feb 9 2023, 6:30 PM
gerritbot added a comment.Feb 14 2023, 9:30 PM

Change 889231 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] quickdatacopy: Add option to show progress during transfer

https://gerrit.wikimedia.org/r/889231

gerritbot added a project: Patch-For-Review.Feb 14 2023, 9:30 PM
gerritbot added a comment.Feb 14 2023, 9:45 PM

Change 889239 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] centrallog: Show transfer progress when using quickdatacopy

https://gerrit.wikimedia.org/r/889239

andrea.denisse updated the task description. (Show Details)Feb 14 2023, 9:47 PM
gerritbot added a comment.Feb 15 2023, 2:57 PM

Change 889239 merged by Andrea Denisse:

[operations/puppet@production] centrallog: Show transfer progress when using quickdatacopy

https://gerrit.wikimedia.org/r/889239

andrea.denisse updated the task description. (Show Details)Feb 15 2023, 3:03 PM
andrea.denisse updated the task description. (Show Details)Feb 15 2023, 3:05 PM
andrea.denisse closed this task as Resolved.Feb 21 2023, 4:40 PM
andrea.denisse updated the task description. (Show Details)
Jclark-ctr closed subtask T328803: Decommission centrallog1001 as Resolved.Mar 15 2023, 7:47 PM
andrea.denisse added a parent task: T324725: Observability Bullseye upgrades.Mar 21 2023, 11:03 PM
andrea.denisse mentioned this in T324725: Observability Bullseye upgrades.
• lmata moved this task from Inbox to Done on the SRE Observability (FY2022/2023-Q3) board.May 2 2023, 1:48 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits