Page MenuHomePhabricator

Unresponsive management for cp2035.mgmt:22
Closed, ResolvedPublic

Description

Common information

  • alertname: ManagementSSHDown
  • instance: cp2035.mgmt:22
  • job: probes/mgmt
  • module: ssh_banner
  • prometheus: ops
  • rack: C2
  • severity: task
  • site: codfw
  • source: prometheus
  • team: dcops

Firing alerts


Related Objects

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

@Dzahn @Jclark-ctr @Jhancock.wm @Papaul @VRiley-WMF this is how the management alerts for example look like

Note that there might be duplicates with the old titles during the transition. Likewise, when changing the summary in the alert also new tasks will be open. Please LMK what do you think

i prefer this. the individual issues makes it easier to communicate and easier to track how many are happening at once.

@ssingh this came up again. it had this issue in august of last year. T372160. the same remedy should apply. I'm hesitant to do anything more intensive since the server is going to be refreshed this quarter. T389840.
i'll need to power cycle the server. Let me know when would be a good time for you to depool it.

Mentioned in SAL (#wikimedia-operations) [2025-04-01T13:44:05Z] <sukhe@puppetserver1001> conftool action : set/pooled=no; selector: name=cp2035.codfw.wmnet [reason: T390658]

@ssingh this came up again. it had this issue in august of last year. T372160. the same remedy should apply. I'm hesitant to do anything more intensive since the server is going to be refreshed this quarter. T389840.
i'll need to power cycle the server. Let me know when would be a good time for you to depool it.

Thanks Jen! The server has been depooled so you can work on it when desired.

@ssingh server has been power cycled. can ssh into it and mgmt/network both ping.

ssingh claimed this task.

Thanks for the help @Jhancock.wm!

Mentioned in SAL (#wikimedia-operations) [2025-04-01T14:51:39Z] <sukhe@puppetserver1001> conftool action : set/pooled=yes; selector: name=cp2035.codfw.wmnet [reason: finished T390658]

@fgiunchedi thanks! looks good to me:) Later, let's do the same for tickets generated for failed systemd timers.