Page MenuHomePhabricator
Create Task
Maniphest T313144

SystemdUnitDownForLong cloudcontrol1003:9100 Unit keystone_sync_keys_to_cloudcontrol1004.wikimedia.org.service on node cloudcontrol1003 has been down for long.
Closed, ResolvedPublic
Actions

Description

Common information

  • alertname: SystemdUnitDownForLong
  • cluster: wmcs
  • instance: cloudcontrol1003:9100
  • job: node
  • name: keystone_sync_keys_to_cloudcontrol1004.wikimedia.org.service
  • prometheus: ops
  • severity: task
  • site: eqiad
  • source: prometheus
  • state: failed
  • team: wmcs
  • type: simple

Firing alerts

Event Timeline

phaultfinder created this task.Jul 16 2022, 2:01 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 16 2022, 2:01 AM
dcaro changed the task status from Open to In Progress.Jul 18 2022, 10:04 AM
dcaro claimed this task.
dcaro moved this task from To refine to Doing on the User-dcaro board.
dcaro added a project: User-dcaro.
dcaro added a project: Cloud-Services-Origin-Alert.
dcaro added a project: Cloud-Services-Worktype-Unplanned.
dcaro added a comment.Jul 18 2022, 10:07 AM

This seemed to be a temporary issue, the service is now up and running, if it happens again will re-open/investigate more in depth.
Some files vanished while rsyncing.

Jul 18 00:00:00 cloudcontrol1003 systemd[1]: keystone_sync_keys_to_cloudcontrol1004.wikimedia.org.service: Succeeded.
root@cloudcontrol1003:~# journalctl -u keystone_sync_keys_to_cloudcontrol1004.wikimedia.org
-- Journal begins at Fri 2022-07-15 09:30:06 UTC, ends at Mon 2022-07-18 10:06:05 UTC. --
Jul 16 00:00:01 cloudcontrol1003 systemd[1]: Started Sync keys for Keystone fernet tokens to cloudcontrol1004.wikimedia.org.
Jul 16 00:00:01 cloudcontrol1003 rsync[3521095]: file has vanished: "/etc/keystone/fernet-keys/1869"
Jul 16 00:00:01 cloudcontrol1003 rsync[3521095]: rsync warning: some files vanished before they could be transferred (code 24) at main.c(1819) [generator=3.2.3]
Jul 16 00:00:01 cloudcontrol1003 systemd[1]: keystone_sync_keys_to_cloudcontrol1004.wikimedia.org.service: Main process exited, code=exited, status=24/n/a
Jul 16 00:00:01 cloudcontrol1003 systemd[1]: keystone_sync_keys_to_cloudcontrol1004.wikimedia.org.service: Failed with result 'exit-code'.


Jul 16 12:07:50 cloudcontrol1003 systemd[1]: Started Sync keys for Keystone fernet tokens to cloudcontrol1004.wikimedia.org.
Jul 16 12:07:50 cloudcontrol1003 systemd[1]: keystone_sync_keys_to_cloudcontrol1004.wikimedia.org.service: Succeeded.
Jul 17 00:00:01 cloudcontrol1003 systemd[1]: Started Sync keys for Keystone fernet tokens to cloudcontrol1004.wikimedia.org.
Jul 17 00:00:01 cloudcontrol1003 systemd[1]: keystone_sync_keys_to_cloudcontrol1004.wikimedia.org.service: Succeeded.
Jul 18 00:00:00 cloudcontrol1003 systemd[1]: Started Sync keys for Keystone fernet tokens to cloudcontrol1004.wikimedia.org.
Jul 18 00:00:00 cloudcontrol1003 systemd[1]: keystone_sync_keys_to_cloudcontrol1004.wikimedia.org.service: Succeeded.
dcaro closed this task as Resolved.Jul 18 2022, 10:07 AM
dcaro moved this task from Doing to Done on the User-dcaro board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL