Page MenuHomePhabricator

asw-a1-codfw spontaneous reboot
Closed, ResolvedPublic

Description

asw-a1-codfw rebooted earlier today (2016-03-02 16:54 UTC). Total downtime was approximately 3 minutes and affecting only A1 hosts (es2001, es2014, mc2019, ms-be2001). Cause so far remains unknown; nothing stands out in the logs and no datacenter work was occuring at the time.

Mar  2 16:54:03  asw-a-codfw vccpd[1635]: VCCPD_PROTOCOL_ADJDOWN: Lost adjacency to dc38.e1d4.1b00 on vcp-255/0/48.32768,
Mar  2 16:54:03  asw-a-codfw vccpd[1635]: interface vcp-255/0/48 went down
Mar  2 16:54:03  asw-a-codfw fpc3 [EX-BCM PIC] ex_bcm_linkscan_handler: Link 54 DOWN
Mar  2 16:54:03  asw-a-codfw vccpd[1635]: Member 2, interface vcp-255/0/48.32768 went down
Mar  2 16:54:03  asw-a-codfw vccpd[1635]: Member 3, interface vcp-255/1/0.32768 went down
Mar  2 16:54:03  asw-a-codfw vccpd[1635]: Member 8, interface vcp-255/1/0.32768 went down
Mar  2 16:54:03  asw-a-codfw chassisd[1627]: CHASSISD_VCHASSIS_MEMBER_UPDATE_NOTICE: Membership update: Member 2->2, Mode Master->Master, 2M 7B, Master Unchanged, Members Changed
Mar  2 16:54:03  asw-a-codfw chassisd[1627]: CHASSISD_VCHASSIS_MEMBER_LIST_NOTICE: Members: 2M 3L 4L 5L 6L 7B 8L
Mar  2 16:54:03  asw-a-codfw chassisd[1627]: CHASSISD_VCHASSIS_MEMBER_OP_NOTICE: Member change: vc delete of member 1
[…]
Mar  2 16:56:38  asw-a-codfw fpc3 [EX-BCM PIC] ex_bcm_linkscan_handler: Link 54 UP
Mar  2 16:56:38  asw-a-codfw vccpd[1635]: interface vcp-255/0/48 came up
Mar  2 16:56:38  asw-a-codfw fpc8 [EX-BCM PIC] ex_bcm_linkscan_handler: Link 54 UP
Mar  2 16:56:38  asw-a-codfw vccpd[1635]: Member 2, interface vcp-255/0/48.32768 came up
Mar  2 16:56:38  asw-a-codfw vccpd[1635]: Member 3, interface vcp-255/1/0.32768 came up
Mar  2 16:56:38  asw-a-codfw vccpd[1635]: JTASK_SIGNAL_UNKNOWN: Ignoring unknown signal SIGVTALRM (26)
Mar  2 16:56:39  asw-a-codfw vccpd[1635]: Member 8, interface vcp-255/1/0.32768 came up
Mar  2 16:56:39  asw-a-codfw vccpd[1635]: JTASK_SIGNAL_UNKNOWN: Ignoring unknown signal SIGVTALRM (26)
Mar  2 16:56:46  asw-a-codfw chassisd[1627]: CHASSISD_VCHASSIS_MEMBER_UPDATE_NOTICE: Membership update: Member 2->2, Mode Master->Master, 2M 7B, Master Unchanged, Members Changed
Mar  2 16:56:46  asw-a-codfw chassisd[1627]: CHASSISD_VCHASSIS_MEMBER_LIST_NOTICE: Members: 1L 2M 3L 4L 5L 6L 7B 8L
Mar  2 16:56:46  asw-a-codfw chassisd[1627]: CHASSISD_VCHASSIS_MEMBER_OP_NOTICE: Member change: vc add of member 1

Event Timeline

faidon created this task.Mar 2 2017, 5:32 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 2 2017, 5:32 PM
elukey added a subscriber: elukey.Mar 2 2017, 5:33 PM

No results for show system core-dumps too.

faidon closed this task as Resolved.Apr 3 2017, 2:01 PM
faidon claimed this task.

Logs didn't show anything and it hasn't happened in a month. Let's resolve for now.