Page MenuHomePhabricator

cp1087 needed a powercycle
Closed, ResolvedPublic

Description

-------------------------------------------------------------------------------                                                                       
Record:      125                                                                                                                                      
Date/Time:   01/27/2021 16:35:07                                                                                                                      
Source:      system                                                                                                                                   
Severity:    Ok                                                                                                                                       
Description: An OEM diagnostic event occurred.                                                                                                        
-------------------------------------------------------------------------------                                                                       
Record:      126                                                                                                                                      
Date/Time:   01/27/2021 16:35:07                                                                                                                      
Source:      system                                                                                                                                   
Severity:    Ok                                                                                                                                       
Description: An OEM diagnostic event occurred.                                                                                                        
-------------------------------------------------------------------------------                                                                       
Record:      127                                                                                                                                      
Date/Time:   01/27/2021 16:35:07                                                                                                                      
Source:      system                                                                                                                                   
Severity:    Ok                                                                                                                                       
Description: An OEM diagnostic event occurred.                                                                                                        
-------------------------------------------------------------------------------                                                                       
Record:      128                                                                                                                                      
Date/Time:   01/27/2021 16:35:07                                                                                                                      
Source:      system                                                                                                                                   
Severity:    Ok                                                                                                                                       
Description: An OEM diagnostic event occurred.                                                                                                        
-------------------------------------------------------------------------------                                                                       
Record:      129                                                                                                                                      
Date/Time:   01/27/2021 16:35:07                                                                                                                      
Source:      system                                                                                                                                   
Severity:    Ok                                                                                                                                       
Description: An OEM diagnostic event occurred.                                                                                                        
-------------------------------------------------------------------------------                                                                       
Record:      130                                                                                                                                      
Date/Time:   01/27/2021 16:35:08                                                                                                                      
Source:      system                                                                                                                                   
Severity:    Ok                                                                                                                                       
Description: An OEM diagnostic event occurred.                                                                                                        
-------------------------------------------------------------------------------                                                                       
Record:      131                                                                                                                                      
Date/Time:   01/27/2021 16:35:08                                                                                                                      
Source:      system                                                                                                                                   
Severity:    Critical

The racadm serial console was not useful (blank screen, no tty usable). I have depooled the host and powercycled it, waiting for the Traffic team to check :)

Event Timeline

elukey triaged this task as High priority.Jan 28 2021, 7:26 AM
elukey created this task.

Mentioned in SAL (#wikimedia-operations) [2021-01-28T08:20:57Z] <vgutierrez> restart purged on cp1087 - T273153

Vgutierrez claimed this task.
Vgutierrez subscribed.

Everything looking good.. purged had some troubles going through the backlog of PURGE requests... especially with varnish-fe. Considering that ats-be eventually caught up with the backlog and that varnish-fe cache was empty due to the server powercycle I triggered a purged restart and things went back to normal:

image.png (1×1 px, 132 KB)