integration-slave-trusty-1012, -1013, and -1015 unresponsive
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Krinkle
	Jun 7 2015, 8:42 PM

Description

These three instances have been unresponsive for the last 24 hours. I'm also not able to establish an SSH connection.

integration-slave-precise-1011.eqiad.wmflabs: 1
integration-slave-precise-1012.eqiad.wmflabs: 1
integration-slave-precise-1013.eqiad.wmflabs: 1
integration-slave-precise-1014.eqiad.wmflabs: 1
integration-slave-trusty-1011.eqiad.wmflabs: 1
integration-slave-trusty-1014.eqiad.wmflabs: 1
integration-slave-trusty-1016.eqiad.wmflabs: 1
integration-slave-trusty-1017.eqiad.wmflabs: 1

Connection closed:

integration-slave-trusty-1012.eqiad.wmflabs
integration-slave-trusty-1013.eqiad.wmflabs
integration-slave-trusty-1015.eqiad.wmflabs

YuviPanda: Krinkle: I get "Connection closed by Unknown", usually this means something OOM'd
YuviPanda: Krinkle: no output in 'get console output' due to wikitech flakiness, I presume.

Related Objects

Duplicates Merged Here: T101653: MediaWiki Jenkins jobs stuck for 20 minutes

Event Timeline

Krinkle created this task.Jun 7 2015, 8:42 PM

Krinkle raised the priority of this task from to Unbreak Now!.

Krinkle updated the task description. (Show Details)

Krinkle added a project: Continuous-Integration-Infrastructure.

Krinkle added subscribers: Krinkle, hashar, yuvipanda.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 7 2015, 8:42 PM

https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm/19823/consoleFull was on integration-slave-trusty-1012 and timed out after 30 minutes. Should we depool the stuck trusty slaves for now?

I marked all 3 slaves as offline in jenkins.

Seems it was some a transient labs issue. I have rebooted the three instances and repooled them:

integration-slave-trusty-1012.eqiad.wmflabs
integration-slave-trusty-1013.eqiad.wmflabs
integration-slave-trusty-1015.eqiad.wmflabs

hashar merged a task: T101653: MediaWiki Jenkins jobs stuck for 20 minutes.Jun 8 2015, 9:03 AM

hashar moved this task from Untriaged to Done on the Continuous-Integration-Infrastructure board.Jun 23 2015, 9:08 AM

• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:50 PM

Restricted Application added a project: Release-Engineering-Team (Kanban). · View Herald TranscriptJun 7 2017, 6:50 PM

Restricted Application added subscribers: Jay8g, TerraCodes. · View Herald Transcript

• Phabricator_maintenance edited projects, added RelEng-Archive-FY201718-Q1; removed Release-Engineering-Team (Kanban).Sep 26 2017, 11:47 PM

integration-slave-trusty-1012, -1013, and -1015 unresponsiveClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

integration-slave-trusty-1012, -1013, and -1015 unresponsive
Closed, ResolvedPublic
Actions