Page MenuHomePhabricator
Create Task
Maniphest T216513

compiler1002.puppet-diffs.eqiad.wmflabs instance is down
Closed, ResolvedPublic
Actions

Description

Aftermath of WMCS outage last week, the instance compiler1002.puppet-diffs.eqiad.wmflabs is unreachable.

The CI Jenkins can not reach it over ssh https://integration.wikimedia.org/ci/computer/compiler1002.puppet-diffs.eqiad.wmflabs/

Horizon page: https://horizon.wikimedia.org/project/instances/b65c0c7d-280f-407b-bf13-fb4b16523c2c/

Logs:

Request IDActionStart TimeUser IDMessage
req-10c22ffc-0340-4bd6-9b28-749b103d02afStopFeb. 13, 2019, 6:39 p.m.novaadmin-
req-28587666-19d4-4034-846c-98a44af7793eStartFeb. 13, 2019, 6:37 p.m.novaadmin-
req-307304fc-ba65-494e-8671-80a9288bea54StopFeb. 13, 2019, 5:01 p.m.novaadmin-
req-2923193e-53ad-4392-891e-721ae8d528c9StartFeb. 13, 2019, 3:33 p.m.novaadmin-
req-71da83a0-32d5-4f38-a0c3-d8e87edf4943StopFeb. 13, 2019, 3:29 p.m.--
req-dbd3ed92-ca73-44e5-bb97-a1e1b240f8ebStartFeb. 13, 2019, 2:24 p.m.novaadmin-
req-df32d30e-a324-4f97-9f90-2a27364885b0StopFeb. 13, 2019, 1:38 p.m.--
req-aea6fa12-5834-4328-a644-8f976eafc6c7RebootNov. 9, 2018, 3:58 a.m.novaadmin-

Event Timeline

hashar created this task.Feb 19 2019, 3:25 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 19 2019, 3:25 PM
Stashbot subscribed.Feb 19 2019, 3:25 PM

Mentioned in SAL (#wikimedia-operations) [2019-02-19T15:25:47Z] <hashar> Started instance compiler1002.puppet-diffs.eqiad.wmflabs via Horizon. It was in shutoff state | T216513

hashar closed this task as Resolved.Feb 19 2019, 3:26 PM
hashar claimed this task.

I started the instance via Horizon and it got attached to Jenkins again.

bd808 removed a project: cloud-services-team.Feb 19 2019, 3:53 PM
hashar reopened this task as Open.Feb 19 2019, 4:56 PM
hashar added a subscriber: elukey.

The last builds on that instances all failed https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/buildTimeTrend

Restricted Application added a project: Release-Engineering-Team (Kanban). · View Herald TranscriptFeb 19 2019, 4:56 PM
herron subscribed.Feb 19 2019, 4:59 PM
Stashbot added a comment.Feb 19 2019, 5:00 PM

Mentioned in SAL (#wikimedia-operations) [2019-02-19T17:00:18Z] <hashar> Offlined compiler1002.puppet-diffs.eqiad.wmflabs from Jenkins. Its disk is corrupt | T216513

hashar added a comment.Feb 19 2019, 5:10 PM

Joe said that it is /var/lib/catalog-differ being corrupted. That might be easy to
recover :)

herron added a comment.Feb 19 2019, 5:34 PM

I've re-created compiler1002 from scratch and am working to bring the puppet compiler service up on the host and validate a few builds locally. I estimate this will take until tonight (eastern time) or tomorrow morning since the local puppetdb takes a while to populate.

herron added a comment.Feb 19 2019, 8:52 PM

Compiler1002 is back online and successfully ran through a few local test catalog compiles. populate-puppetdb is running now so we should be in good shape to re-enable this host tomorrow morning. Will follow up when that completes.

hashar added a comment.Feb 19 2019, 9:26 PM

Wonderful, thank you :-]

herron added a comment.Feb 19 2019, 9:49 PM

That finished a bit faster than I was expecting! Ready to re-enable in the morning. And FWIW here's an example of a successful manual run https://puppet-compiler.wmflabs.org/compiler1002/2/.

hashar reassigned this task from hashar to herron.Feb 20 2019, 10:21 AM

@herron restored the instance, not me :] Will be verified in a few hours and added back to CI pool if it works fine.

hashar triaged this task as High priority.Feb 20 2019, 10:22 AM
hashar moved this task from Untriaged to In-progress on the Continuous-Integration-Infrastructure board.
hashar moved this task from Backlog to In-progress on the Release-Engineering-Team (Kanban) board.
herron reassigned this task from herron to hashar.Feb 21 2019, 4:33 PM

compiler1002 is ready to be re-enabled at your earliest convenience

Stashbot added a comment.Feb 21 2019, 4:35 PM

Mentioned in SAL (#wikimedia-releng) [2019-02-21T16:35:40Z] <hashar> Adjusting IP of https://integration.wikimedia.org/ci/computer/compiler1002.puppet-diffs.eqiad.wmflabs/ and adding it back # T216513

herron closed this task as Resolved.Feb 21 2019, 4:40 PM

Test build succeeded https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/14773/console, I think we're in good shape now.

hashar reassigned this task from hashar to herron.Feb 21 2019, 10:25 PM

Thank you Keith for the complete rebuild! ;-]

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits