Page MenuHomePhabricator
Create Task
Maniphest T292814

Service restarts of cloudelastic for Java security updates (Aug 2021)
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

Status
  • Restarted cloudelastic

Event Timeline

RKemper created this task.Oct 8 2021, 3:51 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 8 2021, 3:51 AM
Stashbot added a comment.Oct 8 2021, 3:58 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-08T03:58:14Z] <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814

Stashbot added a comment.Oct 8 2021, 3:58 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-08T03:58:21Z] <ryankemper@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814

Stashbot added a comment.Oct 8 2021, 4:20 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-08T04:20:48Z] <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814

Stashbot added a comment.Oct 8 2021, 4:20 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-08T04:20:56Z] <ryankemper@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814

Stashbot added a comment.Oct 8 2021, 4:31 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-08T04:31:24Z] <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814

Stashbot added a comment.Oct 8 2021, 4:32 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-08T04:32:05Z] <ryankemper> T292814 Beginning rolling restart of cloudelastic: sudo -i cookbook sre.elasticsearch.rolling-operation cloudelastic "cloudelastic restart" --nodes-per-run 1 --start-datetime 2021-10-08T03:53:49 --task-id T292814 on ryankemper@cumin1001 tmux elastic

Stashbot added a comment.Oct 8 2021, 5:35 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-08T05:35:17Z] <ryankemper@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814

Stashbot added a comment.Oct 8 2021, 11:10 PM

Mentioned in SAL (#wikimedia-operations) [2021-10-08T23:10:22Z] <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814

Stashbot added a comment.Oct 9 2021, 12:12 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-09T00:12:10Z] <ryankemper@cumin1001> END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814

Stashbot added a comment.Oct 9 2021, 12:13 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-09T00:13:17Z] <ryankemper> T292814 Write queue stuck at 133 events in partition 1 of topic codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite, will try again at another time

Stashbot added a comment.Oct 9 2021, 12:28 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-09T00:28:11Z] <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814

Stashbot added a comment.Oct 9 2021, 1:32 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-09T01:32:16Z] <ryankemper@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814

Stashbot added a comment.Oct 14 2021, 10:17 PM

Mentioned in SAL (#wikimedia-operations) [2021-10-14T22:17:40Z] <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814

Stashbot added a comment.Oct 14 2021, 11:21 PM

Mentioned in SAL (#wikimedia-operations) [2021-10-14T23:21:56Z] <ryankemper@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814

MPhamWMF set the point value for this task to 3.Oct 18 2021, 3:51 PM
MPhamWMF moved this task from Incoming to Ready for Dev -- SWE on the Discovery-Search (Current work) board.
MoritzMuehlenhoff closed this task as Resolved.Feb 1 2022, 3:05 PM
MoritzMuehlenhoff subscribed.

This can be closed, all cloudelastic* nodes are running the current JRE.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL