Page MenuHomePhabricator

Service restarts of cloudelastic for Java security updates (Aug 2021)
Open, Needs TriagePublic3 Estimated Story Points

Description

Status
  • Restarted cloudelastic

Event Timeline

Mentioned in SAL (#wikimedia-operations) [2021-10-08T03:58:14Z] <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814

Mentioned in SAL (#wikimedia-operations) [2021-10-08T03:58:21Z] <ryankemper@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814

Mentioned in SAL (#wikimedia-operations) [2021-10-08T04:20:48Z] <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814

Mentioned in SAL (#wikimedia-operations) [2021-10-08T04:20:56Z] <ryankemper@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814

Mentioned in SAL (#wikimedia-operations) [2021-10-08T04:31:24Z] <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814

Mentioned in SAL (#wikimedia-operations) [2021-10-08T04:32:05Z] <ryankemper> T292814 Beginning rolling restart of cloudelastic: sudo -i cookbook sre.elasticsearch.rolling-operation cloudelastic "cloudelastic restart" --nodes-per-run 1 --start-datetime 2021-10-08T03:53:49 --task-id T292814 on ryankemper@cumin1001 tmux elastic

Mentioned in SAL (#wikimedia-operations) [2021-10-08T05:35:17Z] <ryankemper@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814

Mentioned in SAL (#wikimedia-operations) [2021-10-08T23:10:22Z] <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814

Mentioned in SAL (#wikimedia-operations) [2021-10-09T00:12:10Z] <ryankemper@cumin1001> END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814

Mentioned in SAL (#wikimedia-operations) [2021-10-09T00:13:17Z] <ryankemper> T292814 Write queue stuck at 133 events in partition 1 of topic codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite, will try again at another time

Mentioned in SAL (#wikimedia-operations) [2021-10-09T00:28:11Z] <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814

Mentioned in SAL (#wikimedia-operations) [2021-10-09T01:32:16Z] <ryankemper@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814

Mentioned in SAL (#wikimedia-operations) [2021-10-14T22:17:40Z] <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814

Mentioned in SAL (#wikimedia-operations) [2021-10-14T23:21:56Z] <ryankemper@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814