toolforge k8s control plane freezing and other stability issues
Open, HighPublic
Actions

Assigned To

None

Authored By

	hashar
	Apr 4 2023, 9:22 AM

Description

Promoting wikis to a new version on the Wikimedia cluster is done by using scap deploy-promote. That depends on the toolforge service train-blockers to find the current Phabricator task for this week train. Running the commands yields a 500 from toolforge:

09:18:37 deploy-promote failed: <HTTPError> 500 Server Error: Internal Server Error for url: https://train-blockers.toolforge.org/api.php

Also stashbot has vanished from #wikimedia-operations:

09:17:36 ⇐ •stashbot quit (~stashbot@wikimedia/bot/stashbot) Ping timeout: 255 seconds

Which sounds like an issue with Toolforge

Related Objects
Search...

Status	Assigned	Task
Open	None	T333922 toolforge k8s control plane freezing and other stability issues
Resolved	taavi	T333929 toolforge: Move k8s control plane nodes to larger instance flavors
Resolved	Andrew	T333931 toolforge: Move k8s etcd nodes to larger instance flavors
Open	None	T333934 toolforge: scale up coredns replicas
Open	None	T239404 toolforge: new k8s: evaluate DNS (coredns) autoscale options
Resolved	None	T333370 Toolforge k8s: network connetivity problems
Resolved	aborrero	T333379 toolforge kubernetes: create roll-reboot cookbook
Open	None	T356969 toolforge k8s automation: introduce option to force reboot each node directly
Open	None	T356970 toolforge k8s automation: introduce option to reboot a node if the uptime is higher than XYZ
Resolved	taavi	T333972 Toolforge: scale down grid engine nodes, scale up k8s workers (april 2023)

Event Timeline

hashar created this task.Apr 4 2023, 9:22 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 4 2023, 9:22 AM

hashar added a parent task: T330209: 1.41.0-wmf.3 deployment blockers.Apr 4 2023, 9:23 AM

taavi edited projects, added Toolforge; removed Cloud-Services.Apr 4 2023, 9:23 AM

This is blocking the MediaWiki train (T330209) and is thus an Unbreak Now! priority.

The toolforge bot at https://train-blockers.toolforge.org/api.php gave me an error after several seconds about a DNS failure:

<br />
<b>Warning</b>:  mysqli::__construct(): php_network_getaddresses: getaddrinfo failed: Temporary failure in name resolution in <b>/data/project/train-blockers/tool-train-blockers/utils.php</b> on line <b>43</b><br />
<br />
<b>Warning</b>:  mysqli::__construct(): (HY000/2002): php_network_getaddresses: getaddrinfo failed: Temporary failure in name resolution in <b>/data/project/train-blockers/tool-train-blockers/utils.php</b> on line <b>43</b><br />
{"error":"Connection failed: php_network_getaddresses: getaddrinfo failed: Temporary failure in name resolution"}

I don't know about stashbot.

We probably shouldn't have a dependency on toolforge in scap either right ? Maybe a try/catch with a prompt abort/continue should be added around those functions.

hashar mentioned this in T333924: scap deploy-promote depends on Toolforge service.Apr 4 2023, 9:38 AM

In T333922#8753582, @TheDJ wrote:

We probably shouldn't have a dependency on toolforge in snap either right ? Maybe a try/catch with a prompt abort/continue should be added around those functions.

Yes definitely and I have filed it as T333924.

Meanwhile stashbot came back:

09:24:53 → stashbot joined (~stashbot@wikimedia/bot/stashbot)

And https://train-blockers.toolforge.org/api.php gives result again, I am thus lowering the priority and it is no more a train blocker.

@aborrero and @taavi on IRC mentioned it is something related to Kubernetes. I will let them report the actions / debugging etc to mark this one resolved.

hashar removed a parent task: T330209: 1.41.0-wmf.3 deployment blockers.Apr 4 2023, 9:40 AM

I confirm something happened in the toolforge kubernetes cluster that required us to reboot the control nodes.

We are still investigating the root cause.

taavi renamed this task from toolforge tools not reachable (stashdot, train-blockers at least) to toolforge k8s control plane freezing.Apr 4 2023, 10:25 AM

taavi mentioned this in T333929: toolforge: Move k8s control plane nodes to larger instance flavors.

taavi mentioned this in T333931: toolforge: Move k8s etcd nodes to larger instance flavors.

taavi added subtasks: T333929: toolforge: Move k8s control plane nodes to larger instance flavors, T333931: toolforge: Move k8s etcd nodes to larger instance flavors.

taavi mentioned this in T333934: toolforge: scale up coredns replicas.Apr 4 2023, 10:37 AM

taavi added a subtask: T333934: toolforge: scale up coredns replicas.

aborrero renamed this task from toolforge k8s control plane freezing to toolforge k8s control plane freezing and other stability issues.Apr 4 2023, 11:48 AM

aborrero added a subtask: T333370: Toolforge k8s: network connetivity problems.

aborrero added a subtask: T333379: toolforge kubernetes: create roll-reboot cookbook.Apr 4 2023, 11:50 AM

komla subscribed.Apr 4 2023, 3:06 PM

aborrero added a subtask: T333972: Toolforge: scale down grid engine nodes, scale up k8s workers (april 2023).Apr 4 2023, 3:44 PM

aborrero changed the status of subtask T333379: toolforge kubernetes: create roll-reboot cookbook from Open to In Progress.Apr 5 2023, 11:31 AM

taavi closed subtask T333972: Toolforge: scale down grid engine nodes, scale up k8s workers (april 2023) as Resolved.Apr 5 2023, 2:56 PM

taavi closed subtask T333929: toolforge: Move k8s control plane nodes to larger instance flavors as Resolved.Apr 7 2023, 2:35 PM

Andrew closed subtask T333931: toolforge: Move k8s etcd nodes to larger instance flavors as Resolved.May 5 2023, 3:17 PM

aborrero changed the status of subtask T333379: toolforge kubernetes: create roll-reboot cookbook from In Progress to Stalled.Jun 7 2023, 10:40 AM

aborrero closed subtask T333379: toolforge kubernetes: create roll-reboot cookbook as Resolved.Feb 8 2024, 10:01 AM