Toolforge grid deployment/management automation
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	aborrero
	Jan 11 2022, 9:34 AM

Description

This task is to track work to automate several tasks related to grid deployment and maintenance (mostly by means of spicerack/cookbooks).

run grid-configurator (cookbook wmcs.toolforge.grid.reconfigure)
create instance (cookbook wmcs.toolforge.grid.add_node_to_cluster)
handle all grid exec node types
bastion deployment ??
operations in batches

Details

Subject	Repo	Branch	Lines +/-
wmcs: toolforge: grid: node: join: add support for nodeset query syntax	operations/cookbooks	wmcs	+64 -27
wmcs: toolforge: grid: pool: introduce support for nodeset-syntax host selection	operations/cookbooks	wmcs	+47 -17
wmcs: toolforge: introduce cookbook to run tests	operations/cookbooks	wmcs	+75 -0
wmcs: toolforge: grid: introduce cookbook to repool a node	operations/cookbooks	wmcs	+130 -10
toolforge: deploy test suite	operations/puppet	production	+101 -0
wmcs: relocate start_instance_with_prefix cookbook	operations/cookbooks	wmcs	+34 -32
wmcs: toolforge: relocate some node-specific cookbooks	operations/cookbooks	wmcs	+18 -18
wmcs: toolforge: add cookbooks to scale the grid with each node type	operations/cookbooks	wmcs	+259 -1
wmcs: toolforge: grid: factorized node creation cookbook	operations/cookbooks	wmcs	+193 -49

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
		Restricted Task
		Restricted Task
Resolved	• Bstorm	T169289 Tool Labs 2017-06-29 Labstore100[45] kernel upgrade issues
Resolved	MoritzMuehlenhoff	T169290 New anti-stackclash (4.9.25-1~bpo8+3 ) kernel super bad for NFS
Resolved	• Bstorm	T203254 labstore1004 and labstore1005 high load issues following upgrades
Open	None	T257945 NFS v4.1/2 as possible fix for elevated load and lock contention on our NFS servers
Resolved	Andrew	T289883 cloud-vps: actively deprecate/phase out use of Debian Stretch
Resolved	taavi	T306099 Cloud VPS "tools" project Stretch deprecation
Resolved	taavi	T306100 Cloud VPS "toolsbeta" project Stretch deprecation
Resolved	taavi	T275864 Toolforge: migrate to Debian Buster or later
Resolved	hashar	T249949 Stop using integration/composer and then archive the repo
Resolved	taavi	T287900 Switch Toolforge installation of "composer" to use the Debian package
Stalled	None	T305834 Cloud VPS: drop wmflabs names from profile::resolving::domain_search
Resolved	aborrero	T277653 Toolforge: add Debian Buster to the grid and eliminate Debian Stretch
Resolved	aborrero	T298948 Toolforge grid deployment/management automation
Declined	None	T300032 spicerack: introduce GridEngine controller

Event Timeline

aborrero triaged this task as Medium priority.Jan 11 2022, 9:34 AM

aborrero created this task.

aborrero moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.

aborrero updated the task description. (Show Details)Jan 11 2022, 10:19 AM

Change 753006 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/cookbooks@wmcs] wmcs: toolforge: grid: add cookbook to create an exec node

https://gerrit.wikimedia.org/r/753006

gerritbot added a project: Patch-For-Review.Jan 11 2022, 10:20 AM

Change 753017 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/cookbooks@wmcs] wmcs: toolforge: grid: factorized node creation cookbook

https://gerrit.wikimedia.org/r/753017

Change 753018 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/cookbooks@wmcs] wmcs: relocate start_instance_with_prefix cookbook

https://gerrit.wikimedia.org/r/753018

Change 753027 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/cookbooks@wmcs] wmcs: toolforge: relocate some node-specific cookbooks

https://gerrit.wikimedia.org/r/753027

Change 753017 merged by Arturo Borrero Gonzalez:

[operations/cookbooks@wmcs] wmcs: toolforge: grid: factorized node creation cookbook

https://gerrit.wikimedia.org/r/753017

Change 753006 merged by Arturo Borrero Gonzalez:

[operations/cookbooks@wmcs] wmcs: toolforge: add cookbooks to scale the grid with each node type

https://gerrit.wikimedia.org/r/753006

Change 753027 merged by Arturo Borrero Gonzalez:

[operations/cookbooks@wmcs] wmcs: toolforge: relocate some node-specific cookbooks

https://gerrit.wikimedia.org/r/753027

Change 753018 merged by Arturo Borrero Gonzalez:

[operations/cookbooks@wmcs] wmcs: relocate start_instance_with_prefix cookbook

https://gerrit.wikimedia.org/r/753018

Change 754555 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/cookbooks@wmcs] wmcs: toolforge: grid: introduce cookbook to repool a node

https://gerrit.wikimedia.org/r/754555

Change 754944 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/cookbooks@wmcs] wmcs: toolforge: grid: introduce cookbook to verify basic grid health

https://gerrit.wikimedia.org/r/754944

Change 755321 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] toolforge: deploy test suite

https://gerrit.wikimedia.org/r/755321

Change 755321 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] toolforge: deploy test suite

https://gerrit.wikimedia.org/r/755321

Change 754555 merged by Arturo Borrero Gonzalez:

[operations/cookbooks@wmcs] wmcs: toolforge: grid: introduce cookbook to repool a node

https://gerrit.wikimedia.org/r/754555

Change 754944 merged by Arturo Borrero Gonzalez:

[operations/cookbooks@wmcs] wmcs: toolforge: introduce cookbook to run tests

https://gerrit.wikimedia.org/r/754944

aborrero renamed this task from Toolforge grid deployment automation to Toolforge grid deployment/management automation.Jan 27 2022, 1:20 PM

aborrero updated the task description. (Show Details)

aborrero added a subtask: T300032: spicerack: introduce GridEngine controller.Feb 7 2022, 12:14 PM

aborrero changed the status of subtask T300032: spicerack: introduce GridEngine controller from Open to Stalled.

Change 761440 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/cookbooks@wmcs] wmcs: toolforge: grid: pool: introduce support for nodeset-syntax host selection

https://gerrit.wikimedia.org/r/761440

Change 761440 merged by Arturo Borrero Gonzalez:

[operations/cookbooks@wmcs] wmcs: toolforge: grid: pool: introduce support for nodeset-syntax host selection

https://gerrit.wikimedia.org/r/761440

Change 761618 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/cookbooks@wmcs] wmcs: toolforge: grid: node: join: add support for nodeset query syntax

https://gerrit.wikimedia.org/r/761618

Change 761618 merged by Arturo Borrero Gonzalez:

[operations/cookbooks@wmcs] wmcs: toolforge: grid: node: join: add support for nodeset query syntax

https://gerrit.wikimedia.org/r/761618

The most common operations are now automated.

This patch https://gerrit.wikimedia.org/r/761618 includes a way to use nodeset query syntax for batch operations. We could refactor that code if we want to introduce it into other cookbooks.

I decided that automation for bastion provisioning is not worth the effort as of today.

dcaro mentioned this in rCCKBc6edfbbc1f48: wmcs: toolforge: grid: factorized node creation cookbook.Dec 14 2022, 3:32 PM

dcaro mentioned this in rCCKB888b91f5c5c4: wmcs: toolforge: add cookbooks to scale the grid with each node type.

dcaro mentioned this in rCCKB1113f5c3c3ed: wmcs: toolforge: relocate some node-specific cookbooks.

dcaro mentioned this in rCCKBa67329260ee3: wmcs: relocate start_instance_with_prefix cookbook.

dcaro mentioned this in rCCKB60b90fbe1163: wmcs: toolforge: grid: introduce cookbook to repool a node.

dcaro mentioned this in rCCKBda2e489cec28: wmcs: toolforge: introduce cookbook to run tests.

dcaro mentioned this in rCCKBd42569c2ab4f: wmcs: toolforge: grid: pool: introduce support for nodeset-syntax host selection.

dcaro mentioned this in rCCKB8ac57ac4d692: wmcs: toolforge: grid: node: join: add support for nodeset query syntax.

taavi closed subtask T300032: spicerack: introduce GridEngine controller as Declined.Feb 8 2024, 3:34 PM

Maintenance_bot removed a project: Patch-For-Review.Feb 8 2024, 4:31 PM