Page MenuHomePhabricator

Upgrade Toolforge (Elastic|Open)Search cluster to Debian Bullseye
Closed, ResolvedPublic

Event Timeline

taavi added a subscriber: bd808.

Tentatively claiming. Seems like we have packages and Puppetization available for OpenSearch 2 on Bookworm, will need to check how to migrate data from the existing ES7 cluster to that.

T236606: Rebuild Toolforge elasticsearch cluster with Stretch or Buster was the last migration. Also cc-ing @bd808 in case you have a tool that could be used as a canary here.

Also cc-ing @bd808 in case you have a tool that could be used as a canary here.

Stashbot + sal.toolforge.org + bash.toolforge.org are the main tools I have in Toolforge using this cluster. Pointing a test deploy of sal or bash at new nodes should be relatively easy.

Another big user of the cluster is the https://csp-report.toolforge.org/. I would actually expect that it is the dominant writer to the current cluster.

taavi removed taavi as the assignee of this task.Jun 12 2024, 11:35 AM
Andrew renamed this task from Upgrade Toolforge (Elastic|Open)Search cluster to Debian Bullseye to Upgrade Toolforge (Elastic|Open)Search cluster to Debian Bookworm.Jun 27 2024, 7:58 PM
Andrew claimed this task.
Andrew renamed this task from Upgrade Toolforge (Elastic|Open)Search cluster to Debian Bookworm to Upgrade Toolforge (Elastic|Open)Search cluster to Debian Bullseye.Jul 1 2024, 11:11 PM

My plan to jump to bookworm was premature; sticking with elasticsearch and moving to Bullseye is the easier choice for now.

Change #1051216 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Toolforge elasticsearch haproxy: update CORS syntax for modern haproxy

https://gerrit.wikimedia.org/r/1051216

There are now three Bullseye nodes in the cluster, with working ES but non-working haproxy.

This seems to be a reasonable guide for removing the old nodes from the ES cluster without data loss

https://opster.com/guides/elasticsearch/operations/elasticsearch-remove-node/

dcaro triaged this task as High priority.Jul 2 2024, 1:23 PM

Change #1051216 merged by Andrew Bogott:

[operations/puppet@production] Toolforge elasticsearch haproxy: update CORS syntax for modern haproxy

https://gerrit.wikimedia.org/r/1051216

Mentioned in SAL (#wikimedia-cloud) [2024-07-02T17:16:40Z] <andrewbogott> draining (I hope) tools-elastic-3 and tools-elastic-1 for T311905

Change #1051444 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] profile::toolforge::elasticsearch::keepalived: keepalived interface from hiera

https://gerrit.wikimedia.org/r/1051444

Change #1051444 merged by Andrew Bogott:

[operations/puppet@production] profile::toolforge::elasticsearch::keepalived: keepalived interface from hiera

https://gerrit.wikimedia.org/r/1051444