⚓ T431826 July 2026 Bookworm/Trixie reboots: Data Platform SRE

bking created this task.Fri, Jul 10, 1:12 PM

bking changed the task status from Open to In Progress.

bking claimed this task.

bking triaged this task as Medium priority.

bking edited projects, added: Data-Platform-SRE (2026-07-03 - 2026-07-31); removed: Data-Platform-SRE.Fri, Jul 10, 1:19 PM

bking updated the task description. (Show Details)Fri, Jul 10, 1:21 PM

bking added subscribers: klausman, DPogorzelski-WMF.

colewhite removed a project: SRE Observability.Fri, Jul 10, 4:49 PM

Jelto removed a project: collaboration-services.Mon, Jul 13, 6:59 AM

BTullis moved this task from Backlog - project to Tracking on the Data-Platform-SRE (2026-07-03 - 2026-07-31) board.Tue, Jul 14, 2:25 PM

BTullis updated the task description. (Show Details)Wed, Jul 15, 11:51 AM

brouberol updated the task description. (Show Details)Wed, Jul 15, 2:26 PM

bking updated the task description. (Show Details)Wed, Jul 15, 2:49 PM

brouberol updated the task description. (Show Details)Wed, Jul 15, 2:50 PM

klausman updated the task description. (Show Details)Wed, Jul 15, 3:08 PM

bking updated the task description. (Show Details)Thu, Jul 16, 7:00 PM

Hey folks! IIUC this task is to track both Bookworm's and Trixie's updates, but I don't see a lot of hosts mentioned in T431659 (and a lot of ML ones, CC @klausman @DPogorzelski-WMF)

In T431826#12131542, @elukey wrote:

Hey folks! IIUC this task is to track both Bookworm's and Trixie's updates, but I don't see a lot of hosts mentioned in T431659 (and a lot of ML ones, CC @klausman @DPogorzelski-WMF)

Good catch, I have copied over the additional hosts and will update this ticket and the parent as I get to rebooting them.

taavi removed a project: cloud-services-team.Mon, Jul 20, 12:24 PM

ssingh removed a project: Traffic.Mon, Jul 20, 1:39 PM

bking updated the task description. (Show Details)Mon, Jul 20, 2:03 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-20T14:06:20Z] <bking@cumin2003> START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply security updates - bking@cumin2003 - T431826

Mentioned in SAL (#wikimedia-operations) [2026-07-20T14:59:01Z] <bking@cumin2003> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply security updates - bking@cumin2003 - T431826

bking updated the task description. (Show Details)Mon, Jul 20, 3:16 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-20T15:28:03Z] <bking@cumin2003> START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826

Mentioned in SAL (#wikimedia-operations) [2026-07-20T16:58:08Z] <bking@cumin2003> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826

Mentioned in SAL (#wikimedia-operations) [2026-07-20T17:50:45Z] <bking@cumin2003> START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826

Mentioned in SAL (#wikimedia-operations) [2026-07-20T18:50:18Z] <bking@cumin2003> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826

Thanks for pointing that out, @elukey . I've added the hosts from that ticket as well.

40/55 CODFW cirrussearch hosts have been rebooted. We'll update the checklist above when they're completely finished.

Mentioned in SAL (#wikimedia-operations) [2026-07-21T13:07:15Z] <bking@cumin2003> START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826

Mentioned in SAL (#wikimedia-operations) [2026-07-21T15:27:15Z] <bking@cumin2003> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826

Mentioned in SAL (#wikimedia-operations) [2026-07-21T15:29:56Z] <bking@cumin2003> START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826

Mentioned in SAL (#wikimedia-operations) [2026-07-21T16:42:43Z] <bking@cumin2003> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826

Mentioned in SAL (#wikimedia-operations) [2026-07-21T17:11:14Z] <bking@cumin2003> START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826

bking updated the task description. (Show Details)Tue, Jul 21, 5:51 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-21T20:22:20Z] <bking@cumin2003> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826

Mentioned in SAL (#wikimedia-operations) [2026-07-21T20:30:04Z] <bking@cumin2003> START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826

bking updated the task description. (Show Details)Tue, Jul 21, 8:39 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-21T22:23:19Z] <bking@cumin2003> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826

Mentioned in SAL (#wikimedia-operations) [2026-07-22T13:08:25Z] <bking@cumin2003> START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826

Mentioned in SAL (#wikimedia-operations) [2026-07-22T13:35:44Z] <bking@cumin2003> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826

Mentioned in SAL (#wikimedia-operations) [2026-07-22T14:39:34Z] <bking@cumin2003> START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826

Mentioned in SAL (#wikimedia-operations) [2026-07-22T15:11:20Z] <bking@cumin2003> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826

Mentioned in SAL (#wikimedia-operations) [2026-07-22T15:46:51Z] <bking@cumin2003> START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826

Mentioned in SAL (#wikimedia-operations) [2026-07-22T16:01:01Z] <bking@cumin2003> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826

bking updated the task description. (Show Details)Wed, Jul 22, 4:39 PM

bking moved this task from Tracking to In Progress on the Data-Platform-SRE (2026-07-03 - 2026-07-31) board.Thu, Jul 23, 7:37 PM

klausman updated the task description. (Show Details)Fri, Jul 24, 9:34 AM

All ML etcd VMs have been rebooted, will continue with ctrl plane VMs

klausman updated the task description. (Show Details)Fri, Jul 24, 10:10 AM

Also rebooted all of the ML/LW control plane VMs. Will do the staging workers next, but defer the prod workers until next week.

klausman updated the task description. (Show Details)Tue, Jul 28, 11:11 AM

LW staging workers (ml-staging2xxx) are all done, proceeding woth prod-codfw

LW/ML eqiad workers all done.

LW prod workers all done, leaving only the build and lab machines.

klausman updated the task description. (Show Details)Tue, Jul 28, 5:01 PM

As is tradition during reboots, I also found two machines with wobbly RAM (ml-serve2002/2004):

https://phabricator.wikimedia.org/T433476
https://phabricator.wikimedia.org/T433478

klausman updated the task description. (Show Details)Wed, Jul 29, 1:51 PM

		Status	Subtype	Assigned	Task
					Restricted Task
		In Progress		bking	T431826 July 2026 Bookworm/Trixie reboots: Data Platform SRE

July 2026 Bookworm/Trixie reboots: Data Platform SRE
Open, In Progress, MediumPublic
Actions

Description

Data Platform hosts

Machine Learning hosts

Related Objects
Search...

Event Timeline

July 2026 Bookworm/Trixie reboots: Data Platform SREOpen, In Progress, MediumPublicActions