Page MenuHomePhabricator
Create Task
Maniphest T381565

Move maps servers to Bookworm
Closed, ResolvedPublic
Actions

Assigned To
MoritzMuehlenhoff
Authored By
MoritzMuehlenhoff
Dec 5 2024, 11:33 AM
Tags
Referenced Files
F67979074: download.png
Oct 28 2025, 2:37 PM
F67979061: download.png
Oct 28 2025, 2:37 PM
F66747148: diff.png
Oct 13 2025, 8:31 AM
F66747146: b.png
Oct 13 2025, 8:31 AM
F66747144: a.png
Oct 13 2025, 8:31 AM
F66747137: diff.png
Oct 13 2025, 8:31 AM
F66747135: b.png
Oct 13 2025, 8:31 AM
F66747132: a.png
Oct 13 2025, 8:31 AM
View All 25 Files
Subscribers
Aklapper
elukey
Jgiannelos
MoritzMuehlenhoff
MSantos
Muehlenhoff
TheDJ
Waddie96

Description

kartotherian is being moved to wikikube. When that is complete, the server backend will be moved to Bookworm.

We'll be reusing six former ganeti nodes a test cluster to test the new stack (setup at T380144) until we eventually reimage the maps* nodes.

The high level plan is:

  • install a master bookworm node and fix all issues
  • install a replica bookworm node and fix all issues
  • the remaing replicas
  • test the OSM import on the new cluster
  • point wikikube/staging to use the bookworm cluster
  • if this works fine, we can enable expiration events for bookworm and disable it for buster
  • move the prod pods to the new cluster and eventually
  • failover production traffic in codfw to the boomworm-test cluster
  • Install the new eqiad/maps nodes with Bookworm
  • Install the new codfw/maps nodes with Bookworm
  • Failover to the new nodes
  • Decom the old servers

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
sre.maps.roll-restart-reboot: Adapt for Bookwormoperations/cookbooksmaster+1 -1
sre.maps.roll-restart-reboot-master: Adapt to Bookworm changesoperations/cookbooksmaster+1 -1
postgis: Remove support for busteroperations/puppetproduction+2 -12
Remove obsolete stub secretslabs/privatemaster+0 -2
maps::osm_replica: Explicitly pass the replication passwordoperations/puppetproduction+1 -0
Update secrets for tilerator->tegola renamelabs/privatemaster+1 -1
kubernetes: add maps-staging-codfw IPsoperations/puppetproduction+1 -0
osm_replica: Fix Hiera variableoperations/puppetproduction+1 -1
osm_master: Store kartotherian and tegola passwordsoperations/puppetproduction+20 -0
Remove the now unused tilerator_passoperations/puppetproduction+0 -3
Properly rename tilerator_pass variableoperations/puppetproduction+3 -1
Turn paging on for kartotherianoperations/puppetproduction+1 -1
DNM: Prep patch for removal of old maps rolesoperations/puppetproduction+0 -78
Switch maps-test2001 to maps::stagingoperations/puppetproduction+1 -1
Remove tilerator-admin groupoperations/puppetproduction+1 -8
Remove obsolete grants fileoperations/puppetproduction+0 -11
Remove a lot of historical stub secretslabs/privatemaster+0 -124
Fix secret namelabs/privatemaster+2 -2
Fix aliasoperations/puppetproduction+1 -1
osm_replica: Remove support for pre Bookwormoperations/puppetproduction+12 -32
Fix cumin alias for mapsoperations/puppetproduction+1 -1
osm_master: Remove support for pre Bookwormoperations/puppetproduction+21 -92
Fix Cumin aliases for maps following removal of buster nodesoperations/puppetproduction+3 -4
preseed: Remove old maps nodesoperations/puppetproduction+0 -3
Remove maps-admins groupoperations/puppetproduction+1 -9
Remove kartotherian-admin groupoperations/puppetproduction+1 -13
Remove legacy maps rolesoperations/puppetproduction+0 -69
Remove old maps nodes from site.pp and Hieraoperations/puppetproduction+0 -90
Enable tile invalidation for the new maps nodes in codfwoperations/puppetproduction+1 -1
Add separate role for single-node staging DBoperations/puppetproduction+44 -0
maps/bookworm: Re-enable monitoringoperations/puppetproduction+0 -2
Re-enable monitoring for maps/bookwormoperations/puppetproduction+0 -2
maps: Stop installing osm2pgsql and osmborderoperations/puppetproduction+0 -2
osm_sync_lag.sh: Fix default to current directoryoperations/puppetproduction+1 -1
Shift tile eqiad invalidation to the bookworm masteroperations/puppetproduction+3 -2
osm: Remove obsolete spec filesoperations/puppetproduction+0 -54
imposm-initial-import: Read the permissions from a fileoperations/puppetproduction+2 -2
role::maps::master_bookworm: enable tile invalidation in eqiadoperations/puppetproduction+0 -2
services: move tegola eqiad to the 'tegola' usernameoperations/deployment-chartsmaster+1 -9
kubernetes: add the maps bookworm eqiad external service configoperations/puppetproduction+2 -0
services: move tegola and kartotherian's eqiad configs to the new stackoperations/deployment-chartsmaster+11 -18
role::maps::master_bookworm: fix EG stream name in eqiadoperations/puppetproduction+3 -0
role::maps: increase max-conns and shared buffers on Bookwormoperations/puppetproduction+14 -1
Syncronise the Hiera settings for the bookworm maps replicasoperations/puppetproduction+1 -4
Syncronise the Hiera settings for the bookworm maps masters to the role settingsoperations/puppetproduction+12 -39
Remove maps roles from maps-test*operations/puppetproduction+2 -2
services: reduce tegola's cronjob paralleismoperations/deployment-chartsmaster+1 -1
Enable tiles invalidation for maps2011operations/puppetproduction+0 -2
osm: refactor swift scripts and make event-template dynamicoperations/puppetproduction+87 -64
Disable tiles invalidation on maps-test2001operations/puppetproduction+1 -0
Apply replica role to maps1012-1014operations/puppetproduction+11 -1
osm_master: Create /etc/wikimedia directoryoperations/puppetproduction+6 -0
Also Add maps1012-maps1014 as maps nodesoperations/puppetproduction+3 -0
charts: add separate requests/limits for tegola's cronjoboperations/deployment-chartsmaster+15 -3
Add maps1012 to maps1014 as replicasoperations/puppetproduction+8 -1
profile::maps::osm_master: refactor postgres grantsoperations/puppetproduction+5 -12
role::maps::master: enable planet sync in codfwoperations/puppetproduction+1 -1
imposm-initial-import: Set service passwordsoperations/puppetproduction+11 -1
Don't set profile::maps::osm_master::tilerator_pass in role defaultoperations/puppetproduction+1 -1
services: move tegola's codfw postgres config to the new stackoperations/deployment-chartsmaster+3 -7
services: move kartotherian and tegola to the new codfw stackoperations/deployment-chartsmaster+9 -21
Add maps2012-maps2014 as replicasoperations/puppetproduction+13 -1
Enable OSM and waterlines sync on maps1011operations/puppetproduction+2 -2
Reapply master_bookworm role to maps1011operations/puppetproduction+1 -1
Disable replication timers in eqiad/bookwormoperations/puppetproduction+2 -2
Reset maps1011operations/puppetproduction+1 -1
maps2011: Re-enable imports of OSM and waterlinesoperations/puppetproduction+2 -2
Re-re-add maps2011 as maps masteroperations/puppetproduction+15 -1
Reset maps2011operations/puppetproduction+1 -15
Re-add maps2011operations/puppetproduction+15 -1
imposm-initial-import: Fix check whether imposm is runningoperations/puppetproduction+2 -3
Make maps2012-2014 replica nodesoperations/puppetproduction+14 -2
Enable the regular imports of the OSM updates and water lines on maps2011operations/puppetproduction+2 -2
imposm-initial-import: Add the reindex step to the scriptoperations/puppetproduction+8 -0
maps2011: Enable timers for OSM sync and waterlines importoperations/puppetproduction+2 -2
maps1011: Enable timers for OSM sync and waterlines importoperations/puppetproduction+2 -2
maps1011: Fix typooperations/puppetproduction+1 -1
Setup maps1011 as master node for new maps/eqiad serversoperations/puppetproduction+19 -1
Setup maps2011 as master node for new maps/codfw serversoperations/puppetproduction+19 -1
role::maps: fix tegola_swift_container for codfwoperations/puppetproduction+1 -1
maps: Move the setting for planet_sync_hours to the common role settingoperations/puppetproduction+2 -1
maps: Remove disable_tile_generation_timeroperations/puppetproduction+15 -44
maps: Moving the setting to disable paging to the role leveloperations/puppetproduction+8 -23
maps: Remove unused Hiera optionoperations/puppetproduction+0 -4
services: exclude postgres masters from confs in tegola/kartotherianoperations/deployment-chartsmaster+13 -33
Also update partman recipe for new maps/eqiad nodesoperations/puppetproduction+1 -1
Update partman config for the new maps nodes in codfwoperations/puppetproduction+1 -1
Add EFI variant of raid5-4dev.cfgoperations/puppetproduction+15 -0
Stop applying maps-admins to maps Bookworm rolesoperations/puppetproduction+0 -2
role::maps::master: enable import times on maps-testoperations/puppetproduction+8 -8
services: move kartotherian codfw to the maps-test postgres clusteroperations/deployment-chartsmaster+8 -8
services: set user tegola for Tegola's codfwoperations/deployment-chartsmaster+4 -0
services: add missing port to Tegola's staging tcp proxy configoperations/deployment-chartsmaster+1 -0
services: fix Tegola's staging postgres configoperations/deployment-chartsmaster+7 -12
services: move tegola in staging to maps-test2*operations/deployment-chartsmaster+16 -7
services: configure tegola in codfw to use maps-testoperations/deployment-chartsmaster+9 -13
maps: Add tegola user in DB, mark tilerator for removaloperations/puppetproduction+44 -15
admin: Remove tilerator/tileratorui system usersoperations/puppetproduction+0 -2
admin: Empty out kartotherian-adminoperations/puppetproduction+2 -1
deployment: Remove tilerator from scap::sourcesoperations/puppetproduction+0 -3
mtail: Remove tilerator from testsoperations/puppetproduction+7 -3
DNM: tilerator: Remove as much as possible of the last cruftoperations/puppetproduction+0 -40
profile::maps::osm_master: allow tilerator for kubepods on Bookwormoperations/puppetproduction+5 -7
EventStreamConfig: add the maps.tiles_change_bookworm streamoperations/mediawiki-configmaster+9 -0
profile::maps: add default privileges for kartotherianoperations/puppetproduction+1 -0
Move Kartotherian/staging to the new Bookworm nodesoperations/deployment-chartsmaster+7 -7
Add maps-bookworm aliasoperations/puppetproduction+1 -0
kubernetes: add maps-test codfw as external serviceoperations/puppetproduction+2 -0
Enable the remaining two maps nodes as replicasoperations/puppetproduction+1 -5
Add two more maps-replicas on Bookwormoperations/puppetproduction+2 -2
Add maps-replica2002 as Bookworm maps replicaoperations/puppetproduction+5 -1
maps: update grants-db-bookworm.sqloperations/puppetproduction+2 -0
maps/osm: Make create_layers_functions independent of Kartotherian checkoutoperations/puppetproduction+924 -4
maps/osm: Move Kartotherian SQL config into Puppetoperations/puppetproduction+751 -17
imposm-initial-import: Enable the osmupdater DB permissions earlieroperations/puppetproduction+1 -1
maps: Delete obsolete osm-initial-import scriptoperations/puppetproduction+0 -154
imposm-initial-import: Follow 302 redirects when fetching the checksumoperations/puppetproduction+1 -1
Reapply maps_bookworm role to maps-test2001operations/puppetproduction+1 -1
maps/test: Adapt partman recipeoperations/puppetproduction+1 -1
Add raid5-4dev.cfg Partman recipeoperations/puppetproduction+15 -0
Enable maps-test2003 to maps-test2006 as additional maps bookworm replicasoperations/puppetproduction+13 -10
Apply maps/replica role to maps-test2002operations/puppetproduction+1 -1
Apply maps master role to maps-test2001operations/puppetproduction+1 -1
Reimage maps-test2001/2002 to insetup roleoperations/puppetproduction+2 -2
postgresl/osm_master: Make postgresql listen on ipv4 on Bookwormoperations/puppetproduction+13 -1
Move maps-test2002 to replica_bookwormoperations/puppetproduction+1 -1
Switch maps-test2001 to master_bookwormoperations/puppetproduction+1 -1
Move maps-test2001/2002 back to insetupoperations/puppetproduction+2 -2
osm: Handle new requirements for Postgres replication slotsoperations/puppetproduction+29 -13
maps/bookworm: Cleanup confusing Hiera settings for postgresql replication slotsoperations/puppetproduction+2 -4
osm_master: Tighten replication slot name on Bookwormoperations/puppetproduction+5 -1
Make maps-test2002 a bookworm maps replicaoperations/puppetproduction+10 -3
Add stub secrets for bookworm replica rolelabs/privatemaster+4 -0
Make maps-test2001 a bookworm maps master nodeoperations/puppetproduction+6 -2
maps: Add a separate Hiera option to control the waterlines importoperations/puppetproduction+15 -8
maps_bookworm: Initially disable replication/tile gen timersoperations/puppetproduction+2 -2
osm_master: Provide a dummy variable for tilerator on bookworm rolesoperations/puppetproduction+1 -0
profile::maps::osm_master: Make tilerator_pass optionaloperations/puppetproduction+1 -1
Add stub secrets for new master_bookworm roleslabs/privatemaster+8 -0
Extend commentoperations/puppetproduction+2 -0
Manage tile invalidation with a separate Hiera settingoperations/puppetproduction+9 -1
Don't setup database config for tilerator on bookwormoperations/puppetproduction+61 -32
Add missing Hiera settings for new bookworm master rolesoperations/puppetproduction+16 -0
Add separate maps master/replica roles for the new Bookworm setupoperations/puppetproduction+53 -0
Remove profile::java from maps hostsoperations/puppetproduction+0 -8
postgresql: Add support for Bookwormoperations/puppetproduction+3 -4
maps::osm_master: Inline osm classoperations/puppetproduction+5 -10
osmborder: Build for Bookworm and bump debhelper compat to 13operations/debs/osmbordermaster+8 -3
osm: On Bookworm create OSM users using system::sysuseroperations/puppetproduction+29 -15
planet_sync: Remove obsolete optionsoperations/puppetproduction+0 -16
planet_sync: Cleanup time handlingoperations/puppetproduction+42 -44
maps::master Add toggle for planet syncoperations/puppetproduction+2 -9
Configure new maps nodes with nftablesoperations/puppetproduction+6 -0
maps/postgresql: Support bookwormoperations/puppetproduction+4 -14
maps: Allow disabling the installation of kartotherianoperations/puppetproduction+12 -11
osm_master: Avoid Ferm-specific syntaxoperations/puppetproduction+3 -5
maps: Remove support for osm2pgsql as OSM engineoperations/puppetproduction+17 -271
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
elukey added a comment.Oct 21 2025, 2:19 PM

Of course I am stupid, I re-executed the grants on maps2011 that was already ok. This was the correct action:

elukey@maps1011:~$ sudo -u postgres psql -f /usr/local/bin/maps-grants-gis.sql -d gis
ALTER ROLE
GRANT
GRANT
GRANT
GRANT
ALTER DEFAULT PRIVILEGES
GRANT
GRANT
GRANT
GRANT
GRANT
ALTER DEFAULT PRIVILEGES
GRANT
GRANT
GRANT
GRANT
ALTER DEFAULT PRIVILEGES
ALTER DEFAULT PRIVILEGES
elukey added a comment.Oct 21 2025, 2:43 PM

Started the cache warm-up for eqiad, using tegola-swift-codfw-v003 as reference.

To check progress:

  • Kafka events consumption from tegola pregen (starts only once for each day): dashboard
gerritbot added a comment.Oct 25 2025, 7:57 AM

Change #1198614 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] role::maps::master_bookworm: enable tile invalidation in eqiad

https://gerrit.wikimedia.org/r/1198614

elukey added a comment.Oct 25 2025, 8:00 AM

Bootstrap completed, and it looks good:

root@thanos-fe1004:~# swift stat tegola-swift-codfw-v003 | grep Objects
                      Objects: 95190783
root@thanos-fe1004:~# swift stat tegola-swift-eqiad-v003 | grep Objects
                      Objects: 94959489

Now we just need to re-enabled tile invalidation and let it catch up.

gerritbot added a comment.Oct 25 2025, 8:05 AM

Change #1198614 merged by Elukey:

[operations/puppet@production] role::maps::master_bookworm: enable tile invalidation in eqiad

https://gerrit.wikimedia.org/r/1198614

gerritbot added a comment.Oct 27 2025, 8:44 AM

Change #1198908 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] imposm-initial-import: Read the permissions from a file

https://gerrit.wikimedia.org/r/1198908

gerritbot added a comment.Oct 27 2025, 9:34 AM

Change #1198908 merged by Muehlenhoff:

[operations/puppet@production] imposm-initial-import: Read the permissions from a file

https://gerrit.wikimedia.org/r/1198908

gerritbot added a comment.Oct 27 2025, 3:49 PM

Change #1199005 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] osm_master: Remove support for pre Bookworm

https://gerrit.wikimedia.org/r/1199005

gerritbot added a comment.Oct 28 2025, 11:27 AM

Change #1199260 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] osm: Remove obsolete spec files

https://gerrit.wikimedia.org/r/1199260

gerritbot added a comment.Oct 28 2025, 11:36 AM

Change #1199260 merged by Muehlenhoff:

[operations/puppet@production] osm: Remove obsolete spec files

https://gerrit.wikimedia.org/r/1199260

gerritbot added a comment.Oct 28 2025, 11:47 AM

Change #1199265 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] osm_sync_lag.sh: Fix default to current directory

https://gerrit.wikimedia.org/r/1199265

gerritbot added a comment.Oct 28 2025, 12:08 PM

Change #1199271 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] maps: Stop installing osm2pgsql and osmborder

https://gerrit.wikimedia.org/r/1199271

gerritbot added a comment.Oct 28 2025, 1:10 PM

Change #1195717 abandoned by Muehlenhoff:

[operations/puppet@production] Shift tile eqiad invalidation to the bookworm master

Reason:

Old patch, no longer needed

https://gerrit.wikimedia.org/r/1195717

elukey added a comment.Oct 28 2025, 2:37 PM

Ran the diff testing tool between eqiad and codfw:

|      |     ssim |
|-----:|---------:|
| 0.05 | 0.974994 |
| 0.1  | 0.990161 |
| 0.2  | 0.998943 |
| 0.25 | 0.999358 |
| 0.5  | 0.999917 |
| 0.75 | 1        |
| 0.9  | 1        |
| 0.95 | 1        |
| 0.99 | 1        |

|      |   diff_latency |
|-----:|---------------:|
| 0.1  |     -2332.79   |
| 0.2  |     -1574.56   |
| 0.25 |     -1373.6    |
| 0.5  |      -601.896  |
| 0.75 |      -120.024  |
| 0.9  |        69.9375 |
| 0.95 |       178.929  |
| 0.99 |       596.683  |

download.png (433×571 px, 13 KB)

download.png (833×1 px, 61 KB)

The diff in latency can be probably explained by the fact that codfw is currently handling the whole load, so its performances are affected. Overall I think we are good!

elukey added a comment.Oct 28 2025, 3:06 PM

@TheDJ Hi! As FYI we now have eqiad and codfw on the new stack, both eqiad and codfw are pooled :)

elukey added a comment.Oct 29 2025, 10:10 AM

I just moved all the traffic to eqiad depooling codfw. This is the last test to make sure the new stack can handle all traffic in case it is needed.

gerritbot added a comment.Oct 29 2025, 11:45 AM

Change #1199265 merged by Muehlenhoff:

[operations/puppet@production] osm_sync_lag.sh: Fix default to current directory

https://gerrit.wikimedia.org/r/1199265

gerritbot added a comment.Oct 29 2025, 11:52 AM

Change #1199271 merged by Muehlenhoff:

[operations/puppet@production] maps: Stop installing osm2pgsql and osmborder

https://gerrit.wikimedia.org/r/1199271

elukey added a comment.Oct 30 2025, 8:12 AM

Repooled codfw after the eqiad-only test, I think we are good!

We'll wait a couple more days to be sure, but from next week we should start decomming the old hardware on Buster.

elukey updated the task description. (Show Details)Oct 30 2025, 9:19 AM
gerritbot added a comment.Oct 30 2025, 11:28 AM

Change #1200030 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Re-enable monitoring for maps/bookworm

https://gerrit.wikimedia.org/r/1200030

gerritbot added a comment.Nov 3 2025, 8:09 AM

Change #1200030 merged by Muehlenhoff:

[operations/puppet@production] Re-enable monitoring for maps/bookworm

https://gerrit.wikimedia.org/r/1200030

TheDJ added a comment.Nov 3 2025, 10:24 AM

Not sure if this font issue T408884 is related, but it was reported around the switch to the new services, so might be worth double checking if the k8s images have the full font stack that we usually use on media servers.

elukey added a comment.Nov 3 2025, 10:38 AM
In T381565#11334866, @TheDJ wrote:

Not sure if this font issue T408884 is related, but it was reported around the switch to the new services, so might be worth double checking if the k8s images have the full font stack that we usually use on media servers.

@TheDJ thanks for reporting! In theory tegola and kartotherian, running on k8s, have been the same for ages, we just switched the postgres databases. I'll try to follow up in the task, but I'd need more data about what you mean with "media servers" to track down the problem.

gerritbot added a comment.Nov 3 2025, 2:35 PM

Change #1201077 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add separate role for single-node staging DB

https://gerrit.wikimedia.org/r/1201077

gerritbot added a comment.Nov 3 2025, 4:19 PM

Change #1185048 abandoned by Muehlenhoff:

[operations/puppet@production] maps/bookworm: Re-enable monitoring

Reason:

Duplicate, variant already merged

https://gerrit.wikimedia.org/r/1185048

MoritzMuehlenhoff added a comment.Nov 4 2025, 1:02 PM

We'll keep maps-test2001 around as a separate staging system (single PG master, no replicas, a separate role::maps::staging will be used (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1201077)) to eventually point the wikikube staging endpoint to it. This will allow us to test changes like https://gerrit.wikimedia.org/r/c/mediawiki/services/kartotherian/+/1201020 or https://phabricator.wikimedia.org/T407491 more systematically w/o risk to the main production setup.

It's an old Ganeti node, but it should be good for the remainder of the FY and we can refresh it in the next one.

gerritbot added a comment.Nov 4 2025, 2:25 PM

Change #1201077 merged by Muehlenhoff:

[operations/puppet@production] Add separate role for single-node staging DB

https://gerrit.wikimedia.org/r/1201077

gerritbot added a comment.Nov 4 2025, 2:30 PM

Change #1201690 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Switch maps-test2001 to maps::staging

https://gerrit.wikimedia.org/r/1201690

elukey moved this task from Backlog to Waiting for others on the User-Elukey board.Nov 4 2025, 3:28 PM
gerritbot added a comment.Nov 4 2025, 4:28 PM

Change #1188345 abandoned by Muehlenhoff:

[operations/puppet@production] Enable tile invalidation for the new maps nodes in codfw

Reason:

Duplicate, different patch was merged

https://gerrit.wikimedia.org/r/1188345

Stashbot added a comment.Nov 5 2025, 10:06 AM

Mentioned in SAL (#wikimedia-operations) [2025-11-05T10:06:22Z] <moritzm> disabling Puppet on buster maps nodes for pending decom T381565

ops-monitoring-bot added a comment.Nov 5 2025, 1:41 PM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: maps2010.codfw.wmnet

  • maps2010.codfw.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
ops-monitoring-bot added a comment.Nov 5 2025, 1:53 PM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: maps2005.codfw.wmnet

  • maps2005.codfw.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
MoritzMuehlenhoff added a subtask: T409291: decommission maps2005/maps2006/maps2007/maps2008/maps2009/map2010.Nov 5 2025, 1:56 PM
gerritbot added a comment.Nov 5 2025, 3:29 PM

Change #1202176 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove old maps nodes from site.pp and Hiera

https://gerrit.wikimedia.org/r/1202176

MoritzMuehlenhoff added a subtask: T409399: decommission maps1005/maps1006/maps1007/maps1008/maps1009/map1010.Nov 6 2025, 8:25 AM
gerritbot added a comment.Nov 6 2025, 9:58 AM

Change #1202176 merged by Muehlenhoff:

[operations/puppet@production] Remove old maps nodes from site.pp and Hiera

https://gerrit.wikimedia.org/r/1202176

gerritbot added a comment.Nov 6 2025, 10:22 AM

Change #1202664 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] osm_replica: Remove support for pre Bookworm

https://gerrit.wikimedia.org/r/1202664

MoritzMuehlenhoff updated the task description. (Show Details)Nov 6 2025, 10:29 AM
gerritbot added a comment.Nov 6 2025, 12:21 PM

Change #1202686 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove legacy maps roles

https://gerrit.wikimedia.org/r/1202686

gerritbot added a comment.Nov 6 2025, 12:25 PM

Change #1202687 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove kartotherian-admin group

https://gerrit.wikimedia.org/r/1202687

gerritbot added a comment.Nov 6 2025, 12:30 PM

Change #1202689 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove maps-admins group

https://gerrit.wikimedia.org/r/1202689

gerritbot added a comment.Nov 6 2025, 2:08 PM

Change #1202686 merged by Muehlenhoff:

[operations/puppet@production] Remove legacy maps roles

https://gerrit.wikimedia.org/r/1202686

gerritbot added a comment.Nov 6 2025, 2:11 PM

Change #1202687 merged by Muehlenhoff:

[operations/puppet@production] Remove kartotherian-admin group

https://gerrit.wikimedia.org/r/1202687

gerritbot added a comment.Nov 6 2025, 2:17 PM

Change #1202689 merged by Muehlenhoff:

[operations/puppet@production] Remove maps-admins group

https://gerrit.wikimedia.org/r/1202689

MoritzMuehlenhoff mentioned this in T409528: Setup a maps staging DB.Nov 7 2025, 8:03 AM
MoritzMuehlenhoff added a subtask: T409529: decommission maps-test2002/maps-test2003/maps-test2004/maps-test2005/maps-test2006.Nov 7 2025, 8:18 AM
gerritbot added a comment.Nov 7 2025, 1:27 PM

Change #1203013 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] preseed: Remove old maps nodes

https://gerrit.wikimedia.org/r/1203013

gerritbot added a comment.Nov 7 2025, 2:16 PM

Change #1203013 merged by Muehlenhoff:

[operations/puppet@production] preseed: Remove old maps nodes

https://gerrit.wikimedia.org/r/1203013

gerritbot added a comment.Nov 7 2025, 2:18 PM

Change #1203023 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Fix Cumin aliases for maps following removal of buster nodes

https://gerrit.wikimedia.org/r/1203023

gerritbot added a comment.Nov 7 2025, 3:49 PM

Change #1203023 merged by Muehlenhoff:

[operations/puppet@production] Fix Cumin aliases for maps following removal of buster nodes

https://gerrit.wikimedia.org/r/1203023

Jclark-ctr closed subtask T409399: decommission maps1005/maps1006/maps1007/maps1008/maps1009/map1010 as Resolved.Nov 7 2025, 7:24 PM
gerritbot added a comment.Nov 10 2025, 9:30 AM

Change #1199005 merged by Muehlenhoff:

[operations/puppet@production] osm_master: Remove support for pre Bookworm

https://gerrit.wikimedia.org/r/1199005

gerritbot added a comment.Nov 10 2025, 9:50 AM

Change #1203390 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Fix cumin alias for maps

https://gerrit.wikimedia.org/r/1203390

gerritbot added a comment.Nov 11 2025, 2:12 PM

Change #1203390 merged by Muehlenhoff:

[operations/puppet@production] Fix cumin alias for maps

https://gerrit.wikimedia.org/r/1203390

gerritbot added a comment.Nov 11 2025, 2:24 PM

Change #1202664 merged by Muehlenhoff:

[operations/puppet@production] osm_replica: Remove support for pre Bookworm

https://gerrit.wikimedia.org/r/1202664

gerritbot added a comment.Nov 11 2025, 3:11 PM

Change #1203835 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Turn paging on for kartotherian

https://gerrit.wikimedia.org/r/1203835

Jhancock.wm closed subtask T409529: decommission maps-test2002/maps-test2003/maps-test2004/maps-test2005/maps-test2006 as Resolved.Nov 12 2025, 3:57 PM
Jhancock.wm closed subtask T409291: decommission maps2005/maps2006/maps2007/maps2008/maps2009/map2010 as Resolved.Nov 12 2025, 4:56 PM
gerritbot added a comment.Nov 13 2025, 12:24 PM

Change #1204844 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Fix alias

https://gerrit.wikimedia.org/r/1204844

gerritbot added a comment.Nov 13 2025, 12:35 PM

Change #1204844 merged by Muehlenhoff:

[operations/puppet@production] Fix alias

https://gerrit.wikimedia.org/r/1204844

gerritbot added a comment.Nov 13 2025, 3:36 PM

Change #1204900 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Properly rename tilerator_pass variable

https://gerrit.wikimedia.org/r/1204900

gerritbot added a comment.Nov 13 2025, 4:00 PM

Change #1204913 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[labs/private@master] Remove a lot of historical stub secrets

https://gerrit.wikimedia.org/r/1204913

gerritbot added a comment.Nov 13 2025, 4:02 PM

Change #1204914 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove the new unused tilerator_pass

https://gerrit.wikimedia.org/r/1204914

gerritbot added a comment.Nov 13 2025, 4:06 PM

Change #1204916 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove obsolete grants file

https://gerrit.wikimedia.org/r/1204916

gerritbot added a comment.Nov 14 2025, 7:11 AM

Change #1205007 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[labs/private@master] Fix secret name

https://gerrit.wikimedia.org/r/1205007

gerritbot added a comment.Nov 14 2025, 7:13 AM

Change #1205007 merged by Muehlenhoff:

[labs/private@master] Fix secret name

https://gerrit.wikimedia.org/r/1205007

gerritbot added a comment.Nov 14 2025, 8:44 AM

Change #1204913 merged by Muehlenhoff:

[labs/private@master] Remove a lot of historical stub secrets

https://gerrit.wikimedia.org/r/1204913

gerritbot added a comment.Nov 14 2025, 9:57 AM

Change #1205089 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove tilerator-admin group

https://gerrit.wikimedia.org/r/1205089

gerritbot added a comment.Mon, Nov 17, 11:27 AM

Change #1204916 merged by Muehlenhoff:

[operations/puppet@production] Remove obsolete grants file

https://gerrit.wikimedia.org/r/1204916

gerritbot added a comment.Mon, Nov 17, 12:24 PM

Change #1205089 merged by Muehlenhoff:

[operations/puppet@production] Remove tilerator-admin group

https://gerrit.wikimedia.org/r/1205089

gerritbot added a comment.Mon, Nov 17, 12:53 PM

Change #1201690 merged by Muehlenhoff:

[operations/puppet@production] Switch maps-test2001 to maps::staging

https://gerrit.wikimedia.org/r/1201690

gerritbot added a comment.Mon, Nov 17, 5:47 PM

Change #1169636 abandoned by Alexandros Kosiaris:

[operations/puppet@production] DNM: Prep patch for removal of old maps roles

Reason:

No longer applicable.

https://gerrit.wikimedia.org/r/1169636

gerritbot added a comment.Tue, Nov 25, 9:48 AM

Change #1204900 merged by Muehlenhoff:

[operations/puppet@production] Properly rename tilerator_pass variable

https://gerrit.wikimedia.org/r/1204900

gerritbot added a comment.Tue, Nov 25, 10:18 AM

Change #1204914 merged by Muehlenhoff:

[operations/puppet@production] Remove the now unused tilerator_pass

https://gerrit.wikimedia.org/r/1204914

gerritbot added a comment.Tue, Nov 25, 11:20 AM

Change #1128891 merged by Muehlenhoff:

[operations/puppet@production] osm_replica: Fix Hiera variable

https://gerrit.wikimedia.org/r/1128891

• bd808 mentioned this in T411004: No Puppet resources found on instance deployment-maps-master02 on project deployment-prep.Tue, Nov 25, 10:54 PM
• bd808 added a subtask: T411004: No Puppet resources found on instance deployment-maps-master02 on project deployment-prep.Tue, Nov 25, 11:27 PM
gerritbot added a comment.Wed, Nov 26, 8:59 AM

Change #1211606 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] kubernetes: add maps-staging-codfw IPs

https://gerrit.wikimedia.org/r/1211606

gerritbot added a comment.Wed, Nov 26, 9:02 AM

Change #1211606 merged by Elukey:

[operations/puppet@production] kubernetes: add maps-staging-codfw IPs

https://gerrit.wikimedia.org/r/1211606

gerritbot added a comment.Wed, Nov 26, 10:11 AM

Change #1211621 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] maps::osm_replica: Explicitly pass the replication password

https://gerrit.wikimedia.org/r/1211621

gerritbot added a comment.Wed, Nov 26, 10:43 AM

Change #1211625 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[labs/private@master] Update secrets for tilerator->tegola rename

https://gerrit.wikimedia.org/r/1211625

gerritbot added a comment.Wed, Nov 26, 10:48 AM

Change #1211625 merged by Muehlenhoff:

[labs/private@master] Update secrets for tilerator->tegola rename

https://gerrit.wikimedia.org/r/1211625

gerritbot added a comment.Wed, Nov 26, 11:16 AM

Change #1211621 merged by Muehlenhoff:

[operations/puppet@production] maps::osm_replica: Explicitly pass the replication password

https://gerrit.wikimedia.org/r/1211621

gerritbot added a comment.Wed, Nov 26, 2:17 PM

Change #1211684 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[labs/private@master] Remove obsolete stub secrets

https://gerrit.wikimedia.org/r/1211684

MoritzMuehlenhoff closed this task as Resolved.Wed, Nov 26, 4:33 PM
MoritzMuehlenhoff claimed this task.

This is complete. Luca and myself made a total of 122 commits to puppet.git (plus surely a few where me missed to tag the task) for:

  • After initial tests on repurposed old Ganeti nodes: An installation on new servers using standard server types (and reduced from six to four nodes per site due to more powerful specs) was made and the old nodes decommissioned
  • Karthotherian was moved to Wikikube and the new nodes no longer have any traces of it, allowing all future updates to be less complex
  • The new nodes use Bookworm instead of Buster, refreshing the OS stack in many regards, e.g. moving Postgresql from 11 to 15 and postgis from 3.1 to 3.5. This required several fixups, e.g.
    • Postgresql is now stricter in the configuration of replication slots and max_wal_senders needs to be identical on masters and replicas.
    • Grants needed to updated: On Postgres 15 it is no longer allowed by default to create tables in the public namespace.
    • Starting with Postgres 15 the hashing changed from md5 (for which we could precompute the hash in the postgresql user defines) to scram-sha1, which gets salted with a value we've so far been unable to extract to compare the hash in Puppet, so password changes are currently not supported until T326325 is resolved. To address that, these are now configured as part of the initial OSM import.
  • Installed with UEFI, sotftware RAID and standardised Partman recipes (the old nodes used hardware RAID)
  • Migrated to nftables as the firewall provider
  • We fixed the setup to allow the nodes to have proper AAAA records in DNS. This also needed configuration tweaks in Postgresql since the grants are only based on IPV4 addresses.
  • We updated imposm to the latest version (and tracked down a data corruption bug which got introduced in October via new OSM updates)
  • The Swift script were refactored (https://github.com/wikimedia/operations-puppet/commit/e3ebe216c19ce604a306b3255a7c1bd4b47d9af36)
  • We tweaked the Postgres performance settings to increase max-conns and the shared buffer size since we have beefier hardware and increase performance
  • Some slow queries were idenfified (to be indexed when we have a staging host): T407491

Lots of things were also cleaned up in Puppet:

gerritbot added a comment.Thu, Nov 27, 7:51 AM

Change #1211684 merged by Muehlenhoff:

[labs/private@master] Remove obsolete stub secrets

https://gerrit.wikimedia.org/r/1211684

gerritbot added a comment.Thu, Nov 27, 8:43 AM

Change #1212062 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] postgis: Remove support for buster

https://gerrit.wikimedia.org/r/1212062

gerritbot added a comment.Thu, Nov 27, 10:12 AM

Change #1212062 merged by Muehlenhoff:

[operations/puppet@production] postgis: Remove support for buster

https://gerrit.wikimedia.org/r/1212062

gerritbot added a comment.Thu, Nov 27, 10:58 AM

Change #1212099 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/cookbooks@master] sre.maps.roll-restart-reboot-master: Adapt to Bookworm changes

https://gerrit.wikimedia.org/r/1212099

gerritbot added a comment.Thu, Nov 27, 11:00 AM

Change #1212100 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/cookbooks@master] sre.maps.roll-restart-reboot: Adapt for Bookworm

https://gerrit.wikimedia.org/r/1212100

gerritbot added a comment.Thu, Nov 27, 11:14 AM

Change #1212099 merged by Muehlenhoff:

[operations/cookbooks@master] sre.maps.roll-restart-reboot-master: Adapt to Bookworm changes

https://gerrit.wikimedia.org/r/1212099

gerritbot added a comment.Thu, Nov 27, 2:09 PM

Change #1212100 merged by Muehlenhoff:

[operations/cookbooks@master] sre.maps.roll-restart-reboot: Adapt for Bookworm

https://gerrit.wikimedia.org/r/1212100

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits