Page MenuHomePhabricator
Create Task
Maniphest T381565

Move maps servers to Bookworm
Open, Needs TriagePublic
Actions

Description

kartotherian is being moved to wikikube. When that is complete, the server backend will be moved to Bookworm.

We'll be reusing six former ganeti nodes a test cluster to test the new stack (setup at T380144) until we eventually reimage the maps* nodes.

The high level plan is:

  • install a master bookworm node and fix all issues
  • install a replica bookworm node and fix all issues
  • the remaing replicas
  • test the OSM import on the new cluster
  • point wikikube/staging to use the bookworm cluster
  • if this works fine, we can enable expiration events for bookworm and disable it for buster
  • move the prod pods to the new cluster and eventually
  • reimage the maps* from buster to bookworm
  • failover production traffic to the actual maps* cluster

Details

SubjectRepoBranchLines +/-
osm_replica: Fix Hiera variableoperations/puppetproduction+1 -1
osm_master: Tighten replication slot name on Bookwormoperations/puppetproduction+5 -1
Enable maps-test2003 to maps-test2006 as additional maps bookworm replicasoperations/puppetproduction+13 -10
Make maps-test2002 a bookworm maps replicaoperations/puppetproduction+10 -3
Add stub secrets for bookworm replica rolelabs/privatemaster+4 -0
Make maps-test2001 a bookworm maps master nodeoperations/puppetproduction+6 -2
maps: Add a separate Hiera option to control the waterlines importoperations/puppetproduction+15 -8
maps_bookworm: Initially disable replication/tile gen timersoperations/puppetproduction+2 -2
osm_master: Provide a dummy variable for tilerator on bookworm rolesoperations/puppetproduction+1 -0
profile::maps::osm_master: Make tilerator_pass optionaloperations/puppetproduction+1 -1
Add stub secrets for new master_bookworm roleslabs/privatemaster+8 -0
Extend commentoperations/puppetproduction+2 -0
Manage tile invalidation with a separate Hiera settingoperations/puppetproduction+9 -1
Don't setup database config for tilerator on bookwormoperations/puppetproduction+61 -32
Add missing Hiera settings for new bookworm master rolesoperations/puppetproduction+16 -0
Add separate maps master/replica roles for the new Bookworm setupoperations/puppetproduction+53 -0
Remove profile::java from maps hostsoperations/puppetproduction+0 -8
postgresql: Add support for Bookwormoperations/puppetproduction+3 -4
maps::osm_master: Inline osm classoperations/puppetproduction+5 -10
osmborder: Build for Bookworm and bump debhelper compat to 13operations/debs/osmbordermaster+8 -3
osm: On Bookworm create OSM users using system::sysuseroperations/puppetproduction+29 -15
planet_sync: Remove obsolete optionsoperations/puppetproduction+0 -16
planet_sync: Cleanup time handlingoperations/puppetproduction+42 -44
maps::master Add toggle for planet syncoperations/puppetproduction+2 -9
Configure new maps nodes with nftablesoperations/puppetproduction+6 -0
maps/postgresql: Support bookwormoperations/puppetproduction+4 -14
maps: Allow disabling the installation of kartotherianoperations/puppetproduction+12 -11
osm_master: Avoid Ferm-specific syntaxoperations/puppetproduction+3 -5
maps: Remove support for osm2pgsql as OSM engineoperations/puppetproduction+17 -271
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

MoritzMuehlenhoff created this task.Dec 5 2024, 11:33 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 5 2024, 11:33 AM
gerritbot added a comment.Dec 5 2024, 11:49 AM

Change #1100784 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] maps: Remove support for osm2pgsql as OSM engine

https://gerrit.wikimedia.org/r/1100784

gerritbot added a project: Patch-For-Review.Dec 5 2024, 11:49 AM
gerritbot added a comment.Dec 5 2024, 12:03 PM

Change #1100788 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] osm_master: Avoid Ferm-specific syntax

https://gerrit.wikimedia.org/r/1100788

gerritbot added a comment.Dec 9 2024, 7:41 AM

Change #1100784 merged by Muehlenhoff:

[operations/puppet@production] maps: Remove support for osm2pgsql as OSM engine

https://gerrit.wikimedia.org/r/1100784

gerritbot added a comment.Dec 9 2024, 7:52 AM

Change #1100788 merged by Muehlenhoff:

[operations/puppet@production] osm_master: Avoid Ferm-specific syntax

https://gerrit.wikimedia.org/r/1100788

gerritbot added a comment.Dec 9 2024, 8:23 AM

Change #1100456 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] maps: Allow disabling the installation of kartotherian

https://gerrit.wikimedia.org/r/1100456

gerritbot added a comment.Dec 9 2024, 8:42 AM

Change #1100456 merged by Muehlenhoff:

[operations/puppet@production] maps: Allow disabling the installation of kartotherian

https://gerrit.wikimedia.org/r/1100456

Maintenance_bot removed a project: Patch-For-Review.Dec 9 2024, 9:30 AM
gerritbot added a comment.Dec 9 2024, 9:53 AM

Change #1101465 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] maps/postgresql: Support bookworm

https://gerrit.wikimedia.org/r/1101465

gerritbot added a project: Patch-For-Review.Dec 9 2024, 9:53 AM
gerritbot added a comment.Dec 9 2024, 2:25 PM

Change #1101465 merged by Muehlenhoff:

[operations/puppet@production] maps/postgresql: Support bookworm

https://gerrit.wikimedia.org/r/1101465

Maintenance_bot removed a project: Patch-For-Review.Dec 9 2024, 2:30 PM
gerritbot added a comment.Dec 10 2024, 1:40 PM

Change #1101864 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Configure new maps nodes with nftables

https://gerrit.wikimedia.org/r/1101864

gerritbot added a project: Patch-For-Review.Dec 10 2024, 1:40 PM
gerritbot added a comment.Dec 13 2024, 7:58 AM

Change #1101864 merged by Muehlenhoff:

[operations/puppet@production] Configure new maps nodes with nftables

https://gerrit.wikimedia.org/r/1101864

Maintenance_bot removed a project: Patch-For-Review.Dec 13 2024, 8:30 AM
gerritbot added a comment.Dec 20 2024, 7:29 AM

Change #1105836 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] maps::master Add toggle for planet sync

https://gerrit.wikimedia.org/r/1105836

gerritbot added a project: Patch-For-Review.Dec 20 2024, 7:29 AM
gerritbot added a comment.Dec 20 2024, 7:50 AM

Change #1105836 abandoned by Muehlenhoff:

[operations/puppet@production] maps::master Add toggle for planet sync

Reason:

Functionality already exists, just confusingly named

https://gerrit.wikimedia.org/r/1105836

gerritbot added a comment.Dec 20 2024, 8:03 AM

Change #1105875 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] planet_sync: Cleanup time handling

https://gerrit.wikimedia.org/r/1105875

gerritbot added a comment.Dec 20 2024, 9:02 AM

Change #1105876 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] planet_sync: Remove obsolete options

https://gerrit.wikimedia.org/r/1105876

Stashbot added a comment.Dec 20 2024, 9:43 AM

Mentioned in SAL (#wikimedia-operations) [2024-12-20T09:43:53Z] <moritzm> imported imposm3 0.11.1-1+deb12u1 to apt.wikimedia.org T381565

gerritbot added a comment.Jan 6 2025, 9:18 AM

Change #1105875 merged by Muehlenhoff:

[operations/puppet@production] planet_sync: Cleanup time handling

https://gerrit.wikimedia.org/r/1105875

gerritbot added a comment.Jan 6 2025, 9:36 AM

Change #1105876 merged by Muehlenhoff:

[operations/puppet@production] planet_sync: Remove obsolete options

https://gerrit.wikimedia.org/r/1105876

Maintenance_bot removed a project: Patch-For-Review.Jan 6 2025, 10:30 AM
bd808 added a subtask: T361381: Replace deployment-maps-master01 with a Bullseye or Bookworm instance.Jan 6 2025, 8:54 PM
gerritbot added a comment.Jan 7 2025, 1:14 PM

Change #1108745 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] osm: On Bookworm create OSM users using system::sysuser

https://gerrit.wikimedia.org/r/1108745

gerritbot added a project: Patch-For-Review.Jan 7 2025, 1:14 PM
gerritbot added a comment.Jan 7 2025, 3:20 PM

Change #1108745 merged by Muehlenhoff:

[operations/puppet@production] osm: On Bookworm create OSM users using system::sysuser

https://gerrit.wikimedia.org/r/1108745

Maintenance_bot removed a project: Patch-For-Review.Jan 7 2025, 3:31 PM
gerritbot added a comment.Jan 7 2025, 3:35 PM

Change #1108773 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] maps::osm_master: Inline osm class

https://gerrit.wikimedia.org/r/1108773

gerritbot added a project: Patch-For-Review.Jan 7 2025, 3:35 PM
gerritbot added a comment.Jan 7 2025, 3:49 PM

Change #1108776 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/debs/osmborder@master] osmborder: Build for Bookworm and bump debhelper compat to 12

https://gerrit.wikimedia.org/r/1108776

gerritbot added a comment.Jan 8 2025, 8:46 AM

Change #1108776 merged by Muehlenhoff:

[operations/debs/osmborder@master] osmborder: Build for Bookworm and bump debhelper compat to 13

https://gerrit.wikimedia.org/r/1108776

Stashbot added a comment.Jan 8 2025, 10:04 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-08T10:04:41Z] <moritzm> imported osmborder 0.1.0+wmf12u1 to apt.wikimedia.org/bookworm T381565

gerritbot added a comment.Jan 8 2025, 11:31 AM

Change #1108773 merged by Muehlenhoff:

[operations/puppet@production] maps::osm_master: Inline osm class

https://gerrit.wikimedia.org/r/1108773

Maintenance_bot removed a project: Patch-For-Review.Jan 8 2025, 12:30 PM
gerritbot added a comment.Jan 15 2025, 2:46 PM

Change #1111634 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Make maps-test2001 a bookworm maps master node (WIP)

https://gerrit.wikimedia.org/r/1111634

gerritbot added a project: Patch-For-Review.Jan 15 2025, 2:46 PM
gerritbot added a comment.Jan 15 2025, 3:00 PM

Change #1111639 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] postgresql: Add support for Bookworm

https://gerrit.wikimedia.org/r/1111639

gerritbot added a comment.Jan 15 2025, 3:17 PM

Change #1111639 merged by Muehlenhoff:

[operations/puppet@production] postgresql: Add support for Bookworm

https://gerrit.wikimedia.org/r/1111639

gerritbot added a comment.Jan 15 2025, 3:36 PM

Change #1111647 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove profile::java from maps hosts

https://gerrit.wikimedia.org/r/1111647

gerritbot added a comment.Jan 15 2025, 4:11 PM

Change #1111647 merged by Muehlenhoff:

[operations/puppet@production] Remove profile::java from maps hosts

https://gerrit.wikimedia.org/r/1111647

gerritbot added a comment.Jan 15 2025, 4:27 PM

Change #1111659 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add separate maps master/replica roles for the new Bookworm setup (WIP)

https://gerrit.wikimedia.org/r/1111659

gerritbot added a comment.Jan 17 2025, 8:12 AM

Change #1112167 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Don't setup database config for tilerator on bookworm

https://gerrit.wikimedia.org/r/1112167

gerritbot added a comment.Jan 17 2025, 2:07 PM

Change #1111659 merged by Muehlenhoff:

[operations/puppet@production] Add separate maps master/replica roles for the new Bookworm setup

https://gerrit.wikimedia.org/r/1111659

gerritbot added a comment.Jan 17 2025, 2:22 PM

Change #1112218 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add missing Hiera settings for new bookworm master roles

https://gerrit.wikimedia.org/r/1112218

gerritbot added a comment.Jan 20 2025, 7:05 AM

Change #1112218 merged by Muehlenhoff:

[operations/puppet@production] Add missing Hiera settings for new bookworm master roles

https://gerrit.wikimedia.org/r/1112218

gerritbot added a comment.Jan 20 2025, 2:02 PM

Change #1112167 merged by Muehlenhoff:

[operations/puppet@production] Don't setup database config for tilerator on bookworm

https://gerrit.wikimedia.org/r/1112167

gerritbot added a comment.Jan 22 2025, 12:06 PM

Change #1113442 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Manage tile invalidation with a separate Hiera setting

https://gerrit.wikimedia.org/r/1113442

MoritzMuehlenhoff updated the task description. (Show Details)Jan 22 2025, 2:22 PM
gerritbot added a comment.Jan 22 2025, 3:05 PM

Change #1113442 merged by Muehlenhoff:

[operations/puppet@production] Manage tile invalidation with a separate Hiera setting

https://gerrit.wikimedia.org/r/1113442

gerritbot added a comment.Jan 22 2025, 3:56 PM

Change #1113487 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Extend comment

https://gerrit.wikimedia.org/r/1113487

gerritbot added a comment.Jan 23 2025, 7:44 AM

Change #1113487 merged by Muehlenhoff:

[operations/puppet@production] Extend comment

https://gerrit.wikimedia.org/r/1113487

gerritbot added a comment.Jan 23 2025, 7:49 AM

Change #1113740 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[labs/private@master] Add stub secrets for new master_bookworm roles

https://gerrit.wikimedia.org/r/1113740

gerritbot added a comment.Jan 23 2025, 8:00 AM

Change #1113740 merged by Muehlenhoff:

[labs/private@master] Add stub secrets for new master_bookworm roles

https://gerrit.wikimedia.org/r/1113740

gerritbot added a comment.Jan 23 2025, 8:52 AM

Change #1113746 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] profile::maps::osm_master: Make tilerator_pass optional

https://gerrit.wikimedia.org/r/1113746

gerritbot added a comment.Jan 23 2025, 1:47 PM

Change #1113746 merged by Muehlenhoff:

[operations/puppet@production] profile::maps::osm_master: Make tilerator_pass optional

https://gerrit.wikimedia.org/r/1113746

gerritbot added a comment.Jan 23 2025, 1:58 PM

Change #1113810 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] osm_master: Provide a dummy variable for tilerator on bookworm roles

https://gerrit.wikimedia.org/r/1113810

gerritbot added a comment.Jan 23 2025, 4:21 PM

Change #1113810 merged by Muehlenhoff:

[operations/puppet@production] osm_master: Provide a dummy variable for tilerator on bookworm roles

https://gerrit.wikimedia.org/r/1113810

gerritbot added a comment.Jan 28 2025, 3:10 PM

Change #1114755 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] maps_bookworm: Initially disable replication/tile gen timers

https://gerrit.wikimedia.org/r/1114755

gerritbot added a comment.Jan 28 2025, 3:37 PM

Change #1114755 merged by Muehlenhoff:

[operations/puppet@production] maps_bookworm: Initially disable replication/tile gen timers

https://gerrit.wikimedia.org/r/1114755

gerritbot added a comment.Jan 28 2025, 3:51 PM

Change #1114769 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add a separate Hiera option to control the waterlines import

https://gerrit.wikimedia.org/r/1114769

gerritbot added a comment.Jan 29 2025, 1:42 PM

Change #1114769 merged by Muehlenhoff:

[operations/puppet@production] maps: Add a separate Hiera option to control the waterlines import

https://gerrit.wikimedia.org/r/1114769

gerritbot added a comment.Jan 30 2025, 11:56 AM

Change #1111634 merged by Muehlenhoff:

[operations/puppet@production] Make maps-test2001 a bookworm maps master node

https://gerrit.wikimedia.org/r/1111634

ops-monitoring-bot added a comment.Jan 30 2025, 12:45 PM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host maps-test2001.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 30 2025, 3:11 PM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host maps-test2001.codfw.wmnet with OS bookworm executed with errors:

  • maps-test2001 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501301309_jmm_4089887_maps-test2001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console maps-test2001.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jan 30 2025, 3:33 PM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host maps-test2001.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 30 2025, 4:21 PM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host maps-test2001.codfw.wmnet with OS bookworm completed:

  • maps-test2001 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501301558_jmm_4144943_maps-test2001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Jan 31 2025, 1:16 PM

Change #1115846 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[labs/private@master] Add stub secrets for bookworm replica role

https://gerrit.wikimedia.org/r/1115846

gerritbot added a comment.Jan 31 2025, 1:20 PM

Change #1115846 merged by Muehlenhoff:

[labs/private@master] Add stub secrets for bookworm replica role

https://gerrit.wikimedia.org/r/1115846

gerritbot added a comment.Jan 31 2025, 1:23 PM

Change #1115850 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Make maps-test2002 a bookworm maps replica

https://gerrit.wikimedia.org/r/1115850

gerritbot added a comment.Jan 31 2025, 1:57 PM

Change #1115863 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Enable maps-test2003 to maps-test2006 as additional maps bookworm replicas

https://gerrit.wikimedia.org/r/1115863

gerritbot added a comment.Feb 6 2025, 10:53 AM

Change #1115850 merged by Muehlenhoff:

[operations/puppet@production] Make maps-test2002 a bookworm maps replica

https://gerrit.wikimedia.org/r/1115850

gerritbot added a comment.Wed, Mar 12, 2:58 PM

Change #1127042 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] osm_master: Tighten replication slot name on Bookworm

https://gerrit.wikimedia.org/r/1127042

gerritbot added a comment.Wed, Mar 12, 3:55 PM

Change #1127042 merged by Muehlenhoff:

[operations/puppet@production] osm_master: Tighten replication slot name on Bookworm

https://gerrit.wikimedia.org/r/1127042

gerritbot added a comment.Tue, Mar 18, 3:26 PM

Change #1128891 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] osm_replica: Fix Hiera variable

https://gerrit.wikimedia.org/r/1128891

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits