Page MenuHomePhabricator
Create Task
Maniphest T331699

Migrate the r/w LDAP servers to Trixie and MDB storage
Open, MediumPublic
Actions

Description

This can also be used to address some other issues alongside via fresh installs (previously serpens/seaborgium had been dist-upgraded in place)

  • Rename away from the legacy naming scheme towards ldap-rw1001/ldap-rw2001
  • Move away from the deprecated BDB (T292942)
  • Create new ldap-rw1001/ldap-rw2001 VMs using Bookworm and set profile::openldap::storage_backend to "mdb" and configure them as a synchronisation pair
  • slapcat the existing data from serpens to an LDIF (ACLs, LDAP extensions are all distributed via Puppet)
  • slapadd the LDIF on ldap-rw1001 and let it sync towards ldap-rw2001
  • Create four additional ldap-replica VMs running Trixie and sync them against ldap-rw1001/2001
  • Test the new setup
  • When everything works as expected in the parallel setup, revert the new Trixie hosts to a clean state
  • Setup a window (1-2 hours) during which no r/w changes are possible (disable Bitu temporarily, tell SREs to avoid LDAP changes, disable Horizon)
  • Repeat the same import as above with current data, if all is well:
  • Point ldap-rw.codfw.w.o to ldap-rw2001
  • Point ldap-rw.eqiad.w.o to ldap-rw1001
  • Depool all older readonly replicas in favour of the new Trixie ones
  • If there are unforeseen issues we can simply revert to serpens/seaborgium/old replicas
  • If all is well, decom serpens/seaborgium and the old replicas

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
acmechief: Remove static LDAP replica namesMuehlenhoffoperations/puppetproduction+0 -4
acmechief: Add additional SNIs for ldap-ro-test.[eqiad|codfw]Muehlenhoffoperations/puppetproduction+2 -0
idp-test: Configure to use the new Trixie/MDB LDAP servers for testsMuehlenhoffoperations/puppetproduction+10 -0
Revert LDAP settings for idp-test1005/2005Muehlenhoffoperations/puppetproduction+2 -14
Add explicit replica hostnames to LDAP cert SNIsMuehlenhoffoperations/puppetproduction+4 -0
idp-test: Configure to use the new Trixie/MDB LDAP servers for testsMuehlenhoffoperations/puppetproduction+12 -1
Use the new MDB/Trixie servers for testsMuehlenhoffoperations/puppetproduction+5 -0
Assign openldap::replica_mdb role to ldap-replica200[78]Muehlenhoffoperations/puppetproduction+1 -1
Assign openldap::replica_mdb role to ldap-replica200[78]Muehlenhoffoperations/puppetproduction+1 -1
Assign server IDs for new LDAP replicasMuehlenhoffoperations/puppetproduction+6 -0
Add explicit hostnames to LDAP cert SNIsMuehlenhoffoperations/puppetproduction+2 -2
Remove unused profile::openldap::hostnameMuehlenhoffoperations/puppetproduction+0 -13
Create a separate role openldap::replica_mdbMuehlenhoffoperations/puppetproduction+37 -0
Setup a syncrepl cluster on trixie/MDBMuehlenhoffoperations/puppetproduction+6 -10
Stub secrets for new Trixie OpenLDAP rolesMuehlenhofflabs/privatemaster+2 -0
Additional necessary settings for the openldap::rw_mdb roleMuehlenhoffoperations/puppetproduction+4 -0
New r/o LDAP replicas on Trixie/MDBMuehlenhoffoperations/puppetproduction+8 -0
Reimage ldap-rw[12]001Muehlenhoffoperations/puppetproduction+2 -2
Use nftables for openldap::rw_mdbMuehlenhoffoperations/puppetproduction+2 -0
Create a separate role openldap::rw_mdbMuehlenhoffoperations/puppetproduction+14 -0
Add new Bookworm LDAP replicasMuehlenhoffoperations/puppetproduction+48 -2
conftool: Add new LDAP replicasMuehlenhoffoperations/puppetproduction+4 -0
slapd: introduce new slapd.conf template for ldap >= 2.5Andrew Bogottoperations/puppetproduction+172 -10
On Bookworm ship ppolicy.schema via PuppetMuehlenhoffoperations/puppetproduction+11 -2
Only install ppolicy.schema with OpenLDAP < 2.5Muehlenhoffoperations/puppetproduction+11 -0
Fix Hiera entries for ldap master nodesMuehlenhoffoperations/puppetproduction+2 -2
Configure ldap-rw1001/2001 as LDAP serversMuehlenhoffoperations/puppetproduction+12 -2
Extend acmechief config with new names of Bookworm hostsMuehlenhoffoperations/puppetproduction+6 -2
Create DNS records for new LDAP Bookworm clusterMuehlenhoffoperations/dnsmaster+2 -0
Add new LDAP replicasMuehlenhoffoperations/puppetproduction+10 -1
Add ldap-rw[12]001 to site.ppMuehlenhoffoperations/puppetproduction+8 -0
Add partman config for ldap-rw* hostsMuehlenhoffoperations/puppetproduction+1 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Sep 26 2023, 9:14 AM

Change 961047 merged by Muehlenhoff:

[operations/puppet@production] Fix Hiera entries for ldap master nodes

https://gerrit.wikimedia.org/r/961047

gerritbot added a comment.Sep 26 2023, 10:22 AM

Change 961066 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] On Bookworm ship ppolicy.schema via Puppet

https://gerrit.wikimedia.org/r/961066

gerritbot added a comment.Sep 26 2023, 6:57 PM

Change 961188 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] slapd: introduce new slapd.conf template for ldap >= 2.5

https://gerrit.wikimedia.org/r/961188

• jcrespo subscribed.Sep 27 2023, 9:48 AM

I got an alert about ldap-rw2001 failing its backups (probably expected during setup), but wanted to give a heads up.

gerritbot added a comment.Sep 28 2023, 1:06 PM

Change 961796 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Only install ppolicy.schema with OpenLDAP < 2.5

https://gerrit.wikimedia.org/r/961796

gerritbot added a comment.Sep 28 2023, 6:14 PM

Change 961796 merged by Andrew Bogott:

[operations/puppet@production] Only install ppolicy.schema with OpenLDAP < 2.5

https://gerrit.wikimedia.org/r/961796

gerritbot added a comment.Sep 29 2023, 5:00 AM

Change 961066 abandoned by Muehlenhoff:

[operations/puppet@production] On Bookworm ship ppolicy.schema via Puppet

Reason:

Correct fix was https://gerrit.wikimedia.org/r/c/operations/puppet/+/961796

https://gerrit.wikimedia.org/r/961066

gerritbot added a comment.Oct 23 2023, 2:34 PM

Change 961188 abandoned by Andrew Bogott:

[operations/puppet@production] slapd: introduce new slapd.conf template for ldap >= 2.5

Reason:

Save change accomplished in https://gerrit.wikimedia.org/r/c/operations/puppet/+/961796

https://gerrit.wikimedia.org/r/961188

MoritzMuehlenhoff mentioned this in T364823: Upgrade r/w LDAP servers to Bullseye.May 14 2024, 8:12 AM
ops-monitoring-bot added a comment.May 17 2024, 11:48 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: ldap-replica2007.wikimedia.org

  • ldap-replica2007.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
ops-monitoring-bot added a comment.May 17 2024, 12:07 PM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: ldap-replica2008.wikimedia.org

  • ldap-replica2008.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
ops-monitoring-bot added a comment.May 17 2024, 12:27 PM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: ldap-replica1005.wikimedia.org

  • ldap-replica1005.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
ops-monitoring-bot added a comment.May 17 2024, 12:46 PM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: ldap-replica1006.wikimedia.org

  • ldap-replica1006.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
gerritbot added a comment.May 28 2024, 11:59 AM

Change #959215 abandoned by Muehlenhoff:

[operations/puppet@production] conftool: Add new LDAP replicas

Reason:

No longer needed, first move to private hostnames for the replicas

https://gerrit.wikimedia.org/r/959215

gerritbot added a comment.May 28 2024, 2:51 PM

Change #959213 abandoned by Muehlenhoff:

[operations/puppet@production] Add new Bookworm LDAP replicas

Reason:

Not neeedd at this point, we'll first move the replicas to private IPs

https://gerrit.wikimedia.org/r/959213

Maintenance_bot removed a project: Patch-For-Review.May 28 2024, 3:31 PM
MoritzMuehlenhoff removed MoritzMuehlenhoff as the assignee of this task.May 22 2025, 11:59 AM
MoritzMuehlenhoff renamed this task from Migrate the r/w LDAP servers to Bookworm and MDB storage to Migrate the r/w LDAP servers to Trixie and MDB storage (and private IPs).Mar 20 2026, 12:19 PM
MoritzMuehlenhoff renamed this task from Migrate the r/w LDAP servers to Trixie and MDB storage (and private IPs) to Migrate the r/w LDAP servers to Trixie and MDB storage.Aug 14 2026, 12:02 PM
MoritzMuehlenhoff updated the task description. (Show Details)
ops-monitoring-bot added a comment.Aug 14 2026, 12:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2003 for host ldap-rw1001.wikimedia.org with OS trixie

ops-monitoring-bot added a comment.Aug 14 2026, 12:45 PM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2003 for host ldap-rw1001.wikimedia.org with OS trixie completed:

  • ldap-rw1001 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202608141224_jmm_1640145_ldap-rw1001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Aug 17 2026, 7:28 AM

Mentioned in SAL (#wikimedia-operations) [2026-08-17T07:28:17Z] <moritzm> extend the disk of ldap-rw1001 by 80G T331699

ops-monitoring-bot added a comment.Aug 17 2026, 8:19 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2003 for host ldap-rw1001.wikimedia.org with OS trixie

ops-monitoring-bot added a comment.Aug 17 2026, 9:01 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2003 for host ldap-rw1001.wikimedia.org with OS trixie completed:

  • ldap-rw1001 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202608170838_jmm_2390368_ldap-rw1001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Aug 19 2026, 7:45 AM

Mentioned in SAL (#wikimedia-operations) [2026-08-19T07:45:41Z] <moritzm> extend the disk of ldap-rw2001 by 80G T331699

ops-monitoring-bot added a comment.Aug 19 2026, 9:06 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2003 for host ldap-rw2001.wikimedia.org with OS trixie

ops-monitoring-bot added a comment.Aug 19 2026, 9:52 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2003 for host ldap-rw2001.wikimedia.org with OS trixie completed:

  • ldap-rw2001 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202608190933_jmm_3020559_ldap-rw2001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Sep 3 2026, 11:39 AM

Change #1334839 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Create a separate role openldap::rw_mdb

https://gerrit.wikimedia.org/r/1334839

gerritbot added a project: Patch-For-Review.Sep 3 2026, 11:39 AM
gerritbot added a comment.Sep 3 2026, 12:09 PM

Change #1334839 merged by Muehlenhoff:

[operations/puppet@production] Create a separate role openldap::rw_mdb

https://gerrit.wikimedia.org/r/1334839

Maintenance_bot removed a project: Patch-For-Review.Sep 3 2026, 12:31 PM
gerritbot added a comment.Sep 3 2026, 1:08 PM

Change #1334883 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Use nftables for openldap::rw_mdb

https://gerrit.wikimedia.org/r/1334883

gerritbot added a project: Patch-For-Review.Sep 3 2026, 1:08 PM
gerritbot added a comment.Sep 3 2026, 2:00 PM

Change #1334883 merged by Muehlenhoff:

[operations/puppet@production] Use nftables for openldap::rw_mdb

https://gerrit.wikimedia.org/r/1334883

gerritbot added a comment.Sep 3 2026, 2:07 PM

Change #1334906 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Reimage ldap-rw[12]001

https://gerrit.wikimedia.org/r/1334906

gerritbot added a comment.Sep 4 2026, 6:34 AM

Change #1334906 merged by Muehlenhoff:

[operations/puppet@production] Reimage ldap-rw[12]001

https://gerrit.wikimedia.org/r/1334906

ops-monitoring-bot added a comment.Sep 4 2026, 6:40 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2003 for host ldap-rw1001.wikimedia.org with OS trixie

ops-monitoring-bot added a comment.Sep 4 2026, 7:13 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2003 for host ldap-rw1001.wikimedia.org with OS trixie completed:

  • ldap-rw1001 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202609040657_jmm_3271167_ldap-rw1001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Maintenance_bot removed a project: Patch-For-Review.Sep 4 2026, 7:30 AM
ops-monitoring-bot added a comment.Sep 4 2026, 7:55 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2003 for host ldap-rw2001.wikimedia.org with OS trixie

ops-monitoring-bot added a comment.Sep 4 2026, 8:33 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2003 for host ldap-rw2001.wikimedia.org with OS trixie completed:

  • ldap-rw2001 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202609040817_jmm_3288794_ldap-rw2001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Sep 4 2026, 12:20 PM

Change #1335825 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Setup a syncrepl cluster on trixie/MDB

https://gerrit.wikimedia.org/r/1335825

gerritbot added a project: Patch-For-Review.Sep 4 2026, 12:20 PM
gerritbot added a comment.Sep 4 2026, 12:28 PM

Change #1335830 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] New r/o LDAP replicas on Trixie/MDB

https://gerrit.wikimedia.org/r/1335830

gerritbot added a comment.Sep 4 2026, 1:29 PM

Change #1335860 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Additional necessary settings for the openldap::rw_mdb role

https://gerrit.wikimedia.org/r/1335860

gerritbot added a comment.Sep 7 2026, 8:14 AM

Change #1335830 merged by Muehlenhoff:

[operations/puppet@production] New r/o LDAP replicas on Trixie/MDB

https://gerrit.wikimedia.org/r/1335830

ops-monitoring-bot added a comment.Sep 7 2026, 8:38 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin1004 for host ldap-replica1005.wikimedia.org with OS trixie

ops-monitoring-bot added a comment.Sep 7 2026, 9:10 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin1004 for host ldap-replica1005.wikimedia.org with OS trixie completed:

  • ldap-replica1005 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202609070856_jmm_858122_ldap-replica1005.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 7 2026, 9:57 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin1004 for host ldap-replica1006.wikimedia.org with OS trixie

ops-monitoring-bot added a comment.Sep 7 2026, 10:33 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin1004 for host ldap-replica1006.wikimedia.org with OS trixie completed:

  • ldap-replica1006 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202609071018_jmm_873441_ldap-replica1006.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 7 2026, 11:39 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin1004 for hosts: ldap-replica1006.wikimedia.org

  • ldap-replica1006.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
ops-monitoring-bot added a comment.Sep 7 2026, 11:46 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin1004 for host ldap-replica2007.wikimedia.org with OS trixie

ops-monitoring-bot added a comment.Sep 7 2026, 12:22 PM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin1004 for host ldap-replica2007.wikimedia.org with OS trixie completed:

  • ldap-replica2007 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202609071207_jmm_890657_ldap-replica2007.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 7 2026, 1:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin1004 for host ldap-replica2008.wikimedia.org with OS trixie

ops-monitoring-bot added a comment.Sep 7 2026, 1:44 PM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin1004 for host ldap-replica2008.wikimedia.org with OS trixie completed:

  • ldap-replica2008 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202609071328_jmm_902804_ldap-replica2008.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 8 2026, 7:27 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin1004 for host ldap-replica1006.wikimedia.org with OS trixie

ops-monitoring-bot added a comment.Sep 8 2026, 7:58 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin1004 for host ldap-replica1006.wikimedia.org with OS trixie completed:

  • ldap-replica1006 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202609080743_jmm_1031970_ldap-replica1006.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Sep 8 2026, 9:13 AM

Change #1335860 merged by Muehlenhoff:

[operations/puppet@production] Additional necessary settings for the openldap::rw_mdb role

https://gerrit.wikimedia.org/r/1335860

gerritbot added a comment.Sep 8 2026, 11:27 AM

Change #1337901 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[labs/private@master] Stub secrets for new Trixie OpenLDAP roles

https://gerrit.wikimedia.org/r/1337901

gerritbot added a comment.Sep 8 2026, 11:30 AM

Change #1337901 merged by Muehlenhoff:

[labs/private@master] Stub secrets for new Trixie OpenLDAP roles

https://gerrit.wikimedia.org/r/1337901

gerritbot added a comment.Sep 8 2026, 12:10 PM

Change #1337910 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Create a separate role openldap::replica_mdb

https://gerrit.wikimedia.org/r/1337910

gerritbot added a comment.Sep 9 2026, 12:36 PM

Change #1335825 merged by Muehlenhoff:

[operations/puppet@production] Setup a syncrepl cluster on trixie/MDB

https://gerrit.wikimedia.org/r/1335825

gerritbot added a comment.Fri, Sep 11, 9:46 AM

Change #1337910 merged by Muehlenhoff:

[operations/puppet@production] Create a separate role openldap::replica_mdb

https://gerrit.wikimedia.org/r/1337910

Maintenance_bot removed a project: Patch-For-Review.Fri, Sep 11, 10:31 AM
gerritbot added a comment.Fri, Sep 11, 12:31 PM

Change #1339732 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove unused profile::openldap::hostname

https://gerrit.wikimedia.org/r/1339732

gerritbot added a project: Patch-For-Review.Fri, Sep 11, 12:31 PM
gerritbot added a comment.Wed, Sep 16, 2:07 PM

Change #1339732 merged by Muehlenhoff:

[operations/puppet@production] Remove unused profile::openldap::hostname

https://gerrit.wikimedia.org/r/1339732

Maintenance_bot removed a project: Patch-For-Review.Wed, Sep 16, 2:31 PM
gerritbot added a comment.Thu, Sep 17, 12:30 PM

Change #1342660 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add explicit hostnames to LDAP cert SNIs

https://gerrit.wikimedia.org/r/1342660

gerritbot added a project: Patch-For-Review.Thu, Sep 17, 12:30 PM
gerritbot added a comment.Mon, Sep 21, 6:38 AM

Change #1342660 merged by Muehlenhoff:

[operations/puppet@production] Add explicit hostnames to LDAP cert SNIs

https://gerrit.wikimedia.org/r/1342660

Maintenance_bot removed a project: Patch-For-Review.Mon, Sep 21, 7:31 AM
MoritzMuehlenhoff added a comment.Mon, Sep 21, 1:18 PM

The current openLDAP 2.4 installation uses GNUTLS, while OpenLDAP 2.6 in Trixie uses OpenSSL. There were various TLS errors, which made the replication fail (since we had configured STARTLS errors as critical, which make slapd fail), but after https://gerrit.wikimedia.org/r/c/operations/puppet/+/1342660, the reissued ACME certs are working fine and replication now works in the r/w replica pair:

jmm@ldap-rw1001:~$ cat change-repl.ldif
dn: uid=jmm,ou=people,dc=wikimedia,dc=org
changetype: modify
replace: description
description: repl-test
jmm@ldap-rw1001:~$ ldapmodify -H ldap://localhost:389  -x -D "cn=admin,dc=wikimedia,dc=org" -W -f change-repl.ldif
jmm@ldap-rw1001:~$
jmm@ldap-rw2001:~$ ldapsearch -x -H ldap://localhost uid=jmm description
# extended LDIF
#
# LDAPv3
# base <dc=wikimedia,dc=org> (default) with scope subtree
# filter: uid=jmm
# requesting: description
#

# jmm, people, wikimedia.org
dn: uid=jmm,ou=people,dc=wikimedia,dc=org
description: repl-test

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1
gerritbot added a comment.Wed, Sep 23, 11:12 AM

Change #1344236 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Assign server IDs for new LDAP replicas

https://gerrit.wikimedia.org/r/1344236

gerritbot added a project: Patch-For-Review.Wed, Sep 23, 11:12 AM
gerritbot added a comment.Wed, Sep 23, 12:29 PM

Change #1344236 merged by Muehlenhoff:

[operations/puppet@production] Assign server IDs for new LDAP replicas

https://gerrit.wikimedia.org/r/1344236

gerritbot added a comment.Wed, Sep 23, 12:32 PM

Change #1344259 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Assign openldap::replica_mdb role to ldap-replica200[78]

https://gerrit.wikimedia.org/r/1344259

gerritbot added a comment.Thu, Sep 24, 10:40 AM

Change #1344259 merged by Muehlenhoff:

[operations/puppet@production] Assign openldap::replica_mdb role to ldap-replica200[78]

https://gerrit.wikimedia.org/r/1344259

gerritbot added a comment.Thu, Sep 24, 11:06 AM

Change #1344632 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Assign openldap::replica_mdb role to ldap-replica200[78]

https://gerrit.wikimedia.org/r/1344632

gerritbot added a comment.Thu, Sep 24, 11:53 AM

Change #1344632 merged by Muehlenhoff:

[operations/puppet@production] Assign openldap::replica_mdb role to ldap-replica200[78]

https://gerrit.wikimedia.org/r/1344632

Maintenance_bot removed a project: Patch-For-Review.Thu, Sep 24, 12:31 PM
gerritbot added a comment.Thu, Sep 24, 12:48 PM

Change #1344656 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Use the new MDB/Trixie servers for tests

https://gerrit.wikimedia.org/r/1344656

gerritbot added a project: Patch-For-Review.Thu, Sep 24, 12:48 PM
gerritbot added a comment.Thu, Sep 24, 1:13 PM

Change #1344665 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] idp-test: Configure to use the new Trixie/MDB LDAP servers for tests

https://gerrit.wikimedia.org/r/1344665

gerritbot added a comment.Tue, Sep 29, 12:12 PM

Change #1344656 merged by Muehlenhoff:

[operations/puppet@production] Use the new MDB/Trixie servers for tests

https://gerrit.wikimedia.org/r/1344656

gerritbot added a comment.Tue, Sep 29, 12:36 PM

Change #1344665 merged by Muehlenhoff:

[operations/puppet@production] idp-test: Configure to use the new Trixie/MDB LDAP servers for tests

https://gerrit.wikimedia.org/r/1344665

Maintenance_bot removed a project: Patch-For-Review.Tue, Sep 29, 1:33 PM
gerritbot added a comment.Wed, Sep 30, 7:14 AM

Change #1346374 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add explicit replica hostnames to LDAP cert SNIs

https://gerrit.wikimedia.org/r/1346374

gerritbot added a project: Patch-For-Review.Wed, Sep 30, 7:14 AM
gerritbot added a comment.Wed, Sep 30, 8:18 AM

Change #1346374 merged by Muehlenhoff:

[operations/puppet@production] Add explicit replica hostnames to LDAP cert SNIs

https://gerrit.wikimedia.org/r/1346374

gerritbot added a comment.Wed, Sep 30, 10:03 AM

Change #1346495 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Revert LDAP settings for idp-test1005/2005

https://gerrit.wikimedia.org/r/1346495

gerritbot added a comment.Wed, Sep 30, 10:05 AM

Change #1346495 merged by Muehlenhoff:

[operations/puppet@production] Revert LDAP settings for idp-test1005/2005

https://gerrit.wikimedia.org/r/1346495

gerritbot added a comment.Wed, Oct 7, 1:58 PM

Change #1352525 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] idp-test: Configure to use the new Trixie/MDB LDAP servers for tests

https://gerrit.wikimedia.org/r/1352525

gerritbot added a comment.Thu, Oct 8, 7:35 AM

Change #1352525 merged by Muehlenhoff:

[operations/puppet@production] idp-test: Configure to use the new Trixie/MDB LDAP servers for tests

https://gerrit.wikimedia.org/r/1352525

gerritbot added a comment.Thu, Oct 8, 7:47 AM

Change #1352988 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] acmechief: Add additional SNIs for ldap-ro-test.[eqiad|codfw]

https://gerrit.wikimedia.org/r/1352988

gerritbot added a comment.Thu, Oct 8, 9:04 AM

Change #1352988 merged by Muehlenhoff:

[operations/puppet@production] acmechief: Add additional SNIs for ldap-ro-test.[eqiad|codfw]

https://gerrit.wikimedia.org/r/1352988

gerritbot added a comment.Thu, Oct 8, 10:51 AM

Change #1353104 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] acmechief: Remove static LDAP replica names

https://gerrit.wikimedia.org/r/1353104

gerritbot added a comment.Thu, Oct 8, 12:22 PM

Change #1353104 merged by Muehlenhoff:

[operations/puppet@production] acmechief: Remove static LDAP replica names

https://gerrit.wikimedia.org/r/1353104

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits