Page MenuHomePhabricator

Retire ldap-corp cluster
Closed, ResolvedPublic

Description

With the changes deployed to the mail servers following the LDAP rework within ITS, these should no longer be needed.

To validate nothing else is hitting these, I'll still tcpdump incoming queries for a while, but eventually we should be able to decom the VMs and retire the related puppet code.

Event Timeline

I ran tcpdump on both hosts for about a week and aside from random scanning scatter, all remaining connections were to ldap1.corp.wikimedia.org and alert[12]001.org. I'll be stopping puppet and stop/mask slapd on these as a next step.

Mentioned in SAL (#wikimedia-operations) [2023-01-19T12:06:05Z] <moritzm> stopping/masking slapd on ldap-corp1001/ldap-corp2001 T323820

Change 882573 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove openldap_corp role from ldap-corp*

https://gerrit.wikimedia.org/r/882573

Change 882573 merged by Muehlenhoff:

[operations/puppet@production] Remove openldap_corp role from ldap-corp*

https://gerrit.wikimedia.org/r/882573

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: ldap-corp2001.wikimedia.org

  • ldap-corp2001.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox

Change 884276 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/dns@master] Remove ldap-corp-related CNAMES

https://gerrit.wikimedia.org/r/884276

Change 884276 merged by Muehlenhoff:

[operations/dns@master] Remove ldap-corp-related CNAMES

https://gerrit.wikimedia.org/r/884276

Change 884282 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] exim: Remove leftovers of ldap-corp setup

https://gerrit.wikimedia.org/r/884282

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: ldap-corp1001.wikimedia.org

  • ldap-corp1001.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox

Change 884290 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove Puppet references for ldap-corp1001/2001

https://gerrit.wikimedia.org/r/884290

Change 884290 merged by Muehlenhoff:

[operations/puppet@production] Remove Puppet references for ldap-corp1001/2001

https://gerrit.wikimedia.org/r/884290

Change 884295 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove role::openldap_corp and related profiles/templates

https://gerrit.wikimedia.org/r/884295

Change 884295 merged by Muehlenhoff:

[operations/puppet@production] Remove role::openldap_corp and related profiles/templates

https://gerrit.wikimedia.org/r/884295

The two VMs have been decommissioned and the Puppet code/certs/secrets removed. I've also sent ITS a headsup that this has been shut down on the SRE end.

I've synched up with ITS, they will shut down the ldap1.corp.wikimedia.org server that we synched against next calendar year.

Change 884282 merged by Muehlenhoff:

[operations/puppet@production] exim: Remove leftovers of ldap-corp setup

https://gerrit.wikimedia.org/r/884282