Page MenuHomePhabricator

Migrate LDAP replicas to Buster
Closed, ResolvedPublic

Description

The RO replicas are still running Stretch. The eqiad servers currently use a non-standard naming scheme, they will be fixed to also use ldap-replica100[12].

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 2 2020, 7:50 AM
MoritzMuehlenhoff triaged this task as Medium priority.Oct 2 2020, 7:53 AM

Change 632648 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Install ldap-replica200[34] as additional LDAP replicas

https://gerrit.wikimedia.org/r/632648

Change 632648 merged by Muehlenhoff:
[operations/puppet@production] Install ldap-replica200[34] as additional LDAP replicas

https://gerrit.wikimedia.org/r/632648

Mentioned in SAL (#wikimedia-operations) [2020-10-19T13:26:22Z] <moritzm> import prometheus-openldap-exporter 0+git20171128-2+deb10u1 for buster-wikimedia T264388

Change 634974 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] acmechief: Also allow ldap-replica2003/2004

https://gerrit.wikimedia.org/r/634974

Change 634974 merged by Muehlenhoff:
[operations/puppet@production] acmechief: Also allow ldap-replica2003/2004

https://gerrit.wikimedia.org/r/634974

Change 635499 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] acmechief: Add ldap-replica1001/1002

https://gerrit.wikimedia.org/r/635499

Change 635528 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Add ldap-replica2003/2004 to conftool-data

https://gerrit.wikimedia.org/r/635528

Change 635528 merged by Muehlenhoff:
[operations/puppet@production] Add ldap-replica2003/2004 to conftool-data

https://gerrit.wikimedia.org/r/635528

Change 635499 merged by Muehlenhoff:
[operations/puppet@production] acmechief: Add ldap-replica1001/1002

https://gerrit.wikimedia.org/r/635499

Mentioned in SAL (#wikimedia-operations) [2020-10-21T13:21:12Z] <moritzm> pooling ldap-replica2003 T264388

Change 635542 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Install ldap-replica100[12] as additional LDAP replicas

https://gerrit.wikimedia.org/r/635542

Change 635542 merged by Muehlenhoff:
[operations/puppet@production] Install ldap-replica100[12] as additional LDAP replicas

https://gerrit.wikimedia.org/r/635542

Mentioned in SAL (#wikimedia-operations) [2020-10-21T15:07:59Z] <moritzm> imported prometheus-openldap-exporter 0+git20171128-3 to buster-wikimedia T264388

Mentioned in SAL (#wikimedia-operations) [2020-10-22T13:01:39Z] <moritzm> pooling ldap-replica2004 T264388

Mentioned in SAL (#wikimedia-operations) [2020-10-22T13:10:15Z] <moritzm> depooling ldap-replica2001/2002 T264388

Mentioned in SAL (#wikimedia-operations) [2020-10-22T13:41:06Z] <moritzm> pooling ldap-replica1001/1002 T264388

Mentioned in SAL (#wikimedia-operations) [2020-10-22T13:55:13Z] <moritzm> depooling ldap-eqiad-replica01/ldap-eqiad-replica02 T264388

All new buster replicas are now pooled and the stretch ones have been depooled. I'll keep them around for another week just in case, then they are going to be removed.

Change 635951 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove Stretch-based LDAP replicas from conftool

https://gerrit.wikimedia.org/r/635951

Change 635951 merged by Muehlenhoff:
[operations/puppet@production] Remove Stretch-based LDAP replicas from conftool

https://gerrit.wikimedia.org/r/635951

Mentioned in SAL (#wikimedia-operations) [2020-10-23T09:51:28Z] <moritzm> masking slapd on the old Stretch replicas to uncover potential direct access outside of the LVSes T264388

cookbooks.sre.hosts.decommission executed by jmm@cumin1001 for hosts: ldap-eqiad-replica01.wikimedia.org

  • ldap-eqiad-replica01.wikimedia.org (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox

Change 637500 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove ldap-eqiad-replica0[12] from acmechief config

https://gerrit.wikimedia.org/r/637500

cookbooks.sre.hosts.decommission executed by jmm@cumin1001 for hosts: ldap-eqiad-replica02.wikimedia.org

  • ldap-eqiad-replica02.wikimedia.org (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox

Change 637503 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove Puppet references for ldap-eqiad*

https://gerrit.wikimedia.org/r/637503

Change 637503 merged by Muehlenhoff:
[operations/puppet@production] Remove Puppet references for ldap-eqiad*

https://gerrit.wikimedia.org/r/637503

cookbooks.sre.hosts.decommission executed by jmm@cumin2001 for hosts: ldap-replica2001.wikimedia.org

  • ldap-replica2001.wikimedia.org (WARN)
    • Failed downtime host on Icinga (likely already removed)
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
  • COMMON_STEPS (WARN)
    • Not all affected DC(s) have been migrated to automatic DNS, a manual patch to the operations/dns repository is required

cookbooks.sre.hosts.decommission executed by jmm@cumin2001 for hosts: ldap-replica2002.wikimedia.org

  • ldap-replica2002.wikimedia.org (WARN)
    • Failed downtime host on Icinga (likely already removed)
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
  • COMMON_STEPS (WARN)
    • Not all affected DC(s) have been migrated to automatic DNS, a manual patch to the operations/dns repository is required

Change 638067 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/dns@master] Remove ldap-replica2001/2002 from DNS

https://gerrit.wikimedia.org/r/638067

Change 638067 merged by Muehlenhoff:
[operations/dns@master] Remove ldap-replica2001/2002 from DNS

https://gerrit.wikimedia.org/r/638067

Change 637500 merged by Muehlenhoff:
[operations/puppet@production] Remove ldap-eqiad-replica0[12] from acmechief config

https://gerrit.wikimedia.org/r/637500

MoritzMuehlenhoff closed this task as Resolved.Mon, Nov 2, 11:55 AM

ldap-replica1001/1002/2003/2004 are now running Buster, old Stretch instances have been removed.