Page MenuHomePhabricator
Create Task
Maniphest T264388

Migrate LDAP replicas to Buster
Closed, ResolvedPublic
Actions

Description

The RO replicas are still running Stretch. The eqiad servers currently use a non-standard naming scheme, they will be fixed to also use ldap-replica100[12].

Details

SubjectRepoBranchLines +/-
Remove ldap-eqiad-replica0[12] from acmechief configoperations/puppetproduction+0 -1
Remove ldap-replica2001/2002 from DNSoperations/dnsmaster+0 -4
Remove Puppet references for ldap-eqiad*operations/puppetproduction+0 -20
Remove Stretch-based LDAP replicas from conftooloperations/puppetproduction+0 -4
Install ldap-replica100[12] as additional LDAP replicasoperations/puppetproduction+3 -1
acmechief: Add ldap-replica1001/1002operations/puppetproduction+1 -0
Add ldap-replica2003/2004 to conftool-dataoperations/puppetproduction+2 -0
acmechief: Also allow ldap-replica2003/2004operations/puppetproduction+1 -1
Install ldap-replica200[34] as additional LDAP replicasoperations/puppetproduction+3 -5
Customize query in gerrit

Event Timeline

MoritzMuehlenhoff created this task.Oct 2 2020, 7:50 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 2 2020, 7:50 AM
MoritzMuehlenhoff triaged this task as Medium priority.Oct 2 2020, 7:53 AM
gerritbot added a comment.Oct 7 2020, 7:56 AM

Change 632648 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Install ldap-replica200[34] as additional LDAP replicas

https://gerrit.wikimedia.org/r/632648

gerritbot added a project: Patch-For-Review.Oct 7 2020, 7:56 AM
gerritbot added a comment.Oct 19 2020, 1:10 PM

Change 632648 merged by Muehlenhoff:
[operations/puppet@production] Install ldap-replica200[34] as additional LDAP replicas

https://gerrit.wikimedia.org/r/632648

Maintenance_bot removed a project: Patch-For-Review.Oct 19 2020, 1:10 PM
Stashbot added a comment.Oct 19 2020, 1:26 PM

Mentioned in SAL (#wikimedia-operations) [2020-10-19T13:26:22Z] <moritzm> import prometheus-openldap-exporter 0+git20171128-2+deb10u1 for buster-wikimedia T264388

gerritbot added a comment.Oct 19 2020, 1:40 PM

Change 634974 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] acmechief: Also allow ldap-replica2003/2004

https://gerrit.wikimedia.org/r/634974

gerritbot added a project: Patch-For-Review.Oct 19 2020, 1:40 PM
gerritbot added a comment.Oct 21 2020, 7:07 AM

Change 634974 merged by Muehlenhoff:
[operations/puppet@production] acmechief: Also allow ldap-replica2003/2004

https://gerrit.wikimedia.org/r/634974

Maintenance_bot removed a project: Patch-For-Review.Oct 21 2020, 7:10 AM
gerritbot added a comment.Oct 21 2020, 8:09 AM

Change 635499 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] acmechief: Add ldap-replica1001/1002

https://gerrit.wikimedia.org/r/635499

gerritbot added a project: Patch-For-Review.Oct 21 2020, 8:09 AM
gerritbot added a comment.Oct 21 2020, 11:33 AM

Change 635528 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Add ldap-replica2003/2004 to conftool-data

https://gerrit.wikimedia.org/r/635528

gerritbot added a comment.Oct 21 2020, 12:55 PM

Change 635528 merged by Muehlenhoff:
[operations/puppet@production] Add ldap-replica2003/2004 to conftool-data

https://gerrit.wikimedia.org/r/635528

gerritbot added a comment.Oct 21 2020, 12:56 PM

Change 635499 merged by Muehlenhoff:
[operations/puppet@production] acmechief: Add ldap-replica1001/1002

https://gerrit.wikimedia.org/r/635499

Stashbot added a comment.Oct 21 2020, 1:21 PM

Mentioned in SAL (#wikimedia-operations) [2020-10-21T13:21:12Z] <moritzm> pooling ldap-replica2003 T264388

gerritbot added a comment.Oct 21 2020, 1:39 PM

Change 635542 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Install ldap-replica100[12] as additional LDAP replicas

https://gerrit.wikimedia.org/r/635542

gerritbot added a comment.Oct 21 2020, 2:06 PM

Change 635542 merged by Muehlenhoff:
[operations/puppet@production] Install ldap-replica100[12] as additional LDAP replicas

https://gerrit.wikimedia.org/r/635542

Stashbot added a comment.Oct 21 2020, 3:08 PM

Mentioned in SAL (#wikimedia-operations) [2020-10-21T15:07:59Z] <moritzm> imported prometheus-openldap-exporter 0+git20171128-3 to buster-wikimedia T264388

Stashbot added a comment.Oct 22 2020, 1:01 PM

Mentioned in SAL (#wikimedia-operations) [2020-10-22T13:01:39Z] <moritzm> pooling ldap-replica2004 T264388

Stashbot added a comment.Oct 22 2020, 1:10 PM

Mentioned in SAL (#wikimedia-operations) [2020-10-22T13:10:15Z] <moritzm> depooling ldap-replica2001/2002 T264388

Stashbot added a comment.Oct 22 2020, 1:41 PM

Mentioned in SAL (#wikimedia-operations) [2020-10-22T13:41:06Z] <moritzm> pooling ldap-replica1001/1002 T264388

Stashbot added a comment.Oct 22 2020, 1:55 PM

Mentioned in SAL (#wikimedia-operations) [2020-10-22T13:55:13Z] <moritzm> depooling ldap-eqiad-replica01/ldap-eqiad-replica02 T264388

MoritzMuehlenhoff added a comment.Oct 22 2020, 2:58 PM

All new buster replicas are now pooled and the stretch ones have been depooled. I'll keep them around for another week just in case, then they are going to be removed.

gerritbot added a comment.Oct 23 2020, 8:55 AM

Change 635951 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove Stretch-based LDAP replicas from conftool

https://gerrit.wikimedia.org/r/635951

gerritbot added a comment.Oct 23 2020, 9:43 AM

Change 635951 merged by Muehlenhoff:
[operations/puppet@production] Remove Stretch-based LDAP replicas from conftool

https://gerrit.wikimedia.org/r/635951

Stashbot added a comment.Oct 23 2020, 9:51 AM

Mentioned in SAL (#wikimedia-operations) [2020-10-23T09:51:28Z] <moritzm> masking slapd on the old Stretch replicas to uncover potential direct access outside of the LVSes T264388

ops-monitoring-bot added a comment.Oct 29 2020, 2:26 PM

cookbooks.sre.hosts.decommission executed by jmm@cumin1001 for hosts: ldap-eqiad-replica01.wikimedia.org

  • ldap-eqiad-replica01.wikimedia.org (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
gerritbot added a comment.Oct 29 2020, 2:29 PM

Change 637500 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove ldap-eqiad-replica0[12] from acmechief config

https://gerrit.wikimedia.org/r/637500

ops-monitoring-bot added a comment.Oct 29 2020, 2:33 PM

cookbooks.sre.hosts.decommission executed by jmm@cumin1001 for hosts: ldap-eqiad-replica02.wikimedia.org

  • ldap-eqiad-replica02.wikimedia.org (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
gerritbot added a comment.Oct 29 2020, 2:35 PM

Change 637503 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove Puppet references for ldap-eqiad*

https://gerrit.wikimedia.org/r/637503

gerritbot added a comment.Oct 29 2020, 2:48 PM

Change 637503 merged by Muehlenhoff:
[operations/puppet@production] Remove Puppet references for ldap-eqiad*

https://gerrit.wikimedia.org/r/637503

ops-monitoring-bot added a comment.Nov 2 2020, 10:57 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2001 for hosts: ldap-replica2001.wikimedia.org

  • ldap-replica2001.wikimedia.org (WARN)
    • Failed downtime host on Icinga (likely already removed)
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
  • COMMON_STEPS (WARN)
    • Not all affected DC(s) have been migrated to automatic DNS, a manual patch to the operations/dns repository is required
ops-monitoring-bot added a comment.Nov 2 2020, 11:13 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2001 for hosts: ldap-replica2002.wikimedia.org

  • ldap-replica2002.wikimedia.org (WARN)
    • Failed downtime host on Icinga (likely already removed)
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
  • COMMON_STEPS (WARN)
    • Not all affected DC(s) have been migrated to automatic DNS, a manual patch to the operations/dns repository is required
gerritbot added a comment.Nov 2 2020, 11:14 AM

Change 638067 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/dns@master] Remove ldap-replica2001/2002 from DNS

https://gerrit.wikimedia.org/r/638067

gerritbot added a comment.Nov 2 2020, 11:16 AM

Change 638067 merged by Muehlenhoff:
[operations/dns@master] Remove ldap-replica2001/2002 from DNS

https://gerrit.wikimedia.org/r/638067

gerritbot added a comment.Nov 2 2020, 11:35 AM

Change 637500 merged by Muehlenhoff:
[operations/puppet@production] Remove ldap-eqiad-replica0[12] from acmechief config

https://gerrit.wikimedia.org/r/637500

MoritzMuehlenhoff closed this task as Resolved.Nov 2 2020, 11:55 AM

ldap-replica1001/1002/2003/2004 are now running Buster, old Stretch instances have been removed.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits