Page MenuHomePhabricator
Create Task
Maniphest T224557

Migrate ldap/corp replicas to Stretch/Buster
Closed, ResolvedPublic
Actions

Description

dubnium.wikimedia.org and pollux.wikimedia.org are the OpenLDAP replicas of the OIT directory. Given that these are Ganeti instances and only have a single user it's probably best to create a new cluster, test it and then switch the MXes to use it instead of dubnium/pollux. Let's maybe also use a name like ldapcorp1001 to make the DC naming more visible.

Details

SubjectRepoBranchLines +/-
Remove DNS entries for dubnium/polluxoperations/dnsmaster+0 -4
Remove remaining Puppet refs of old jessie LDAP corp serversoperations/puppetproduction+2 -31
Turn old LDAP replicas into sparesoperations/puppetproduction+1 -1
Switch ldap-corp.eqiad.wikimedia.org to ldap-corp1001operations/dnsmaster+1 -1
Switch ldap-corp.codfw.wikimedia.org to ldap-corp2001operations/dnsmaster+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

• MoritzMuehlenhoff created this task.May 29 2019, 11:03 AM
• MoritzMuehlenhoff mentioned this in T224549: Track remaining jessie systems in production.
ArielGlenn triaged this task as Medium priority.Jun 11 2019, 7:58 AM
• MoritzMuehlenhoff claimed this task.Sep 27 2019, 2:10 PM
gerritbot added a comment.Nov 27 2019, 11:46 AM

Change 553323 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/dns@master] Switch ldap-corp.codfw.wikimedia.org to ldap-corp2001

https://gerrit.wikimedia.org/r/553323

gerritbot added a project: Patch-For-Review.Nov 27 2019, 11:46 AM
gerritbot added a comment.Dec 5 2019, 12:17 PM

Change 553323 merged by Muehlenhoff:
[operations/dns@master] Switch ldap-corp.codfw.wikimedia.org to ldap-corp2001

https://gerrit.wikimedia.org/r/553323

Maintenance_bot removed a project: Patch-For-Review.Dec 5 2019, 1:10 PM
gerritbot added a comment.Dec 5 2019, 1:22 PM

Change 554852 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/dns@master] Switch ldap-corp.eqiad.wikimedia.org to ldap-corp1001

https://gerrit.wikimedia.org/r/554852

gerritbot added a project: Patch-For-Review.Dec 5 2019, 1:22 PM
gerritbot added a comment.Dec 9 2019, 9:10 AM

Change 554852 merged by Muehlenhoff:
[operations/dns@master] Switch ldap-corp.eqiad.wikimedia.org to ldap-corp1001

https://gerrit.wikimedia.org/r/554852

Maintenance_bot removed a project: Patch-For-Review.Dec 9 2019, 10:10 AM
• MoritzMuehlenhoff mentioned this in T231015: eqiad/codfw: 2 VMs for corp LDAP replicas.Dec 9 2019, 4:52 PM
gerritbot added a comment.Dec 9 2019, 4:52 PM

Change 555990 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Turn old LDAP replicas into spares

https://gerrit.wikimedia.org/r/555990

gerritbot added a project: Patch-For-Review.Dec 9 2019, 4:52 PM
gerritbot added a comment.Dec 10 2019, 9:13 AM

Change 555990 merged by Muehlenhoff:
[operations/puppet@production] Turn old LDAP replicas into spares

https://gerrit.wikimedia.org/r/555990

Maintenance_bot removed a project: Patch-For-Review.Dec 10 2019, 10:11 AM
Stashbot added a comment.Dec 10 2019, 10:13 AM

Mentioned in SAL (#wikimedia-operations) [2019-12-10T10:13:52Z] <moritzm> stopping slapd on dubnium/pollux following application of the spare role T224557

ops-monitoring-bot added a comment.Dec 18 2019, 2:37 PM

cookbooks.sre.hosts.decommission executed by jmm@cumin2001 for hosts: pollux.wikimedia.org

  • pollux.wikimedia.org (FAIL)
    • Downtimed host on Icinga
    • No management interface found (likely a VM)
    • Unable to connect to the host, wipe of bootloaders will not be performed: Cumin execution failed (exit_code=2)
    • Failed to shutdown, manual intervention required: Cumin execution failed (exit_code=2)
    • Set Netbox status on VM not yet supported: manual intervention required
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

ERROR: some step on some host failed, check the bolded items above

Stashbot added a comment.Dec 23 2019, 11:04 AM

Mentioned in SAL (#wikimedia-operations) [2019-12-23T11:04:19Z] <moritzm> removing dubnium in ganeti T224557

ops-monitoring-bot added a comment.Dec 23 2019, 11:06 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin1001 for hosts: dubnium.wikimedia.org

  • dubnium.wikimedia.org (FAIL)
    • Downtimed host on Icinga
    • No management interface found (likely a VM)
    • Unable to connect to the host, wipe of bootloaders will not be performed: Cumin execution failed (exit_code=2)
    • Failed to shutdown, manual intervention required: Cumin execution failed (exit_code=2)
    • Set Netbox status on VM not yet supported: manual intervention required
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

ERROR: some step on some host failed, check the bolded items above

gerritbot added a comment.Dec 23 2019, 11:13 AM

Change 560383 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove remaining Puppet refs of old jessie LDAP corp servers

https://gerrit.wikimedia.org/r/560383

gerritbot added a project: Patch-For-Review.Dec 23 2019, 11:13 AM
gerritbot added a comment.Dec 23 2019, 12:14 PM

Change 560383 merged by Muehlenhoff:
[operations/puppet@production] Remove remaining Puppet refs of old jessie LDAP corp servers

https://gerrit.wikimedia.org/r/560383

• MoritzMuehlenhoff closed this task as Resolved.Dec 23 2019, 1:06 PM

This is complete. The new systems based on buster are ldap-corp1001 and ldap-corp2001 and dubnium/pollux have been decomissioned.

Maintenance_bot removed a project: Patch-For-Review.Dec 23 2019, 1:11 PM
ayounsi reopened this task as Open.May 18 2020, 2:10 PM
ayounsi subscribed.

Not sure if I'm re-opening the proper task, but looks relevant.

dubnium/pollux are still present in DNS while I don't think they should (and they don't reply to pings).

gerritbot added a comment.May 18 2020, 3:14 PM

Change 597099 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/dns@master] Remove DNS entries for dubnium/pollux

https://gerrit.wikimedia.org/r/597099

gerritbot added a project: Patch-For-Review.May 18 2020, 3:14 PM
gerritbot added a comment.May 18 2020, 3:52 PM

Change 597099 merged by Muehlenhoff:
[operations/dns@master] Remove DNS entries for dubnium/pollux

https://gerrit.wikimedia.org/r/597099

• MoritzMuehlenhoff closed this task as Resolved.May 18 2020, 3:53 PM
In T224557#6144902, @ayounsi wrote:

Not sure if I'm re-opening the proper task, but looks relevant.

dubnium/pollux are still present in DNS while I don't think they should (and they don't reply to pings).

Thanks, good catch! Fixed.

Maintenance_bot removed a project: Patch-For-Review.May 18 2020, 4:12 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits