Page MenuHomePhabricator
Create Task
Maniphest T339869

codfw1dev: LDAP database content seems to have lost years of content
Closed, InvalidPublic
Actions

Description

When working on T338778: cloudservices2004-dev: reimage into new network setup cloudservices2005-dev was made LDAP primary, which apparently now has old data (or at least, missing accounts).

Example:

aborrero@cloudservices2005-dev:~ $ sudo ldapsearch -H ldapi:/// -Y EXTERNAL | grep -i andrew
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
uniqueMember: uid=andrew,ou=people,dc=wikimedia,dc=org
member: uid=andrew,ou=people,dc=wikimedia,dc=org
member: uid=andrew,ou=people,dc=wikimedia,dc=org
member: uid=andrew,ou=people,dc=wikimedia,dc=org
member: uid=labtestandrew,ou=people,dc=wikimedia,dc=org
aborrero@cloudservices2005-dev:~  $ sudo ldapsearch -H ldapi:/// -Y EXTERNAL | grep -i aborrero
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0

Explore backup restoration options.

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolved• aborreroT296411 cloud: decide on general idea for having cloud-dedicated hardware provide service in the cloud realm & the internet
 Resolved• aborreroT297596 have cloud hardware servers in the cloud realm using a dedicated LB layer
 Resolved• aborreroT324992 cloudlb: create PoC on codfw
 Resolved• aborreroT327908 rename cloudgw2001-dev into cloudlb2001-dev
 ResolvedayounsiT327930 Is Vlan 2122 cloud-support1-b-codfw required?
 ResolvedcmooneyT327919 Configure cloudsw1-b1-codfw and migrate cloud hosts in codfw B1 to it
 ResolvedPapaulT331470 Run 2x1G links from asw-b1-codfw to cloudsw1-b1-codfw
Restricted Task
 ResolvedcmooneyT333316 Homer unable to commit config to cloudsw1-b1-codfw (QFX5120 21.4R3.16)
 ResolvedcmooneyT336368 cloudgw: review security policy for edge network
 Resolved• aborreroT332153 cloudlb PoC: prepare backends
 Resolved• aborreroT336963 cloudcontrol2001-dev can't reach cloud-vps public IPs
 Resolved• aborreroT335759 cloud-private subnet: introduce new domain
 ResolvedtaaviT336566 cloud: failures resolving some `wikimedia.cloud` domains
 OpencmooneyT346428 Netbox: Add support for our complex host network setups in provision script
 Resolved• aborreroT346410 need private IPs for cloudvirt200[4-6]-dev.codfw.wmnet
 OpencmooneyT347375 Netbox device location information not available on the first Puppet run of a device
 Resolved• aborreroT335760 Modify Bird module to allow source IP to be passed to template
 Resolved• aborreroT336071 cloudlb: figure out routing
 Resolved• aborreroT337758 puppet: interface::route resource does not persists settings
 Resolved• aborreroT338936 cloudlb: figure out plans for eqiad1
 Resolved• aborreroT338937 cloudlb: review swift/radosgw status
ResolvedAndrewT341380 Open swift port (28080) to the public internet
 ResolvedAndrewT341484 Horizon object store UI doesn't allow uploading of files
 ResolvedAndrewT341640 Move Horizon deploy to a Docker container
 ResolvedAndrewT341509 radosgw+keystone chokes on projects with '-' in their id
 ResolvedAndrewT343158 Support OpenStack projects where project name != project id
 ResolvedAndrewT366679 openstack-browser support for projects where id != name
 ResolvedAndrewT347927 Better swift error messages for projects with invalid chars
 ResolvedAndrewT366301 Document and enforce restrictions on openstack project names
 Resolved• aborreroT340536 cloud-private: figure out, implement and test cross-DC traffic
 OpenNoneT317177 [tracking] Don't keep on the public vlans hosts that don't require it
 ResolvedNoneT340446 Cloud VPS Designate setup improvements
 Resolved• aborreroT307357 Move cloud vps ns-recursor IPs to host/row-independent addressing
 ResolvedcmooneyT336587 cloudservices[2004/2005]-dev & cloudweb2002-dev: connect them to cloudsw so they can have cloud-private vlan
 Resolved• rookT121478 i18n support
 ResolvedNoneT343054 [bug] PAWS has problems loading
 Resolved• rookT338981 [Rollback] paws to jupyterlab 4.0
 Resolved• aborreroT338778 cloudservices2004-dev: reimage into new network setup
 InvalidNoneT339869 codfw1dev: LDAP database content seems to have lost years of content

Event Timeline

• aborrero created this task.Jun 19 2023, 4:40 PM
• aborrero added a comment.Jun 20 2023, 8:42 AM

I will try with this procedure: https://wikitech.wikimedia.org/wiki/Bacula#Restore_from_a_non-existent_host_(missing_private_key)

• aborrero added a comment.Jun 20 2023, 8:47 AM

This is the list of cloud hosts that receive backups:

50: cloudcontrol1003.wikimedia.org-fd
51: cloudcontrol1004.wikimedia.org-fd
52: cloudcontrol1005.wikimedia.org-fd
53: cloudcontrol1006.wikimedia.org-fd
54: cloudcontrol1007.wikimedia.org-fd
55: cloudcontrol2001-dev.codfw.wmnet-fd
56: cloudcontrol2001-dev.wikimedia.org-fd
57: cloudcontrol2003-dev.wikimedia.org-fd
58: cloudcontrol2004-dev.codfw.wmnet-fd
59: cloudcontrol2004-dev.wikimedia.org-fd
60: cloudcontrol2005-dev.codfw.wmnet-fd
61: cloudcontrol2005-dev.wikimedia.org-fd
62: cloudcumin1001.eqiad.wmnet-fd
63: cloudcumin2001.codfw.wmnet-fd
64: cloudmetrics1001.eqiad.wmnet-fd
65: cloudmetrics1002.eqiad.wmnet-fd
66: cloudmetrics1003.eqiad.wmnet-fd
67: cloudmetrics1004.eqiad.wmnet-fd
68: cloudweb1003.wikimedia.org-fd
69: cloudweb1004.wikimedia.org-fd
70: cloudweb2001-dev.wikimedia.org-fd
71: cloudweb2002-dev.wikimedia.org-fd

Apparently, cloudservices hosts don't receive backups, which is very unfortunate.

• aborrero closed this task as Invalid.Jun 20 2023, 9:00 AM

Two things here:

  • first, somehow the DB is apparently still present. It was the query that was misleading me:
aborrero@cloudservices2005-dev:~$ sudo ldapsearch -H ldapi:/// -x uid=aborrero | grep -i aborrero
[..]
dn: uid=aborrero,ou=people,dc=wikimedia,dc=org
uid: aborrero
sn: Aborrero
cn: Aborrero

(thanks @taavi for the pointer)

• aborrero removed a subtask: T339894: cloudservices: codfw1dev: fix backups.Jun 26 2023, 2:49 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits