Page MenuHomePhabricator

codfw1dev: LDAP database content seems to have lost years of content
Closed, InvalidPublic

Description

When working on T338778: cloudservices2004-dev: reimage into new network setup cloudservices2005-dev was made LDAP primary, which apparently now has old data (or at least, missing accounts).

Example:

aborrero@cloudservices2005-dev:~ $ sudo ldapsearch -H ldapi:/// -Y EXTERNAL | grep -i andrew
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
uniqueMember: uid=andrew,ou=people,dc=wikimedia,dc=org
member: uid=andrew,ou=people,dc=wikimedia,dc=org
member: uid=andrew,ou=people,dc=wikimedia,dc=org
member: uid=andrew,ou=people,dc=wikimedia,dc=org
member: uid=labtestandrew,ou=people,dc=wikimedia,dc=org
aborrero@cloudservices2005-dev:~  $ sudo ldapsearch -H ldapi:/// -Y EXTERNAL | grep -i aborrero
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0

Explore backup restoration options.

Related Objects

StatusSubtypeAssignedTask
Resolvedaborrero
Resolvedaborrero
Resolvedaborrero
Resolvedaborrero
Resolvedayounsi
Resolvedcmooney
ResolvedPapaul
Resolvedcmooney
Resolvedcmooney
Resolvedaborrero
Resolvedaborrero
Resolvedaborrero
Resolvedtaavi
Opencmooney
Resolvedaborrero
Opencmooney
Resolvedaborrero
Resolvedaborrero
Resolvedaborrero
Resolvedaborrero
Resolvedaborrero
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
OpenAndrew
OpenAndrew
ResolvedAndrew
Resolvedaborrero
OpenNone
OpenNone
Resolvedaborrero
Resolvedcmooney
Resolvedrook
ResolvedNone
Resolvedrook
Resolvedaborrero
InvalidNone

Event Timeline

This is the list of cloud hosts that receive backups:

50: cloudcontrol1003.wikimedia.org-fd
51: cloudcontrol1004.wikimedia.org-fd
52: cloudcontrol1005.wikimedia.org-fd
53: cloudcontrol1006.wikimedia.org-fd
54: cloudcontrol1007.wikimedia.org-fd
55: cloudcontrol2001-dev.codfw.wmnet-fd
56: cloudcontrol2001-dev.wikimedia.org-fd
57: cloudcontrol2003-dev.wikimedia.org-fd
58: cloudcontrol2004-dev.codfw.wmnet-fd
59: cloudcontrol2004-dev.wikimedia.org-fd
60: cloudcontrol2005-dev.codfw.wmnet-fd
61: cloudcontrol2005-dev.wikimedia.org-fd
62: cloudcumin1001.eqiad.wmnet-fd
63: cloudcumin2001.codfw.wmnet-fd
64: cloudmetrics1001.eqiad.wmnet-fd
65: cloudmetrics1002.eqiad.wmnet-fd
66: cloudmetrics1003.eqiad.wmnet-fd
67: cloudmetrics1004.eqiad.wmnet-fd
68: cloudweb1003.wikimedia.org-fd
69: cloudweb1004.wikimedia.org-fd
70: cloudweb2001-dev.wikimedia.org-fd
71: cloudweb2002-dev.wikimedia.org-fd

Apparently, cloudservices hosts don't receive backups, which is very unfortunate.

Two things here:

  • first, somehow the DB is apparently still present. It was the query that was misleading me:
aborrero@cloudservices2005-dev:~$ sudo ldapsearch -H ldapi:/// -x uid=aborrero | grep -i aborrero
[..]
dn: uid=aborrero,ou=people,dc=wikimedia,dc=org
uid: aborrero
sn: Aborrero
cn: Aborrero

(thanks @taavi for the pointer)