Page MenuHomePhabricator

login.toolforge.org (tools-sgebastion-07) sssd not recognizing new user mdipietro
Closed, ResolvedPublic

Description

The main tools bastion is behaving oddly for use mdipietro. I've cleared sssd cache sever times and rebooted, and it still doesn't recognize the user, though ldapsearch shows it and dev.toolforge.org works fine.

Previous to reboot errors were coming up in sssd:

Jul 27 20:48:01 tools-sgebastion-07 sssd[30538]: More users have the same name [mdipietro@wikimedia.org@wikimedia.org] in SSSD cache. SSSD will not work correctly.

Issues with corrupt caches also showed up then but went away on reboot and clearing caches.
SSH simply says the user is unknown, which the rest of the system agrees on.

root@tools-sgebastion-07:~# id mdipietro
id: ‘mdipietro’: no such user

This will probably need some debug logging turned on for sssd. I am somewhat worried it won't pick up other new users at this rate.

Event Timeline

Bstorm triaged this task as Medium priority.

This works fine on the toolsbeta bastion. I think his account was in a strange state, attempted login on bastion-07, and now that host specifically is not recovering well.

bstorm@toolsbeta-sgebastion-04:~$ id mdipietro
uid=32260(mdipietro) gid=500(wikidev) groups=50380(project-tools),51051(tools.admin),500(wikidev)

It's an odd error. Unfortunately, troubleshooting it will probably require some downtime for the main bastion (or possibly just replacing it with another host behind the DNS name) if it doesn't fix itself soon.

/var/lib/sss/db/cache_wikimedia.org.ldb still seems to have the old (duplicate) account.. I imagine next step would be to stop sssd, clear that file (or folder) and start it again to see what happens.

Andrew claimed this task.
Andrew subscribed.

This was a persistent sssd cache issue. Strangely, sss_cache -E did NOT resolve the problem but rm'ing the cache files did.

I extended the docs at

https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Refresh_sssd_caches#Fixing

to reflect that this might be needed.