Possibly related:

$ ldapsearch -xLLL '(objectclass=groupofnames)'|grep precise
dn: cn=tools.test-lighttpd-precise,ou=servicegroups,dc=wikimedia,dc=org
cn: tools.test-lighttpd-precise
Size limit exceeded (4)

This isn't the exact LDAP search that the bot is doing. I'll track that down and see if it is being truncated.

The real LDAP query and logic is basically:

$ ldapsearch -xLLL -b ou=servicegroups,dc=wikimedia,dc=org -s sub '(objectclass=groupofnames)' cn | grep 'cn:' | awk '{print $2}

This seems to run without truncation, returns 1649 results, and includes tools.precise-tools.

I added a bit of live logging to stashbot to track this down and restarted the bot. That restart seems to have 'fixed' the problem which is not entirely helpful.

[18:38]  <    bd808>	!log tools.precise-tools test
[18:38]  < stashbot>	Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.precise-tools/SAL

This makes it seem likely that there is some problem with the caching mechanism or a way that the LDAP search becomes broken over time. I'm going to restart again with even more extra logging which may or may not help if we can recreate the problem.

Mentioned in SAL (#wikimedia-labs) [2017-01-31T01:49:41Z] <bd808> Restarted to pick up some temporary logging added to debug T156652

Mentioned in SAL (#wikimedia-labs) [2017-02-03T19:35:41Z] <bd808> Updated to 6f45ed3 Fix LDAP cache expiration check (T156652)