Logging at this and the log "swift" via fenari, it seems like these requests are not even hitting swift (or if they are, must be dying hard enough that no reponse is given and nothing is logged).

The error log has died down after ms-be3 was pulled out. This might come back easily...until the replacement hardware is up.

I improved the errors for auth requests and they are now "Couldn't resolve host 'ms-fe.pmtpa.wmnet'", so these are all the same error.

Switched auth URL to an IP to avoid dns lookups for auth requests. I'll see if this works around the dns problems or pushes them down the road.

(In reply to comment #4)

Switched auth URL to an IP to avoid dns lookups for auth requests. I'll see if
this works around the dns problems or pushes them down the road.

Can is down the road :)

This really needs an ops person to look at.

Sorry for not updating the ticket earlier -- I've actually attempted to debug this and have chatted with Aaron last week or the one before that.

I've verified that at the time errors were spawned, DNS replies were coming into the system. Also, it's peculiar how no other infrastructure seems to be affected, not even the application server ones (this apparently affects only job runners). It's also something that has manifested recently, possibly after the precise upgrade.

I have some suspections that it may be curl-related (curl has an internal DNS cache that is enabled by default, so it's not just simple libc resolver calls).

I've asked Aaron to isolate the code in question and produce some kind of script that we can run repetively, reproduce and run under strace/gdb, rather than trying to attach them on random jobrunners and hope we catch it. The issue happens on jobrunners, so it's under php cli anyway, so the environment won't be that different anyway.

(In reply to comment #6 by Faidon)

I've asked Aaron to isolate the code in question and produce some kind of
script that we can run repetively, reproduce and run under strace/gdb, rather
than trying to attach them on random jobrunners and hope we catch it.

Faidon / Aaron: Has this happened yet?

(In reply to comment #6 by Faidon)

I've asked Aaron to isolate the code in question and produce some kind of
script that we can run repetively, reproduce and run under strace/gdb, rather
than trying to attach them on random jobrunners and hope we catch it.

Faidon / Aaron: Has this happened yet?

(In reply to comment #6 by Faidon)

I've asked Aaron to isolate the code in question and produce some kind of
script that we can run repetively, reproduce and run under strace/gdb, rather
than trying to attach them on random jobrunners and hope we catch it.

Faidon / Aaron: Has this happened yet?

Tried that a long time ago, didn't work.

This is still occurring from time to time :(

$ zgrep -c 'resolve host' swift-backend.log-201312*
swift-backend.log-20131201.gz:0
swift-backend.log-20131202.gz:0
swift-backend.log-20131203.gz:0
swift-backend.log-20131204.gz:0
swift-backend.log-20131205.gz:0
swift-backend.log-20131206.gz:0
swift-backend.log-20131207.gz:0
swift-backend.log-20131208.gz:0
swift-backend.log-20131209.gz:0
swift-backend.log-20131210.gz:51
swift-backend.log-20131211.gz:115
swift-backend.log-20131212.gz:35
swift-backend.log-20131213.gz:0
swift-backend.log-20131214.gz:0
swift-backend.log-20131215.gz:0
$

Not seeing these anymore