I just found out that there seems to be a bug in HHVM's fsockopen implementation, so that when you try to connect to a dead host, it will not respect the timeout you pass it from the invokation.

As a test I ran:

php -r 'var_dump(fsockopen("kafka1012.eqiad.wmnet", 9092, $errno, $errstr, 0.1));'

Which is supposed to return an error in 0.1 seconds; it does so on zend php, but it doesn't seem to be the case with HHVM, where the request hangs for several seconds.

My (very early) testing seems to show that HHVM 3.11 behaves correctly in this case.

So the workaround while we deploy a new HHVM package (which will take some time) is to depool kafka servers whenever we need to reboot them, or one goes down for good.

This is far from ideal, of course.

Hmm, I think we could put kafka clusters into pybal, and use LVS for bootstrapping/metadata lookups. Then clients could use something like kafka.analytics-eqiad.svc.eqiad.wmnet for bootstrapping. This might make pooling/depooling and automatically failing over bootstrap meta data requests easier.

We'd have to confirm that this would work with all Kafka clients, but I'm pretty sure it would.

In T125084#1974286, @Ottomata wrote:

Hmm, I think we could put kafka clusters into pybal, and use LVS for bootstrapping/metadata lookups. Then clients could use something like kafka.analytics-eqiad.svc.eqiad.wmnet for bootstrapping. This might make pooling/depooling and automatically failing over bootstrap meta data requests easier.

We'd have to confirm that this would work with all Kafka clients, but I'm pretty sure it would.

That might have helped here but it's not really a solution to _this_ problem. Even if e.g. *all* brokers are down, the outage should not cascade to the (API) appserver fleet.

Reminder to delete workaround documentation here and here when this ticket is fixed.

Edited task detail to point to T88732: Decouple logging infrastructure failures from MediaWiki logging which is the log/redis issue faidon was referring to. Got solved by using syslog/UDP for transport.

Added also a note in https://wikitech.wikimedia.org/wiki/Service_restarts

For the reference, upstream fix: https://github.com/facebook/hhvm/commit/88e9ca810d1af78b63cf1668841fa38b2b0a01ba

Change 267196 had a related patch set uploaded (by MaxSem):
KafkaHandler: allow customizing timeouts

https://gerrit.wikimedia.org/r/267196

Change 267200 had a related patch set uploaded (by MaxSem):
Reduce Kafka timeouts

https://gerrit.wikimedia.org/r/267200

I think we should just backport this patch to our current package while we are confident releasing a new one. This is absolutely devastating as a single server going down (not just kafka, but anything we connect to using fsockopen) can cause a site-wide outage.

Thanks @MaxSem for taking the time of finding the upstream fix.

Change 267228 had a related patch set uploaded (by Giuseppe Lavagetto):
Add support for float timeouts in socket streams

https://gerrit.wikimedia.org/r/267228

Patch applied and new package built.

The package was installed on labs and my test of fsockopen now shows that the timeout is respected.

I am going to install the newbuilt package on the canaries now.

Change 267228 merged by Giuseppe Lavagetto:
Add support for float timeouts in socket streams

https://gerrit.wikimedia.org/r/267228

Change 267196 merged by jenkins-bot:
KafkaHandler: allow customizing timeouts

https://gerrit.wikimedia.org/r/267196

All appservers have been upgraded, I'll perform some tests tomorrow to ensure this is solved.

Change 267200 merged by jenkins-bot:
Reduce Kafka timeouts

https://gerrit.wikimedia.org/r/267200

Change 273488 had a related patch set uploaded (by Elukey):
Add kafka1012.eqiad.wmnet back to the media-wiki config.

https://gerrit.wikimedia.org/r/273488

Change 273488 merged by jenkins-bot:
Add kafka1012.eqiad.wmnet back to the media-wiki config.

https://gerrit.wikimedia.org/r/273488