@Jgreen
I'm putting this task in the current sprint, not expecting to finish it but hoping we can take some steps forward. Feel free to push back on the timeline, whatever works for you.

@Jgreen
I'm not feeling very warm nor fuzzy about the bleeding-edge components that I've been able to line up for this project. So far,

• WMF will probably be using a 3rd-party Kafka package, and we'd have to follow suit if we want to be compatible. There are big architectural changes between the stable packages and the 0.9 series that we're talking about using. I'm not sure whether we're better off with 0.8, or if there's something to gain from authentication, getting rid of ZooKeeper, and possibly reusing production puppet modules.
• The PHP client library I'm considering is feeble, lack packaging, and don't compile for HHVM yet.
• We would be pioneering PHP consumption of Kafka streams at the WMF.

It would be great to chat and to get your feedback on how harebrained this looks. And where to go from here.

Happy to chat too if you all are having trouble.

Here's a crazy idea...don't use PHP for backend job processing stuff? :D

Should we have another meeting?

@Ottomata
Thanks! I've been toying with the idea of separating the client from the processing jobs, but haven't come up with an architecture that makes sense. In theory, it could be really nice: something abstract feeds our jobs a simple list of records, and we no longer need any queue cruft in the job proper. Transactionality seems hard, though--if a record fails, we can requeue it, but if the whole job fails, we have to reprocess everything, ensure idempotence, etc.

We're tragically overinvested in PHP for processing, so will need to feed the data into existing scripts somehow.

Have you talked to Services folks about this? This sounds very similar to the job queuing stuff they are doing for their change propagation service.

Also, have you seen this library?

https://github.com/arnaud-lb/php-rdkafka

It looks fairly up to date: #define PHP_RDKAFKA_VERSION "0.9.1"

But, ja, maybe that will be extra hard to build for HHVM, not sure. Your backend jobs run on HHVM?

@Ottomata
Yes, that one was my top choice, see T131268 for some explanation.

Thanks to the tip about Services, T88459 is a great resource for deduplicating work!

To clarify the changes to the task title and description, we've decided to use Redis rather that Kafka as the replacement queue, see T130304 for the discussion.