Page MenuHomePhabricator

Provision Redis cluster for Fundraising
Closed, ResolvedPublic

Description

Set up a simple master -> slave replicated Redis server. Redis 2 is fine. We will connect to this server from the payments boxes (please enforce write-only permissions), the listeners, and from the Civi jobs.

Authentication would be prudent.

IMO we should use both of the disk backing options, RDB and AOF.

Setting up a staging server would be ideal, but not required. This can probably live on the staging box.

Event Timeline

awight created this task.Mar 17 2016, 8:56 PM
awight updated the task description. (Show Details)Mar 17 2016, 9:21 PM
awight renamed this task from [Epic] Provision high-availability Redis cluster for Fundraising to [Epic] Provision high-availability Redis or Kafka cluster for Fundraising.Mar 24 2016, 10:48 PM
awight updated the task description. (Show Details)
awight renamed this task from [Epic] Provision high-availability Redis or Kafka cluster for Fundraising to [Epic] Provision high-availability Kafka cluster for Fundraising.Mar 30 2016, 6:18 PM
awight updated the task description. (Show Details)
atgo removed a subscriber: atgo.Mar 30 2016, 8:29 PM

@Jgreen
I'm putting this task in the current sprint, not expecting to finish it but hoping we can take some steps forward. Feel free to push back on the timeline, whatever works for you.

awight updated the task description. (Show Details)Apr 14 2016, 11:39 PM
awight updated the task description. (Show Details)Apr 14 2016, 11:46 PM

@Jgreen
I'm not feeling very warm nor fuzzy about the bleeding-edge components that I've been able to line up for this project. So far,

  • WMF will probably be using a 3rd-party Kafka package, and we'd have to follow suit if we want to be compatible. There are big architectural changes between the stable packages and the 0.9 series that we're talking about using. I'm not sure whether we're better off with 0.8, or if there's something to gain from authentication, getting rid of ZooKeeper, and possibly reusing production puppet modules.
  • The PHP client library I'm considering is feeble, lack packaging, and don't compile for HHVM yet.
  • We would be pioneering PHP consumption of Kafka streams at the WMF.

It would be great to chat and to get your feedback on how harebrained this looks. And where to go from here.

Restricted Application added a subscriber: TerraCodes. · View Herald TranscriptApr 19 2016, 6:39 PM

Happy to chat too if you all are having trouble.

Here's a crazy idea...don't use PHP for backend job processing stuff? :D

Should we have another meeting?

@Ottomata
Thanks! I've been toying with the idea of separating the client from the processing jobs, but haven't come up with an architecture that makes sense. In theory, it could be really nice: something abstract feeds our jobs a simple list of records, and we no longer need any queue cruft in the job proper. Transactionality seems hard, though--if a record fails, we can requeue it, but if the whole job fails, we have to reprocess everything, ensure idempotence, etc.

We're tragically overinvested in PHP for processing, so will need to feed the data into existing scripts somehow.

Ottomata added a comment.EditedApr 20 2016, 5:49 PM

Have you talked to Services folks about this? This sounds very similar to the job queuing stuff they are doing for their change propagation service.

Also, have you seen this library?

https://github.com/arnaud-lb/php-rdkafka

It looks fairly up to date: #define PHP_RDKAFKA_VERSION "0.9.1"

But, ja, maybe that will be extra hard to build for HHVM, not sure. Your backend jobs run on HHVM?

@Ottomata
Yes, that one was my top choice, see T131268 for some explanation.

Thanks to the tip about Services, T88459 is a great resource for deduplicating work!

awight renamed this task from [Epic] Provision high-availability Kafka cluster for Fundraising to Provision Redis cluster for Fundraising.Jul 13 2016, 6:30 PM
awight updated the task description. (Show Details)
awight removed subscribers: Ottomata, Joe.

To clarify the changes to the task title and description, we've decided to use Redis rather that Kafka as the replacement queue, see T130304 for the discussion.

awight assigned this task to Jgreen.Jul 20 2016, 10:44 PM
Ejegg closed this task as Resolved.Aug 8 2016, 11:12 PM
Ejegg moved this task from Backlog to Done on the Fundraising Sprint Octopus Untangling board.
Ejegg added a subscriber: Ejegg.

We are using this!

mmodell removed a subscriber: awight.Jun 22 2017, 9:50 PM