Page MenuHomePhabricator

Setup rsyncable git fat store to host Logstash plugins
Closed, DeclinedPublic

Description

Logstash 1.5.x has dropped the logstash-contrib package and replaced it with a plugin management script that installs non-core plugins via Ruby's bundler.

The plugin install script can be told to install a plugin from a local gem file rather than fetching from rubyforge, so I can use a Puppet define to automate this and install gems from a Trebuchet managed repo. Putting blobs in git is yuck so I'd like to use git-fat similarly to the way it is used for elasticsearch plugins. Gem files are not java jars however so the gems should be stored somewhere other than archiva.

We need to create an rsync module that is accessible from labs and production that can hold the gem binaries as well as a procedure for adding new gems there in a similar manner to the process for updating Archiva.

Event Timeline

bd808 created this task.Jul 28 2015, 12:30 AM
bd808 claimed this task.
bd808 raised the priority of this task from to Normal.
bd808 updated the task description. (Show Details)
bd808 added subscribers: bd808, Ottomata.
Restricted Application added subscribers: Matanya, Aklapper. · View Herald TranscriptJul 28 2015, 12:30 AM
hashar added a subscriber: hashar.EditedJul 28 2015, 9:12 AM

wrote before I realized Logstash apparently uses JRuby 1.9 so should not be an issue

A gotcha is the gem binaries are compiled against a ruby version. If you deploy on different Linux distributions / vary ruby version, you end up with an incompatible ABI. So you would need to namespace the binaries modules by ABI version :-( Potentially has branches in operations/software/logstash/plugins.git.

In ruby world you could use rake-compiler ( https://github.com/rake-compiler/rake-compiler#generating-native-rubygems ) which creates native / pre compiled gems.

Or easier, bundle package package all your gems under vendor/cache.

Both have the same issue, you need to build the .gem on a staging machine that matches the ruby version of the deployment target.


The scap Deployments is being improved to deploy MediaWiki service backends, but that uses git fetch. Tin has a rsync server though, so maybe you can reuse it ?

The rsync common container serves /srv/mediawiki-staging setup via rsync::server::module. Maybe package { provider => Trebuchet } could be made to setup an appropriate rsync container or have the deployment role set it up via rsync::server::module.

Not sure how many Gems we're talking here and how often they change, if the numbers are low, it would also be an option to use gem2deb to create a Debian source package and build that for trusty/jessie:
https://wiki.debian.org/Teams/Ruby/Packaging
https://packages.debian.org/sid/gem2deb

One problem is that logstash moves quickly when it comes to dependency and there is a lot to package or the distro versions would not match the requirements :(

For v1.5.3 see Gemfile and Gemfile.jruby-1.9.lock.

Interesting!

Rather than using git-fat + rsync, perhaps it would be better to do the deploy-repo module, where there is a git-submodule that includes committed dependencies, including binaries. This isn't ideal, but at least it keeps the binaries out of your main source repository.

Maybe package { provider => Trebuchet } could be made to setup an appropriate rsync container

It can already do this via 'gitfat_enabled' => true in the repo config in deployment.pp

bd808 added a comment.Jul 28 2015, 3:04 PM

Not sure how many Gems we're talking here and how often they change, if the numbers are low, it would also be an option to use gem2deb to create a Debian source package and build that for trusty/jessie:
https://wiki.debian.org/Teams/Ruby/Packaging
https://packages.debian.org/sid/gem2deb

For the initial 1.5.3 deploy we only need one non-core plugin: https://rubygems.org/gems/logstash-filter-prune/versions/0.1.5. This is a pure ruby gem (https://github.com/logstash-plugins/logstash-filter-prune) so the ABI issue is moot at this time.

The tricky bit about deb packaging however is that logstash doesn't use the system ruby or any standard gem locations. It's actually a jruby app and their new plugin management system seems to rely on reading the Gemfile and Gemfile.lock in the deploy directory (https://github.com/elastic/logstash/issues/3540).

I can't say that I like how they are changing things upstream, but arguing with Jordan Sissel of fpm fame about package management practices has not been historically useful.

bd808 closed this task as Declined.Aug 4 2015, 4:48 AM

I'm going to keep things simple and just ship a repo of unpacked plugins. Support for this was restored in logstash 1.5.3 and seems unlikely to be removed again based on the large number of bug reports against 1.5.0 for removing it.