Page MenuHomePhabricator

Add a easy way to run a ruby webservice on tools
Open, MediumPublic

Description

Should support using a way of building ruby libraries that are tool local (similar to virtualenv in python - bundler maybe?) and a default nice web server.

Event Timeline

Restricted Application added a project: Cloud-Services. · View Herald TranscriptJul 26 2016, 6:59 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

I'll share my current set up for https://tools.wmflabs.org/musikanimal

  • rbenv to install the Ruby version
  • Manually install gems with gem install --user-install (no bundler)
  • Use jstart to load my httpserver.sh on trusty
  • The httpserver script exports the Ruby paths and sets the Ruby with rbenv, then uses exec portgrabber to load the Unicorn server config
  • Unicorn takes roughly 5 minutes to start

The only real complaint from me is that it takes forever to restart, and each release requires a restart. Bundler would be nice but it's not a huge pain to manually install gems.

MusikBot works the same way, only there is no webserver.

bd808 added a subscriber: bd808.Sep 21 2016, 4:39 PM

@MusikAnimal Debian Jessie has Ruby 2.1.5 packaged. Would that be new enough to replace your use of rbenv? What Ruby version are you using?

I think Ruby could be used with our uwsgi webservice backend, but it shouldn't be too hard to add a gunicorn backend that would look for a config.ru file.

Change 312033 had a related patch set uploaded (by BryanDavis):
Add ruby images

https://gerrit.wikimedia.org/r/312033

@bd808 Yeah that should suffice. I'm OK with something other than Unicorn too, especially if it means it will take less than 5 minutes to restart :)

scfc assigned this task to bd808.Nov 26 2016, 11:55 PM
scfc triaged this task as Medium priority.
scfc moved this task from Triage to Waiting for code review on the Toolforge board.
Restricted Application added a project: User-bd808. · View Herald TranscriptNov 26 2016, 11:55 PM

Change 312033 merged by Yuvipanda:
Add ruby images

https://gerrit.wikimedia.org/r/312033

bd808 edited projects, added Kubernetes; removed Toolforge.Jul 28 2017, 11:01 PM
bd808 removed bd808 as the assignee of this task.Jan 10 2018, 12:57 AM
bd808 removed projects: User-bd808, Patch-For-Review.

Unlicking this cookie for now.

Krinkle edited projects, added Cloud-Services; removed Toolforge.
Krinkle edited projects, added Toolforge; removed Cloud-Services.

T250118: Request creation of wlmitaly VPS project led me to look at https://github.com/ferdi2005/concorsi-locali and think about what it would take to make this run easily on our Kubernetes cluster. This app has the standard config.ru entry point for a Rack ruby webapp. It also needs a number of gems and expects them to be managed by Bundler.

I'm a ruby novice, but reading things online makes me think this is a very similar set of conventions to those that we have established for Python WSGI applications. This makes me hopeful that this both a common pattern in the Ruby world and also something we can create convention over configuration patterns for in Toolforge.

I'm thinking that we can try to create a container and associated config in webservice that expects a $HOME/www/ruby/src/config.ru entrypoint and an optional $HOME/www/ruby/bundle Bundler managed collection of gems. The container itself would have ruby installed via Debian packages, some rack compliant hosting container (uWSGI, unicorn, passenger, ...), and configuration managed by some combination of webservice and files in $HOME that make webservice --backend=kubernetes ruby25 start launch the container, activate the bundle, start the rack app, and handle inbound HTTP requests.

I looked at the dashboard for ruby usage on our Kubernetes cluster and found that nobody is using the ruby25 image at all and only 2 tools are using the older ruby21 image.

position-holder-history

  • Maintained by @chrismytton, @mhl20, and Sa.ge.pe (not sure what Phab user if any)
  • Uses a locally compiled ruby 2.4.1 binary
  • Has a startup shell script that runs bundle install ... each time the webservice is started
  • Ultimately runs bundle exec rackup ... to run the app
  • Has a config.ru entry point for rackup
  • Has a cron job that runs on the grid using the same custom compiled ruby and bundle

prompter

  • Same maintainers as position-holder-history (do we only have 5 folks using ruby in total?!)
  • Pretty much the same core setup as position-holder-history, unsurprisingly

Looking at these makes me more confident of the possible solution I sketched out in T141388#6057259.

  • $HOME/www/ruby/vendor might be a better name for the bundler managed gems location.
Bstorm added a subscriber: Bstorm.EditedApr 15 2020, 5:47 PM

Using a locally-compiled ruby is very common. I've used rbenv and chruby on production servers to stay ahead of security patching. I cannot imagine running with the runtime on NFS makes it go faster, though, so it'd be good if folks can adopt using OS ruby. It is likely that people will assume they need to build ruby, so we might want to expressly discourage it for performance while tolerating it as inevitable anyway.

Modern rails apps have puma as an included webserver and only really need a reverse proxy in front of that (since it does concurrency and all that with a little config). We've got at least 3 reverse proxies in front, so that's covered, lol. Most local development will be done using puma, and it's faster than unicorn in any test I've done. That said, it only supports http1.1 (you'd need falcon for http2, and not sure how much of rails is compliant anyway), and the wsgi config for rails doesn't look all that weird. My one concern with using uwsgi is that it's linked to the OS ruby (https://packages.debian.org/buster/uwsgi-plugin-rack-ruby2.5). That seems like it'll cramp the style of a ruby dev, potentially. We can install puma in the image if we choose https://packages.debian.org/stable/source/puma. I'm just suggesting it if we want to go native in that ecosystem. We'd be in-line with nearly every recent tutorial using puma and a proxy (unless people are expecting phusion or something). It compares very favorably with unicorn https://scoutapm.com/blog/which-ruby-app-server-is-right-for-you, and it would start quite fast. I could try to be more useful and dust off how we could configure that in a webservice to be deploy-ready. It's probably pretty simple and would be easier for people to troubleshoot for themselves since it's familiar.

The other piece of rails dev is that people expect to have npm or yarn for their js assets. It's possible to split all that out into another container/tool, but I'd consider it a more advanced option.

I just read over the parent task. I think the general issue here is that we need instructions for using bundler and yarn/npm in Toolforge K8s. Nobody should have to learn k8s to launch a rails app in TF, and puma should be plenty without adding wsgi or anything else. Let me see if I can try some things. The image might need update, but most of the tools are likely already in place.

Change 589086 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/docker-images/toollabs-images@master] rails: ruby needs nodejs and bundler to be useful

https://gerrit.wikimedia.org/r/589086

I'll add some dopey simple rails tool to one of my test tools to validate that once we've built it.

bd808 added a comment.Apr 15 2020, 8:52 PM

Would it be reasonable to install something similar to https://github.com/evanphx/puma-heroku/blob/master/lib/puma/plugin/heroku.rb in the image too to configure puma?

For our Python images, uWSGI is configured with some defaults set by webservice when it runs inside the container (listen on tcp port 8080, chdir to $HOME/www/python/src, log to $HOME/uwsgi.log, 4 worker threads, die on sig TERM). It also checks for $HOME/www/python/venv and adds that as the active virtual environment if found. Finally it checks for a $HOME/www/python/uwsgi.ini and reads it if present. There are some things don't love about this (mostly that uwsgi.ini can't override the logging location), but it seems to be pretty easy for most folks to adapt to using.

It's not clear to me exactly what the Puma equivalent would be to get the bundle mounted. I think bundle exec puma ... is what would be expected, but that would mean that everyone needs to include Puma in their bundle right? That kind of feels like a recipe for problems.

That's likely going to be inside a rails app's committed code. You'd run the command with bundle exec puma -C config/puma.rb We can provide a recommended config, but it may change between rails and, say, sinatra (which would likely be a whole lot simpler). Overall, some settings would need to be there for it to work.

Right now, I've started to focus on getting bundler to a state where it can actually install rails in our image (using bundler exactly like we use virtualenv, basically). However, I'm amazed at how good the debian distro is at breaking dev toolchains out of the box. Our ruby image as installed would have a really hard time running nearly anything.

Btw, if we run this as a generic webservice, then the user could choose to install something like falcon if they are doing some fancy http/2 things instead of using puma. They could also install unicorn if they really wanted to. Someone is surely going to cause a thread leak, but kubernetes might be able to contain it.

Change 589086 merged by jenkins-bot:
[operations/docker-images/toollabs-images@master] rails: ruby needs nodejs and bundler to be useful

https://gerrit.wikimedia.org/r/589086

Mentioned in SAL (#wikimedia-cloud) [2020-04-15T23:20:55Z] <bd808> Building ruby25-sssd/base and children (T141388, T250118)

Got my rails app working in Toolforge https://bstorm-tool.toolforge.org/
It wouldn't work last night because the sassc gem was killed during compile. It turns out, ruby gems doesn't run make clean when you just try again on a failed compile of native extensions. 😅

Just going to double check the mysql gem before calling the image ready.

tools.bstorm-tool@interactive:~$ gem install --user-install mysql2
Fetching: mysql2-0.5.3.gem (100%)
WARNING:  You don't have /data/project/bstorm-tool/.gem/ruby/2.5.0/bin in your PATH,
	  gem executables will not run.
Building native extensions. This could take a while...
Successfully installed mysql2-0.5.3
Parsing documentation for mysql2-0.5.3
Installing ri documentation for mysql2-0.5.3
Done installing documentation for mysql2 after 1 seconds
1 gem installed

It works fine in this image. If anyone does mapping in rails, we might need some pg library to compile against, but I think I'm willing to worry about that when we actually see it.

One thing here: we share out static assets from a particular folder using tools-static, right? If so, the recommendation should be to place static rails assets in that folder. Rails serves static assets slowly, and they recommend you use a CDN or external server to handle them.

bd808 added a comment.Apr 16 2020, 4:21 PM

One thing here: we share out static assets from a particular folder using tools-static, right? If so, the recommendation should be to place static rails assets in that folder. Rails serves static assets slowly, and they recommend you use a CDN or external server to handle them.

Yes, $HOME/www/static for any tool is then served at tools-static.wmflabs.org/$TOOL/ (example: https://tools-static.wmflabs.org/toolforge/).

It should also be possible to setup the ingress for a tool to serve static assets, but that might end up being a drag on the shared ingress system. Right now I don't think that layer in our Kubernetes cluster actually has to touch NFS files.

That should be perfect. Instructions should just include adding some kind of symlink or rails config to place assets there.

My app is configured to serve static assets (and yours might be), and that may be fine for many tools. However, I'm sure ruby devs would want a different server for the statics if they can get one.

bd808 added a comment.Apr 29 2020, 1:29 AM

One thing here: we share out static assets from a particular folder using tools-static, right? If so, the recommendation should be to place static rails assets in that folder. Rails serves static assets slowly, and they recommend you use a CDN or external server to handle them.

This seems to work as hoped:

$ ln -s $HOME/www/ruby/src/public $HOME/www/static
$ echo "config.action_controller.asset_host = 'https://tools-static.wmflabs.org/${USER##tools.}'" >> www/ruby/src/config/environments/production.rb
$ webservice restart

Alternately, stick gem 'rails_serve_static_assets' in your Gemfile and let puma handle serving them.

Next step here is making some tutorial on wikitech