Page MenuHomePhabricator

Create a gateway in kubernetes for the execution of our "lambdas"
Open, MediumPublic

Description

We plan to run a series of small services as "support lambdas" for MediaWiki, using the php microservice described at T260330 as a basis.

Given we want the setup to be fast, we certainly don't want to have to go through the hassle of creating a new load-balanced service for each of them, given there could be several dozens of such very small services.

The workflow I envision for installing one of those php-backed services would be:

  • Create a new namespace (optional)
  • Create a new deployment definition for helmfile
  • Create the deployment/namespace
  • Wire the new deployment to be accessed under a url prefix on (say) lambdaiod.discovery.wmnet

The gateway (codename "lambdaoid", to keep up with the noble wikimedia tradition of service naming) itself will need to be accessible by the load-balancers as any other service on k8s, but the single deployments won't need to.

So, summing up, the gateway to the lambdas will need to have the following characteristics:

  • FLOSS
  • actively developed
  • can work cross-namespaces
  • zero or little configuration to wire in a new service
  • Uses a boring technology (i.e. one we know already)

Nice to haves are:

  • telemetry
  • logging to logstash
  • dynamic configuration reloading

Event Timeline

jijiki triaged this task as Medium priority.Aug 26 2020, 3:05 PM

@Joe is everything in this ticket now covered by Shellbox?

@Joe is everything in this ticket now covered by Shellbox?

No, this ticket is about adding an Ingress controller in front of the various shellbox deployments.

Looking at https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/, I see three broad categories of ingresses that use a technology we know well.

nginx-based

https://github.com/kubernetes/ingress-nginx is the "officially supported" FLOSS ingress. It is based on an open core model because nginx is, so it bears all the limitations of nginx in its FLOSS version. It should check most of the boxes above, and has a quite comprehensive documentation https://kubernetes.github.io/ingress-nginx/. It allows running on bare metal and has a full section of the documentation about it.

Bonus points: it can also proxy to a fcgi backend.

envoy-based

Envoy is in many ways a superior http gateway compared to nginx, especially the free version. So it's natural that many ingresses based on it were created:

https://projectcontour.io/ - Contour is a CNCF incubator project, completely FLOSS, with a pretty active community it seems. The documentation is pretty ample but I found some of it confusing. AIUI it can be installed using an operator or helm, which we'd prefer, but then there are some missing details in that case as well. It allows running as NodePort as well.

HAProxy based

https://haproxy-ingress.github.io/#td-block-1 - A community effort, pretty actively developed (https://github.com/jcmoraisjr/haproxy-ingress), completely FLOSS. Seems to do a few things differently than the other ingress controllers but seems to have a solid community. Documentation seems to be decently written.

https://github.com/haproxytech/kubernetes-ingress#readme is the official ingress by haproxy. I couldn't find big differences in popularity or support between this and the other HAProxy-based ingress, besides a better documentation of the community project.

There are many more solutions listed on the k8s ingress page, but I've excluded them all because of low activity, being open core, lacking even basic documentation, or implementing the kubernetes ingress functionality as an afterthought.

We also talked about using Istio Ingress in the past (envoy-based) which could be a good fit as well and we could share technology and resources with ML (regarding Istio components). AIUI it should be possible to use Istio ingress without going full-istio immediately.

We also talked about using Istio Ingress in the past (envoy-based) which could be a good fit as well and we could share technology and resources with ML (regarding Istio components). AIUI it should be possible to use Istio ingress without going full-istio immediately.

Yes I had a conversation with Luca this morning, and I looked into it a bit. I'll create a subtask to report my current analisys.

...given there could be several dozens of such very small services

I searched through exec.log*, and it appears that we'll want Shellboxes for:

  • Score (lilypond, ghostscript, fluidsynth, lame) - already using LVS
  • Timeline (perl/ploticus)
  • SyntaxHighlight (Python3/pygments)
  • Various image things: PagedTiffHandler (tiffinfo) and PdfHandler (pdftext/pdfinfo), djvutxt/djvudump, rsvg-convert

Plus one more Shellbox for Wikidata constraints (T285104).

So in reality we're looking at more like 5 Shellboxes for now, not several dozens. We still do need proper ingress, but I don't think we should necessarily block the deployment of more Shellboxes on the lack of a proper ingress solution.

* ignoring CodeReview's svn shellouts which are already broken and ffmpeg/fluidsynth from videoscalers, which AIUI will be managed differently

...given there could be several dozens of such very small services

I searched through exec.log*, and it appears that we'll want Shellboxes for:

  • Score (lilypond, ghostscript, fluidsynth, lame) - already using LVS
  • Timeline (perl/ploticus)
  • SyntaxHighlight (Python3/pygments)
  • Various image things: PagedTiffHandler (tiffinfo) and PdfHandler (pdftext/pdfinfo), djvutxt/djvudump, rsvg-convert

Plus one more Shellbox for Wikidata constraints (T285104).

So in reality we're looking at more like 5 Shellboxes for now, not several dozens. We still do need proper ingress, but I don't think we should necessarily block the deployment of more Shellboxes on the lack of a proper ingress solution.

* ignoring CodeReview's svn shellouts which are already broken and ffmpeg/fluidsynth from videoscalers, which AIUI will be managed differently

I looked at the query you linked, and I don't think you should exclude the scripts/ directory, or am I missing something?

I looked at the query you linked, and I don't think you should exclude the scripts/ directory, or am I missing something?

Only Score shells out to paths with scripts/ AFAIS. Here's the query with the explicit names of those two scripts instead of the directory name: https://logstash.wikimedia.org/goto/83eb2a76aa8eddfaca51a7166b55b6a8

I looked at the query you linked, and I don't think you should exclude the scripts/ directory, or am I missing something?

Only Score shells out to paths with scripts/ AFAIS. Here's the query with the explicit names of those two scripts instead of the directory name: https://logstash.wikimedia.org/goto/83eb2a76aa8eddfaca51a7166b55b6a8

Oh that's great. But this makes me wonder... if score runs those scripts locally, we're actually executing ccommands on the appservers still, which won't work on kubernetes. There is some work to do there I guess?

Only Score shells out to paths with scripts/ AFAIS. Here's the query with the explicit names of those two scripts instead of the directory name: https://logstash.wikimedia.org/goto/83eb2a76aa8eddfaca51a7166b55b6a8

Oh that's great. But this makes me wonder... if score runs those scripts locally, we're actually executing ccommands on the appservers still, which won't work on kubernetes. There is some work to do there I guess?

Sorry, I meant it "Shellboxes out" to those scripts, they're copied over to the Shellbox and executed only there, not locally. But the path is kept intact in logs, which is kind of nice.

I love "shellboxes out". Thanks I remember we discussed managing scripts explicitly when we introduced shellbox, so I was wondering why they were executed locally, and yes it is nice to have the full path indeed.

...given there could be several dozens of such very small services

I searched through exec.log*, and it appears that we'll want Shellboxes for:
<snip>

I missed SecurePoll, which shells out to gnupg, since it only gets used like once or twice a year. I don't think migrating that to Shellbox is a good idea, I'll work on a different proposal for that one.