Page MenuHomePhabricator

Deploy the MP Instrumentation Configuration application to the DSE k8s cluster
Open, HighPublic

Description

The Metrics Platform Instrumentation Configuration app (as described in T360647) will need to be deployed in the DSE Kubernetes cluster.

As @WDoranWMF explained, it will need to:

  • be publicly accessible under a mpic.wikimedia.org domain (specific domain to be confirmed)
  • have a staging and production deployment
  • have SSO enabled (at least for production, with a potential bypass for staging at first)
  • have access to the the database setup in T331516

This ticket will break down the work into infrastructure-related tasks that should be accomplished by Data Platform SREs, and application-related tasks, that should be handled by Metrics Platform, with help from Data Platform SREs.

Event Timeline

@brouberol just posting the Qs we have from Slack here for visibility (cc @Sfaci):

  1. Is there a standard way to use config files for storing environment variables (db connection creds)?
  1. Currently there is a frontend app (vuejs) silo'd in a subdirectory of the backend app (stripped down version of service-template-node) - both are served from different ports:
    • Is it reasonable to assume that we can map the different ports to separate domains or subdomains?
    • Should frontend + backend be in the same container or different containers?
    • Can the FE/BE container(s) be configured with a blubber file?

Is there a standard way to use config files for storing environment variables (db connection creds)?

TLDR: there is a way to commit secrets and have them be usable within your charts.

What we did for Superset was define a DB connection URI containing a <PASSWORD> placeholder that we later-on replace with the actual password. Because the password is secret, we only commit it to the private puppet repo, via which we render a special YAML file on the deployment server containing secrets/passwords only.

Is it reasonable to assume that we can map the different ports to separate domains or subdomains?

Do you mean DNS (sub)domains? If so, there is a way, but it requires just a bit of elbow grease. It can be done though.

However, what I'd do is slightly different, and it mirrors what was done for Superset. We have a single DNS domain (in our case, superset.wikimedia.org), and traffic reaches an nginx reverse proxy, that either proxies the traffic to the frontend container if the url path starts with e.g. /static and to the backend for anything else.

This is what this nginx config does.

Should frontend + backend be in the same container or different containers?

I'd tend to say different containers.

Can the FE/BE container(s) be configured with a blubber file?

Yes! In fact, we did exactly that for Superset

On slack, @Sfaci mentioned

About how the application is configured, at this moment, we have only a config.yaml file per environment but I don't know if we should declare some environment variables.

I would advise for a single way of configuring the application: either a config file _or_ environment variables. It'd be easier to maintain and override over time. Having a yaml file as a source of configuration is perfectly reasonable!