Page MenuHomePhabricator

RFC: Container path conventions
Closed, ResolvedPublic

Description

To simplify human understanding and tooling, we need to establish some path conventions for our containers.

Assumptions / considerations

  • Only run a single service per container
  • Prioritize keeping things simple. Don't require path templating by service name.
    • Downside: Makes it harder to reuse scripts outside of containers.

Paths

Service Code

Where the service's code will live.

  • /srv/service -- Follows srv convention.
  • /opt/service (used currently in service-runner-based Dockerfiles)
  • /src -- Fairly common in other containers.
  • /srv -- Standard location for site-specific data.

Entry Point

Regular service startup executable or script.

Proposals:

  • /bin/run_service
  • /bin/docker_entrypoint
  • /docker_entrypoint

Test Entry Point

Run tests in container

  • /bin/run_tests

Data

Data directories. Typically externally mounted volumes.

  • /data -- Fairly common in docker containers.

Configuration files

Read-only, often from Kubernetes config maps.

  • /conf -- Common in docker containers.
  • /etc/service -- Follows etc convention.

Related Objects

StatusSubtypeAssignedTask
ResolvedNone
InvalidNone
Invalidmobrovac
Resolvedmobrovac
InvalidNone
InvalidNone
ResolvedNone
Resolveddduvall
InvalidNone
DeclinedNone
Resolveddduvall
Resolveddduvall
Resolveddduvall
Resolveddduvall
Resolveddduvall
ResolvedNone
Resolveddduvall
Resolvedthcipriani
Resolveddduvall
Resolveddduvall
Resolvedthcipriani
Resolveddduvall
Resolvedthcipriani
ResolvedJoe
Resolveddduvall
Resolveddduvall
Resolveddduvall
Resolveddduvall
Resolvedakosiaris
Resolvedthcipriani
DeclinedNone
Resolvedthcipriani
Resolveddduvall
Resolveddduvall
InvalidNone
Resolveddduvall
Resolveddduvall
Resolveddduvall
InvalidNone
InvalidNone
Resolvedthcipriani
DeclinedNone
OpenNone
DeclinedNone
Declinedthcipriani
Resolveddduvall
Resolveddduvall
Resolveddduvall
DeclinedNone

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 7 2017, 4:23 PM
GWicke updated the task description. (Show Details)Jul 7 2017, 4:31 PM

We should also take into consideration the development cycle with mounted paths. In case of development, the code is over mounted on top of the sources from inside the container, and the entry point changes to install new node_modules (or update old already installed) and then start the service with the config, that's also over-mounted from the cloned repo. So we need something like /bin/reinstall_deps_and_start

Also, it would be nice to install dependencies in the dev environment to some of the minikube persistent volumes so that when the user restarts minikube they don't need to reinstall all the dependencies again.

mobrovac triaged this task as Medium priority.Jul 10 2017, 9:47 PM
mobrovac updated the task description. (Show Details)
Joe added a subscriber: Joe.EditedAug 3 2017, 10:46 AM

I think the proposal is pretty sound - with a couple of suggestions to keep things more "familiar" for ops, and more in line with what we use for our production environment:

  1. Use /srv/service instead of /opt
  2. Use /etc/service-config instead of /config
  3. I would pick /bin/run_service, but maybe even /usr/local/bin/run_service, which adheres more to the FHS.

I agree with the assmuptions and I don't want us to have to template the service name everywhere - having the exact same hierarchy in every container is a good idea.

I do think we should expand on this proposal - but this is a good starting point.

I would add that run_service and run_tests should be a link to a base script that will change depending on the specific language/frramework used, and should be included in our base images.

GWicke updated the task description. (Show Details)Aug 8 2017, 5:14 PM

Added @Joe's proposals to the description. Except for /etc/service-config, which I shortened to just /etc/service.

Overall, there seem to be two broad directions emerging:

  1. Follow the Filesystem Hierarchy Standard, and use the generic "service" as the service name.
    • Config in /etc/service, code in /srv/service, binary in /bin/run_service and /bin/run_tests`, possibly data in /var/service.
  2. Follow external Docker conventions.
    • Config in /conf/, code in /src/, binary in /entrypoint, data in /data/.

I personally am fine with either option, but wouldn't be too keen on using a mix between the two. One minor niggle with the first option is the need to use run_service and run_tests instead of the plain service name. I am wondering if we can find a better service name that avoids these issues, and lets us use consistent binary names.

FTR, I would really prefer us to stay close to FHS. I 've had to debug containers that did not follow it already and the time lost trying to figure out where things were was pretty significant. We can make the argument that we would get used to a non FHS solution, but if we add the time it would take every new person to get accustomed to it I am willing to bet it would end up being considerable. Given we also want to create an environment that would be welcoming to new developers we should probably avoid extra surprises. They would already have enough new things to get accustomed to, let's not add more.

I am also not fully sold on the idea that we should not template some things. I do expect e.g. mathoid's configuration to be under /etc/mathoid and not under /etc/service but I am I guess this is something I can work around.

FTR, I would really prefer us to stay close to FHS. I 've had to debug containers that did not follow it already and the time lost trying to figure out where things were was pretty significant. We can make the argument that we would get used to a non FHS solution, but if we add the time it would take every new person to get accustomed to it I am willing to bet it would end up being considerable. Given we also want to create an environment that would be welcoming to new developers we should probably avoid extra surprises. They would already have enough new things to get accustomed to, let's not add more.

I don't believe that to be such an overhead, but in general I'm +1 on going with FHS, it just feels more natural.

I am also not fully sold on the idea that we should not template some things. I do expect e.g. mathoid's configuration to be under /etc/mathoid and not under /etc/service but I am I guess this is something I can work around.

Why not? This to me is the big advantage of containers - the fact that we don't have to template names and worry about exact locations, but stick to a pre-defined contract. What would be the point of putting Mathoid's configuration in /etc/mathoid/ over /etc/service/ when we know that that is going to be the only service running inside the container?

I am also not fully sold on the idea that we should not template some things. I do expect e.g. mathoid's configuration to be under /etc/mathoid and not under /etc/service but I am I guess this is something I can work around.

Why not?

Me finding it instantly and not having to do the indirection of "oh wait.. we are talking about a WMF containerized service, these guys do it some other way, what is it again?<some time later>ah, here it is" But as I said, I 'll survive that part.

This to me is the big advantage of containers - the fact that we don't have to template names and worry about exact locations, but stick to a pre-defined contract. What would be the point of putting Mathoid's configuration in /etc/mathoid/ over /etc/service/ when we know that that is going to be the only service running inside the container?

To me it is not, in fact this is the biggest problem of containers. Encouraging shipping code somehow somewhere inside a tar file alongside a few steps "here's how to run me", is not the big selling point for me. Not that I 've ever seen any container company actively encouraging that practice. Anyway, It's the orchestration that is made possible, or the decoupling of developer's code from system provided library versions increasing portability and extensibility that make containers appealing.

To me it is not, in fact this is the biggest problem of containers.

I meant in the context of path layout naming.

Pchelolo closed this task as Resolved.Jul 16 2019, 7:34 PM

Given that we now have Release Pipeline (Blubber) and default scaffolding for deployment charts, I would consider this resolved.