Page MenuHomePhabricator

Auto-generate Debian packages for service-runner services
Closed, DeclinedPublic

Description

The https://github.com/wikimedia/service-runner project establishes a standard interface for node services. This means that each service (or collection of services) has the same commandline arguments, uses the same config format & starts up the same way.

We can exploit this regularity to write a tool that auto-generates a simple Debian package based on standard templates. A lot of the information needed for the package is already available in package.json and config.yaml. We can also add more packaging information in config.yaml as needed. For this we should consider whether we need to separate these dependencies between distros / releases, or whether we can share them at least for debian-based distros.

Event Timeline

GWicke raised the priority of this task from to Needs Triage.
GWicke updated the task description. (Show Details)
GWicke subscribed.
GWicke set Security to None.

Is the goal here to have them in Debian binary package format as a for-our-use deployment-medium only? Is there any value in having the individual project-specific versioned source, (for use outside of wmf for example)?

@Eevans, main priority from my POV is

  • binary (often actually arch-independent) distribution for third-party devs from our apt repository, and possibly
  • internal use for deployments, if opsens warm up to the idea.

Source debs shouldn't be too hard to generate too, but aren't quite as exciting if you can just run the script in the git repo to generate the entire package from scratch.

...you can just run the script in the git repo to generate the entire package from scratch.

Sure, but that isn't idiomatic for Debian packaged software projects.

FWIW, modern versions of debhelper are extensible to the point that it should be possible to reduce debian/rules to a bit of boilerplate that can be copied from one project to the next. That would leave you with nothing but the rfc822 metadata to maintain (e.g. debian/control).

I don't know that I feel strongly about this one way or another, which is why I asked; If we do care about promoting/encouraging use of RESTBase outside of Wikimedia, then a proper Debian package would be a nice thing to have.

...you can just run the script in the git repo to generate the entire package from scratch.

Sure, but that isn't idiomatic for Debian packaged software projects.

FWIW, modern versions of debhelper are extensible to the point that it should be possible to reduce debian/rules to a bit of boilerplate that can be copied from one project to the next. That would leave you with nothing but the rfc822 metadata to maintain (e.g. debian/control).

The interesting thing is that this metadata is mostly already available in package.json, so we don't need to add much information in there (or in config.yaml) in order to be able to auto-generate debian/control too. For most node projects, no additional information is needed at all. We have been copying our debianization between projects so far, but doing so has a lot of disadvantages in terms of consistency, duplication of effort, sharing improvements etc.

I don't know that I feel strongly about this one way or another, which is why I asked; If we do care about promoting/encouraging use of RESTBase outside of Wikimedia, then a proper Debian package would be a nice thing to have.

We have had a Debian package for Parsoid for a while now, and people have been using it very happily, despite it being hosted in an apt repo on labs. Main priorities for me are:

  • Timely updates: Could offer nightly or bi-weekly releases and stable + timely security updates for unattended-upgrades use.
  • Reasonable cost / benefit ratio.

Requirements for a proper Debian package (which could go into Debian proper) aren't very aligned with this (ex: no bundled dependencies, slow updates), so I see that as lower priority unless we can find a way to make it both timely and cheap.

  • internal use for deployments, if opsens warm up to the idea.

I'm not sure why this is even mentioned here. This is clearly not going to happen (which is the response you've gotten all of the hundreds of times you've proposed it), so I don't think it deserves to be called a "main priority" for anyone.

This is clearly not going to happen

@faidon: I think you are arguing against the strawman of developers getting root by deploying arbitrary debian packages here. Nobody is proposing that.

My main concern here is about avoiding duplication of effort & keeping things simple. A standard template for init scripts / systemd units / logrotate setups as needed for the tool discussed here will make it easier to share things between our config / deployment system and packages, which is a good thing in my book.

This is clearly not going to happen

@faidon: I think you are arguing against the strawman of developers getting root by deploying arbitrary debian packages here. Nobody is proposing that.

No, I'm talking about using .debs to deploy services in the Wikimedia fleet. And I'm not really arguing, we've already established that we're not going to do that.

Closing, as the focus is now on containers, and not Debian packages.