Page MenuHomePhabricator

Create a standard service template / init / logging / package setup
Closed, ResolvedPublic

Description

Between node services, we have developed fairly standard solutions for common service issues:

  • init scripts
  • logging (logstash & local with rotation using bunyan)
  • txstatsd / graphite metric reporting
  • config loading
  • muti-process clustering
  • Debian packaging (mostly a combination of the above)

We currently share these solutions manually, but it would be great if we could extract the common code into a template repository or skeleton initialization tool.

Event Timeline

GWicke raised the priority of this task from to Needs Triage.
GWicke updated the task description. (Show Details)
GWicke added a project: Services.
GWicke added a subscriber: GWicke.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 4 2015, 6:50 PM
GWicke updated the task description. (Show Details)Feb 4 2015, 6:53 PM
GWicke set Security to None.
GWicke removed a subscriber: Aklapper.

Can basic testing setup be added to the template/skeleton project as well?

At $DAYJOB-1 we made a skeleton project and used Phing (our PHP build tool of choice) to make it easy to clone the skeleton and run a build command that prompted you for various things to change in the template files. Once it was done, you had a fully functional "hello world" app that included stub config files, logging, and a build script to run tests, create release packages and configure a local deployment.

+1 also I think it'd make sense to capture what metadata we need in a (per-repo?) file so templates and other artifacts can be regenerated at will

Joe added a subscriber: Joe.Feb 5 2015, 11:16 AM
Dzahn triaged this task as Medium priority.Feb 13 2015, 10:41 PM
Dzahn added a subscriber: Dzahn.

Can basic testing setup be added to the template/skeleton project as well?
At $DAYJOB-1 we made a skeleton project and used Phing (our PHP build tool of choice) to make it easy to clone the skeleton and run a build command that prompted you for various things to change in the template files. Once it was done, you had a fully functional "hello world" app that included stub config files, logging, and a build script to run tests, create release packages and configure a local deployment.

That sounds neat.

Lets not go over board with templating though. I think we should try hard to use shared modules with stable interfaces, and keep the actual template bit to an absolute minimum. This is pretty much what we have already been doing, but there are still some utilities that aren't in a convenient package yet.

+1 also I think it'd make sense to capture what metadata we need in a (per-repo?) file so templates and other artifacts can be regenerated at will

+1 for metadata and tools over templates that are then hacked. For example, I think that for most node services we should be able to generate Debian packages using just information from package.json and perhaps some other metadata file (port defaults come to mind) as long as the entry points and wrappers are standardized.

GWicke raised the priority of this task from Medium to High.Feb 13 2015, 11:12 PM

We have started to package our existing tools around nodejs clustering, logging, statsd reporting, config loading in https://github.com/wikimedia/supervisoid. By establishing conventions for how a service is started and configured, this will let us build a generic packaging tool & service puppet module.

gerritbot added a subscriber: gerritbot.

Change 190708 had a related patch set uploaded (by KartikMistry):
Enable jenkins for operations/debs/contenttranslation

https://gerrit.wikimedia.org/r/190708

Patch-For-Review

A basic, preliminary v0 of the service template is available at https://github.com/d00rman/service-template-node. We plan to switch to swagger for driving the API route creation process, thus auto-documenting API routes, allowing us to visualise the API intricacies of each service more easily.

GWicke added a comment.EditedFeb 21 2015, 3:54 AM

Now merged & slightly improved at https://github.com/wikimedia/service-template-node.

I think there is still some room to pare it down further to just the functionality needed for a simple but solid service with less than a handful of end points. Maybe it makes sense to add a 'simple' branch?

GWicke moved this task from Backlog to In-progress on the Services board.Mar 17 2015, 8:03 PM
ssastry moved this task from Backlog to Non-Parsoid Tasks on the Parsoid board.Apr 21 2015, 3:56 AM
Restricted Application added a subscriber: Matanya. · View Herald TranscriptJul 25 2015, 2:28 PM
GWicke lowered the priority of this task from High to Medium.Jan 27 2016, 6:34 PM
Jcook added a subscriber: Jcook.Aug 19 2016, 11:50 AM

Hello,

I wondered if you ever considered using something like yeoman for creating a generic service template? e.g)
http://yeoman.io/authoring/index.html

Then rather than using the current service-template-node you could even go as far as to create new projects with routes etc.

It was just an idea/suggestion
Thanks

GWicke closed this task as Resolved.Oct 12 2016, 11:00 PM
GWicke claimed this task.
GWicke edited projects, added Services (done); removed Services (doing).

@Jcook, we considered it when we started work on this, but decided to go with a repository. So far that has worked well for us. One advantage over yeoman we have seen is better support for updating / rebasing services on top of a newer version of the template. Generators are more of a one-shot operation.

Granted, this is currently more important than it should be because the template includes so much functionality. I would like us to move more of it into libraries & minimizing the template itself.

Since the core of this task is done, I am going to resolve it. Lets continue the discussion about moving out more functionality into libraries in T144954.