Page MenuHomePhabricator

Meeting: Automatic deployment of backend services on beta cluster
Closed, DeclinedPublic

Description

On the Beta-Cluster-Infrastructure, only parsoid and cxserver are automatically deployed on post merge. More services are being added and we need a common solution to auto deploy them on post merge.

Things to talk about:

  • explain how it is done currently (beta cluster instances are made Jenkins slave)
  • what to deploy? master branch of source repo or the stable deploy repos
  • switch to git-deploy/Trebuchet to match production?
  • how to setup and use git-deploy on labs/beta-cluster

Outcomes

  • We aren't going to use git-deploy to deploy services (automatically) in Beta Cluster
  • Services will all run from master in Beta Cluster (not their "deploy" branch).
  • RelEng will setup Jenkins Jobs to update all services we already have.
  • Marko work with Alexandros on deciding whether to put all services on SCA01 or separately (choose one)
  • Figure out LDAP search to map which service is running on which instance and the which instance is running which service
  • Update (Beta Cluster?) documentation on Wikitech on how to add a service and how to update a service
  • npm modules should be specified/fixed in packages.json and use npm install on Beta Cluster
    • auto commit that set of dependencies ???

Event Timeline

hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)

Session happening on Sunday at 9am

greg added a subscriber: Services.
greg subscribed.

One agreed upon outcome: Services will all run from master in Beta Cluster (not their "deploy" branch).

greg triaged this task as Medium priority.May 24 2015, 8:07 AM

Notes from the meeting below:

Deployment service labs

How is code deployed on Beta?

  • MediaWiki = Scap
  • Services (Parsoid, Mathoid, Graphoid, etc)

Parsoid/CXServer

  • deployment-parsoid05
  • Checkout Master/Sudo service restart

Other Services (Mathoid, Graphoid, RESTBase)

  • Manual
  • Postmerge
  • CRON

Git Deploy

  • Postmerge
  • Current state
  • Would be nice to have since it's production
  • Requires refactor to work
  • Not going to use

Via Jenkins Slave

  • puppet role
  • validate the service restart
  • Pull branch and restart via bash
  • Valid model

DONE What branch do we want to run on beta cluser

CLOSED: [2015-05-24 Sun 01:02]

  • Master needs NPM
  • Deploy has node~modules~ w/submodules
  • MASTER on Beta Cluster
  • Use \rm -rf -- node~modules~ && npm install\

Staging environment (canary) for deploy branch

  • cookie detection for staging
  • Maybe already done (ori may know)
  • Testing staging services against everything else production
  • Need possible hardware for staging services

Find good solution for new services

  • Antoine will do a lot of copy-n-paste :)
  • More future plans

TODO RelEng Setup Jenkins for all current services

TODO Update Documentation on wikitech

TODO Talk with ops, services, and releng about Canary or Production Staging

Qgil claimed this task.

This is not solved, still need to:

  • create an epic
  • list the actionables as tasks
  • get in sync with services teams
hashar removed hashar as the assignee of this task.Jun 1 2015, 7:18 PM
hashar moved this task from To Triage to Next: Feature on the Beta-Cluster-Infrastructure board.

This should be paired within Release-Engineering-Team . Probably enough work to do to qualify it as an Epic.

I have added it for our weekly team meeting on June 2nd 2015.

I found probably mention here the auto mode I implemented for git-deploy, which would allow its use for beta deploys.

We are trying to help all open tasks listed under "Work continues after Lyon" at the Wikimedia Hackathon 2015 workboard finding their best way forward. * If you are participating in Wikimania, consider adding the #Wikimania-Hackathon-2015 project to get this task in that loop, which is about to start. * If you think this project could welcome help from a dedicated Google Summer of Code or Outreachy intern, or from an Individual Engagement Grant, add the Possible-Tech-Projects project. * If you would like to receive some other type of support (organizing a Tech Talk, establishing contacts with existing developer teams in Wikimedia or elsewhere, travel sponsorship for a related activity... you name it), please create a subtask explaining your request and associate it with #Engineering-Community (or you can start by commenting here if you prefer). * Keeping the description, priority and assigned fields up to date always helps. :) For some context about this message, see T101151: Evaluate which projects showcased at the Wikimedia Hackathon should be supported further. It is the last communication related to Wikimedia-Hackathon-2015 that we will post here.

From T95652: On beta cluster citoid should self update and reload after change is merged:

@mobrovac and I had some pairing last week to train him up on JJB usage. The informal .plan is to revisit each of the mediawiki services backend and have them all configured alike. Namely:

  • proper source/deploy jenkins jobs
  • self update via Jenkins jobs on postmerge
GWicke edited projects, added Services (attic); removed Services.

This task is obsolete. The idea was to have CI to manage the deployment of the services on the Beta-Cluster-Infrastructure . It got forgotten, we moved to other duties and had troubled times. Nowadays the services are build with the release Pipeline and the deployment is done manually by pulling the Docker image and restarting the container. That seems to work.