Use debian packages instead of salt to deploy elasticsearch plugins
Closed, ResolvedPublic

Description

Today the process to deploy elasticsearch plugins is complex and depends on salt.
It is based on trebuchet/git-fat and archiva. Drawbacks are:

  • Every jar present in a plugin zip file needs to be uploaded manually to archiva.wikimedia.org (note that archiva has no review process)
  • a patch to operations/software/elasticsearch/plugins needs to be made
  • someone must verify that the md5 of every files can be retrieved from archiva.wikimedia.org
  • Some jar are no longer official maven dependencies and are not available outside the zip file itself (e.g. analysis-icu-5.2.1.jar in analysis-icu-5.2.1.zip)
  • Vagrant uses a completely different approach based on bash scripts and needs to be updated manually

This is not straightforward, error-prone, dangerous and time-consuming.

We should find a process that is more fluent/automatic and not dependent on salt/trebuchet that will be removed soon.

The solution is to create a debian package to deploy the plugins. the process is still manual but involves a review process for third party plugins.

Full detail of the process here: https://gerrit.wikimedia.org/r/#/c/352170/8/README.txt

Benefits:

  • adding a new plugin is simply a new line in a file
  • version dependencies between the plugins and elastic itself will be managed by apt/dpkg (currently causes vagrant to fail if the plugins are not updated at the very same time)
  • the deb package could be used for production and vagrant (single process for both)
  • possibility to use the experimental apt repo to deploy new elastic versions and plugins without breaking production or keep unmerged patches in gerrit
  • no need to deploy artifacts to archiva
  • java binaries finally reviewed (by at least downloading them from 2 different connections + verifying checksums and gpg keys)

Drawbacks:

  • not fully automatic yet: ops must prepare the build on his machine and scp to copper
dcausse created this task.Feb 20 2017, 12:55 PM
Restricted Application added a project: Discovery. · View Herald TranscriptFeb 20 2017, 12:55 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Packaging a deb with the plugins seems doable, it will involve some manual steps to upload the jars to copper but this can be automated with a bash script.
FTR here is the raw discussion

10:44 <moritzm> the packaging itself is fairly straightforward:
10:44 <dcausse> I'd prefer to write a simple build script that download the zips from its original source, explode them then build the deb 
10:44 <moritzm> - a minimal rules file with dh
10:44 <moritzm> - a debian/control file with the basic parameters like package name
10:44 <moritzm> - a debian/foo.install referencing the jars
10:45 <moritzm> then build it on copper
10:45 <gehel> moritzm: and where do you get the .jars? Download them directly from upstream?
10:45 <moritzm> (that's the part that gehel would need to take care of, since copper is restricted)
10:46 <moritzm> gehel: I was under the impression upstream ships these jar? but I don't know much about elastic plugins
10:46 <dcausse> moritzm: it's a zip file with jars in them
10:47 <gehel> moritzm: my understanding was that we don't want network access during build, so we need to bring those jars to copper in some way...
10:47 <moritzm> hey provide their debs via secure apt (i.e. with signed hashes), maybe they offer signed hashes for the ZIP files as well
10:47 <moritzm> gehel: extract the zip file into the root directory of your debian package tree
10:48 <moritzm> and reference the jars in the .install files
10:48 <moritzm> the scp the entire packge tree to copper, build and upload on install1002
10:48 <gehel> so the download and extract the .zip is a manual step (or at least a step before packaging itself)
10:49 <moritzm> then you have all the ameneties of a Debian package, like using apt, debdeploy, internal repo etc.
10:49 <moritzm> gehel: yes, that would be manual, but if it only happens 4/5 times a year not worth automating
10:50 <gehel> and it can be automated with ~5 lines of bash...
10:50 <moritzm> yep
debt triaged this task as Normal priority.Mar 2 2017, 11:13 PM
debt moved this task from Needs triage to This Quarter on the Discovery-Search board.
dcausse raised the priority of this task from Normal to High.EditedMay 4 2017, 12:00 PM

raising prio, our process is still too painful. Given we will add even more plugins we really need to get away from the current git/git-fat/trebuchet deploy process:

  • some plugins are broken, zip file is not compliant with elastic format (needs a lot of hacks in both vagrant puppet and our mvnhelper)
  • some plugins will need additional resources deployed (hebmorph)
  • our mvnhelper works with dependencies but not all jars are proper dependencies (bundle), we have some conflicts (something wants morphologik-fsa-2.1.1 but ukrainian wants morphologik-fsa-2.1.0, that is impossile to fix with a maven process)

imo a deb with a build stage where we list zip file locations + sig files if they exists for gpg check. Few hacks in the bash for dealing with broken zip structures.
Advantages:

  • consistent dependency: elastic-version <> plugins-version will be managed by apt
  • can use the experimental repo consistently: no need to deploy an uncommitted patch
  • consistent with vagrant: no need to have 2 methods to install plugins prod vs vagrant
TJones updated the task description. (Show Details)May 4 2017, 2:23 PM
TJones awarded a token.

Change 352170 had a related patch set uploaded (by DCausse; owner: DCausse):
[operations/software/elasticsearch/plugins@master] [WIP] Switch this repo to a deb package

https://gerrit.wikimedia.org/r/352170

Salt will be removed from production in Q1 (2017-2018) so this task must be addressed before then.

dcausse renamed this task from Simplify how we deploy elasticsearch plugins to Use debian packages instead of salt to deploy elasticsearch plugins.May 31 2017, 7:21 AM
dcausse moved this task from Backlog to Needs review on the Discovery-Search (Current work) board.
Gehel claimed this task.Jun 27 2017, 5:00 PM
TJones added a subscriber: TJones.Jun 27 2017, 5:52 PM

I don't have anything useful to add and I already gave this a token, but here's another thumbs up: 👍

Thanks again for working on this!

dcausse updated the task description. (Show Details)Jul 17 2017, 2:10 PM

@dcausse : I built a package using the new approach (with the two changes I mentioned wrt gpg->gnupg and setting DISTRIBUTION to jessie) and that worked well, result is here in case you want to test further: https://people.wikimedia.org/~jmm/elastic/

@MoritzMuehlenhoff I tested the package you generated and it seems to work well, thanks!
(I amended the patch with your suggestions)

Change 375812 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch - deploy plugins with debian package instead of trebuchet

https://gerrit.wikimedia.org/r/375812

Change 352170 merged by Gehel:
[operations/software/elasticsearch/plugins@master] Switch this repo to a deb package

https://gerrit.wikimedia.org/r/352170

Mentioned in SAL (#wikimedia-operations) [2017-09-12T08:38:07Z] <gehel> deploying elasticsearch plugins on relforge - T158560 (wrong ticket number before)

Gehel added a comment.Tue, Sep 12, 8:51 AM

Deployment on relforge:

  • stop puppet
  • rm -rf /srv/deployment/elasticsearch/plugins/
  • rm -rf /usr/share/elasticsearch/plugins
  • apt-get install wmf-elasticsearch-search-plugins
  • reinstall kuromoji plugin

Mentioned in SAL (#wikimedia-operations) [2017-09-12T14:02:18Z] <gehel> switching elasticsearch plugin deployment to .deb instead of scap on cirrus/codfw - T158560

Mentioned in SAL (#wikimedia-operations) [2017-09-12T14:05:59Z] <gehel> restarting elasticsearch on elastic2001 to validate new plugin deployment - T158560

Mentioned in SAL (#wikimedia-operations) [2017-09-12T14:07:25Z] <gehel> switching elasticsearch plugin deployment to .deb instead of scap on cirrus/eqiad - T158560

Mentioned in SAL (#wikimedia-operations) [2017-09-12T14:10:16Z] <gehel> restarting elasticsearch on elastic1020 to validate new plugin deployment - T158560

Change 375812 merged by Gehel:
[operations/puppet@production] elasticsearch - deploy plugins with debian package instead of trebuchet

https://gerrit.wikimedia.org/r/375812

Mentioned in SAL (#wikimedia-operations) [2017-09-12T15:35:39Z] <gehel> restart elasticsearch on relforge (last check of new plugin deployment) - T158560

New plugins are deployed, elasticsearch has been restarted on 2 nodes to validate. I'm waiting for the elasticsearch 5.5.x upgrade to restart the full cluster.

debt closed this task as Resolved.Fri, Sep 15, 8:18 PM