Change Details

Today the process to deploy elasticsearch plugins is complex and requires a lot of manual steps. - Every jar present in a plugin zip file needs to be uploaded manually to archiva.wikimedia.org - a patch to operations/software/elasticsearch/plugins needs to be made - someone must verify that the md5 of every files can be retrieved from archiva.wikimedia.org - Some jar are no longer official maven dependencies and are not available outside the zip file itself (e.g. analysis-icu-5.2.1.jar in analysis-icu-5.2.1.zip) The steps to upload artifacts to archiva.wikimedia.org is a pain, all plugin dependencies need to be uploaded manually, e.g. for analysis-icu: - upload com.ibm.icu:icu4j - upload org.apache.lucene:lucene-analysis-icu - upload a fake artifact (auto generate pom) for org.elasticsearch.plugin:analysis-icu (Every upload require filling 4 text boxes) This is not straightforward, error-prone and time-consuming. We should find a process that is more fluent/automatic. Possible solutions: - create a custom deb package to bundle all the required plugins, ideally working directly with zip distributions.

Today the process to deploy elasticsearch plugins is complex and depends on salt. It is based on trebuchet/git-fat and archiva. Drawbacks are: - Every jar present in a plugin zip file needs to be uploaded manually to archiva.wikimedia.org (note that archiva has no review process) - a patch to operations/software/elasticsearch/plugins needs to be made - someone must verify that the md5 of every files can be retrieved from archiva.wikimedia.org - Some jar are no longer official maven dependencies and are not available outside the zip file itself (e.g. analysis-icu-5.2.1.jar in analysis-icu-5.2.1.zip) - Vagrant uses a completely different approach based on bash scripts and needs to be updated manually This is not straightforward, error-prone, dangerous and time-consuming. We should find a process that is more fluent/automatic and not dependent on salt/trebuchet that will be removed soon. The solution is to create a debian package to deploy the plugins. the process is still manual but involves a review process for third party plugins. Full detail of the process here: https://gerrit.wikimedia.org/r/#/c/352170/8/README.txt Benefits: - adding a new plugin is simply a new line in a file - version dependencies between the plugins and elastic itself will be managed by apt/dpkg (currently causes vagrant to fail if the plugins are not updated at the very same time) - the deb package could be used for production and vagrant (single process for both) - possibility to use the experimental apt repo to deploy new elastic versions and plugins without breaking production or keep unmerged patches in gerrit - no need to deploy artifacts to archiva - java binaries finally reviewed (by at least downloading them from 2 different connections + verifying checksums and gpg keys) Drawbacks: - not fully automatic yet: ops must prepare the build on his machine and scp to copper

Today the process to deploy elasticsearch plugins is complex and requires a lot of manual stepsdepends on salt. It is based on trebuchet/git-fat and archiva. Drawbacks are: - Every jar present in a plugin zip file needs to be uploaded manually to archiva.wikimedia.org (note that archiva has no review process) - a patch to operations/software/elasticsearch/plugins needs to be made - someone must verify that the md5 of every files can be retrieved from archiva.wikimedia.org - Some jar are no longer official maven dependencies and are not available outside the zip file itself (e.g. analysis-icu-5.2.1.jar in analysis-icu-5.2.1.zip) The steps to upload artifacts to archiva.wikimedia.org is a pain, all plugin dependencies need to be uploaded manually, e.g. for analysis-icu: - upload com.ibm.icu:icu4j - upload org.apache.lucene:lucene-analysis-icu - upload a fake artifact (auto generate pom) for org.elasticsearch.plugin:analysis-icuVagrant uses a completely different approach based on bash scripts and needs to be updated manually This is not straightforward, error-prone, dangerous and time-consuming. (Every upload require filling 4 text boxes)We should find a process that is more fluent/automatic and not dependent on salt/trebuchet that will be removed soon. ThisThe solution is not straightforward,to create a debian package to deploy the plugins. error-prone and time-consumingthe process is still manual but involves a review process for third party plugins. We should find aFull detail of the process that is more fluent/automatic.here: https://gerrit.wikimedia.org/r/#/c/352170/8/README.txt Possible solutions:Benefits: - adding a new plugin is simply a new line in a file - create a custom deb package to bundle all the required plugins, ideally working directly with zip distributions.version dependencies between the plugins and elastic itself will be managed by apt/dpkg (currently causes vagrant to fail if the plugins are not updated at the very same time) - the deb package could be used for production and vagrant (single process for both) - possibility to use the experimental apt repo to deploy new elastic versions and plugins without breaking production or keep unmerged patches in gerrit - no need to deploy artifacts to archiva - java binaries finally reviewed (by at least downloading them from 2 different connections + verifying checksums and gpg keys) Drawbacks: - not fully automatic yet: ops must prepare the build on his machine and scp to copper