Upgrade our logstash-gelf package to latest available upstream version
Closed, ResolvedPublic

Description

The version of logstash-gelf we use (1.5.3) is fairly old. It has issues when running in elasticsearch with a security manager. Upgrading to latest version (1.11.0) fixes the issue. We should upgrade our debian package.

Gehel created this task.Nov 10 2016, 9:41 AM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptNov 10 2016, 9:41 AM
Deskana moved this task from Backlog to In progress on the Discovery-Search (Current work) board.
Deskana added a subscriber: Deskana.

I think this is in progress, so I've marked it as such.

Change 320992 had a related patch set uploaded (by Gehel):
New upstream version: 1.11.0

https://gerrit.wikimedia.org/r/320992

Change 321468 had a related patch set uploaded (by Gehel):
package_builder: add javahelper to standard build dependencies

https://gerrit.wikimedia.org/r/321468

Change 321468 merged by Gehel:
package_builder: add javahelper to standard build dependencies

https://gerrit.wikimedia.org/r/321468

Gehel added a comment.Nov 17 2016, 4:14 PM

Building the package on copper fails as some dependencies are still being fetched from maven central, even if we patch the pom.xml of the project to disable central and use only archiva.

Steps taken:

  • add USENETWORK=yes to ~/.bpuilderrc (access to archiva.wikimedia.org is required)
  • build with GIT_PBUILDER_AUTOCONF=no DIST=trusty git-buildpackage -sa -us -uc --git-builder=git-pbuilder

last portion of the build logs

I: Copying back the cached apt archive contents
I: Copying source file
I: copying [/home/gehel/logstash-gelf_1.11.0-1wmf1.dsc]
I: copying [/home/gehel/logstash-gelf_1.11.0.orig.tar.gz]
I: copying [/home/gehel/logstash-gelf_1.11.0-1wmf1.debian.tar.xz]
I: Extracting source
dpkg-source: warning: extracting unsigned source package (logstash-gelf_1.11.0-1wmf1.dsc)
dpkg-source: info: extracting logstash-gelf in logstash-gelf-1.11.0
dpkg-source: info: unpacking logstash-gelf_1.11.0.orig.tar.gz
dpkg-source: info: unpacking logstash-gelf_1.11.0-1wmf1.debian.tar.xz
dpkg-source: info: applying 0001-no-clean-during-test.patch
dpkg-source: info: applying 0002-downgrade-redis.patch
dpkg-source: info: applying 0003-downgrade-publisher.patch
dpkg-source: info: applying 0004-use-wmf-archiva.patch
I: Building the package
W: no hooks of type A found -- ignoring
I: Running cd tmp/buildd/*/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin" dpkg-buildpackage -us -uc    '-sa' '-us' '-uc'   '-sa' '-us' '-uc' -rfakeroot
dpkg-buildpackage: source package logstash-gelf
dpkg-buildpackage: source version 1.11.0-1wmf1
dpkg-buildpackage: source distribution trusty-wikimedia
dpkg-buildpackage: source changed by Guillaume Lederrey <glederrey@wikimedia.org>
 dpkg-source --before-build logstash-gelf-1.11.0
dpkg-buildpackage: host architecture amd64
dpkg-source: info: using options from logstash-gelf-1.11.0/debian/source/options: --extend-diff-ignore=^\.gitreview$
 fakeroot debian/rules clean
dh clean --with javahelper --with jh_maven_repo_helper
   dh_testdir
   debian/rules override_dh_auto_clean
make[1]: Entering directory `/tmp/buildd/logstash-gelf-1.11.0'
#rm -rf target/
mh_clean
jh_clean
dh_auto_clean
make[1]: Leaving directory `/tmp/buildd/logstash-gelf-1.11.0'
   jh_clean
   mh_clean
   dh_clean
 dpkg-source -b logstash-gelf-1.11.0
dpkg-source: info: using options from logstash-gelf-1.11.0/debian/source/options: --extend-diff-ignore=^\.gitreview$
dpkg-source: info: using source format `3.0 (quilt)'
dpkg-source: info: building logstash-gelf using existing ./logstash-gelf_1.11.0.orig.tar.gz
dpkg-source: info: building logstash-gelf in logstash-gelf_1.11.0-1wmf1.debian.tar.gz
dpkg-source: info: building logstash-gelf in logstash-gelf_1.11.0-1wmf1.dsc
 debian/rules build
dh build --with javahelper --with jh_maven_repo_helper
   dh_testdir
   dh_auto_configure
   jh_linkjars
   debian/rules override_dh_auto_build
make[1]: Entering directory `/tmp/buildd/logstash-gelf-1.11.0'
mvn package
[INFO] Scanning for projects...
Downloading: https://archiva.wikimedia.org/repository/mirrored/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
6K downloaded  (oss-parent-9.pom)
Downloading: https://archiva.wikimedia.org/repository/mirrored/org/jboss/arquillian/arquillian-bom/1.1.5.Final/arquillian-bom-1.1.5.Final.pom
10K downloaded  (arquillian-bom-1.1.5.Final.pom)
Downloading: http://repo1.maven.org/maven2/org/jboss/shrinkwrap/shrinkwrap-bom/1.2.2/shrinkwrap-bom-1.2.2.pom
[WARNING] Unable to get resource 'org.jboss.shrinkwrap:shrinkwrap-bom:pom:1.2.2' from repository central (http://repo1.maven.org/maven2): Error transferring file: Connection timed out
[INFO] ------------------------------------------------------------------------
[ERROR] FATAL ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Error building POM (may not be this project's POM).


Project ID: org.jboss.shrinkwrap:shrinkwrap-bom

Reason: POM 'org.jboss.shrinkwrap:shrinkwrap-bom' not found in repository: Unable to download the artifact from any repository

  org.jboss.shrinkwrap:shrinkwrap-bom:pom:1.2.2

from the specified remote repositories:
  central (http://repo1.maven.org/maven2)

 for project org.jboss.shrinkwrap:shrinkwrap-bom


[INFO] ------------------------------------------------------------------------
[INFO] Trace
org.apache.maven.reactor.MavenExecutionException: POM 'org.jboss.shrinkwrap:shrinkwrap-bom' not found in repository: Unable to download the artifact from any repository

  org.jboss.shrinkwrap:shrinkwrap-bom:pom:1.2.2

from the specified remote repositories:
  central (http://repo1.maven.org/maven2)

 for project org.jboss.shrinkwrap:shrinkwrap-bom
	at org.apache.maven.DefaultMaven.getProjects(DefaultMaven.java:404)
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:272)
	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:138)
	at org.apache.maven.cli.MavenCli.main(MavenCli.java:362)
	at org.apache.maven.cli.compat.CompatibleMain.main(CompatibleMain.java:60)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.codehaus.classworlds.Launcher.launchEnhanced(Launcher.java:315)
	at org.codehaus.classworlds.Launcher.launch(Launcher.java:255)
	at org.codehaus.classworlds.Launcher.mainWithExitCode(Launcher.java:430)
	at org.codehaus.classworlds.Launcher.main(Launcher.java:375)
Caused by: org.apache.maven.project.ProjectBuildingException: POM 'org.jboss.shrinkwrap:shrinkwrap-bom' not found in repository: Unable to download the artifact from any repository

  org.jboss.shrinkwrap:shrinkwrap-bom:pom:1.2.2

from the specified remote repositories:
  central (http://repo1.maven.org/maven2)

 for project org.jboss.shrinkwrap:shrinkwrap-bom
	at org.apache.maven.project.DefaultMavenProjectBuilder.findModelFromRepository(DefaultMavenProjectBuilder.java:605)
	at org.apache.maven.project.DefaultMavenProjectBuilder.buildFromRepository(DefaultMavenProjectBuilder.java:251)
	at org.apache.maven.project.DefaultMavenProjectBuilder.mergeManagedDependencies(DefaultMavenProjectBuilder.java:1456)
	at org.apache.maven.project.DefaultMavenProjectBuilder.processProjectLogic(DefaultMavenProjectBuilder.java:999)
	at org.apache.maven.project.DefaultMavenProjectBuilder.buildInternal(DefaultMavenProjectBuilder.java:880)
	at org.apache.maven.project.DefaultMavenProjectBuilder.buildFromRepository(DefaultMavenProjectBuilder.java:255)
	at org.apache.maven.project.DefaultMavenProjectBuilder.mergeManagedDependencies(DefaultMavenProjectBuilder.java:1456)
	at org.apache.maven.project.DefaultMavenProjectBuilder.processProjectLogic(DefaultMavenProjectBuilder.java:999)
	at org.apache.maven.project.DefaultMavenProjectBuilder.buildInternal(DefaultMavenProjectBuilder.java:880)
	at org.apache.maven.project.DefaultMavenProjectBuilder.buildFromSourceFileInternal(DefaultMavenProjectBuilder.java:508)
	at org.apache.maven.project.DefaultMavenProjectBuilder.build(DefaultMavenProjectBuilder.java:200)
	at org.apache.maven.DefaultMaven.getProject(DefaultMaven.java:604)
	at org.apache.maven.DefaultMaven.collectProjects(DefaultMaven.java:487)
	at org.apache.maven.DefaultMaven.getProjects(DefaultMaven.java:391)
	... 12 more
Caused by: org.apache.maven.artifact.resolver.ArtifactNotFoundException: Unable to download the artifact from any repository

  org.jboss.shrinkwrap:shrinkwrap-bom:pom:1.2.2

from the specified remote repositories:
  central (http://repo1.maven.org/maven2)


	at org.apache.maven.artifact.resolver.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:228)
	at org.apache.maven.artifact.resolver.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:90)
	at org.apache.maven.project.DefaultMavenProjectBuilder.findModelFromRepository(DefaultMavenProjectBuilder.java:558)
	... 25 more
Caused by: org.apache.maven.wagon.ResourceDoesNotExistException: Unable to download the artifact from any repository
	at org.apache.maven.artifact.manager.DefaultWagonManager.getArtifact(DefaultWagonManager.java:404)
	at org.apache.maven.artifact.resolver.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:216)
	... 27 more
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2 minutes 8 seconds
[INFO] Finished at: Thu Nov 17 15:45:55 UTC 2016
[INFO] Final Memory: 5M/119M
[INFO] ------------------------------------------------------------------------
make[1]: *** [override_dh_auto_build] Error 1
make[1]: Leaving directory `/tmp/buildd/logstash-gelf-1.11.0'
make: *** [build] Error 2
dpkg-buildpackage: error: debian/rules build gave error exit status 2
E: Failed autobuilding of package
W: no hooks of type C found -- ignoring
I: unmounting dev/pts filesystem
I: unmounting run/shm filesystem
I: unmounting proc filesystem
 -> Cleaning COW directory
  forking: rm -rf /var/cache/pbuilder/build//cow.7425 
gbp:error: 'git-pbuilder -sa -us -uc' failed: it exited with 1
Gehel added a subscriber: hashar.Nov 17 2016, 4:15 PM

@hashar : you probably have some experience in building .debs from maven projects, any chance you'd know how to make this work? The repo from which I'm trying to build is https://gerrit.wikimedia.org/r/#/c/320992

It might fail due to not using the web proxy?

@Gehel wrote:

@hashar : you probably have some experience in building .debs from maven projects [...]

.debs ?? Maven ??

`
██████╗ ██╗   ██╗███╗   ██╗███████╗     █████╗ ███╗   ██╗██████╗               
██╔══██╗██║   ██║████╗  ██║██╔════╝    ██╔══██╗████╗  ██║██╔══██╗              
██████╔╝██║   ██║██╔██╗ ██║███████╗    ███████║██╔██╗ ██║██║  ██║              
██╔══██╗██║   ██║██║╚██╗██║╚════██║    ██╔══██║██║╚██╗██║██║  ██║              
██║  ██║╚██████╔╝██║ ╚████║███████║    ██║  ██║██║ ╚████║██████╔╝              
╚═╝  ╚═╝ ╚═════╝ ╚═╝  ╚═══╝╚══════╝    ╚═╝  ╚═╝╚═╝  ╚═══╝╚═════╝               

                                                                               
███████╗██████╗ ███████╗ █████╗ ██╗  ██╗███████╗     ██████╗ ██╗   ██╗████████╗
██╔════╝██╔══██╗██╔════╝██╔══██╗██║ ██╔╝██╔════╝    ██╔═══██╗██║   ██║╚══██╔══╝
█████╗  ██████╔╝█████╗  ███████║█████╔╝ ███████╗    ██║   ██║██║   ██║   ██║   
██╔══╝  ██╔══██╗██╔══╝  ██╔══██║██╔═██╗ ╚════██║    ██║   ██║██║   ██║   ██║   
██║     ██║  ██║███████╗██║  ██║██║  ██╗███████║    ╚██████╔╝╚██████╔╝   ██║   
╚═╝     ╚═╝  ╚═╝╚══════╝╚═╝  ╚═╝╚═╝  ╚═╝╚══════╝     ╚═════╝  ╚═════╝    ╚═╝

I thought at first you referred to the Jenkins job that attempt to build the .deb. It fails due to lack of customization of git buidpackage via debian/gbp.conf: upstream/1.11.0 is not a valid treeish. But that is not relevant.

If I remember properly, for analytics there was legitimate concern about including in our .deb random artifacts from Maven Central over HTTP. Hence we have a controlled artifact repository archiva.wikimedia.org which is apparently reachable from what ever machine you have build the package on:

Downloading: https://archiva.wikimedia.org/repository/mirrored/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
6K downloaded  (oss-parent-9.pom)

Then the pom.xml reference org.jboss.shrinkwrap which points somehow to maven.org and that one is not reachable:

Downloading: http://repo1.maven.org/maven2/org/jboss/shrinkwrap/shrinkwrap-bom/1.2.2/shrinkwrap-bom-1.2.2.pom
[WARNING] Unable to get resource 'org.jboss.shrinkwrap:shrinkwrap-bom:pom:1.2.2' from repository central (http://repo1.maven.org/maven2): Error transferring file: Connection timed out

It is listed on our archiva with version 1.2.2 though: https://archiva.wikimedia.org/#artifact/org.jboss.shrinkwrap/shrinkwrap-parent

Looking at debian/patches/0004-use-wmf-archiva.patch it disables maven central in favor of our archiva.

Maven first downloaded file is https://archiva.wikimedia.org/repository/mirrored/org/jboss/arquillian/arquillian-bom/1.1.5.Final/arquillian-bom-1.1.5.Final.pom which references org.jboss.shrinkwrap. It defines some repositories, might be that overrides our statement to disable maven central repo or somehow fallback to its internal setting to use https://repo1.maven.org/maven2 that can surely be set globally.

I took a look at this too and couldn't understand why the override didn't work, anyways it looks like maven 3 is needed now by logstash-gelf (I discovered this by manually changing the global maven preferences for repos). Using maven 3 needs a Build-Dep on maven-debian-helper (>= 2) and BACKPORTS=yes on the command line. With that the build gets past the error above but there are still missing dependencies (from archiva itself I think)

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 06:36 min
[INFO] Finished at: 2016-11-24T21:04:16+00:00
[INFO] Final Memory: 15M/148M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project logstash-gelf: Could not resolve dependencies for project biz.paluch.logging:logstash-
gelf:jar:1.11.0: Failed to collect dependencies at org.wildfly.arquillian:wildfly-arquillian-common:jar:1.0.0.Alpha5 -> org.wild
fly.arquillian:wildfly-arquillian-testenricher-msc:jar:1.0.0.Alpha5 -> org.wildfly.core:wildfly-core-feature-pack:pom:1.0.0.Alph
a17 -> org.jboss.slf4j:slf4j-jboss-logmanager:jar:1.0.3.GA: Failed to read artifact descriptor for org.jboss.slf4j:slf4j-jboss-l
ogmanager:jar:1.0.3.GA: Could not transfer artifact org.jboss.slf4j:slf4j-jboss-logmanager:pom:1.0.3.GA from/to jboss-public-rep
ository-group (http://repository.jboss.org/nexus/content/groups/public/): Connect to repository.jboss.org:80 [repository.jboss.o
rg/209.132.182.97] failed: Connection timed out -> [Help 1]

Coming back to this after a long break...

So it seems that we need to add a few dependencies from JBoss to archiva. I would normally add a proxied repo, but it looks like we only proxy central and cloudera. It looks like we do upload a few non WMF artifacts to our "releases" repository. It also looks like we mirror all external repos to a single "mirrored" managed repo, which might lead to namespace collisions. It also looks like we don't use repository groups.

The simplest thing to do would be to upload manually the few required dependencies to our "releases" repo. Probably the best thing to do would be to add a "jboss-mirrored" managed repo, which would mirror jboss. I'm especially not keen on mixing stuff from the JBoss repo in a global repo as JBoss is known to contain some strange and very much JBoss specific things.

@Ottomata: it seems that you know a few things about how we use archiva here. Do you have an opinion on the subject?

And not directly related to this task, but I would have expected the "mirrored" repo to be a repositoy group, which would group our main mirrors. And I would expect each mirror to be mirrored to a separate managed repo.

After a discussion with @Smalyshev, it seems that we usually upload individual jars to the "mirrored" repository when those are not available in central. I'm going to do just that with the following:

  • org.jboss.slf4j:slf4j-jboss-logmanager:1.0.3.GA
  • org.slf4j:slf4j-api:1.7.7.jbossorg-1
  • org.slf4j:slf4j-parent:1.7.7.jbossorg-1
  • xerces:xercesImpl:2.9.1-jbossas-2

The updated liblogstash-gelf package now build correctly for jessie (build on copper with BACKPORTS=yes GIT_PBUILDER_AUTOCONF=no DIST=jessie git-buildpackage -sa -us -uc --git-builder=git-pbuilder). After consulting with @MoritzMuehlenhoff, and given that this package is simply a .jar file with no runtime dependencies and that we are planning to migrate elasticsearch to jessie (T151326), it is fine to copy the jessie build to trusty.

Great! Yeah, if your number of dependencies is small enough, it is easiest to just manually upload them to Archiva. If your list is larger, then we temporarily add a proxied mirror and just pull in those dependencies to Archiva to freeze them there.

Repo groups? I am a n00b java package d00d, so ¯\_(ツ)_/¯

Gehel added a subscriber: elukey.Jan 3 2017, 11:08 AM

Hadoop also uses liblogstash-gelf. It would make sense to test and upgrade it as well... Testing will be done in coordination with @elukey.

Mentioned in SAL (#wikimedia-operations) [2017-01-03T11:16:06Z] <gehel> upgrade lilogstash-gelf on relforge - T150408

Mentioned in SAL (#wikimedia-operations) [2017-01-03T11:27:24Z] <gehel> upgrade liblogstash-gelf on deployment-elastic* - T150408

Gehel added a comment.Jan 3 2017, 11:33 AM

relforge* and deployment-elastic* have the new logstash-gelf version deployed and are sending logs to logstash again. @elukey we can test on hadoop whenever you want.

Change 320992 merged by Gehel:
New upstream version: 1.11.0

https://gerrit.wikimedia.org/r/320992

Hadoop also uses liblogstash-gelf

Sort of. Early on in the Analytics Cluster lifetime, it was really very difficult for users to figure out what why their jobs failed. We tried to get Hadoop job logs into logstash via gelf. It turned out to be not very useful, and we had to go through a hugely hacky process of using a puppet exec to manually unpack and patch a jar, and then re-jar it in order for Hadoop to pick up the changes. We logstash logging a long time ago, and have never turned it back on.

Mentioned in SAL (#wikimedia-operations) [2017-01-03T14:43:17Z] <gehel> upgrade liblogstash-gelf on elastic* - T150408

Gehel added a comment.Jan 3 2017, 2:45 PM

liblogstash-gelf-java is now up to date on all elasticsearch servers. Production clusters have not been restarted yet, the actual activation will occur at the next cluster restart (which will be all too soon :)

Since hadoop does not seem to ever actually use logstash-gelf, all relevant servers have been updated.

hashar removed a subscriber: hashar.Jan 4 2017, 10:15 AM
Deskana closed this task as Resolved.Jan 6 2017, 10:53 PM