The version of logstash-gelf we use (1.5.3) is fairly old. It has issues when running in elasticsearch with a security manager. Upgrading to latest version (1.11.0) fixes the issue. We should upgrade our debian package.
Description
Description
Details
Details
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Gehel | T136696 Elasticsearch logs are not send to logstash after 2.3.3 upgrade | |||
| Resolved | Gehel | T150408 Upgrade our logstash-gelf package to latest available upstream version |
Event Timeline
Comment Actions
Change 320992 had a related patch set uploaded (by Gehel):
New upstream version: 1.11.0
Comment Actions
Change 321468 had a related patch set uploaded (by Gehel):
package_builder: add javahelper to standard build dependencies
Comment Actions
Change 321468 merged by Gehel:
package_builder: add javahelper to standard build dependencies
Comment Actions
Building the package on copper fails as some dependencies are still being fetched from maven central, even if we patch the pom.xml of the project to disable central and use only archiva.
Steps taken:
- add USENETWORK=yes to ~/.bpuilderrc (access to archiva.wikimedia.org is required)
- build with GIT_PBUILDER_AUTOCONF=no DIST=trusty git-buildpackage -sa -us -uc --git-builder=git-pbuilder
last portion of the build logs
I: Copying back the cached apt archive contents I: Copying source file I: copying [/home/gehel/logstash-gelf_1.11.0-1wmf1.dsc] I: copying [/home/gehel/logstash-gelf_1.11.0.orig.tar.gz] I: copying [/home/gehel/logstash-gelf_1.11.0-1wmf1.debian.tar.xz] I: Extracting source dpkg-source: warning: extracting unsigned source package (logstash-gelf_1.11.0-1wmf1.dsc) dpkg-source: info: extracting logstash-gelf in logstash-gelf-1.11.0 dpkg-source: info: unpacking logstash-gelf_1.11.0.orig.tar.gz dpkg-source: info: unpacking logstash-gelf_1.11.0-1wmf1.debian.tar.xz dpkg-source: info: applying 0001-no-clean-during-test.patch dpkg-source: info: applying 0002-downgrade-redis.patch dpkg-source: info: applying 0003-downgrade-publisher.patch dpkg-source: info: applying 0004-use-wmf-archiva.patch I: Building the package W: no hooks of type A found -- ignoring I: Running cd tmp/buildd/*/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin" dpkg-buildpackage -us -uc '-sa' '-us' '-uc' '-sa' '-us' '-uc' -rfakeroot dpkg-buildpackage: source package logstash-gelf dpkg-buildpackage: source version 1.11.0-1wmf1 dpkg-buildpackage: source distribution trusty-wikimedia dpkg-buildpackage: source changed by Guillaume Lederrey <glederrey@wikimedia.org> dpkg-source --before-build logstash-gelf-1.11.0 dpkg-buildpackage: host architecture amd64 dpkg-source: info: using options from logstash-gelf-1.11.0/debian/source/options: --extend-diff-ignore=^\.gitreview$ fakeroot debian/rules clean dh clean --with javahelper --with jh_maven_repo_helper dh_testdir debian/rules override_dh_auto_clean make[1]: Entering directory `/tmp/buildd/logstash-gelf-1.11.0' #rm -rf target/ mh_clean jh_clean dh_auto_clean make[1]: Leaving directory `/tmp/buildd/logstash-gelf-1.11.0' jh_clean mh_clean dh_clean dpkg-source -b logstash-gelf-1.11.0 dpkg-source: info: using options from logstash-gelf-1.11.0/debian/source/options: --extend-diff-ignore=^\.gitreview$ dpkg-source: info: using source format `3.0 (quilt)' dpkg-source: info: building logstash-gelf using existing ./logstash-gelf_1.11.0.orig.tar.gz dpkg-source: info: building logstash-gelf in logstash-gelf_1.11.0-1wmf1.debian.tar.gz dpkg-source: info: building logstash-gelf in logstash-gelf_1.11.0-1wmf1.dsc debian/rules build dh build --with javahelper --with jh_maven_repo_helper dh_testdir dh_auto_configure jh_linkjars debian/rules override_dh_auto_build make[1]: Entering directory `/tmp/buildd/logstash-gelf-1.11.0' mvn package [INFO] Scanning for projects... Downloading: https://archiva.wikimedia.org/repository/mirrored/org/sonatype/oss/oss-parent/9/oss-parent-9.pom 6K downloaded (oss-parent-9.pom) Downloading: https://archiva.wikimedia.org/repository/mirrored/org/jboss/arquillian/arquillian-bom/1.1.5.Final/arquillian-bom-1.1.5.Final.pom 10K downloaded (arquillian-bom-1.1.5.Final.pom) Downloading: http://repo1.maven.org/maven2/org/jboss/shrinkwrap/shrinkwrap-bom/1.2.2/shrinkwrap-bom-1.2.2.pom [WARNING] Unable to get resource 'org.jboss.shrinkwrap:shrinkwrap-bom:pom:1.2.2' from repository central (http://repo1.maven.org/maven2): Error transferring file: Connection timed out [INFO] ------------------------------------------------------------------------ [ERROR] FATAL ERROR [INFO] ------------------------------------------------------------------------ [INFO] Error building POM (may not be this project's POM). Project ID: org.jboss.shrinkwrap:shrinkwrap-bom Reason: POM 'org.jboss.shrinkwrap:shrinkwrap-bom' not found in repository: Unable to download the artifact from any repository org.jboss.shrinkwrap:shrinkwrap-bom:pom:1.2.2 from the specified remote repositories: central (http://repo1.maven.org/maven2) for project org.jboss.shrinkwrap:shrinkwrap-bom [INFO] ------------------------------------------------------------------------ [INFO] Trace org.apache.maven.reactor.MavenExecutionException: POM 'org.jboss.shrinkwrap:shrinkwrap-bom' not found in repository: Unable to download the artifact from any repository org.jboss.shrinkwrap:shrinkwrap-bom:pom:1.2.2 from the specified remote repositories: central (http://repo1.maven.org/maven2) for project org.jboss.shrinkwrap:shrinkwrap-bom at org.apache.maven.DefaultMaven.getProjects(DefaultMaven.java:404) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:272) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:138) at org.apache.maven.cli.MavenCli.main(MavenCli.java:362) at org.apache.maven.cli.compat.CompatibleMain.main(CompatibleMain.java:60) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.codehaus.classworlds.Launcher.launchEnhanced(Launcher.java:315) at org.codehaus.classworlds.Launcher.launch(Launcher.java:255) at org.codehaus.classworlds.Launcher.mainWithExitCode(Launcher.java:430) at org.codehaus.classworlds.Launcher.main(Launcher.java:375) Caused by: org.apache.maven.project.ProjectBuildingException: POM 'org.jboss.shrinkwrap:shrinkwrap-bom' not found in repository: Unable to download the artifact from any repository org.jboss.shrinkwrap:shrinkwrap-bom:pom:1.2.2 from the specified remote repositories: central (http://repo1.maven.org/maven2) for project org.jboss.shrinkwrap:shrinkwrap-bom at org.apache.maven.project.DefaultMavenProjectBuilder.findModelFromRepository(DefaultMavenProjectBuilder.java:605) at org.apache.maven.project.DefaultMavenProjectBuilder.buildFromRepository(DefaultMavenProjectBuilder.java:251) at org.apache.maven.project.DefaultMavenProjectBuilder.mergeManagedDependencies(DefaultMavenProjectBuilder.java:1456) at org.apache.maven.project.DefaultMavenProjectBuilder.processProjectLogic(DefaultMavenProjectBuilder.java:999) at org.apache.maven.project.DefaultMavenProjectBuilder.buildInternal(DefaultMavenProjectBuilder.java:880) at org.apache.maven.project.DefaultMavenProjectBuilder.buildFromRepository(DefaultMavenProjectBuilder.java:255) at org.apache.maven.project.DefaultMavenProjectBuilder.mergeManagedDependencies(DefaultMavenProjectBuilder.java:1456) at org.apache.maven.project.DefaultMavenProjectBuilder.processProjectLogic(DefaultMavenProjectBuilder.java:999) at org.apache.maven.project.DefaultMavenProjectBuilder.buildInternal(DefaultMavenProjectBuilder.java:880) at org.apache.maven.project.DefaultMavenProjectBuilder.buildFromSourceFileInternal(DefaultMavenProjectBuilder.java:508) at org.apache.maven.project.DefaultMavenProjectBuilder.build(DefaultMavenProjectBuilder.java:200) at org.apache.maven.DefaultMaven.getProject(DefaultMaven.java:604) at org.apache.maven.DefaultMaven.collectProjects(DefaultMaven.java:487) at org.apache.maven.DefaultMaven.getProjects(DefaultMaven.java:391) ... 12 more Caused by: org.apache.maven.artifact.resolver.ArtifactNotFoundException: Unable to download the artifact from any repository org.jboss.shrinkwrap:shrinkwrap-bom:pom:1.2.2 from the specified remote repositories: central (http://repo1.maven.org/maven2) at org.apache.maven.artifact.resolver.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:228) at org.apache.maven.artifact.resolver.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:90) at org.apache.maven.project.DefaultMavenProjectBuilder.findModelFromRepository(DefaultMavenProjectBuilder.java:558) ... 25 more Caused by: org.apache.maven.wagon.ResourceDoesNotExistException: Unable to download the artifact from any repository at org.apache.maven.artifact.manager.DefaultWagonManager.getArtifact(DefaultWagonManager.java:404) at org.apache.maven.artifact.resolver.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:216) ... 27 more [INFO] ------------------------------------------------------------------------ [INFO] Total time: 2 minutes 8 seconds [INFO] Finished at: Thu Nov 17 15:45:55 UTC 2016 [INFO] Final Memory: 5M/119M [INFO] ------------------------------------------------------------------------ make[1]: *** [override_dh_auto_build] Error 1 make[1]: Leaving directory `/tmp/buildd/logstash-gelf-1.11.0' make: *** [build] Error 2 dpkg-buildpackage: error: debian/rules build gave error exit status 2 E: Failed autobuilding of package W: no hooks of type C found -- ignoring I: unmounting dev/pts filesystem I: unmounting run/shm filesystem I: unmounting proc filesystem -> Cleaning COW directory forking: rm -rf /var/cache/pbuilder/build//cow.7425 gbp:error: 'git-pbuilder -sa -us -uc' failed: it exited with 1
Comment Actions
@hashar : you probably have some experience in building .debs from maven projects, any chance you'd know how to make this work? The repo from which I'm trying to build is https://gerrit.wikimedia.org/r/#/c/320992
Comment Actions
@Gehel wrote:
@hashar : you probably have some experience in building .debs from maven projects [...]
.debs ?? Maven ??
`
██████╗ ██╗ ██╗███╗ ██╗███████╗ █████╗ ███╗ ██╗██████╗
██╔══██╗██║ ██║████╗ ██║██╔════╝ ██╔══██╗████╗ ██║██╔══██╗
██████╔╝██║ ██║██╔██╗ ██║███████╗ ███████║██╔██╗ ██║██║ ██║
██╔══██╗██║ ██║██║╚██╗██║╚════██║ ██╔══██║██║╚██╗██║██║ ██║
██║ ██║╚██████╔╝██║ ╚████║███████║ ██║ ██║██║ ╚████║██████╔╝
╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═══╝╚══════╝ ╚═╝ ╚═╝╚═╝ ╚═══╝╚═════╝
███████╗██████╗ ███████╗ █████╗ ██╗ ██╗███████╗ ██████╗ ██╗ ██╗████████╗
██╔════╝██╔══██╗██╔════╝██╔══██╗██║ ██╔╝██╔════╝ ██╔═══██╗██║ ██║╚══██╔══╝
█████╗ ██████╔╝█████╗ ███████║█████╔╝ ███████╗ ██║ ██║██║ ██║ ██║
██╔══╝ ██╔══██╗██╔══╝ ██╔══██║██╔═██╗ ╚════██║ ██║ ██║██║ ██║ ██║
██║ ██║ ██║███████╗██║ ██║██║ ██╗███████║ ╚██████╔╝╚██████╔╝ ██║
╚═╝ ╚═╝ ╚═╝╚══════╝╚═╝ ╚═╝╚═╝ ╚═╝╚══════╝ ╚═════╝ ╚═════╝ ╚═╝Comment Actions
I thought at first you referred to the Jenkins job that attempt to build the .deb. It fails due to lack of customization of git buidpackage via debian/gbp.conf: upstream/1.11.0 is not a valid treeish. But that is not relevant.
If I remember properly, for analytics there was legitimate concern about including in our .deb random artifacts from Maven Central over HTTP. Hence we have a controlled artifact repository archiva.wikimedia.org which is apparently reachable from what ever machine you have build the package on:
Downloading: https://archiva.wikimedia.org/repository/mirrored/org/sonatype/oss/oss-parent/9/oss-parent-9.pom 6K downloaded (oss-parent-9.pom)
Then the pom.xml reference org.jboss.shrinkwrap which points somehow to maven.org and that one is not reachable:
Downloading: http://repo1.maven.org/maven2/org/jboss/shrinkwrap/shrinkwrap-bom/1.2.2/shrinkwrap-bom-1.2.2.pom [WARNING] Unable to get resource 'org.jboss.shrinkwrap:shrinkwrap-bom:pom:1.2.2' from repository central (http://repo1.maven.org/maven2): Error transferring file: Connection timed out
It is listed on our archiva with version 1.2.2 though: https://archiva.wikimedia.org/#artifact/org.jboss.shrinkwrap/shrinkwrap-parent
Comment Actions
Looking at debian/patches/0004-use-wmf-archiva.patch it disables maven central in favor of our archiva.
Maven first downloaded file is https://archiva.wikimedia.org/repository/mirrored/org/jboss/arquillian/arquillian-bom/1.1.5.Final/arquillian-bom-1.1.5.Final.pom which references org.jboss.shrinkwrap. It defines some repositories, might be that overrides our statement to disable maven central repo or somehow fallback to its internal setting to use https://repo1.maven.org/maven2 that can surely be set globally.
Comment Actions
I took a look at this too and couldn't understand why the override didn't work, anyways it looks like maven 3 is needed now by logstash-gelf (I discovered this by manually changing the global maven preferences for repos). Using maven 3 needs a Build-Dep on maven-debian-helper (>= 2) and BACKPORTS=yes on the command line. With that the build gets past the error above but there are still missing dependencies (from archiva itself I think)
[INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 06:36 min [INFO] Finished at: 2016-11-24T21:04:16+00:00 [INFO] Final Memory: 15M/148M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal on project logstash-gelf: Could not resolve dependencies for project biz.paluch.logging:logstash- gelf:jar:1.11.0: Failed to collect dependencies at org.wildfly.arquillian:wildfly-arquillian-common:jar:1.0.0.Alpha5 -> org.wild fly.arquillian:wildfly-arquillian-testenricher-msc:jar:1.0.0.Alpha5 -> org.wildfly.core:wildfly-core-feature-pack:pom:1.0.0.Alph a17 -> org.jboss.slf4j:slf4j-jboss-logmanager:jar:1.0.3.GA: Failed to read artifact descriptor for org.jboss.slf4j:slf4j-jboss-l ogmanager:jar:1.0.3.GA: Could not transfer artifact org.jboss.slf4j:slf4j-jboss-logmanager:pom:1.0.3.GA from/to jboss-public-rep ository-group (http://repository.jboss.org/nexus/content/groups/public/): Connect to repository.jboss.org:80 [repository.jboss.o rg/209.132.182.97] failed: Connection timed out -> [Help 1]
Comment Actions
Coming back to this after a long break...
So it seems that we need to add a few dependencies from JBoss to archiva. I would normally add a proxied repo, but it looks like we only proxy central and cloudera. It looks like we do upload a few non WMF artifacts to our "releases" repository. It also looks like we mirror all external repos to a single "mirrored" managed repo, which might lead to namespace collisions. It also looks like we don't use repository groups.
The simplest thing to do would be to upload manually the few required dependencies to our "releases" repo. Probably the best thing to do would be to add a "jboss-mirrored" managed repo, which would mirror jboss. I'm especially not keen on mixing stuff from the JBoss repo in a global repo as JBoss is known to contain some strange and very much JBoss specific things.
@Ottomata: it seems that you know a few things about how we use archiva here. Do you have an opinion on the subject?
And not directly related to this task, but I would have expected the "mirrored" repo to be a repositoy group, which would group our main mirrors. And I would expect each mirror to be mirrored to a separate managed repo.
Comment Actions
After a discussion with @Smalyshev, it seems that we usually upload individual jars to the "mirrored" repository when those are not available in central. I'm going to do just that with the following:
- org.jboss.slf4j:slf4j-jboss-logmanager:1.0.3.GA
- org.slf4j:slf4j-api:1.7.7.jbossorg-1
- org.slf4j:slf4j-parent:1.7.7.jbossorg-1
- xerces:xercesImpl:2.9.1-jbossas-2
Comment Actions
The updated liblogstash-gelf package now build correctly for jessie (build on copper with BACKPORTS=yes GIT_PBUILDER_AUTOCONF=no DIST=jessie git-buildpackage -sa -us -uc --git-builder=git-pbuilder). After consulting with @MoritzMuehlenhoff, and given that this package is simply a .jar file with no runtime dependencies and that we are planning to migrate elasticsearch to jessie (T151326), it is fine to copy the jessie build to trusty.
Comment Actions
Great! Yeah, if your number of dependencies is small enough, it is easiest to just manually upload them to Archiva. If your list is larger, then we temporarily add a proxied mirror and just pull in those dependencies to Archiva to freeze them there.
Repo groups? I am a n00b java package d00d, so ¯\_(ツ)_/¯
Comment Actions
Hadoop also [[ https://github.com/wikimedia/operations-puppet/blob/production/modules/role/manifests/analytics_cluster/hadoop/logstash.pp#L14-L42 | uses liblogstash-gelf ]]. It would make sense to test and upgrade it as well... Testing will be done in coordination with @elukey.
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2017-01-03T11:16:06Z] <gehel> upgrade lilogstash-gelf on relforge - T150408
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2017-01-03T11:27:24Z] <gehel> upgrade liblogstash-gelf on deployment-elastic* - T150408
Comment Actions
relforge* and deployment-elastic* have the new logstash-gelf version deployed and are sending logs to logstash again. @elukey we can test on hadoop whenever you want.
Comment Actions
Hadoop also uses liblogstash-gelf
Sort of. Early on in the Analytics Cluster lifetime, it was really very difficult for users to figure out what why their jobs failed. We tried to get Hadoop job logs into logstash via gelf. It turned out to be not very useful, and we had to go through a hugely hacky process of using a puppet exec to manually unpack and patch a jar, and then re-jar it in order for Hadoop to pick up the changes. We logstash logging a long time ago, and have never turned it back on.
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2017-01-03T14:43:17Z] <gehel> upgrade liblogstash-gelf on elastic* - T150408
Comment Actions
liblogstash-gelf-java is now up to date on all elasticsearch servers. Production clusters have not been restarted yet, the actual activation will occur at the next cluster restart (which will be all too soon :)
Comment Actions
Since hadoop does not seem to ever actually use logstash-gelf, all relevant servers have been updated.