Page MenuHomePhabricator

Regression test for Logstash filters
Closed, ResolvedPublic

Description

The Logstash filters are currently not tested for regressions, in other words when changing filters it is very easy to also introduce regressions and not be aware.

I have run into https://github.com/magnusbaeck/logstash-filter-verifier and seems promising: test cases can be specified in yaml or json in the "input line" + "expected output" format.

Things to do:

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 5 2020, 11:06 AM

Change 594460 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] profile: initial tests for logstash filters

https://gerrit.wikimedia.org/r/594460

fgiunchedi updated the task description. (Show Details)May 5 2020, 1:02 PM
fgiunchedi updated the task description. (Show Details)

cc Release-Engineering-Team: I'd like to have puppet CI run a golang binary and logstash for logstash regression tests, what's the recommended way to achieve that ? I outlined a couple of options in the task description, what do you think ?

fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.May 5 2020, 1:37 PM
fgiunchedi moved this task from Backlog to In Dev/Progress on the Wikimedia-Logstash board.
fgiunchedi removed a project: observability.
fgiunchedi moved this task from Inbox to In progress on the observability board.

I gave the naive option (add logstash to operations-puppet docker image) and I realize it might be a tall order for the size difference (1.12GB vs 1.69GB) but it would work as expected, logstash tests would of course run only when needed

diff --git a/dockerfiles/operations-puppet/Dockerfile.template b/dockerfiles/operations-puppet/Dockerfile.template
index b6f22382..b67f49c8 100644
--- a/dockerfiles/operations-puppet/Dockerfile.template
+++ b/dockerfiles/operations-puppet/Dockerfile.template
@@ -6,11 +6,20 @@ ENV PUPPET_DIR='/srv/workspace/puppet'
 
 {% set pkgs_to_install = """build-essential bundler  rubygems-integration \
 rake ruby ruby-dev ca-certificates default-libmysqlclient-dev mtail=3.0.0~rc5-1~bpo9+1wmf2 \
-isc-dhcp-server unzip lua-busted file python3-dev python3-pip tox python-dev python-pip python3-wheel python-wheel systemd"""
+isc-dhcp-server unzip lua-busted file python3-dev python3-pip tox python-dev python-pip \
+python3-wheel python-wheel systemd logstash-oss logstash-filter-verifier"""
 %}
 
 ARG PIP_DISABLE_PIP_VERSION_CHECK=1
 
+# Install logstash repositories and JRE. The latter is required for Logstash to
+# not fail at install time if java isn't found or hasn't been configured by
+# dpkg yet. See also https://github.com/elastic/logstash/issues/6394
+USER root
+RUN echo "deb http://apt.wikimedia.org/wikimedia buster-wikimedia thirdparty/elastic74" \
+    > /etc/apt/sources.list.d/wikimedia-elastic.list && \
+    {{ "default-jre-headless" | apt_install }}
+
 USER root
 RUN {{ pkgs_to_install | apt_install }} \
     && pip3 install --no-cache-dir setuptools \
fgiunchedi updated the task description. (Show Details)May 11 2020, 2:48 PM

Change 597227 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[integration/config@master] operations-puppet: ship logstash and logstash-filter-verifier

https://gerrit.wikimedia.org/r/597227

Joe added a subscriber: Joe.May 19 2020, 10:18 AM

I think we're at the point where it would be best if we could change the logic of our testing, and use docker directly, so that we can split tests into different images.

But I think that should be a separated task and should not block you right now.

Change 598703 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Job to run logstash-filter-verifier on puppet.git

https://gerrit.wikimedia.org/r/598703

Change 597227 merged by jenkins-bot:
[integration/config@master] dockerfiles: image for logstash-filter-verifier

https://gerrit.wikimedia.org/r/597227

Mentioned in SAL (#wikimedia-releng) [2020-05-26T12:26:48Z] <hashar> Built docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.1 # T251869

Change 598703 merged by jenkins-bot:
[integration/config@master] Job to run logstash-filter-verifier on puppet.git

https://gerrit.wikimedia.org/r/598703

Change 594460 merged by Filippo Giunchedi:
[operations/puppet@production] profile: initial tests for logstash filters

https://gerrit.wikimedia.org/r/594460

fgiunchedi updated the task description. (Show Details)May 26 2020, 1:03 PM
jijiki added a subscriber: jijiki.May 28 2020, 10:42 AM

Change 608061 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] logstash: add tests for filter-mediawiki

https://gerrit.wikimedia.org/r/608061

Change 608061 merged by Filippo Giunchedi:
[operations/puppet@production] logstash: add tests for filter-mediawiki

https://gerrit.wikimedia.org/r/c/operations/puppet/ /608061

fgiunchedi closed this task as Resolved.Jul 21 2020, 9:27 AM
fgiunchedi claimed this task.

We have the testing scaffolding in place and coverage for basic filters, we can add more tests as needed.