Puppetize clickhouse
Description
Description
Details
Details
Related Changes in Gerrit:
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| [WIP] Yandex ClickHouse puppetization | operations/puppet | production | +427 -0 |
Related Objects
Related Objects
Event Timeline
Comment Actions
clickhouse is a columnar datastore that we are using as an aid to run complex SQL queries on the edit data "lake" that we have as a result of the edit reconstruction project.
It is similar to Druid but faster for complex queries. It cannot however be used as a Druid replacement because it lacks feature ssuch as e a cache in front for repeated queries and does not have a friendly way to do data management when it comes to have data replicated when you add a node. Regardless it is a useful tool to be able to iterate and vet data faster that we might be able to do in Druid.
We are going to deploy a very simple puppetization of clickhouse to the same cluster than druid is deployed to so we have a reproducible setup
Comment Actions
Pkg work, build for our OS distro, import pkgs
uploading packages to wikimedia
puppet code
Test deployment
Comment Actions
Opened https://github.com/yandex/ClickHouse/issues/228 to upstream to ask for some clarification of the debian files.
Comment Actions
I managed to build version 54010 but not more recent "stable" ones, due to various build errors (see https://github.com/yandex/ClickHouse/issues/228 for more info).
Lintian returns some concerns:
elukey@copper:~/clickhouse/ClickHouse$ lintian /var/cache/pbuilder/result/sid-amd64/clickhouse_1.1.54010_amd64.changes E: clickhouse changes: changed-by-name-missing <elukey@yandex-team.ru> W: clickhouse source: diff-contains-git-control-dir .git E: clickhouse source: license-problem-json-evil contrib/libpoco/doc/Acknowledgements.html E: clickhouse source: license-problem-json-evil contrib/libpoco/doc/90100-Acknowledgements.page E: clickhouse source: license-problem-json-evil contrib/libpoco/JSON/include/Poco/JSON/Parser.h W: clickhouse source: maintainer-script-lacks-debhelper-token debian/clickhouse-server-base.postinst W: clickhouse source: changelog-should-mention-nmu W: clickhouse source: source-nmu-has-incorrect-version-number 1.1.54010 W: clickhouse source: debian-rules-missing-recommended-target build-indep W: clickhouse source: debian-rules-missing-recommended-target build-arch W: clickhouse source: no-debian-copyright W: clickhouse source: newer-standards-version 3.9.8 (current is 3.9.6) E: clickhouse-server-common: description-starts-with-package-name E: clickhouse-server-common: description-too-short clickhouse-server-common E: clickhouse-server-common: description-is-pkg-name clickhouse-server-common W: clickhouse-server-common: empty-binary-package E: clickhouse-server-base: description-starts-with-package-name E: clickhouse-server-base: description-too-short clickhouse-server-base E: clickhouse-server-base: description-is-pkg-name clickhouse-server-base W: clickhouse-server-base: empty-binary-package E: clickhouse-server-base: postrm-does-not-call-updaterc.d-for-init.d-script etc/init.d/clickhouse-server W: clickhouse-server-base: init.d-script-not-marked-as-conffile etc/init.d/clickhouse-server E: clickhouse-server-base: init.d-script-not-included-in-package etc/init.d/clickhouse-server
The libpoco JSON license seems to go against https://wiki.debian.org/qa.debian.org/jsonevil
Comment Actions
Summary:
All the Clickhouse dependencies are built statically in one binary that depends only on libc at runtime (verified with ldd).
Available things:
- A Trusty Debian package that can theoretically work on Jessie as is (we are currently using its un-archived/un-tar version on druid100[123] with screen sessions).
- Some automation to create Debian packages (https://github.com/yandex/ClickHouse/blob/master/doc/build.md). I managed to make some changes (outlined in https://github.com/yandex/ClickHouse/issues/228) to build version 54010 with pdebuild and DISTRO=sid (to leverage gcc-6, g++-6, etc.. as suggested by the Clickhouse docs). Lintian is not really happy about the result, also alerting about license-problem-json-evil.
So we have several options available:
- Upload the Clickhouse Trusty deb to reprepro (`jessie thirdparty?)
- Build the package on copper using Debian Sid dependencies and try to solve licensing issues with upstream, eventually uploading the deb to reprepro (thirdparty?)
- Something better that I don't know.
Why we need Clickhouse if we already have Druid/Pivot? Because it is faster on some workloads that the Analytics team needs to perform for the Mediawiki History reconstruction project. We don't really know how much time Clickhouse will be needed, and I'd really like to get it puppetized/productionized correctly.
@faidon: Do you have any suggestion about how to proceed? I am really newbie with Debian packages and some guidance would really help :)
Comment Actions
The better the Debian package is, the better for future maintainability, but I wouldn't say that good Debian packages are a requirement for us, especially for something with an as limited deployment as this one. Depending on how much you like this piece of software and want to help it pick up in use, you might be interested into writing a proper package and pushing it to Debian -- but this certainly is not a requirement for using it at the WMF.
On the technical side, from a quick glance I gave it, it's using debhelper 5 which is super old (current version is 10, or 9 if you're targeting jessie) and the packaging changes are certainly going to be quite invasive. On the plus side, the toolchain has evolved quite a bit since, and an up-to-date package will get *a lot* simpler :)
If you build the package or import a .deb, you shouldn't use the "thirdparty" section of our repository in this case; we've (informally) designated that for use by the automatic import mechanism of reprepro, via the "updates" file. "main" would probably be more appropriate here.
Finally, as far as the JSON license goes… it's complicated. It's not an FSF nor OSI-approved license and we've discussed it here a few times before (e.g. T49431). As it's non-OSI-approved, in theory we shouldn't be using it and for HHVM we went into great lengths to replace its use with the json-c library. However, I'm fairly confident that the Analytics stack is using it in all kinds of places: the Hadoop ecosystem in particular is known to carry quite a few dependencies using this problematic piece of code. Coincidentally, the Apache Foundation is having a complicated discussion as we speak on how to rip out code licensed under the evil-JSON from all their of projects right now. The latest LWN weekly edition has an article all about that (embargoed for subscribers; let me know privately if you need a one-off non-subscriber link or wait a couple of days). My own take would be that a) there's precedence (Hadoop) but b) for Hadoop the fix is in progress, so c) for a non-Hadoop-related piece of code, you should attempt to fix the issue and not introduce another evil-json-licensed code to our infrastructure. Our legal department would be the authoritative point of contact for this kind of thing, though :)
Comment Actions
Change 325797 had a related patch set uploaded (by Elukey):
[WIP] Yandex ClickHouse puppetization
Comment Actions
Thanks to the awesome upstream support the evil json license is gone in the last stable tag, and I was able to build the clickhouse-server deb package with the following:
DAEMONS=clickhouse-server APT_USE_BUILT=yes DIST=sid ARCH=amd64 USE_STATIC_LIBRARIES=0 GIT_PBUILDER_OUTPUT_DIR=/var/cache/pbuilder/result/sid-amd64 git-buildpackage --git-builder=git-pbuilder --git-upstream-branch=stable --git-upstream-tag=v1.1.54112-stable --git-ignore-new
And also the following changes:
diff --git a/contrib/libpoco/Net/CMakeLists.txt b/contrib/libpoco/Net/CMakeLists.txt index 2b79106..1eb3c74 100644 --- a/contrib/libpoco/Net/CMakeLists.txt +++ b/contrib/libpoco/Net/CMakeLists.txt @@ -17,7 +17,7 @@ if(WIN32) set(SYSLIBS ${SYSLIBS} "ws2_32.lib" "iphlpapi.lib") endif() elseif(NOT APPLE) - set(SYSLIBS ${SYSLIBS} "libanl.a") + set(SYSLIBS ${SYSLIBS} "libanl.so") endif(WIN32) add_library( "${LIBNAME}" ${LIB_MODE} ${SRCS} ) diff --git a/debian/control.in b/debian/control.in index e13ac5c..7fba836 100644 --- a/debian/control.in +++ b/debian/control.in @@ -1,6 +1,10 @@ Source: clickhouse Priority: optional Maintainer: Alexey Milovidov <milovidov@yandex-team.ru> -Build-Depends: debhelper (>= 5) -Standards-Version: 3.8.0 +Build-Depends: debhelper (>= 5), cmake, gcc-6, g++-6, libboost-dev, + libicu-dev, libreadline-dev, libmysqlclient-dev, unixodbc-dev, + libboost-thread-dev, libboost-program-options-dev, + libboost-system-dev, libboost-regex-dev, libboost-filesystem-dev, + libglib2.0-dev, libltdl-dev, libssl-dev +Standards-Version: 3.9.8 Section: libs
So next step:
- Follow up with upstream to close the last linking issue
- Upload the debian package to jessie-wikimedia
- Finalize the puppet code
Comment Actions
"Victory" should never be said out loud too soon:
./usr/bin/clickhouse-server: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.23' not found (required by ./usr/bin/clickhouse-server) linux-vdso.so.1 (0x00007ffee77fb000) libreadline.so.7 => not found libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f034f7a1000) libglib-2.0.so.0 => /lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x00007f034f491000) libicui18n.so.57 => not found libicuuc.so.57 => not found libicudata.so.57 => not found libmysqlclient.so.20 => not found libssl.so.1.1 => not found libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f034f274000) libcrypto.so.1.1 => not found libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f034f057000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f034ee4e000) libanl.so.1 => /lib/x86_64-linux-gnu/libanl.so.1 (0x00007f034ec4a000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f034ea46000) libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7 (0x00007f034e83b000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f034e53a000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f034e18f000) /lib64/ld-linux-x86-64.so.2 (0x000055cf6ba35000) libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f034df20000)
This is clearly far from what we want to be, namely a single binary depending only on libc (like the one running on druid100[123] atm in screen):
elukey@druid1001:~$ ldd /home/otto/clickhouse/base/usr/bin/clickhouse-server linux-vdso.so.1 (0x00007fff1858a000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fbe37563000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fbe3735b000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fbe37156000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fbe36e55000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fbe36aaa000) /lib64/ld-linux-x86-64.so.2 (0x0000555c469af000)
If I try to build with USE_STATIC_LIBRARIES=1 the following build error appears:
[100%] Building CXX object dbms/src/Server/CMakeFiles/clickhouse.dir/main.cpp.o [100%] Linking CXX executable clickhouse /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libreadline.a(readline.o): relocation R_X86_64_32 against symbol `rl_get_previous_history' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libreadline.a(vi_mode.o): relocation R_X86_64_32S against `.bss' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libreadline.a(funmap.o): relocation R_X86_64_32 against symbol `rl_abort' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libreadline.a(keymaps.o): relocation R_X86_64_32S against symbol `rl_insert' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libreadline.a(search.o): relocation R_X86_64_32S against symbol `rl_history_substr_search_backward' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libreadline.a(rltty.o): relocation R_X86_64_32S against symbol `rl_vi_unix_word_rubout' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libreadline.a(complete.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC [...]
This seems to be an issue in Cmakes trying to use a .a lib while building a -shared -fPIC library. As far as I know Debian follows the convention of not using relocable code for static libraries, and this does make sense to me. I have no idea how upstream is able to build successfully their debs on Ubuntu.
Comment Actions
It has been decided within the team to wait for upstream changes before proceeding further.
Comment Actions
Updates: upsteam released a fix for the linking issue. This solves the problem that I described above, but another issue remains, namely that building on Debian Sid adds loader/runtime dependency to GLIBC_2.23, meanwhile on Jessie we have 2.19. I naively thought that the GLIB_COMPATIBILITY build flag would have helped but it does not, so I'd need to come up with another trick :)
Comment Actions
Moving this to declined for the moment since there is no clear idea if clickhouse will be used or not in the future of the Analytics team. Will re-open if necessary :)