Page MenuHomePhabricator

build/package mcrouter for Debian Bullseye
Closed, ResolvedPublic


OpenStack designate uses memcached/mcrouter to coordinate between multiple worker nodes. We need to upgrade our OpenStack nodes to Bullseye in order to advance past version Victoria (which is nearly out of support).

I might be the first to need this but I won't be the last.

Event Timeline

Gerrit isn't letting me run review as is. So here is a git diff:

diff --git a/ b/
index 9152b08..665cb33 100755
--- a/
+++ b/
@@ -1,2 +1,2 @@
-docker run --rm -it -v "${PWD}":/build /build/
+docker run --rm -it -v "${PWD}":/build /build/
diff --git a/ b/
index 31f58fa..78fa733 100755
--- a/
+++ b/
@@ -7,7 +7,7 @@ dir=/var/tmp/tmp.AJvPCDTbXG
 export LDFLAGS="-L${install_dir}/lib -ldl -ljemalloc $LDFLAGS"
 export CPPFLAGS="-I${install_dir}/include $CPPFLAGS"
@@ -16,11 +16,11 @@ apt-get update
 # could create an image with these preloaded
 apt-get install -y autoconf binutils-dev bison cmake flex g++ gcc git \
-  libboost1.67-all-dev libbz2-dev libdouble-conversion-dev libevent-dev \
+  libboost1.74-all-dev libbz2-dev libdouble-conversion-dev libevent-dev \
   libgflags-dev libgoogle-glog-dev libjemalloc-dev liblz4-dev liblzma-dev \
   liblzma5 libsnappy-dev libsodium-dev libssl-dev libtool libunwind8-dev \
   libfmt-dev zlib1g-dev libzstd-dev make pkg-config python-dev python-six \
-  dpkg-dev debhelper ragel ca-certificates build-essential
+  dpkg-dev debhelper ragel ca-certificates build-essential libgtest-dev
 function build_git {
@@ -84,10 +84,6 @@ build_git \
 build_git \
   "$(<${mcrouter_base}/WANGLE_COMMIT)" "-DBUILD_TESTS=OFF" "." "wangle/wangle"
-build_git \
-  "rsocket-cpp/build" "-fPIC"
 build_git \
   "$(<${mcrouter_base}/FBTHRIFT_COMMIT)" "" ".."  "fbthrift/build" "-fPIC"

The main issue appeared to have been rsocket, it seemingly isn't needed for mcrouter (Do we need it?), though in order to install it an older version of folly needed to be installed, which caused many different problems. The above however still fails. If run with it will fail with:

[ 57%] Building CXX object thrift/lib/cpp2/CMakeFiles/thriftprotocol.dir/protocol/TableBasedSerializer.cpp.o
/var/tmp/tmp.AJvPCDTbXG/pkgs/fbthrift/thrift/lib/cpp2/protocol/TableBasedSerializer.cpp: In substitution of ‘template<class T> constexpr T apache::thrift::detail::identity(T) [with T = <missing>]’:
/var/tmp/tmp.AJvPCDTbXG/pkgs/fbthrift/thrift/lib/cpp2/protocol/TableBasedSerializer.cpp:37:1:   required from here
/var/tmp/tmp.AJvPCDTbXG/pkgs/fbthrift/thrift/lib/cpp2/protocol/TableBasedSerializer.cpp:32:67: internal compiler error: Segmentation fault
   32 |       reinterpret_cast<VoidFuncPtr>(identity(set<Type, ThriftType>)),  \
      |                                                                   ^
/var/tmp/tmp.AJvPCDTbXG/pkgs/fbthrift/thrift/lib/cpp2/protocol/TableBasedSerializer.cpp:37:1: note: in expansion of macro ‘THRIFT_DEFINE_PRIMITIVE_TYPE_TO_INFO’
      | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-10/README.Bugs> for instructions.
make[2]: *** [thrift/lib/cpp2/CMakeFiles/thriftprotocol.dir/build.make:173: thrift/lib/cpp2/CMakeFiles/thriftprotocol.dir/protocol/TableBasedSerializer.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:896: thrift/lib/cpp2/CMakeFiles/thriftprotocol.dir/all] Error 2
make: *** [Makefile:171: all] Error 2

docker run -it bash
then copying into / and running it will get down to dpkg-buildpackage before failing (as the volume isn't mounted so the changelog and company are not visible to it)
I've yet to identify why running inside the container manually gets all the way through the build.

I tried to build the buster version and I got a consistent (repeteable) error here:

[ 91%] Building CXX object CMakeFiles/ReactiveSocket.dir/rsocket/transports/tcp/TcpConnectionAcceptor.cpp.o
/var/tmp/tmp.AJvPCDTbXG/pkgs/rsocket-cpp/rsocket/transports/tcp/TcpConnectionAcceptor.cpp:46:8: error: ‘void rsocket::TcpConnectionAcceptor::SocketCallback::acceptError(folly::exception_wrapper)’ marked ‘override’, but does not override
   void acceptError(folly::exception_wrapper ex) noexcept override {
/var/tmp/tmp.AJvPCDTbXG/pkgs/rsocket-cpp/rsocket/transports/tcp/TcpConnectionAcceptor.cpp:46:8: warning:   by ‘void rsocket::TcpConnectionAcceptor::SocketCallback::acceptError(folly::exception_wrapper)’ [-Woverloaded-virtual]
In file included from /usr/include/c++/8/memory:80,
                 from /var/tmp/tmp.AJvPCDTbXG/install/include/folly/IPAddress.h:21,
                 from /var/tmp/tmp.AJvPCDTbXG/install/include/folly/SocketAddress.h:26,
                 from /var/tmp/tmp.AJvPCDTbXG/install/include/folly/io/async/AsyncServerSocket.h:19,
                 from /var/tmp/tmp.AJvPCDTbXG/pkgs/rsocket-cpp/rsocket/transports/tcp/TcpConnectionAcceptor.h:17,
                 from /var/tmp/tmp.AJvPCDTbXG/pkgs/rsocket-cpp/rsocket/transports/tcp/TcpConnectionAcceptor.cpp:15:
/usr/include/c++/8/bits/unique_ptr.h: In instantiation of ‘typename std::_MakeUniq<_Tp>::__single_object std::make_unique(_Args&& ...) [with _Tp = rsocket::TcpConnectionAcceptor::SocketCallback; _Args = {std::function<void(std::unique_ptr<rsocket::DuplexConnection, std::default_delete<rsocket::DuplexConnection> >, folly::EventBase&)>&}; typename std::_MakeUniq<_Tp>::__single_object = std::unique_ptr<rsocket::TcpConnectionAcceptor::SocketCallback>]’:
/var/tmp/tmp.AJvPCDTbXG/pkgs/rsocket-cpp/rsocket/transports/tcp/TcpConnectionAcceptor.cpp:83:68:   required from here
/usr/include/c++/8/bits/unique_ptr.h:831:30: error: invalid new-expression of abstract class type ‘rsocket::TcpConnectionAcceptor::SocketCallback’
     { return unique_ptr<_Tp>(new _Tp(std::forward<_Args>(__args)...)); }
/var/tmp/tmp.AJvPCDTbXG/pkgs/rsocket-cpp/rsocket/transports/tcp/TcpConnectionAcceptor.cpp:26:30: note:   because the following virtual functions are pure within ‘rsocket::TcpConnectionAcceptor::SocketCallback’:
 class TcpConnectionAcceptor::SocketCallback
In file included from /var/tmp/tmp.AJvPCDTbXG/pkgs/rsocket-cpp/rsocket/transports/tcp/TcpConnectionAcceptor.h:17,
                 from /var/tmp/tmp.AJvPCDTbXG/pkgs/rsocket-cpp/rsocket/transports/tcp/TcpConnectionAcceptor.cpp:15:
/var/tmp/tmp.AJvPCDTbXG/install/include/folly/io/async/AsyncServerSocket.h:174:18: note: 	‘virtual void folly::AsyncServerSocket::AcceptCallback::acceptError(const std::exception&)’
     virtual void acceptError(const std::exception& ex) noexcept = 0;
make[2]: *** [CMakeFiles/ReactiveSocket.dir/build.make:661: CMakeFiles/ReactiveSocket.dir/rsocket/transports/tcp/TcpConnectionAcceptor.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:937: CMakeFiles/ReactiveSocket.dir/all] Error 2
make: *** [Makefile:163: all] Error 2

Also in the rsocket step.

Per Mcrouter depends on folly, wangle, fizz, and fbthrift.

Building with the above patch while mounting the directory, then copying to / and running there, does result in me getting a deb (after modifying the changelog and rerunning dpkg-buildpackage -us -uc) Though running with ./ still fails on thrifty.

Change 759539 had a related patch set uploaded (by Michael DiPietro; author: Michael DiPietro):

[operations/debs/mcrouter@master] mcrouter for bullseye. This does some unexpected things, in particular doesn't seem to work, errors out with a seg fault. README updated with instructions on how to make it work.

Change 759522 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/debs/mcrouter@master] mcrouter: introduce updates for bullseye build

Change 759522 merged by Arturo Borrero Gonzalez:

[operations/debs/mcrouter@master] mcrouter: introduce updates for Debian Bullseye

Change 759539 abandoned by Michael DiPietro:

[operations/debs/mcrouter@master] mcrouter for bullseye. This does some unexpected things, in particular doesn't seem to work, errors out with a seg fault. README updated with instructions on how to make it work.


759522 solves the same

Mentioned in SAL (#wikimedia-operations) [2022-02-04T17:00:26Z] <arturo> add mcrouter 2022.01.31.00-1 to bullseye-wikimedia (T300578)

aborrero assigned this task to rook.


aborrero@cloudservices2003-dev:~ $ apt-cache policy mcrouter
  Installed: 2022.01.31.00-1
  Candidate: 2022.01.31.00-1
  Version table:
 *** 2022.01.31.00-1 1001
       1001 bullseye-wikimedia/main amd64 Packages
        100 /var/lib/dpkg/status