OpenStack designate uses memcached/mcrouter to coordinate between multiple worker nodes. We need to upgrade our OpenStack nodes to Bullseye in order to advance past version Victoria (which is nearly out of support).
I might be the first to need this but I won't be the last.
Gerrit isn't letting me run review as is. So here is a git diff:
diff --git a/build.sh b/build.sh index 9152b08..665cb33 100755 --- a/build.sh +++ b/build.sh @@ -1,2 +1,2 @@ #!/bin/sh -docker run --rm -it -v "${PWD}":/build docker-registry.wikimedia.org/wikimedia-buster:latest /build/docker_entry.sh +docker run --rm -it -v "${PWD}":/build docker-registry.wikimedia.org/bullseye:latest /build/docker_entry.sh diff --git a/docker_entry.sh b/docker_entry.sh index 31f58fa..78fa733 100755 --- a/docker_entry.sh +++ b/docker_entry.sh @@ -7,7 +7,7 @@ dir=/var/tmp/tmp.AJvPCDTbXG shared_dir="/build" pkg_dir="${dir}/pkgs" install_dir="${dir}/install" -mcrouter_version="v0.41.0-release" +mcrouter_version="v2022.01.31.00" export LDFLAGS="-L${install_dir}/lib -ldl -ljemalloc $LDFLAGS" export CPPFLAGS="-I${install_dir}/include $CPPFLAGS" @@ -16,11 +16,11 @@ apt-get update # could create an image with these preloaded apt-get install -y autoconf binutils-dev bison cmake flex g++ gcc git \ - libboost1.67-all-dev libbz2-dev libdouble-conversion-dev libevent-dev \ + libboost1.74-all-dev libbz2-dev libdouble-conversion-dev libevent-dev \ libgflags-dev libgoogle-glog-dev libjemalloc-dev liblz4-dev liblzma-dev \ liblzma5 libsnappy-dev libsodium-dev libssl-dev libtool libunwind8-dev \ libfmt-dev zlib1g-dev libzstd-dev make pkg-config python-dev python-six \ - dpkg-dev debhelper ragel ca-certificates build-essential + dpkg-dev debhelper ragel ca-certificates build-essential libgtest-dev function build_git { repo=$1 @@ -84,10 +84,6 @@ build_git https://github.com/facebookincubator/fizz \ build_git https://github.com/facebook/wangle \ "$(<${mcrouter_base}/WANGLE_COMMIT)" "-DBUILD_TESTS=OFF" "." "wangle/wangle" -build_git https://github.com/rsocket/rsocket-cpp \ - "" "-DBUILD_BENCHMARKS=OFF -DBUILD_EXAMPLES=OFF -DBUILD_TESTS=OFF" ".." \ - "rsocket-cpp/build" "-fPIC" - build_git https://github.com/facebook/fbthrift \ "$(<${mcrouter_base}/FBTHRIFT_COMMIT)" "" ".." "fbthrift/build" "-fPIC"
The main issue appeared to have been rsocket, it seemingly isn't needed for mcrouter (Do we need it?), though in order to install it an older version of folly needed to be installed, which caused many different problems. The above however still fails. If run with build.sh it will fail with:
[ 57%] Building CXX object thrift/lib/cpp2/CMakeFiles/thriftprotocol.dir/protocol/TableBasedSerializer.cpp.o
/var/tmp/tmp.AJvPCDTbXG/pkgs/fbthrift/thrift/lib/cpp2/protocol/TableBasedSerializer.cpp: In substitution of ‘template<class T> constexpr T apache::thrift::detail::identity(T) [with T = <missing>]’:
/var/tmp/tmp.AJvPCDTbXG/pkgs/fbthrift/thrift/lib/cpp2/protocol/TableBasedSerializer.cpp:37:1: required from here
/var/tmp/tmp.AJvPCDTbXG/pkgs/fbthrift/thrift/lib/cpp2/protocol/TableBasedSerializer.cpp:32:67: internal compiler error: Segmentation fault
32 | reinterpret_cast<VoidFuncPtr>(identity(set<Type, ThriftType>)), \
| ^
/var/tmp/tmp.AJvPCDTbXG/pkgs/fbthrift/thrift/lib/cpp2/protocol/TableBasedSerializer.cpp:37:1: note: in expansion of macro ‘THRIFT_DEFINE_PRIMITIVE_TYPE_TO_INFO’
37 | THRIFT_DEFINE_PRIMITIVE_TYPE_TO_INFO(
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-10/README.Bugs> for instructions.
make[2]: *** [thrift/lib/cpp2/CMakeFiles/thriftprotocol.dir/build.make:173: thrift/lib/cpp2/CMakeFiles/thriftprotocol.dir/protocol/TableBasedSerializer.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:896: thrift/lib/cpp2/CMakeFiles/thriftprotocol.dir/all] Error 2
make: *** [Makefile:171: all] Error 2however...
docker run -it docker-registry.wikimedia.org/bullseye:latest bash
then copying docker_entry.sh into / and running it will get down to dpkg-buildpackage before failing (as the volume isn't mounted so the changelog and company are not visible to it)
I've yet to identify why running inside the container manually gets all the way through the build.
I tried to build the buster version and I got a consistent (repeteable) error here:
[ 91%] Building CXX object CMakeFiles/ReactiveSocket.dir/rsocket/transports/tcp/TcpConnectionAcceptor.cpp.o
/var/tmp/tmp.AJvPCDTbXG/pkgs/rsocket-cpp/rsocket/transports/tcp/TcpConnectionAcceptor.cpp:46:8: error: ‘void rsocket::TcpConnectionAcceptor::SocketCallback::acceptError(folly::exception_wrapper)’ marked ‘override’, but does not override
void acceptError(folly::exception_wrapper ex) noexcept override {
^~~~~~~~~~~
/var/tmp/tmp.AJvPCDTbXG/pkgs/rsocket-cpp/rsocket/transports/tcp/TcpConnectionAcceptor.cpp:46:8: warning: by ‘void rsocket::TcpConnectionAcceptor::SocketCallback::acceptError(folly::exception_wrapper)’ [-Woverloaded-virtual]
In file included from /usr/include/c++/8/memory:80,
from /var/tmp/tmp.AJvPCDTbXG/install/include/folly/IPAddress.h:21,
from /var/tmp/tmp.AJvPCDTbXG/install/include/folly/SocketAddress.h:26,
from /var/tmp/tmp.AJvPCDTbXG/install/include/folly/io/async/AsyncServerSocket.h:19,
from /var/tmp/tmp.AJvPCDTbXG/pkgs/rsocket-cpp/rsocket/transports/tcp/TcpConnectionAcceptor.h:17,
from /var/tmp/tmp.AJvPCDTbXG/pkgs/rsocket-cpp/rsocket/transports/tcp/TcpConnectionAcceptor.cpp:15:
/usr/include/c++/8/bits/unique_ptr.h: In instantiation of ‘typename std::_MakeUniq<_Tp>::__single_object std::make_unique(_Args&& ...) [with _Tp = rsocket::TcpConnectionAcceptor::SocketCallback; _Args = {std::function<void(std::unique_ptr<rsocket::DuplexConnection, std::default_delete<rsocket::DuplexConnection> >, folly::EventBase&)>&}; typename std::_MakeUniq<_Tp>::__single_object = std::unique_ptr<rsocket::TcpConnectionAcceptor::SocketCallback>]’:
/var/tmp/tmp.AJvPCDTbXG/pkgs/rsocket-cpp/rsocket/transports/tcp/TcpConnectionAcceptor.cpp:83:68: required from here
/usr/include/c++/8/bits/unique_ptr.h:831:30: error: invalid new-expression of abstract class type ‘rsocket::TcpConnectionAcceptor::SocketCallback’
{ return unique_ptr<_Tp>(new _Tp(std::forward<_Args>(__args)...)); }
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/var/tmp/tmp.AJvPCDTbXG/pkgs/rsocket-cpp/rsocket/transports/tcp/TcpConnectionAcceptor.cpp:26:30: note: because the following virtual functions are pure within ‘rsocket::TcpConnectionAcceptor::SocketCallback’:
class TcpConnectionAcceptor::SocketCallback
^~~~~~~~~~~~~~
In file included from /var/tmp/tmp.AJvPCDTbXG/pkgs/rsocket-cpp/rsocket/transports/tcp/TcpConnectionAcceptor.h:17,
from /var/tmp/tmp.AJvPCDTbXG/pkgs/rsocket-cpp/rsocket/transports/tcp/TcpConnectionAcceptor.cpp:15:
/var/tmp/tmp.AJvPCDTbXG/install/include/folly/io/async/AsyncServerSocket.h:174:18: note: ‘virtual void folly::AsyncServerSocket::AcceptCallback::acceptError(const std::exception&)’
virtual void acceptError(const std::exception& ex) noexcept = 0;
^~~~~~~~~~~
make[2]: *** [CMakeFiles/ReactiveSocket.dir/build.make:661: CMakeFiles/ReactiveSocket.dir/rsocket/transports/tcp/TcpConnectionAcceptor.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:937: CMakeFiles/ReactiveSocket.dir/all] Error 2
make: *** [Makefile:163: all] Error 2Also in the rsocket step.
Per https://github.com/facebook/mcrouter/blob/main/README.md#installing-from-source Mcrouter depends on folly, wangle, fizz, and fbthrift.
Building with the above patch while mounting the directory, then copying docker_entry.sh to / and running there, does result in me getting a deb (after modifying the changelog and rerunning dpkg-buildpackage -us -uc) Though running with ./build.sh still fails on thrifty.
Change 759539 had a related patch set uploaded (by Michael DiPietro; author: Michael DiPietro):
[operations/debs/mcrouter@master] mcrouter for bullseye. This does some unexpected things, in particular build.sh doesn't seem to work, errors out with a seg fault. README updated with instructions on how to make it work.
Change 759522 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[operations/debs/mcrouter@master] mcrouter: introduce updates for bullseye build
Change 759522 merged by Arturo Borrero Gonzalez:
[operations/debs/mcrouter@master] mcrouter: introduce updates for Debian Bullseye
Change 759539 abandoned by Michael DiPietro:
[operations/debs/mcrouter@master] mcrouter for bullseye. This does some unexpected things, in particular build.sh doesn't seem to work, errors out with a seg fault. README updated with instructions on how to make it work.
Reason:
759522 solves the same
Mentioned in SAL (#wikimedia-operations) [2022-02-04T17:00:26Z] <arturo> add mcrouter 2022.01.31.00-1 to bullseye-wikimedia (T300578)
Done!
aborrero@cloudservices2003-dev:~ $ apt-cache policy mcrouter mcrouter: Installed: 2022.01.31.00-1 Candidate: 2022.01.31.00-1 Version table: *** 2022.01.31.00-1 1001 1001 http://apt.wikimedia.org/wikimedia bullseye-wikimedia/main amd64 Packages 100 /var/lib/dpkg/status