Page MenuHomePhabricator

memkeys segfaults on Debian Stretch
Closed, ResolvedPublic0 Story Points

Description

I have rebuilt memkeys for stretch-wikimedia, uploaded it to reprepro and tried to use it on mw1238 to sniff some memcached traffic. The result was the following segfault in Thread 2 (got the stacktrace using gdb):

(gdb) thread apply all bt

Thread 7 (Thread 0x7fffe77fe700 (LWP 46005)):
#0  0x00007ffff73337fd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
#1  0x000055555556fcba in ?? ()
#2  0x00007ffff705ae6f in std::execute_native_thread_routine (__p=0x555555797f80) at ../../../../../src/libstdc++-v3/src/c++11/thread.cc:83
#3  0x00007ffff732a4a4 in start_thread (arg=0x7fffe77fe700) at pthread_create.c:456
#4  0x00007ffff67cfd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 6 (Thread 0x7fffe7fff700 (LWP 46004)):
#0  0x00007ffff679f28d in nanosleep () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007ffff679f1da in __sleep (seconds=0) at ../sysdeps/posix/sleep.c:55
#2  0x0000555555570322 in ?? ()
#3  0x00007ffff705ae6f in std::execute_native_thread_routine (__p=0x555555797f50) at ../../../../../src/libstdc++-v3/src/c++11/thread.cc:83
#4  0x00007ffff732a4a4 in start_thread (arg=0x7fffe7fff700) at pthread_create.c:456
#5  0x00007ffff67cfd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 5 (Thread 0x7ffff4a6c700 (LWP 46003)):
#0  0x00007ffff73337fd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
#1  0x0000555555579754 in ?? ()
#2  0x00007ffff705ae6f in std::execute_native_thread_routine (__p=0x555555798410) at ../../../../../src/libstdc++-v3/src/c++11/thread.cc:83
#3  0x00007ffff732a4a4 in start_thread (arg=0x7ffff4a6c700) at pthread_create.c:456
#4  0x00007ffff67cfd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 4 (Thread 0x7ffff526d700 (LWP 46002)):
#0  0x00007ffff73337fd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
#1  0x0000555555579754 in ?? ()
#2  0x00007ffff705ae6f in std::execute_native_thread_routine (__p=0x555555797ed0) at ../../../../../src/libstdc++-v3/src/c++11/thread.cc:83
#3  0x00007ffff732a4a4 in start_thread (arg=0x7ffff526d700) at pthread_create.c:456
#4  0x00007ffff67cfd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 3 (Thread 0x7ffff5a6e700 (LWP 46001)):
#0  0x00007ffff73337fd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
#1  0x0000555555579754 in ?? ()
#2  0x00007ffff705ae6f in std::execute_native_thread_routine (__p=0x555555798370) at ../../../../../src/libstdc++-v3/src/c++11/thread.cc:83
#3  0x00007ffff732a4a4 in start_thread (arg=0x7ffff5a6e700) at pthread_create.c:456
#4  0x00007ffff67cfd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 2 (Thread 0x7ffff626f700 (LWP 46000)):
#0  __memmove_sse2_unaligned_erms () at ../sysdeps/x86_64/multiarch/../multiarch/memmove-vec-unaligned-erms.S:242
#1  0x00007ffff70bd52b in std::char_traits<char>::copy (__n=65535, __s2=0x5555557a761c "", __s1=<optimized out>)
    at /build/gcc-6-6.3.0/build/x86_64-linux-gnu/libstdc++-v3/include/bits/char_traits.h:290
#2  std::basic_streambuf<char, std::char_traits<char> >::xsputn (this=0x7ffff626e948, __s=0x5555557a761c "", __n=140737331596120)
    at /build/gcc-6-6.3.0/build/x86_64-linux-gnu/libstdc++-v3/include/bits/streambuf.tcc:90
#3  0x00007ffff70ae58e in std::basic_streambuf<char, std::char_traits<char> >::sputn (__n=140737331596120, __s=0x555555797680 "0uyUUU",
    this=<optimized out>) at /build/gcc-6-6.3.0/build/x86_64-linux-gnu/libstdc++-v3/include/streambuf:451
#4  std::__ostream_write<char, std::char_traits<char> > (__n=140737331596120, __s=0x555555797680 "0uyUUU", __out=...)
    at /build/gcc-6-6.3.0/build/x86_64-linux-gnu/libstdc++-v3/include/bits/ostream_insert.h:50
#5  std::__ostream_insert<char, std::char_traits<char> > (__out=..., __s=0x555555797680 "0uyUUU", __n=140737331596120)
    at /build/gcc-6-6.3.0/build/x86_64-linux-gnu/libstdc++-v3/include/bits/ostream_insert.h:101
#6  0x0000555555563a6e in ?? ()
#7  0x0000555555566fbe in ?? ()
#8  0x00007ffff705ae6f in std::execute_native_thread_routine (__p=0x5555557971b0) at ../../../../../src/libstdc++-v3/src/c++11/thread.cc:83
#9  0x00007ffff732a4a4 in start_thread (arg=0x7ffff626f700) at pthread_create.c:456
#10 0x00007ffff67cfd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 1 (Thread 0x7ffff7fe3740 (LWP 45988)):
#0  0x00007ffff732b6dd in pthread_join (threadid=140737323136768, thread_return=0x0) at pthread_join.c:90
#1  0x00007ffff705b0f7 in __gthread_join (__value_ptr=0x0, __threadid=<optimized out>)
    at /build/gcc-6-6.3.0/build/x86_64-linux-gnu/libstdc++-v3/include/x86_64-linux-gnu/bits/gthr-default.h:668
#2  std::thread::join (this=0x5555557985d0) at ../../../../../src/libstdc++-v3/src/c++11/thread.cc:136
#3  0x0000555555578221 in ?? ()
#4  0x0000555555566429 in ?? ()
#5  0x0000555555578eee in ?? ()

Details

Related Gerrit Patches:

Event Timeline

elukey created this task.May 20 2019, 6:36 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 20 2019, 6:36 AM
elukey added a comment.EditedMay 20 2019, 6:50 AM
elukey@mc1019:~$ dpkg -l | grep stdc
ii  libstdc++6:amd64                     4.9.2-10+deb8u1                  amd64        GNU Standard C++ Library v3

elukey@mw1238:~$ dpkg -l | grep stdc
ii  libstdc++6:amd64                     6.3.0-18+deb9u1                                             amd64        GNU Standard C++ Library v3
ii  libstdc++6-6-dbg:amd64               6.3.0-18+deb9u1                                             amd64        GNU Standard C++ Library v3 (debugging files)

Maybe memkeys needs to be adjusted to use the 6.x std library? Or I have built the package in the wrong way:

GIT_PBUILDER_AUTOCONF=no DIST=stretch WIKIMEDIA=yes gbp buildpackage -sa -us -uc --git-builder=git-pbuilder --git-debian-branch=debian --git-upstream-branch=master
elukey moved this task from Backlog to Mcrouter/Memcached on the User-Elukey board.

Created a pull request https://github.com/bmatheny/memkeys/pull/26/commits

From my tests it fixes the issue, but I am not sure how reactive upstream is in these days. Since there is currently no rush to get this package built for stretch/buster I'll wait to get feedback from upstream. If this doesn't happen, we'll just add a patch in the debian package.

elukey triaged this task as Normal priority.Jul 18 2019, 12:26 PM

Change 542992 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/debs/memkeys@debian] Add upstream patch to avoid segfaults on Debian Stretch

https://gerrit.wikimedia.org/r/542992

Change 542992 merged by Elukey:
[operations/debs/memkeys@debian] Add upstream patch to avoid segfaults on Debian Stretch

https://gerrit.wikimedia.org/r/542992

root@install1002:/srv/wikimedia# reprepro lsbycomponent memkeys
memkeys |         20181031-1 |  jessie-wikimedia | thirdparty | amd64, source
memkeys |  20181031-2+deb9u1 | stretch-wikimedia |       main | amd64, source
memkeys | 20181031-2+deb10u1 |  buster-wikimedia |       main | amd64, source

Mentioned in SAL (#wikimedia-operations) [2019-10-15T08:06:18Z] <elukey> upload new version of memkeys (adding a patch to merged to upstream to avoid segfaults on stretch/buster) to stretch|buster wikimedia apt repos - T223863

elukey closed this task as Resolved.Oct 15 2019, 8:06 AM