Page MenuHomePhabricator
Create Task
Maniphest T223863

memkeys segfaults on Debian Stretch
Closed, ResolvedPublic0 Estimated Story Points
Actions

Description

I have rebuilt memkeys for stretch-wikimedia, uploaded it to reprepro and tried to use it on mw1238 to sniff some memcached traffic. The result was the following segfault in Thread 2 (got the stacktrace using gdb):

(gdb) thread apply all bt

Thread 7 (Thread 0x7fffe77fe700 (LWP 46005)):
#0  0x00007ffff73337fd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
#1  0x000055555556fcba in ?? ()
#2  0x00007ffff705ae6f in std::execute_native_thread_routine (__p=0x555555797f80) at ../../../../../src/libstdc++-v3/src/c++11/thread.cc:83
#3  0x00007ffff732a4a4 in start_thread (arg=0x7fffe77fe700) at pthread_create.c:456
#4  0x00007ffff67cfd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 6 (Thread 0x7fffe7fff700 (LWP 46004)):
#0  0x00007ffff679f28d in nanosleep () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007ffff679f1da in __sleep (seconds=0) at ../sysdeps/posix/sleep.c:55
#2  0x0000555555570322 in ?? ()
#3  0x00007ffff705ae6f in std::execute_native_thread_routine (__p=0x555555797f50) at ../../../../../src/libstdc++-v3/src/c++11/thread.cc:83
#4  0x00007ffff732a4a4 in start_thread (arg=0x7fffe7fff700) at pthread_create.c:456
#5  0x00007ffff67cfd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 5 (Thread 0x7ffff4a6c700 (LWP 46003)):
#0  0x00007ffff73337fd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
#1  0x0000555555579754 in ?? ()
#2  0x00007ffff705ae6f in std::execute_native_thread_routine (__p=0x555555798410) at ../../../../../src/libstdc++-v3/src/c++11/thread.cc:83
#3  0x00007ffff732a4a4 in start_thread (arg=0x7ffff4a6c700) at pthread_create.c:456
#4  0x00007ffff67cfd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 4 (Thread 0x7ffff526d700 (LWP 46002)):
#0  0x00007ffff73337fd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
#1  0x0000555555579754 in ?? ()
#2  0x00007ffff705ae6f in std::execute_native_thread_routine (__p=0x555555797ed0) at ../../../../../src/libstdc++-v3/src/c++11/thread.cc:83
#3  0x00007ffff732a4a4 in start_thread (arg=0x7ffff526d700) at pthread_create.c:456
#4  0x00007ffff67cfd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 3 (Thread 0x7ffff5a6e700 (LWP 46001)):
#0  0x00007ffff73337fd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
#1  0x0000555555579754 in ?? ()
#2  0x00007ffff705ae6f in std::execute_native_thread_routine (__p=0x555555798370) at ../../../../../src/libstdc++-v3/src/c++11/thread.cc:83
#3  0x00007ffff732a4a4 in start_thread (arg=0x7ffff5a6e700) at pthread_create.c:456
#4  0x00007ffff67cfd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 2 (Thread 0x7ffff626f700 (LWP 46000)):
#0  __memmove_sse2_unaligned_erms () at ../sysdeps/x86_64/multiarch/../multiarch/memmove-vec-unaligned-erms.S:242
#1  0x00007ffff70bd52b in std::char_traits<char>::copy (__n=65535, __s2=0x5555557a761c "", __s1=<optimized out>)
    at /build/gcc-6-6.3.0/build/x86_64-linux-gnu/libstdc++-v3/include/bits/char_traits.h:290
#2  std::basic_streambuf<char, std::char_traits<char> >::xsputn (this=0x7ffff626e948, __s=0x5555557a761c "", __n=140737331596120)
    at /build/gcc-6-6.3.0/build/x86_64-linux-gnu/libstdc++-v3/include/bits/streambuf.tcc:90
#3  0x00007ffff70ae58e in std::basic_streambuf<char, std::char_traits<char> >::sputn (__n=140737331596120, __s=0x555555797680 "0uyUUU",
    this=<optimized out>) at /build/gcc-6-6.3.0/build/x86_64-linux-gnu/libstdc++-v3/include/streambuf:451
#4  std::__ostream_write<char, std::char_traits<char> > (__n=140737331596120, __s=0x555555797680 "0uyUUU", __out=...)
    at /build/gcc-6-6.3.0/build/x86_64-linux-gnu/libstdc++-v3/include/bits/ostream_insert.h:50
#5  std::__ostream_insert<char, std::char_traits<char> > (__out=..., __s=0x555555797680 "0uyUUU", __n=140737331596120)
    at /build/gcc-6-6.3.0/build/x86_64-linux-gnu/libstdc++-v3/include/bits/ostream_insert.h:101
#6  0x0000555555563a6e in ?? ()
#7  0x0000555555566fbe in ?? ()
#8  0x00007ffff705ae6f in std::execute_native_thread_routine (__p=0x5555557971b0) at ../../../../../src/libstdc++-v3/src/c++11/thread.cc:83
#9  0x00007ffff732a4a4 in start_thread (arg=0x7ffff626f700) at pthread_create.c:456
#10 0x00007ffff67cfd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 1 (Thread 0x7ffff7fe3740 (LWP 45988)):
#0  0x00007ffff732b6dd in pthread_join (threadid=140737323136768, thread_return=0x0) at pthread_join.c:90
#1  0x00007ffff705b0f7 in __gthread_join (__value_ptr=0x0, __threadid=<optimized out>)
    at /build/gcc-6-6.3.0/build/x86_64-linux-gnu/libstdc++-v3/include/x86_64-linux-gnu/bits/gthr-default.h:668
#2  std::thread::join (this=0x5555557985d0) at ../../../../../src/libstdc++-v3/src/c++11/thread.cc:136
#3  0x0000555555578221 in ?? ()
#4  0x0000555555566429 in ?? ()
#5  0x0000555555578eee in ?? ()

Details

SubjectRepoBranchLines +/-
Add upstream patch to avoid segfaults on Debian Stretchoperations/debs/memkeysdebian+35 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

elukey created this task.May 20 2019, 6:36 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 20 2019, 6:36 AM
elukey added a comment.EditedMay 20 2019, 6:50 AM
elukey@mc1019:~$ dpkg -l | grep stdc
ii  libstdc++6:amd64                     4.9.2-10+deb8u1                  amd64        GNU Standard C++ Library v3

elukey@mw1238:~$ dpkg -l | grep stdc
ii  libstdc++6:amd64                     6.3.0-18+deb9u1                                             amd64        GNU Standard C++ Library v3
ii  libstdc++6-6-dbg:amd64               6.3.0-18+deb9u1                                             amd64        GNU Standard C++ Library v3 (debugging files)

Maybe memkeys needs to be adjusted to use the 6.x std library? Or I have built the package in the wrong way:

GIT_PBUILDER_AUTOCONF=no DIST=stretch WIKIMEDIA=yes gbp buildpackage -sa -us -uc --git-builder=git-pbuilder --git-debian-branch=debian --git-upstream-branch=master
elukey mentioned this in T223647: Investigate increase in GET ops registered by mcrouter for the mediawiki appserver cluster.May 20 2019, 7:41 AM
elukey added a project: User-Elukey.Jul 6 2019, 1:50 PM
elukey moved this task from Backlog to Mcrouter/Memcached on the User-Elukey board.
elukey added a parent task: T213089: Upgrade memcached cluster to Debian Buster.Jul 12 2019, 2:48 PM
elukey added a comment.Jul 15 2019, 8:51 AM

Opened https://github.com/bmatheny/memkeys/issues/25

elukey added a comment.Jul 18 2019, 12:26 PM

Created a pull request https://github.com/bmatheny/memkeys/pull/26/commits

From my tests it fixes the issue, but I am not sure how reactive upstream is in these days. Since there is currently no rush to get this package built for stretch/buster I'll wait to get feedback from upstream. If this doesn't happen, we'll just add a patch in the debian package.

elukey triaged this task as Medium priority.Jul 18 2019, 12:26 PM
gerritbot added a comment.Oct 14 2019, 4:45 PM

Change 542992 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/debs/memkeys@debian] Add upstream patch to avoid segfaults on Debian Stretch

https://gerrit.wikimedia.org/r/542992

gerritbot added a project: Patch-For-Review.Oct 14 2019, 4:45 PM
gerritbot added a comment.Oct 15 2019, 7:00 AM

Change 542992 merged by Elukey:
[operations/debs/memkeys@debian] Add upstream patch to avoid segfaults on Debian Stretch

https://gerrit.wikimedia.org/r/542992

elukey added a comment.Oct 15 2019, 8:05 AM
root@install1002:/srv/wikimedia# reprepro lsbycomponent memkeys
memkeys |         20181031-1 |  jessie-wikimedia | thirdparty | amd64, source
memkeys |  20181031-2+deb9u1 | stretch-wikimedia |       main | amd64, source
memkeys | 20181031-2+deb10u1 |  buster-wikimedia |       main | amd64, source
Stashbot added a comment.Oct 15 2019, 8:06 AM

Mentioned in SAL (#wikimedia-operations) [2019-10-15T08:06:18Z] <elukey> upload new version of memkeys (adding a patch to merged to upstream to avoid segfaults on stretch/buster) to stretch|buster wikimedia apt repos - T223863

elukey closed this task as Resolved.Oct 15 2019, 8:06 AM
jijiki added a parent task: T244852: Upgrade and improve our application object caching service (memcached).Feb 11 2020, 12:54 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits