During Post-merge build e.g.
https://integration.wikimedia.org/ci/job/mwext-doxygen-publish/12245/console
17:39:45 + doxygen --version 17:39:45 1.8.17 17:39:45 + exec doxygen ... 17:39:46 Parsing file /src/repo/maintenance/rebuildTermsSearchKey.php... 17:39:46 Reading /src/repo/maintencorrupted size vs. prev_size 17:39:46 Build step 'Execute shell' marked build as failure
Looks like a glibc error.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • hashar | T253723 mwext-doxygen-publish Doxygen 1.8.17 fails: corrupted size vs. prev_size | |||
| Resolved | • hashar | T253793 Update Doxygen to 1.8.18 | |||
| Resolved | • hashar | T254465 mwext-doxygen-publish Doxygen 1.8.18 fails for Wikibase.git | |||
| Resolved | JMeybohm | T265579 Build and release Doxygen 1.8.19 to apt.wikimedia.org |
We have recently upgraded to Doxygen 1.8.17 T242155
From a quick glance at the build history, it seems to only happen for Wikibase so far.
Change 598980 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Rollback mwext-doxygen-publish to Doxygen 1.8.16
Change 598980 merged by jenkins-bot:
[integration/config@master] Rollback mwext-doxygen-publish to Doxygen 1.8.16
That works with Doxygen 1.8.16 from docker-registry.wikimedia.org/releng/doxygen:0.6.1-s1
But fails with Doxygen 1.8.17 from docker-registry.wikimedia.org/releng/doxygen:0.7.0
:-(
Repro:
$ cd mediawiki/extensions/Wikibase $ git rev-parse HEAD af206ee5d88f82a5dcfc2cd9b289eb3171e12210 $ docker run --rm -it --entrypoint=doxygen --workdir /src --user=$UID -v "$(pwd):/src" docker-registry.wikimedia.org/releng/doxygen:0.7.0 ... Parsing file /src/repo/WikibaseRepo.datatypes.php... corrupted size vs. prev_size
I went to some arbitrary commit from the past and it worked fine. With git bisect the issue comes from: 9c2cc04e28f8dffe0e230c2f3d2c1f01afbc46b2
* [BAD] e450192e11 - (HEAD -> master) Merge "WikibaseValueFormatterBuilders: inject specialized services" (8 days ago) <jenkins-bot> |\ | * [GOOD] 9c2cc04e28 - WikibaseValueFormatterBuilders: inject specialized services (13 days ago) <Jakob Warkotsch> * | [GOOD] 704ccc8368 - Merge "FP: Allow descriptions in ApiPrefetchingTermLookup" (8 days ago) <jenkins-bot>
If I delete repo/WikibaseRepo.datatypes.php it works again.
I then reset to master af206ee5d88f82a5dcfc2cd9b289eb3171e12210 , deleted repo/WikibaseRepo.datatypes.php and this time I get:
Parsing file /src/view/resources.php... double free or corruption (!prev)
The Doxygen package published on apt.wikimedia.org and used in docker-registry.wikimedia.org/releng/doxygen:0.7.0 is broken somehow. It crashes when generating documentation for mediawiki/extensions/Wikibase with errors such as:
| corrupted size vs. prev_size |
| double free or corruption (!prev) |
I have tried on my local Buster using the packages generated by CI for https://gerrit.wikimedia.org/r/#/c/operations/debs/doxygen/+/589416/ and stored at: https://integration.wikimedia.org/ci/job/debian-glue/1876/ and https://integration.wikimedia.org/ci/job/debian-glue/1908/ . They both work.
I went with the package published on apt.wikimedia.org https://apt.wikimedia.org/wikimedia/pool/component/ci/d/doxygen/doxygen_1.8.17-1~deb10%2Bwmf1_amd64.deb and it works fine on my local Buster machine.
Crafted a container with gdb added to it and with the package having the debug symbols
FROM docker-registry.wikimedia.org/releng/doxygen:0.7.0 USER root RUN apt update && apt -y install gdb wget RUN mkdir -p /tmp/doxygen-deb \ && cd /tmp/doxygen-deb \ && wget --quiet https://apt.wikimedia.org/wikimedia/pool/component/ci/d/doxygen/doxygen_1.8.17-1~deb10%2Bwmf1_amd64.deb \ && wget --quiet https://apt.wikimedia.org/wikimedia/pool/component/ci/d/doxygen/doxygen-dbgsym_1.8.17-1~deb10%2Bwmf1_amd64.deb \ && dpkg -i doxygen*.deb USER nobody
docker rmi doxygen docker build -t doxygen . docker run --rm -it --entrypoint=doxygen --workdir /src --user=$UID -v "$(pwd):/src" doxygen
Which results in the error:
Parsing file /src/repo/WikibaseRepo.datatypes.php... corrupted size vs. prev_size
And a core file :)
$ docker run --privileged --rm -it --entrypoint=gdb --workdir /src --user=$UID -v "$(pwd):/src" doxygen -- /usr/bin/doxygen core Reading symbols from /usr/bin/doxygen...Reading symbols from /usr/lib/debug/.build-id/42/6bc7fc13da2a4875a3d9eced38298c895ec951.debug...done. done. [New LWP 1] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `doxygen'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f9fc6ed3611 in abort () from /lib/x86_64-linux-gnu/libc.so.6 (gdb)
(gdb) bt
#0 0x00007f9fc6ed3611 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f9fc6f2a508 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f9fc6f30c1a in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007f9fc6f34e00 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007f9fc6f35d4f in realloc () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000561bd7b5d56f in QCString::LSData::resize (size=17, d=0x561bdbfc06c0)
at ./qtools/qcstring.h:398
#6 QCString::LSData::resize (size=17, d=0x561bdbfc06c0)
at ./qtools/qcstring.h:394
#7 QCString::StringRep::resize (this=0x561bdbfd73c0, newlen=17)
at ./qtools/qcstring.h:665
#8 0x0000561bd7b5d5f9 in QCString::resize (newlen=<optimized out>,
this=0x561bdbfd73c0) at ./qtools/qcstring.h:312
#9 QCString::operator+= (this=0x561bdbfd73c0, str=0x561bd8de3bd5 " ")
at ./qtools/qcstring.h:312
#10 0x0000561bd7e51fec in scannerYYlex (yyscanner=0x561bda040050)
at scanner.l:4135
#11 0x0000561bd7e699b4 in parseMain (filesInSameTranslationUnit=...,
sameTranslationUnit=<optimized out>,
rt=std::shared_ptr<Entry> (use count 2, weak count 0) = {...},
fileBuf=<optimized out>,
fileName=0x561bdbfbf728 "/src/repo/WikibaseRepo.datatypes.php",
yyscanner=0x561bda040050) at scanner.l:7242
--Type <RET> for more, q to quit, c to continue without paging--
#12 COutlineParser::parseInput (this=<optimized out>,
fileName=0x561bdbfbf728 "/src/repo/WikibaseRepo.datatypes.php",
fileBuf=<optimized out>,
root=std::shared_ptr<Entry> (use count 2, weak count 0) = {...},
sameTranslationUnit=<optimized out>, filesInSameTranslationUnit=...)
at scanner.l:7377
#13 0x0000561bd7b4111c in parseFile (parser=...,
root=std::shared_ptr<Entry> (use count 1, weak count 0) = {...},
fd=0x561bda3332e0, fn=<optimized out>, sameTu=<optimized out>,
filesInSameTu=...) at ./qtools/qcstring.h:593
#14 0x0000561bd7b44866 in parseFiles (
root=std::shared_ptr<Entry> (use count 1, weak count 0) = {...})
at ./qtools/qcstring.h:593
#15 0x0000561bd7b577aa in parseInput () at ./src/doxygen.cpp:11073
#16 0x0000561bd7b037f9 in main (argc=1, argv=0x7ffd64cfca88)
at ./src/main.cpp:37If I delete repo/WikibaseRepo.datatypes.php which yields and repeat the above. I get the error double free or corruption (!prev) and in gdb:
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `doxygen'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f62f7a93611 in abort () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007f62f7a93611 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f62f7aea508 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f62f7af0c1a in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007f62f7af273c in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x0000557f5302771c in __gnu_cxx::new_allocator<Argument>::deallocate (
this=0x557f5831a5f0, __p=<optimized out>)
at /usr/include/c++/8/ext/new_allocator.h:116
#5 std::allocator_traits<std::allocator<Argument> >::deallocate (__a=...,
__n=<optimized out>, __p=<optimized out>)
at /usr/include/c++/8/bits/alloc_traits.h:462
#6 std::_Vector_base<Argument, std::allocator<Argument> >::_M_deallocate (
this=0x557f5831a5f0, __n=<optimized out>, __p=<optimized out>)
at /usr/include/c++/8/bits/stl_vector.h:304
#7 std::_Vector_base<Argument, std::allocator<Argument> >::~_Vector_base (
this=0x557f5831a5f0, __in_chrg=<optimized out>)
at /usr/include/c++/8/bits/stl_vector.h:285
#8 std::vector<Argument, std::allocator<Argument> >::~vector (
this=0x557f5831a5f0, __in_chrg=<optimized out>)
at /usr/include/c++/8/bits/stl_vector.h:570
#9 ArgumentList::~ArgumentList (this=0x557f5831a5f0,
__in_chrg=<optimized out>) at ./src/arguments.h:59
#10 Entry::~Entry (this=0x557f5831a550, __in_chrg=<optimized out>)
at ./src/entry.cpp:123
#11 0x0000557f5332bd19 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x557f5831a540) at /usr/include/c++/8/bits/shared_ptr_base.h:148
#12 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (
this=0x557f5831a540) at /usr/include/c++/8/bits/shared_ptr_base.h:148
#13 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (
this=<optimized out>, __in_chrg=<optimized out>)
at /usr/include/c++/8/bits/shared_ptr_base.h:728
#14 std::__shared_ptr<Entry, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (
this=<optimized out>, __in_chrg=<optimized out>)
at /usr/include/c++/8/bits/shared_ptr_base.h:1167
#15 std::__shared_ptr<Entry, (__gnu_cxx::_Lock_policy)2>::operator= (__r=...,
this=<optimized out>) at /usr/include/c++/8/bits/shared_ptr_base.h:1263
#16 std::shared_ptr<Entry>::operator= (__r=..., this=<optimized out>)
at /usr/include/c++/8/bits/shared_ptr.h:335
#17 parseMain (filesInSameTranslationUnit=...,
sameTranslationUnit=<optimized out>,
rt=std::shared_ptr<Entry> (use count 2, weak count 0) = {...},
fileBuf=<optimized out>,
fileName=0x557f58357248 "/src/view/resources.php",
yyscanner=0x557f56393050) at scanner.l:7222
#18 COutlineParser::parseInput (this=<optimized out>,
fileName=0x557f58357248 "/src/view/resources.php",
fileBuf=<optimized out>,
--Type <RET> for more, q to quit, c to continue without paging--
root=std::shared_ptr<Entry> (use count 2, weak count 0) = {...},
sameTranslationUnit=<optimized out>, filesInSameTranslationUnit=...)
at scanner.l:7377
#19 0x0000557f5300311c in parseFile (parser=...,
root=std::shared_ptr<Entry> (use count 1, weak count 0) = {...},
fd=0x557f566993a0, fn=<optimized out>, sameTu=<optimized out>,
filesInSameTu=...) at ./qtools/qcstring.h:593
#20 0x0000557f53006866 in parseFiles (
root=std::shared_ptr<Entry> (use count 1, weak count 0) = {...})
at ./qtools/qcstring.h:593
#21 0x0000557f530197aa in parseInput () at ./src/doxygen.cpp:11073
#22 0x0000557f52fc57f9 in main (argc=1, argv=0x7ffe90110fe8)
at ./src/main.cpp:37None of that explains why it crashes in the Docker container while that works just fine on my local machine :-\
Doxygen 1.8.18 is out. I rebuild a package and I have confirmed the issue no more occurs inside the same Docker container. Guess I will fill a task / rebuild etc. Sorry for the annoyance.
Change 599094 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/debs/doxygen@debian/buster-wikimedia] Merge tag 'debian/1.8.18-1_exp1' into debian/buster-wikimedia
1.8.18 is out
Went there to check if you actually filed an upstream ticket. The word "crash" actually appears twice in the change log so maybe someone found it before us.
Thank you!
Had 1.8.18 failed as well, I would have gone into git bisecting doxygen 1.8.16 .. 1.8.17 to find the faulty patch and indeed would have filed a ticket to upstream for sure :] Luckily 1.8.18 seems to fix it which saves my a day or so of madness.
I haven't thought about comparing the release tags, thanks for the link! I went through https://www.doxygen.nl/manual/changelog.html but it is rather dense and the commit descriptions are not very meaningful unfortunately.
Our post merge builds are green ever since. Guess we can call this. T253793: Update Doxygen to 1.8.18 remains (though I'm not sure the way the ticket dependency expressed is how the phabricator makers envisioned things).
Thanks again.
That could be the subject of a nice debate: When do we close a task?
But indeed the issue for Wikibase got resolved by rolling back to Doxygen 1.8.16 and it seems it will work fine with 1.8.18.
Change 599094 merged by Dzahn:
[operations/debs/doxygen@debian/buster-wikimedia] Merge tag 'debian/1.8.18-1_exp1' into debian/buster-wikimedia
Change 602338 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] jjb: update to Doxygen 1.8.18
Change 602338 merged by jenkins-bot:
[integration/config@master] jjb: update to Doxygen 1.8.18