Page MenuHomePhabricator

mwext-doxygen-publish Doxygen 1.8.17 fails: corrupted size vs. prev_size
Closed, ResolvedPublic

Description

During Post-merge build e.g.
https://integration.wikimedia.org/ci/job/mwext-doxygen-publish/12245/console

17:39:45 + doxygen --version
17:39:45 1.8.17
17:39:45 + exec doxygen
...
17:39:46 Parsing file /src/repo/maintenance/rebuildTermsSearchKey.php...
17:39:46 Reading /src/repo/maintencorrupted size vs. prev_size
17:39:46 Build step 'Execute shell' marked build as failure

Looks like a glibc error.

Event Timeline

We have recently upgraded to Doxygen 1.8.17 T242155

From a quick glance at the build history, it seems to only happen for Wikibase so far.

Change 598980 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Rollback mwext-doxygen-publish to Doxygen 1.8.16

https://gerrit.wikimedia.org/r/598980

Change 598980 merged by jenkins-bot:
[integration/config@master] Rollback mwext-doxygen-publish to Doxygen 1.8.16

https://gerrit.wikimedia.org/r/598980

That works with Doxygen 1.8.16 from docker-registry.wikimedia.org/releng/doxygen:0.6.1-s1

But fails with Doxygen 1.8.17 from docker-registry.wikimedia.org/releng/doxygen:0.7.0

:-(

Repro:

$ cd mediawiki/extensions/Wikibase
$ git rev-parse HEAD
af206ee5d88f82a5dcfc2cd9b289eb3171e12210
$ docker run --rm -it --entrypoint=doxygen --workdir /src --user=$UID -v "$(pwd):/src" docker-registry.wikimedia.org/releng/doxygen:0.7.0
...
Parsing file /src/repo/WikibaseRepo.datatypes.php...
corrupted size vs. prev_size

I went to some arbitrary commit from the past and it worked fine. With git bisect the issue comes from: 9c2cc04e28f8dffe0e230c2f3d2c1f01afbc46b2

*   [BAD] e450192e11 - (HEAD -> master) Merge "WikibaseValueFormatterBuilders: inject specialized services" (8 days ago) <jenkins-bot>
|\  
| * [GOOD] 9c2cc04e28 - WikibaseValueFormatterBuilders: inject specialized services (13 days ago) <Jakob Warkotsch>
* |   [GOOD] 704ccc8368 - Merge "FP: Allow descriptions in ApiPrefetchingTermLookup" (8 days ago) <jenkins-bot>

If I delete repo/WikibaseRepo.datatypes.php it works again.

I then reset to master af206ee5d88f82a5dcfc2cd9b289eb3171e12210 , deleted repo/WikibaseRepo.datatypes.php and this time I get:

Parsing file /src/view/resources.php...
double free or corruption (!prev)

The Doxygen package published on apt.wikimedia.org and used in docker-registry.wikimedia.org/releng/doxygen:0.7.0 is broken somehow. It crashes when generating documentation for mediawiki/extensions/Wikibase with errors such as:

corrupted size vs. prev_size
double free or corruption (!prev)

I have tried on my local Buster using the packages generated by CI for https://gerrit.wikimedia.org/r/#/c/operations/debs/doxygen/+/589416/ and stored at: https://integration.wikimedia.org/ci/job/debian-glue/1876/ and https://integration.wikimedia.org/ci/job/debian-glue/1908/ . They both work.

I went with the package published on apt.wikimedia.org https://apt.wikimedia.org/wikimedia/pool/component/ci/d/doxygen/doxygen_1.8.17-1~deb10%2Bwmf1_amd64.deb and it works fine on my local Buster machine.

Crafted a container with gdb added to it and with the package having the debug symbols

FROM docker-registry.wikimedia.org/releng/doxygen:0.7.0

USER root

RUN apt update && apt -y install gdb wget

RUN mkdir -p /tmp/doxygen-deb \
	&& cd /tmp/doxygen-deb \
	&& wget --quiet https://apt.wikimedia.org/wikimedia/pool/component/ci/d/doxygen/doxygen_1.8.17-1~deb10%2Bwmf1_amd64.deb \
	&& wget --quiet https://apt.wikimedia.org/wikimedia/pool/component/ci/d/doxygen/doxygen-dbgsym_1.8.17-1~deb10%2Bwmf1_amd64.deb \
	&& dpkg -i doxygen*.deb

USER nobody
docker rmi doxygen
docker build -t doxygen .
docker run --rm -it --entrypoint=doxygen --workdir /src --user=$UID -v "$(pwd):/src" doxygen

Which results in the error:

Parsing file /src/repo/WikibaseRepo.datatypes.php...
corrupted size vs. prev_size

And a core file :)

$ docker run --privileged --rm -it --entrypoint=gdb --workdir /src --user=$UID -v "$(pwd):/src" doxygen -- /usr/bin/doxygen core
Reading symbols from /usr/bin/doxygen...Reading symbols from /usr/lib/debug/.build-id/42/6bc7fc13da2a4875a3d9eced38298c895ec951.debug...done.
done.
[New LWP 1]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `doxygen'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f9fc6ed3611 in abort () from /lib/x86_64-linux-gnu/libc.so.6
(gdb)
(gdb) bt
#0  0x00007f9fc6ed3611 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f9fc6f2a508 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f9fc6f30c1a in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007f9fc6f34e00 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x00007f9fc6f35d4f in realloc () from /lib/x86_64-linux-gnu/libc.so.6
#5  0x0000561bd7b5d56f in QCString::LSData::resize (size=17, d=0x561bdbfc06c0)
    at ./qtools/qcstring.h:398
#6  QCString::LSData::resize (size=17, d=0x561bdbfc06c0)
    at ./qtools/qcstring.h:394
#7  QCString::StringRep::resize (this=0x561bdbfd73c0, newlen=17)
    at ./qtools/qcstring.h:665
#8  0x0000561bd7b5d5f9 in QCString::resize (newlen=<optimized out>, 
    this=0x561bdbfd73c0) at ./qtools/qcstring.h:312
#9  QCString::operator+= (this=0x561bdbfd73c0, str=0x561bd8de3bd5 " ")
    at ./qtools/qcstring.h:312
#10 0x0000561bd7e51fec in scannerYYlex (yyscanner=0x561bda040050)
    at scanner.l:4135
#11 0x0000561bd7e699b4 in parseMain (filesInSameTranslationUnit=..., 
    sameTranslationUnit=<optimized out>, 
    rt=std::shared_ptr<Entry> (use count 2, weak count 0) = {...}, 
    fileBuf=<optimized out>, 
    fileName=0x561bdbfbf728 "/src/repo/WikibaseRepo.datatypes.php", 
    yyscanner=0x561bda040050) at scanner.l:7242
--Type <RET> for more, q to quit, c to continue without paging--
#12 COutlineParser::parseInput (this=<optimized out>, 
    fileName=0x561bdbfbf728 "/src/repo/WikibaseRepo.datatypes.php", 
    fileBuf=<optimized out>, 
    root=std::shared_ptr<Entry> (use count 2, weak count 0) = {...}, 
    sameTranslationUnit=<optimized out>, filesInSameTranslationUnit=...)
    at scanner.l:7377
#13 0x0000561bd7b4111c in parseFile (parser=..., 
    root=std::shared_ptr<Entry> (use count 1, weak count 0) = {...}, 
    fd=0x561bda3332e0, fn=<optimized out>, sameTu=<optimized out>, 
    filesInSameTu=...) at ./qtools/qcstring.h:593
#14 0x0000561bd7b44866 in parseFiles (
    root=std::shared_ptr<Entry> (use count 1, weak count 0) = {...})
    at ./qtools/qcstring.h:593
#15 0x0000561bd7b577aa in parseInput () at ./src/doxygen.cpp:11073
#16 0x0000561bd7b037f9 in main (argc=1, argv=0x7ffd64cfca88)
    at ./src/main.cpp:37

If I delete repo/WikibaseRepo.datatypes.php which yields and repeat the above. I get the error double free or corruption (!prev) and in gdb:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `doxygen'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f62f7a93611 in abort () from /lib/x86_64-linux-gnu/libc.so.6

(gdb) bt
#0  0x00007f62f7a93611 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f62f7aea508 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f62f7af0c1a in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007f62f7af273c in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x0000557f5302771c in __gnu_cxx::new_allocator<Argument>::deallocate (
    this=0x557f5831a5f0, __p=<optimized out>)
    at /usr/include/c++/8/ext/new_allocator.h:116
#5  std::allocator_traits<std::allocator<Argument> >::deallocate (__a=..., 
    __n=<optimized out>, __p=<optimized out>)
    at /usr/include/c++/8/bits/alloc_traits.h:462
#6  std::_Vector_base<Argument, std::allocator<Argument> >::_M_deallocate (
    this=0x557f5831a5f0, __n=<optimized out>, __p=<optimized out>)
    at /usr/include/c++/8/bits/stl_vector.h:304
#7  std::_Vector_base<Argument, std::allocator<Argument> >::~_Vector_base (
    this=0x557f5831a5f0, __in_chrg=<optimized out>)
    at /usr/include/c++/8/bits/stl_vector.h:285
#8  std::vector<Argument, std::allocator<Argument> >::~vector (
    this=0x557f5831a5f0, __in_chrg=<optimized out>)
    at /usr/include/c++/8/bits/stl_vector.h:570
#9  ArgumentList::~ArgumentList (this=0x557f5831a5f0, 
    __in_chrg=<optimized out>) at ./src/arguments.h:59
#10 Entry::~Entry (this=0x557f5831a550, __in_chrg=<optimized out>)
    at ./src/entry.cpp:123
#11 0x0000557f5332bd19 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x557f5831a540) at /usr/include/c++/8/bits/shared_ptr_base.h:148
#12 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (
    this=0x557f5831a540) at /usr/include/c++/8/bits/shared_ptr_base.h:148
#13 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (
    this=<optimized out>, __in_chrg=<optimized out>)
    at /usr/include/c++/8/bits/shared_ptr_base.h:728
#14 std::__shared_ptr<Entry, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (
    this=<optimized out>, __in_chrg=<optimized out>)
    at /usr/include/c++/8/bits/shared_ptr_base.h:1167
#15 std::__shared_ptr<Entry, (__gnu_cxx::_Lock_policy)2>::operator= (__r=..., 
    this=<optimized out>) at /usr/include/c++/8/bits/shared_ptr_base.h:1263
#16 std::shared_ptr<Entry>::operator= (__r=..., this=<optimized out>)
    at /usr/include/c++/8/bits/shared_ptr.h:335
#17 parseMain (filesInSameTranslationUnit=..., 
    sameTranslationUnit=<optimized out>, 
    rt=std::shared_ptr<Entry> (use count 2, weak count 0) = {...}, 
    fileBuf=<optimized out>, 
    fileName=0x557f58357248 "/src/view/resources.php", 
    yyscanner=0x557f56393050) at scanner.l:7222
#18 COutlineParser::parseInput (this=<optimized out>, 
    fileName=0x557f58357248 "/src/view/resources.php", 
    fileBuf=<optimized out>, 
--Type <RET> for more, q to quit, c to continue without paging--
    root=std::shared_ptr<Entry> (use count 2, weak count 0) = {...}, 
    sameTranslationUnit=<optimized out>, filesInSameTranslationUnit=...)
    at scanner.l:7377
#19 0x0000557f5300311c in parseFile (parser=..., 
    root=std::shared_ptr<Entry> (use count 1, weak count 0) = {...}, 
    fd=0x557f566993a0, fn=<optimized out>, sameTu=<optimized out>, 
    filesInSameTu=...) at ./qtools/qcstring.h:593
#20 0x0000557f53006866 in parseFiles (
    root=std::shared_ptr<Entry> (use count 1, weak count 0) = {...})
    at ./qtools/qcstring.h:593
#21 0x0000557f530197aa in parseInput () at ./src/doxygen.cpp:11073
#22 0x0000557f52fc57f9 in main (argc=1, argv=0x7ffe90110fe8)
    at ./src/main.cpp:37

None of that explains why it crashes in the Docker container while that works just fine on my local machine :-\

Doxygen 1.8.18 is out. I rebuild a package and I have confirmed the issue no more occurs inside the same Docker container. Guess I will fill a task / rebuild etc. Sorry for the annoyance.

Change 599094 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/debs/doxygen@debian/buster-wikimedia] Merge tag 'debian/1.8.18-1_exp1' into debian/buster-wikimedia

https://gerrit.wikimedia.org/r/599094

1.8.18 is out

Went there to check if you actually filed an upstream ticket. The word "crash" actually appears twice in the change log so maybe someone found it before us.

Thank you!

1.8.18 is out

Went there to check if you actually filed an upstream ticket. The word "crash" actually appears twice in the change log so maybe someone found it before us.

Had 1.8.18 failed as well, I would have gone into git bisecting doxygen 1.8.16 .. 1.8.17 to find the faulty patch and indeed would have filed a ticket to upstream for sure :] Luckily 1.8.18 seems to fix it which saves my a day or so of madness.

I haven't thought about comparing the release tags, thanks for the link! I went through https://www.doxygen.nl/manual/changelog.html but it is rather dense and the commit descriptions are not very meaningful unfortunately.

Our post merge builds are green ever since. Guess we can call this. T253793: Update Doxygen to 1.8.18 remains (though I'm not sure the way the ticket dependency expressed is how the phabricator makers envisioned things).

Thanks again.

That could be the subject of a nice debate: When do we close a task?

But indeed the issue for Wikibase got resolved by rolling back to Doxygen 1.8.16 and it seems it will work fine with 1.8.18.

Change 599094 merged by Dzahn:
[operations/debs/doxygen@debian/buster-wikimedia] Merge tag 'debian/1.8.18-1_exp1' into debian/buster-wikimedia

https://gerrit.wikimedia.org/r/599094

Change 602338 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] jjb: update to Doxygen 1.8.18

https://gerrit.wikimedia.org/r/602338

Change 602338 merged by jenkins-bot:
[integration/config@master] jjb: update to Doxygen 1.8.18

https://gerrit.wikimedia.org/r/602338

I have switched the Jenkins job to use 1.8.18.

hashar renamed this task from mwext-doxygen-publish fails: corrupted size vs. prev_size to mwext-doxygen-publish Doxygen 1.8.17 fails: corrupted size vs. prev_size.Aug 13 2020, 9:33 AM