Page MenuHomePhabricator

mwext-doxygen-publish Doxygen 1.8.18 fails for Wikibase.git
Closed, ResolvedPublic

Description

The Doxygen 1.8.18 parser explodes on Wikibase.git:

00:00:15.310 Reading /src/repo/maintenance/pruneItemsPerSite.php...
00:00:15.763 Parsing file /src/repo/maintenance/pruneItemsPerSite.php..Build step 'Execute shell' marked build as failure
00:00:15.820 Archiving artifacts

It failed with 1.8.17 as well T253723

Repro

cd mediawiki/extensions/Wikibase
docker run --rm -it --entrypoint=doxygen --workdir /src --user=$UID -v "$(pwd):/src" docker-registry.wikimedia.org/releng/doxygen:0.8.0

Though that locally gives me:

Parsing file /src/repo/Wikibase.php...
realloc(): invalid old size

:((

Event Timeline

hashar created this task.Jun 4 2020, 11:55 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 4 2020, 11:55 AM

Change 602345 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] jjb: rollback mwext-doxygen-publish to Doxygen 1.8.16

https://gerrit.wikimedia.org/r/602345

Change 602345 merged by jenkins-bot:
[integration/config@master] jjb: rollback mwext-doxygen-publish to Doxygen 1.8.16

https://gerrit.wikimedia.org/r/602345

hashar updated the task description. (Show Details)Jun 4 2020, 12:02 PM

One that fails on Parsing file /src/view/lib/resources.test.php...

#0  tcache_get (tc_idx=9) at malloc.c:2934
#1  __GI___libc_malloc (bytes=161) at malloc.c:3042
#2  0x000055738e16747c in QCString::LSData::create (size=<optimized out>) at ./qtools/qcstring.h:492
#3  QCString::StringRep::StringRep (this=0x7fff78b75de0, 
    str=0x557392c7e1c0 "\n\\copyright GPL-2.0-or-later\n@author Daniel Werner < daniel.a.r.werner@gmail.com >\n@author H. Snater < mediawiki@snater.com >\n\n@codeCoverageIgnoreStart\n")
    at ./qtools/qcstring.h:492
#4  0x000055738e2c89f5 in QCString::QCString (str=<optimized out>, this=0x7fff78b75de0) at ./qtools/qcstring.h:161
#5  detab (refIndent=<synthetic pointer>: <optimized out>, s=...) at ./src/markdown.cpp:2535
#6  processMarkdown (fileName=..., lineNr=<optimized out>, e=<optimized out>, input=...) at ./src/markdown.cpp:2572
#7  0x000055738e2ca844 in processMarkdownForCommentBlock (comment=..., fileName=..., lineNr=lineNr@entry=3) at ./src/markdown.cpp:2606
#8  0x000055738e45394b in handleCommentBlock (yyscanner=0x557390d2ec80, doc=..., brief=false) at scanner.l:7028
#9  0x000055738e456a30 in scannerYYlex (yyscanner=0x557390d2ec80) at ./qtools/qcstring.h:161
#10 0x000055738e479f43 in parseMain (filesInSameTranslationUnit=..., sameTranslationUnit=<optimized out>, rt=std::shared_ptr<class Entry> (use count 2, weak count 0) = {...}, 
    fileBuf=<optimized out>, fileName=0x557392c74a08 "/src/view/lib/resources.test.php", yyscanner=0x557390d2ec80) at scanner.l:7301
#11 COutlineParser::parseInput (this=<optimized out>, fileName=0x557392c74a08 "/src/view/lib/resources.test.php", fileBuf=<optimized out>, 
    root=std::shared_ptr<class Entry> (use count 2, weak count 0) = {...}, sameTranslationUnit=<optimized out>, filesInSameTranslationUnit=...) at scanner.l:7436
#12 0x000055738e136de2 in parseFile (parser=..., root=std::shared_ptr<class Entry> (use count 1, weak count 0) = {...}, fd=0x557390fe8aa0, fn=<optimized out>, sameTu=<optimized out>, 
    filesInSameTu=...) at ./qtools/qcstring.h:595
#13 0x000055738e150cd6 in parseFiles (root=std::shared_ptr<class Entry> (use count 1, weak count 0) = {...}) at ./qtools/qcstring.h:595
#14 0x000055738e1622fa in parseInput () at ./src/doxygen.cpp:10860
#15 0x000055738e114079 in main (argc=1, argv=0x7fff78b76a08) at ./src/main.cpp:37

Another example choking on view/src/EditSectionGenerator.php

#0  tcache_get (tc_idx=9) at malloc.c:2934
#1  __GI___libc_malloc (bytes=153) at malloc.c:3042
#2  0x00005569d27cf47c in QCString::LSData::create (size=<optimized out>) at ./qtools/qcstring.h:492
#3  QCString::StringRep::StringRep (this=0x7ffcc3ad3b80, 
    str=0x5569d6a0e1d0 "\nReturns HTML allowing to edit label, description and aliases.\n\n@param string $languageCode\n@param EntityId|null $entityId\n\n@return string HTML\n")
    at ./qtools/qcstring.h:492
#4  0x00005569d29309f5 in QCString::QCString (str=<optimized out>, this=0x7ffcc3ad3b80) at ./qtools/qcstring.h:161
#5  detab (refIndent=<synthetic pointer>: <optimized out>, s=...) at ./src/markdown.cpp:2535
#6  processMarkdown (fileName=..., lineNr=<optimized out>, e=<optimized out>, input=...) at ./src/markdown.cpp:2572
#7  0x00005569d2932844 in processMarkdownForCommentBlock (comment=..., fileName=..., lineNr=lineNr@entry=25) at ./src/markdown.cpp:2606
#8  0x00005569d2abb94b in handleCommentBlock (yyscanner=0x5569d4b0ec80, doc=..., brief=false) at scanner.l:7028
#9  0x00005569d2abea30 in scannerYYlex (yyscanner=0x5569d4b0ec80) at ./qtools/qcstring.h:161
#10 0x00005569d2ae164e in parseCompounds (yyscanner=0x5569d4b0ec80, rt=...) at scanner.l:7216
#11 0x00005569d2ae16d9 in parseCompounds (yyscanner=0x5569d4b0ec80, rt=...) at scanner.l:7230
#12 0x00005569d2ae1fb0 in parseMain (filesInSameTranslationUnit=..., sameTranslationUnit=<optimized out>, rt=std::shared_ptr<class Entry> (use count 1, weak count 0) = {...}, 
    fileBuf=<optimized out>, fileName=0x5569d56216a8 "/src/view/src/EditSectionGenerator.php", yyscanner=0x5569d4b0ec80) at scanner.l:7314
#13 COutlineParser::parseInput (this=<optimized out>, fileName=0x5569d56216a8 "/src/view/src/EditSectionGenerator.php", fileBuf=<optimized out>, 
    root=std::shared_ptr<class Entry> (use count 1, weak count 0) = {...}, sameTranslationUnit=<optimized out>, filesInSameTranslationUnit=...) at scanner.l:7436
#14 0x00005569d279ede2 in parseFile (parser=..., root=std::shared_ptr<class Entry> (use count 1, weak count 0) = {...}, fd=0x5569d4dd0420, fn=<optimized out>, sameTu=<optimized out>, 
    filesInSameTu=...) at ./qtools/qcstring.h:595
#15 0x00005569d27b8cd6 in parseFiles (root=std::shared_ptr<class Entry> (use count 1, weak count 0) = {...}) at ./qtools/qcstring.h:595
#16 0x00005569d27ca2fa in parseInput () at ./src/doxygen.cpp:10860
#17 0x00005569d277c079 in main (argc=1, argv=0x7ffcc3ad48a8) at ./src/main.cpp:37

and of course now it no more happens :-\

hashar triaged this task as Medium priority.Jun 5 2020, 7:38 AM

I think the problem has been fixed but not released yet.

Looks promising.

The 1.8.18 segfault on view/lib/resources.test.php apparently came from aca13723a9373a1080ca7f108e7be0905b9ae793 which is also in 1.8.17. From my testing 9e2b6a2dac870b5daabde118c5327281e3f79ba2 resolves it and should be in the future 1.8.19.

1.8.17 also had an issue with Wikibase which was T253723

So we are stuck on 1.8.16 for now.

hashar changed the task status from Open to Stalled.Jun 11 2020, 5:28 PM

Stalled waiting for a new release of Doxygen to be cut.

And I filed a bug to Debian to have 1.8.19 imported in the packaging repo: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=968340

Change 621291 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/debs/doxygen@debian/buster-wikimedia] Merge 'doxygen 1.8.19 release' from Debian

https://gerrit.wikimedia.org/r/621291

The debian/1.8.19-1~exp1 merge proposed at https://gerrit.wikimedia.org/r/c/operations/debs/doxygen/+/621291/ fails to build:

ld: cannot find -lclang-cpp

[100%] Linking CXX executable ../../bin/doxyparse
cd /build/doxygen-1.8.19/build/addon/doxyparse && /usr/bin/cmake -E cmake_link_script CMakeFiles/doxyparse.dir/link.txt --verbose=1

/usr/bin/c++  -Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 -fdebug-prefix-map=/build/doxygen-1.8.19=. -fstack-protector-strong -Wformat -Werror=format-security  -Wl,-z,relro -rdynamic CMakeFiles/doxyparse.dir/doxyparse.cpp.o  -o ../../bin/doxyparse -Wl,-rpath,/usr/lib/llvm-7/lib: ../../lib/libdoxymain.a ../../lib/libqtools.a ../../lib/libmd5.a ../../lib/liblodepng.a ../../lib/libmscgen.a ../../lib/libdoxygen_version.a ../../lib/libdoxycfg.a ../../lib/libvhdlparser.a -lpthread /usr/lib/llvm-7/lib/libclang-7.so.1 -lclang-cpp /usr/lib/llvm-7/lib/libLLVM-7.so.1 
/usr/bin/ld: cannot find -lclang-cpp
collect2: error: ld returned 1 exit status

debian/control has llvm-7-dev, libclang-7-dev, clang-7.

Doxygen 1.8.19 uses libclang-cpp which is available in newer LLVM versions. The backport to Buster has LLVM 7 so we need to do some static linking: https://github.com/doxygen/doxygen/issues/7883

Or in short pass: -Dstatic_libclang=ON

Debian also has 1.8.20 now. But I guess that will be for later.

Change 634080 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/debs/doxygen@debian/buster-wikimedia] Link to static libclang

https://gerrit.wikimedia.org/r/634080

Change 634080 abandoned by Hashar:
[operations/debs/doxygen@debian/buster-wikimedia] Link to static libclang

Reason:
squashed in previous change

https://gerrit.wikimedia.org/r/634080

hashar added a comment.EditedThu, Oct 15, 9:19 AM

I have tested the version built by CI for https://gerrit.wikimedia.org/r/c/operations/debs/doxygen/+/621291/

Dockerfile
FROM docker-registry.wikimedia.org/releng/doxygen:latest

COPY doxygen_1.8.19-1~deb10+wmf1+0~20201015074531.2032+buster+wikimedia~1.gbpc7b1eb_amd64.deb /tmp/doxygen.deb
USER root
RUN dpkg -i /tmp/doxygen.deb
USER nobody
$ docker build -t doxygenew .
...
$ install --dir --mode 777 docs
$ docker run --rm -it -v /home/hashar/projects/mediawiki/extensions/Wikibase:/src -v "$(pwd)/docs:/src/docs" --workdir=/src --entrypoint=doxygen doxygenew:latest
...
lookup cache used 13996/65536 hits=36363 misses=16005
finished...
$

So yeah that works :]

Change 621291 merged by jenkins-bot:
[operations/debs/doxygen@debian/buster-wikimedia] Merge tag 'debian/1.8.19-1' into debian/buster-wikimedia

https://gerrit.wikimedia.org/r/621291

hashar changed the task status from Stalled to Open.Thu, Oct 15, 9:34 AM

Change 634281 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] dockerfiles: upgrade Doxygen to 1.8.19

https://gerrit.wikimedia.org/r/634281

Change 634281 merged by jenkins-bot:
[integration/config@master] dockerfiles: upgrade Doxygen to 1.8.19

https://gerrit.wikimedia.org/r/634281

Mentioned in SAL (#wikimedia-releng) [2020-10-15T16:44:20Z] <hashar> Successfully tagged docker-registry.discovery.wmnet/releng/doxygen:0.9.0 # T254465

Change 634915 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] jjb: use Doyxgen 1.8.19 in mwext-doxygen-publish

https://gerrit.wikimedia.org/r/634915

Verified for Wikibase using the latest merged change https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/634204 and adding it back to the CI postmerge pipeline with:

zuul enqueue --trigger gerrit --pipeline postmerge --project mediawiki/extensions/Wikibase --change 634204,2

https://doc.wikimedia.org/Wikibase/master/php/ states it got generated by Doxygen 1.8.19 ;)

Change 634916 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] jjb: upgrade to Doxygen 1.8.19

https://gerrit.wikimedia.org/r/634916

Change 634915 merged by jenkins-bot:
[integration/config@master] jjb: use Doyxgen 1.8.19 in mwext-doxygen-publish

https://gerrit.wikimedia.org/r/634915

Change 634916 merged by jenkins-bot:
[integration/config@master] jjb: upgrade to Doxygen 1.8.19

https://gerrit.wikimedia.org/r/634916

hashar closed this task as Resolved.Mon, Oct 19, 9:13 AM