Page MenuHomePhabricator
Create Task
Maniphest T254465

mwext-doxygen-publish Doxygen 1.8.18 fails for Wikibase.git
Closed, ResolvedPublic
Actions

Description

The Doxygen 1.8.18 parser explodes on Wikibase.git:

00:00:15.310 Reading /src/repo/maintenance/pruneItemsPerSite.php...
00:00:15.763 Parsing file /src/repo/maintenance/pruneItemsPerSite.php..Build step 'Execute shell' marked build as failure
00:00:15.820 Archiving artifacts

It failed with 1.8.17 as well T253723

Repro

cd mediawiki/extensions/Wikibase
docker run --rm -it --entrypoint=doxygen --workdir /src --user=$UID -v "$(pwd):/src" docker-registry.wikimedia.org/releng/doxygen:0.8.0

Though that locally gives me:

Parsing file /src/repo/Wikibase.php...
realloc(): invalid old size

:((

Details

SubjectRepoBranchLines +/-
jjb: upgrade to Doxygen 1.8.19integration/configmaster+5 -5
jjb: use Doyxgen 1.8.19 in mwext-doxygen-publishintegration/configmaster+1 -1
dockerfiles: upgrade Doxygen to 1.8.19integration/configmaster+6 -0
Merge tag 'debian/1.8.19-1' into debian/buster-wikimediaoperations/debs/doxygendebian/buster-wikimedia+13 -8
Link to static libclangoperations/debs/doxygendebian/buster-wikimedia+10 -0
jjb: rollback mwext-doxygen-publish to Doxygen 1.8.16integration/configmaster+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

hashar created this task.Jun 4 2020, 11:55 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 4 2020, 11:55 AM
gerritbot added a comment.Jun 4 2020, 11:56 AM

Change 602345 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] jjb: rollback mwext-doxygen-publish to Doxygen 1.8.16

https://gerrit.wikimedia.org/r/602345

gerritbot added a project: Patch-For-Review.Jun 4 2020, 11:56 AM
hashar mentioned this in T253723: mwext-doxygen-publish Doxygen 1.8.17 fails: corrupted size vs. prev_size.Jun 4 2020, 11:57 AM
gerritbot added a comment.Jun 4 2020, 11:58 AM

Change 602345 merged by jenkins-bot:
[integration/config@master] jjb: rollback mwext-doxygen-publish to Doxygen 1.8.16

https://gerrit.wikimedia.org/r/602345

Lucas_Werkmeister_WMDE updated the task description. (Show Details)Jun 4 2020, 12:01 PM
Lucas_Werkmeister_WMDE subscribed.
hashar updated the task description. (Show Details)Jun 4 2020, 12:02 PM
Maintenance_bot removed a project: Patch-For-Review.Jun 4 2020, 12:10 PM
hashar added a comment.Jun 4 2020, 12:14 PM

One that fails on Parsing file /src/view/lib/resources.test.php...

#0  tcache_get (tc_idx=9) at malloc.c:2934
#1  __GI___libc_malloc (bytes=161) at malloc.c:3042
#2  0x000055738e16747c in QCString::LSData::create (size=<optimized out>) at ./qtools/qcstring.h:492
#3  QCString::StringRep::StringRep (this=0x7fff78b75de0, 
    str=0x557392c7e1c0 "\n\\copyright GPL-2.0-or-later\n@author Daniel Werner < daniel.a.r.werner@gmail.com >\n@author H. Snater < mediawiki@snater.com >\n\n@codeCoverageIgnoreStart\n")
    at ./qtools/qcstring.h:492
#4  0x000055738e2c89f5 in QCString::QCString (str=<optimized out>, this=0x7fff78b75de0) at ./qtools/qcstring.h:161
#5  detab (refIndent=<synthetic pointer>: <optimized out>, s=...) at ./src/markdown.cpp:2535
#6  processMarkdown (fileName=..., lineNr=<optimized out>, e=<optimized out>, input=...) at ./src/markdown.cpp:2572
#7  0x000055738e2ca844 in processMarkdownForCommentBlock (comment=..., fileName=..., lineNr=lineNr@entry=3) at ./src/markdown.cpp:2606
#8  0x000055738e45394b in handleCommentBlock (yyscanner=0x557390d2ec80, doc=..., brief=false) at scanner.l:7028
#9  0x000055738e456a30 in scannerYYlex (yyscanner=0x557390d2ec80) at ./qtools/qcstring.h:161
#10 0x000055738e479f43 in parseMain (filesInSameTranslationUnit=..., sameTranslationUnit=<optimized out>, rt=std::shared_ptr<class Entry> (use count 2, weak count 0) = {...}, 
    fileBuf=<optimized out>, fileName=0x557392c74a08 "/src/view/lib/resources.test.php", yyscanner=0x557390d2ec80) at scanner.l:7301
#11 COutlineParser::parseInput (this=<optimized out>, fileName=0x557392c74a08 "/src/view/lib/resources.test.php", fileBuf=<optimized out>, 
    root=std::shared_ptr<class Entry> (use count 2, weak count 0) = {...}, sameTranslationUnit=<optimized out>, filesInSameTranslationUnit=...) at scanner.l:7436
#12 0x000055738e136de2 in parseFile (parser=..., root=std::shared_ptr<class Entry> (use count 1, weak count 0) = {...}, fd=0x557390fe8aa0, fn=<optimized out>, sameTu=<optimized out>, 
    filesInSameTu=...) at ./qtools/qcstring.h:595
#13 0x000055738e150cd6 in parseFiles (root=std::shared_ptr<class Entry> (use count 1, weak count 0) = {...}) at ./qtools/qcstring.h:595
#14 0x000055738e1622fa in parseInput () at ./src/doxygen.cpp:10860
#15 0x000055738e114079 in main (argc=1, argv=0x7fff78b76a08) at ./src/main.cpp:37
hashar added a comment.Jun 4 2020, 12:23 PM

Another example choking on view/src/EditSectionGenerator.php

#0  tcache_get (tc_idx=9) at malloc.c:2934
#1  __GI___libc_malloc (bytes=153) at malloc.c:3042
#2  0x00005569d27cf47c in QCString::LSData::create (size=<optimized out>) at ./qtools/qcstring.h:492
#3  QCString::StringRep::StringRep (this=0x7ffcc3ad3b80, 
    str=0x5569d6a0e1d0 "\nReturns HTML allowing to edit label, description and aliases.\n\n@param string $languageCode\n@param EntityId|null $entityId\n\n@return string HTML\n")
    at ./qtools/qcstring.h:492
#4  0x00005569d29309f5 in QCString::QCString (str=<optimized out>, this=0x7ffcc3ad3b80) at ./qtools/qcstring.h:161
#5  detab (refIndent=<synthetic pointer>: <optimized out>, s=...) at ./src/markdown.cpp:2535
#6  processMarkdown (fileName=..., lineNr=<optimized out>, e=<optimized out>, input=...) at ./src/markdown.cpp:2572
#7  0x00005569d2932844 in processMarkdownForCommentBlock (comment=..., fileName=..., lineNr=lineNr@entry=25) at ./src/markdown.cpp:2606
#8  0x00005569d2abb94b in handleCommentBlock (yyscanner=0x5569d4b0ec80, doc=..., brief=false) at scanner.l:7028
#9  0x00005569d2abea30 in scannerYYlex (yyscanner=0x5569d4b0ec80) at ./qtools/qcstring.h:161
#10 0x00005569d2ae164e in parseCompounds (yyscanner=0x5569d4b0ec80, rt=...) at scanner.l:7216
#11 0x00005569d2ae16d9 in parseCompounds (yyscanner=0x5569d4b0ec80, rt=...) at scanner.l:7230
#12 0x00005569d2ae1fb0 in parseMain (filesInSameTranslationUnit=..., sameTranslationUnit=<optimized out>, rt=std::shared_ptr<class Entry> (use count 1, weak count 0) = {...}, 
    fileBuf=<optimized out>, fileName=0x5569d56216a8 "/src/view/src/EditSectionGenerator.php", yyscanner=0x5569d4b0ec80) at scanner.l:7314
#13 COutlineParser::parseInput (this=<optimized out>, fileName=0x5569d56216a8 "/src/view/src/EditSectionGenerator.php", fileBuf=<optimized out>, 
    root=std::shared_ptr<class Entry> (use count 1, weak count 0) = {...}, sameTranslationUnit=<optimized out>, filesInSameTranslationUnit=...) at scanner.l:7436
#14 0x00005569d279ede2 in parseFile (parser=..., root=std::shared_ptr<class Entry> (use count 1, weak count 0) = {...}, fd=0x5569d4dd0420, fn=<optimized out>, sameTu=<optimized out>, 
    filesInSameTu=...) at ./qtools/qcstring.h:595
#15 0x00005569d27b8cd6 in parseFiles (root=std::shared_ptr<class Entry> (use count 1, weak count 0) = {...}) at ./qtools/qcstring.h:595
#16 0x00005569d27ca2fa in parseInput () at ./src/doxygen.cpp:10860
#17 0x00005569d277c079 in main (argc=1, argv=0x7ffcc3ad48a8) at ./src/main.cpp:37
hashar added a comment.Jun 4 2020, 12:32 PM

and of course now it no more happens :-\

• Pablo-WMDE mentioned this in T253793: Update Doxygen to 1.8.18.Jun 4 2020, 1:54 PM
hashar added a project: Upstream.Jun 5 2020, 4:19 AM

Filed at https://github.com/doxygen/doxygen/issues/7832

hashar triaged this task as Medium priority.Jun 5 2020, 7:38 AM
• Pablo-WMDE added a comment.Jun 5 2020, 1:17 PM
In T254465#6195081, @hashar wrote:

Filed at https://github.com/doxygen/doxygen/issues/7832

I think the problem has been fixed but not released yet.

Looks promising.

hashar added a comment.Jun 9 2020, 12:14 PM

The 1.8.18 segfault on view/lib/resources.test.php apparently came from aca13723a9373a1080ca7f108e7be0905b9ae793 which is also in 1.8.17. From my testing 9e2b6a2dac870b5daabde118c5327281e3f79ba2 resolves it and should be in the future 1.8.19.

1.8.17 also had an issue with Wikibase which was T253723

So we are stuck on 1.8.16 for now.

hashar changed the task status from Open to Stalled.Jun 11 2020, 5:28 PM

Stalled waiting for a new release of Doxygen to be cut.

Jdforrester-WMF assigned this task to hashar.Jun 15 2020, 10:37 AM
hashar moved this task from INBOX to Maintenance on the Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)) board.Jun 24 2020, 2:06 PM
thcipriani edited projects, added Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)); removed Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)).Jul 8 2020, 5:34 PM
hashar moved this task from INBOX to New Work on the Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)) board.Jul 27 2020, 1:37 PM
hashar moved this task from Backlog to Patch merged upstream on the Upstream board.Aug 10 2020, 8:15 AM

Doxygen has cut a 1.8.19

https://github.com/doxygen/doxygen/issues/7832#issuecomment-641231186

hashar added a comment.Aug 13 2020, 12:13 PM

And I filed a bug to Debian to have 1.8.19 imported in the packaging repo: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=968340

gerritbot added a comment.Aug 19 2020, 3:12 PM

Change 621291 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/debs/doxygen@debian/buster-wikimedia] Merge 'doxygen 1.8.19 release' from Debian

https://gerrit.wikimedia.org/r/621291

gerritbot added a project: Patch-For-Review.Aug 19 2020, 3:12 PM
hashar edited projects, added Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)); removed Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)).Sep 24 2020, 10:24 AM
hashar moved this task from INBOX to Maintenance on the Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)) board.Oct 14 2020, 8:34 AM
hashar added a comment.Oct 14 2020, 12:27 PM

The debian/1.8.19-1~exp1 merge proposed at https://gerrit.wikimedia.org/r/c/operations/debs/doxygen/+/621291/ fails to build:

ld: cannot find -lclang-cpp

[100%] Linking CXX executable ../../bin/doxyparse
cd /build/doxygen-1.8.19/build/addon/doxyparse && /usr/bin/cmake -E cmake_link_script CMakeFiles/doxyparse.dir/link.txt --verbose=1

/usr/bin/c++  -Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 -fdebug-prefix-map=/build/doxygen-1.8.19=. -fstack-protector-strong -Wformat -Werror=format-security  -Wl,-z,relro -rdynamic CMakeFiles/doxyparse.dir/doxyparse.cpp.o  -o ../../bin/doxyparse -Wl,-rpath,/usr/lib/llvm-7/lib: ../../lib/libdoxymain.a ../../lib/libqtools.a ../../lib/libmd5.a ../../lib/liblodepng.a ../../lib/libmscgen.a ../../lib/libdoxygen_version.a ../../lib/libdoxycfg.a ../../lib/libvhdlparser.a -lpthread /usr/lib/llvm-7/lib/libclang-7.so.1 -lclang-cpp /usr/lib/llvm-7/lib/libLLVM-7.so.1 
/usr/bin/ld: cannot find -lclang-cpp
collect2: error: ld returned 1 exit status

debian/control has llvm-7-dev, libclang-7-dev, clang-7.

hashar mentioned this in T255465: Investigate how to accomplish A/B testing in the iOS app.Oct 14 2020, 8:48 PM
hashar added a comment.Oct 14 2020, 8:54 PM

Doxygen 1.8.19 uses libclang-cpp which is available in newer LLVM versions. The backport to Buster has LLVM 7 so we need to do some static linking: https://github.com/doxygen/doxygen/issues/7883

Or in short pass: -Dstatic_libclang=ON

Debian also has 1.8.20 now. But I guess that will be for later.

gerritbot added a comment.Oct 14 2020, 8:54 PM

Change 634080 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/debs/doxygen@debian/buster-wikimedia] Link to static libclang

https://gerrit.wikimedia.org/r/634080

gerritbot added a comment.Oct 15 2020, 7:40 AM

Change 634080 abandoned by Hashar:
[operations/debs/doxygen@debian/buster-wikimedia] Link to static libclang

Reason:
squashed in previous change

https://gerrit.wikimedia.org/r/634080

hashar added a comment.EditedOct 15 2020, 9:19 AM

I have tested the version built by CI for https://gerrit.wikimedia.org/r/c/operations/debs/doxygen/+/621291/

Dockerfile
FROM docker-registry.wikimedia.org/releng/doxygen:latest

COPY doxygen_1.8.19-1~deb10+wmf1+0~20201015074531.2032+buster+wikimedia~1.gbpc7b1eb_amd64.deb /tmp/doxygen.deb
USER root
RUN dpkg -i /tmp/doxygen.deb
USER nobody
$ docker build -t doxygenew .
...
$ install --dir --mode 777 docs
$ docker run --rm -it -v /home/hashar/projects/mediawiki/extensions/Wikibase:/src -v "$(pwd)/docs:/src/docs" --workdir=/src --entrypoint=doxygen doxygenew:latest
...
lookup cache used 13996/65536 hits=36363 misses=16005
finished...
$

So yeah that works :]

gerritbot added a comment.Oct 15 2020, 9:30 AM

Change 621291 merged by jenkins-bot:
[operations/debs/doxygen@debian/buster-wikimedia] Merge tag 'debian/1.8.19-1' into debian/buster-wikimedia

https://gerrit.wikimedia.org/r/621291

hashar changed the task status from Stalled to Open.Oct 15 2020, 9:34 AM
hashar added a subtask: T265579: Build and release Doxygen 1.8.19 to apt.wikimedia.org.
JMeybohm closed subtask T265579: Build and release Doxygen 1.8.19 to apt.wikimedia.org as Resolved.Oct 15 2020, 2:22 PM
gerritbot added a comment.Oct 15 2020, 4:14 PM

Change 634281 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] dockerfiles: upgrade Doxygen to 1.8.19

https://gerrit.wikimedia.org/r/634281

gerritbot added a comment.Oct 15 2020, 4:21 PM

Change 634281 merged by jenkins-bot:
[integration/config@master] dockerfiles: upgrade Doxygen to 1.8.19

https://gerrit.wikimedia.org/r/634281

Stashbot added a comment.Oct 15 2020, 4:44 PM

Mentioned in SAL (#wikimedia-releng) [2020-10-15T16:44:20Z] <hashar> Successfully tagged docker-registry.discovery.wmnet/releng/doxygen:0.9.0 # T254465

gerritbot added a comment.Oct 19 2020, 8:06 AM

Change 634915 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] jjb: use Doyxgen 1.8.19 in mwext-doxygen-publish

https://gerrit.wikimedia.org/r/634915

hashar added a comment.Oct 19 2020, 8:09 AM

Verified for Wikibase using the latest merged change https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/634204 and adding it back to the CI postmerge pipeline with:

zuul enqueue --trigger gerrit --pipeline postmerge --project mediawiki/extensions/Wikibase --change 634204,2

https://doc.wikimedia.org/Wikibase/master/php/ states it got generated by Doxygen 1.8.19 ;)

gerritbot added a comment.Oct 19 2020, 8:12 AM

Change 634916 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] jjb: upgrade to Doxygen 1.8.19

https://gerrit.wikimedia.org/r/634916

gerritbot added a comment.Oct 19 2020, 8:15 AM

Change 634915 merged by jenkins-bot:
[integration/config@master] jjb: use Doyxgen 1.8.19 in mwext-doxygen-publish

https://gerrit.wikimedia.org/r/634915

gerritbot added a comment.Oct 19 2020, 8:15 AM

Change 634916 merged by jenkins-bot:
[integration/config@master] jjb: upgrade to Doxygen 1.8.19

https://gerrit.wikimedia.org/r/634916

hashar closed this task as Resolved.Oct 19 2020, 9:13 AM
hashar added a comment.Oct 19 2020, 9:18 AM

Announced on wikitech-l https://lists.wikimedia.org/pipermail/wikitech-l/2020-October/093970.html

hashar mentioned this in T242155: Update Doxygen in CI to 1.8.17 or greater.Oct 19 2020, 9:20 AM
hashar added a parent task: T253793: Update Doxygen to 1.8.18.
hashar mentioned this in T233788: Doxygen output should strip @suppress annotations.Oct 19 2020, 9:23 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL