Page MenuHomePhabricator
Create Task
Maniphest T317451

MediaWiki core docs unavailable for MW 1.35 and later
Closed, ResolvedPublic
Actions

Description

https://doc.wikimedia.org/mediawiki-core/1.33.0/php/
https://doc.wikimedia.org/mediawiki-core/1.34.0/

But 1.35.0 and later are missing the Doxygen build under https://doc.wikimedia.org/mediawiki-core/.

I confirmed in the CI config that there are (correctly) JS and PHP doc publishing jobs configured for mediawiki/core:publish pipeline. To debug it, I re-tried it manually from the CLI (given that for tags and branch creations, there is no web page to comment "recheck" on).

krinkle@contint2001
$ zuul enqueue-ref --trigger gerrit --pipeline publish --project mediawiki/core --ref refs/tags/1.38.0 --newrev 975ddea706c4026daa9cbef51a0f668539d650a3 --oldrev 0000000000

https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker-publish/49/

It fails as follows:

git fetch --quiet --update-head-ok --depth 2 git://contint1001.wikimedia.org/mediawiki/core +refs/tags/1.38.0:refs/tags/1.38.0
...
git submodule --quiet update --init --recursive
..
+ exec docker run --entrypoint=/usr/bin/php .. maintenance/mwdocgen.php --output /src/docs --version 1.38.0
..

00:00:39.945 doxygen /tmp/MWDocGen-JSiWIL
00:00:59.802 /src/extensions/ReplaceText/maintenance/replaceAll.php:24: warning: missing argument after '\category'.
00:01:02.523 input buffer overflow, can't enlarge buffer because scanner uses REJECT
---------------------------------------------------
Doxygen execution finished.
Check above for possible errors.

You might want to delete the temporary file:
 /tmp/MWDocGen-JSiWIL
---------------------------------------------------
Something went wrong (exit: 2)

Build step 'Execute shell' marked build as failure

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
build: Restore Doxygen output for MediaWiki release tags (take 3)mediawiki/coreREL1_41+10 -8
build: Restore Doxygen output for MediaWiki release tags (take 3)mediawiki/coreREL1_40+10 -8
build: Restore Doxygen output for MediaWiki release tags (take 3)mediawiki/coreREL1_39+10 -8
build: Restore Doxygen output for MediaWiki release tags (take 3)mediawiki/coremaster+10 -8
build: Restore Doxygen output for MediaWiki release tagsmediawiki/coreREL1_41+5 -12
build: Restore Doxygen output for MediaWiki release tagsmediawiki/coreREL1_40+5 -12
build: Restore Doxygen output for MediaWiki release tagsmediawiki/coreREL1_39+5 -12
build: Restore Doxygen output for MediaWiki release tagsmediawiki/coremaster+5 -12
docs: Apply excludes like 'vendor' and 'tests' to extensions as wellmediawiki/coremaster+5 -4
jjb: Skip submodules in 'mediawiki-core-doxygen-docker-publish'integration/configmaster+4 -2
Customize query in gerrit

Event Timeline

Krinkle created this task.Sep 9 2022, 9:47 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 9 2022, 9:47 PM
Krinkle updated the task description. (Show Details)EditedSep 9 2022, 9:55 PM

Per upstream https://www.doxygen.nl/manual/faq.html#faq_lex this is most likely due to an excessively large file somewhere being scanned for doc comments. Once we identify the file in question we can probably exclude it.

Krinkle added a comment.Sep 9 2022, 10:08 PM

Interestingly, the REL branch builds are unaffected. If the presence of extension submodules played a role, I would have expected those to similarly fail.

Krinkle added a comment.Sep 9 2022, 10:25 PM

The Doxygen version currently deployed in CI (Doxygen 1.9.1) unfortunately does not make mention of the file that is being scanned in its "buffer overflow" erroe message (as per the task description).

However, I see that upstream has improved this in the 1.9.3 release (doxygen#8576). Using that locally with a couple of extensions checked-out I see:

…
mediawiki/extensions/AbuseFilter/vendor/netresearch/jsonmapper/src/JsonMapper.php:17: warning: missing argument after '\category'.
…
mediawiki/extensions/CheckUser/node_modules/doiuse/README.md:83: warning: multiple use of section label 'features-list' 
mediawiki/extensions/CheckUser/node_modules/yargs-parser/README.md:154: warning: multiple use of section label 'configuration'
…
input buffer overflow, can't enlarge buffer because scanner uses REJECT
    lexical analyzer: doxygen/src/scanner.l (for: mediawiki/extensions/CirrusSearch/tests/phpunit/resources/wmf/interwiki.php)

Doxygen execution finished.

The above identifies mediawiki/extensions/CirrusSearch/tests/phpunit/resources/wmf/interwiki.php as the problematic file in question.

gerritbot added a comment.Sep 9 2022, 11:08 PM

Change 831161 had a related patch set uploaded (by Krinkle; author: Krinkle):

[integration/config@master] jjb: Skip submodules in 'mediawiki-core-doxygen-docker-publish'

https://gerrit.wikimedia.org/r/831161

gerritbot added a project: Patch-For-Review.Sep 9 2022, 11:08 PM
gerritbot added a comment.Sep 9 2022, 11:13 PM

Change 831161 abandoned by Krinkle:

[integration/config@master] jjb: Skip submodules in 'mediawiki-core-doxygen-docker-publish'

Reason:

vendor is a submodule in these jobs. Also: why is extenions/* being scanned in the first place given that --extensions isn't passed on mwdocgen.php.

00:00:13.713 Error: You are missing some external dependencies.
00:00:13.713 MediaWiki also has some external dependencies that need to be installed
00:00:13.713 via composer or from a separate git repo. Please see

https://gerrit.wikimedia.org/r/831161

gerritbot added a comment.Sep 9 2022, 11:18 PM

Change 831162 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] docs: Apply excludes like 'vendor' and 'tests' to extensions as well

https://gerrit.wikimedia.org/r/831162

gerritbot added a comment.Sep 13 2022, 12:00 PM

Change 831162 merged by jenkins-bot:

[mediawiki/core@master] docs: Apply excludes like 'vendor' and 'tests' to extensions as well

https://gerrit.wikimedia.org/r/831162

ReleaseTaggerBot added a project: MW-1.40-notes (1.40.0-wmf.2; 2022-09-19).Sep 13 2022, 1:00 PM
thcipriani moved this task from INBOX to Blocking 🧱 on the Release-Engineering-Team board.Sep 14 2022, 4:29 PM
thcipriani edited projects, added Release-Engineering-Team (Blocking 🧱); removed Release-Engineering-Team.
RhinosF1 subscribed.Sep 14 2022, 4:37 PM
TheresNoTime removed a subscriber: RhinosF1.Dec 15 2022, 11:35 PM
Krinkle closed this task as Resolved.Feb 8 2023, 11:20 PM
Krinkle claimed this task.
Krinkle edited projects, added Performance-Team; removed Release-Engineering-Team (Blocking 🧱), MW-1.40-notes (1.40.0-wmf.2; 2022-09-19).
Krinkle reopened this task as Open.Mar 9 2024, 2:31 AM

This has broken again. No MediaWiki core PHP docs for:

https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker-publish/82/console

+ git fetch --quiet --update-head-ok --depth 2 git://contint2002.wikimedia.org/mediawiki/core +refs/tags/1.41.0:refs/tags/1.41.0
+ git submodule --quiet update --init --recursive

+ exec docker run --entrypoint=/usr/bin/php --workdir=/src --volume /srv/jenkins/workspace/mediawiki-core-doxygen-docker-publish/src:/src --volume /srv/jenkins/workspace/mediawiki-core-doxygen-docker-publish/cache:/cache --volume /srv/jenkins/workspace/mediawiki-core-doxygen-docker-publish/log:/log --security-opt seccomp=unconfined --init --rm --label jenkins.job=mediawiki-core-doxygen-docker-publish --label jenkins.build=82 --env-file /dev/fd/63 docker-registry.wikimedia.org/releng/doxygen:1.9.8-s2 maintenance/mwdocgen.php --output /src/docs --version 1.41.0
++ /usr/bin/env
++ egrep -v '^(HOME|SHELL|PATH|LOGNAME|MAIL)='
Executing command:
doxygen /tmp/MWDocGen-wYwZPH
/src/extensions/ReplaceText/maintenance/replaceAll.php:24: warning: missing argument after '\category'.
input buffer overflow, can't enlarge buffer because scanner uses REJECT
    lexical analyzer: /mnt/hgfs/dimitri/doxygen.git/src/scanner.l (for: /src/extensions/SecureLinkFixer/domains.php)
Krinkle added a comment.Mar 9 2024, 2:41 AM
In T317451#8226145, @Krinkle wrote:

Also: why is extenions/* being scanned in the first place given that --extensions isn't passed on mwdocgen.php.

I've looked into this again. I suspect the way we use EXCLUDE_PATTERNS is just wrong.

What we do:

EXCLUDE = /src/cache /src/images
EXCLUDE_PATTERNS = 
                         */node_modules \
                         */resources \
                         */tests \
                         extensions skins

Based on https://doxygen.nl/manual/config.html#cfg_exclude that just seems wrong. It specifcally says you have to use */mydir/* to exclude directories in this way. Historically, these jobs did work and did not include extension output. I'm guessing that:

  • technically it did work, but has the risk of also excluding other directories and files that contain the word "extensions" but not with an actual directory names that way.
  • it filters it after Doxygen first tries to scan it, instead of skipping the directory traversal entirely.

I have no idea why Doxygen would do this, even if it recursed into every sub dir inefficiently skipped it on a per-file basis against this pattern, that would still not make it actually open the file and try to parse it. The answer is, it parses it because it doesn't skip it.

The warning: missing argument after '\category'. output shows this definitely. So my second guess is:

  • This used to work as per my previous guess, but a Doxygen release at some point made our badly written exclude pattern no longer work. So it now gets parsed and fully included.
gerritbot added a comment.Mar 9 2024, 2:45 AM

Change 1009821 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] build: Restore Doxygen output for MediaWiki release tags

https://gerrit.wikimedia.org/r/1009821

Krinkle edited projects, added MediaWiki-Platform-Team (Radar); removed Performance-Team.Mar 10 2024, 2:36 AM
gerritbot added a comment.Mar 11 2024, 3:01 PM

Change 1009821 merged by jenkins-bot:

[mediawiki/core@master] build: Restore Doxygen output for MediaWiki release tags

https://gerrit.wikimedia.org/r/1009821

gerritbot added a comment.Mar 11 2024, 3:25 PM

Change 1010209 had a related patch set uploaded (by Jforrester; author: Krinkle):

[mediawiki/core@REL1_41] build: Restore Doxygen output for MediaWiki release tags

https://gerrit.wikimedia.org/r/1010209

gerritbot added a comment.Mar 11 2024, 3:26 PM

Change 1010210 had a related patch set uploaded (by Jforrester; author: Krinkle):

[mediawiki/core@REL1_40] build: Restore Doxygen output for MediaWiki release tags

https://gerrit.wikimedia.org/r/1010210

gerritbot added a comment.Mar 11 2024, 3:26 PM

Change 1010211 had a related patch set uploaded (by Jforrester; author: Krinkle):

[mediawiki/core@REL1_39] build: Restore Doxygen output for MediaWiki release tags

https://gerrit.wikimedia.org/r/1010211

ReleaseTaggerBot added a project: MW-1.42-notes (1.42.0-wmf.22; 2024-03-12).Mar 11 2024, 7:00 PM
gerritbot added a comment.Mar 12 2024, 5:22 PM

Change 1010211 merged by jenkins-bot:

[mediawiki/core@REL1_39] build: Restore Doxygen output for MediaWiki release tags

https://gerrit.wikimedia.org/r/1010211

gerritbot added a comment.Mar 12 2024, 5:23 PM

Change 1010210 merged by jenkins-bot:

[mediawiki/core@REL1_40] build: Restore Doxygen output for MediaWiki release tags

https://gerrit.wikimedia.org/r/1010210

gerritbot added a comment.Mar 12 2024, 5:27 PM

Change 1010209 merged by jenkins-bot:

[mediawiki/core@REL1_41] build: Restore Doxygen output for MediaWiki release tags

https://gerrit.wikimedia.org/r/1010209

ReleaseTaggerBot added projects: MW-1.39-notes, MW-1.40-notes, MW-1.41-notes.Mar 13 2024, 1:01 AM
Krinkle added a comment.Mar 13 2024, 11:44 PM

Tried it out by creating a tag 1.39.7-alpha-T317451 and.... it still fails.

https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker-publish/83/console

+ exec docker run […] docker-registry.wikimedia.org/releng/doxygen:1.9.8-s2 maintenance/mwdocgen.php --output /src/docs --version 1.39.7-alpha-T317451

[…]

 input buffer overflow, can't enlarge buffer because scanner uses REJECT
     lexical analyzer: /mnt/hgfs/dimitri/doxygen.git/src/scanner.l (for: /src/extensions/SecureLinkFixer/domains.php)
 
 Something went wrong (exit: 2)

It is still indexing the extensions directory.

Reproducing this locally, I see there is a different issue here.

$ cat /tmp/MWDocGen-000000
…
PROJECT_NAME           = MediaWiki
PROJECT_NUMBER         = 1.39.7-alpha-T317451
…
STRIP_FROM_PATH        = /Users/krinkle/Development/mediawiki
…
INPUT                  =  "/Users/krinkle/Development/mediawiki/"
…
EXCLUDE                =  /Users/krinkle/Development/mediawiki/cache /Users/krinkle/Development/mediawiki/images
EXCLUDE_SYMLINKS       = YES
EXCLUDE_PATTERNS       = LocalSettings.php \
                         .svn \
                         */.git/* \
                         */node_modules \
                         */resources \
                         */tests \
                         */vendor \
                         */includes/libs/*/README.md \
                         */maintenance/*/README.md
…

Notice how, despite what the code suggests, EXCLUDE did not gain an extensions and skins entry.

gerritbot added a comment.Mar 13 2024, 11:53 PM

Change 1011003 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] build: Restore Doxygen output for MediaWiki release tags (take 3)

https://gerrit.wikimedia.org/r/1011003

Krinkle triaged this task as Medium priority.Mar 13 2024, 11:54 PM
gerritbot added a comment.Mar 27 2024, 8:32 PM

Change #1011003 merged by jenkins-bot:

[mediawiki/core@master] build: Restore Doxygen output for MediaWiki release tags (take 3)

https://gerrit.wikimedia.org/r/1011003

ReleaseTaggerBot edited projects, added MW-1.42-notes (1.42.0-wmf.25; 2024-04-02); removed MW-1.42-notes (1.42.0-wmf.22; 2024-03-12).Mar 27 2024, 9:00 PM
gerritbot added a comment.Mar 27 2024, 10:55 PM

Change #1015057 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@REL1_39] build: Restore Doxygen output for MediaWiki release tags (take 3)

https://gerrit.wikimedia.org/r/1015057

gerritbot added a comment.Mar 27 2024, 11:08 PM

Change #1015057 merged by jenkins-bot:

[mediawiki/core@REL1_39] build: Restore Doxygen output for MediaWiki release tags (take 3)

https://gerrit.wikimedia.org/r/1015057

Krinkle added a comment.Mar 27 2024, 11:45 PM

Yay! https://doc.wikimedia.org/mediawiki-core/1.39.7-alpha-T317451.2/php/

gerritbot added a comment.Mar 27 2024, 11:48 PM

Change #1015061 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@REL1_41] build: Restore Doxygen output for MediaWiki release tags (take 3)

https://gerrit.wikimedia.org/r/1015061

gerritbot added a comment.Mar 27 2024, 11:49 PM

Change #1015062 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@REL1_40] build: Restore Doxygen output for MediaWiki release tags (take 3)

https://gerrit.wikimedia.org/r/1015062

• apaskulin awarded a token.Mar 28 2024, 12:51 AM
gerritbot added a comment.Mar 28 2024, 1:07 AM

Change #1015062 merged by jenkins-bot:

[mediawiki/core@REL1_40] build: Restore Doxygen output for MediaWiki release tags (take 3)

https://gerrit.wikimedia.org/r/1015062

gerritbot added a comment.Mar 28 2024, 1:54 AM

Change #1015061 merged by jenkins-bot:

[mediawiki/core@REL1_41] build: Restore Doxygen output for MediaWiki release tags (take 3)

https://gerrit.wikimedia.org/r/1015061

Krinkle closed this task as Resolved.Mar 28 2024, 11:39 PM

Marking as resolved. Besides the test run above, it also worked fully automatically after today's security release, all by itself :)

• apaskulin subscribed.Mar 28 2024, 11:50 PM

Thanks for all your work on this, @Krinkle!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits