Page MenuHomePhabricator

Replace jsduck with JSDoc3 across all Wikimedia code bases
Open, HighPublic

Description

JSDuck is formally unmaintained and seems abandonware by now. It is hurting our ability to properly document our code.

  1. Examples of problems:
    • No support for @interface
    • No support for namespaced events (change:title is invalid)
    • No support for ES6 (T156469)
  2. Links that show it is unmaintained:
  3. Proposed alternatives:

Requirements for alternative

What are blockers for adopting JSDoc3 and/or a custom theme? In other words, compared to viewing code comments directly, what added value do we want our generated documentation to have for consumers?

TDB, see T187672: Create Wikimedia theme for JSDoc3 (version 1)

AC

  • Usage of jsduck is dropped in favor of jsdoc in MobileFrontend
  • Usage of jsduck is dropped in favor of jsdoc in Minerva
  • Usage of jsduck is dropped in favor of jsdoc in core
  • Usage of jsduck is dropped in favor of jsdoc in VisualEditor
  • Usage of jsduck is dropped in favor of jsdoc in ContentTranslation
  • Usage of jsduck is dropped in favor of jsdoc in Wikibase

Checklist for each migration

  • Code documentation works fine, fix any problems
  • Docs get properly published to docs.wikimedia.org e.g. https://doc.wikimedia.org/MobileFrontend/master/js/
  • CI lints documentation properly with jenkins-bot on patches
  • Documentation about how to run documentation in mw.org, README, etc is updated to point to the new instructions if any change

Migration guide

JSDoc configuration:

  • Filename is jsdoc.json.
  • Don't forget the tabs.
  • Update destination to wherever the autogenerated JavaScript documentation output is. This should be ignored in a .gitignore.
  • Try hard to enable pedantic.
  • If you have a JS-specific readme, update the readme option.
  • If you do not have many externals, try to enable the Wikimedia template. See T250022 if this is blocking adoption. It's better to have pedantic than a theme.
  • Update source.include as needed.
  • Output should most probably be available on doc.wikimedia.org since you've taken the trouble to write it.
jsdoc.json
{
	"opts": {
		"destination": "docs/js",
		"package": "package.json",
		"pedantic": true,
		"readme": "README.md",
		"recurse": true,
		"template": "node_modules/jsdoc-wmf-theme"
	},
	"plugins": [
		"plugins/markdown"
	],
	"source": {
		"include": [ "resources", "src" ]
	},
	"templates": {
		"cleverLinks": true,
		"default": {
			"useLongnameInNav": true
		}
	}
}

NPM configuration:

  • If a separate NPM script is added, the script should be called jsdoc to avoid confusion with other languages like PHP. Invocation can be inline though too.
  • Invocation looks like jsdoc -c jsdoc.json.
  • Script should be invoked as part of test like ... && npm -s run jsdoc && ....
  • Add the latest JSDoc release (v3.6.4 at time of writing) and jsdoc-wmf-theme (v0.0.3, at time of writing) to devDependencies.
  • All jsduck references should be clean (e.g., rg jsduck).
Example package.json
{
  ...,
  "scripts": {
    ...,
    "test": "npm -s run build && npm -s run doc",
    "doc": "jsdoc -c jsdoc.json && npm run build-storybook",
    ...,
  },
  "devDependencies": {
    ...,
    "jsdoc": "3.6.4",
    "jsdoc-wmf-theme": "0.0.3",
    ...,
  }
}

JSDuck to JSDoc mapping:

JSDuck tagJSDoc tag and notes
@abstract@abstract
@accessor Unsupported. Only used by ExtJSBase.
@alias prefix.name@alias <aliasNamepath>
@alternateClassName OtherClassName What to do? @alias <aliasNamepath>? @typedef [<type>] <namepath>? Sparse usage.
@aside <name> Unsupported. No usage.
@author Some name...@author <name> [<emailAddress>]
@cfg, @cfg name, @cfg {Type} name, @cfg {Type} [name="default value"], @cfg {Type} name.subproperty What to do?
@chainable What to do? Custom tag?
@class, @class ClassName@class [<type> <name>]
@constructor@return [{type}] [description].
@deprecated@deprecated.
@docauthor Some name... Unsupported and unused.
@enum, @enum {Type}, @enum {Type} EnumName, @enum [EnumName=alias.*]@enum [<type>]
@event, @event name@event <className>#[event:]<eventName>.
@evented Unsupported. Only used by ExtJSBase.
@example@example.
@experimental, @experimental 2.0 Some description...@experimental
@extends ParentClassName@extends <namepath>
@fires eventName (in 5.x beta)@fires <className>#[event:]<eventName>
@ftype name Unsupported. No usage.
@hide Unsupported. Only used by ExtJSBase.
@ignore What to do? @ignore but move to end of block?
@inheritable What to do? Delete?
@inheritdoc, @inheritdoc ClassName, @inheritdoc #memberName, @inheritdoc ClassName#memberName, @inheritdoc ClassName#static-type-memberNameWhat to do? If no value, @inheritdoc. If value, @extends <value>?
@localdoc This documentation is only visible inside this class/member. (in 5.x beta) What to do? Delete?
@markdown Unsupported. No usage.
@member ClassName What to do? @member [<type>] [<name>], @memberof <parentNamepath> @memberof! <parentNamepath>, or something else?
@method, @method name@method [<FunctionName>]
@mixins ClassName@mixin [<MixinName>]
@new Unsupported. No usage.
@override OverriddenClassName@override
@param name, @param {Type} name, @param {Type} [name], @param {Type} [name="default-value"], @param {Type} name.subproperty@param [<type>] [<name>] [<description>
@preventable Unsupported. Only used by ExtJSBase.
@private@private [{typeExpression}]
@property, @property name, @property {Type} name, @property {Type} [name="default value"], @property {Type} name.subproperty@property [<type>] [<name>] [<description>
@protected@protected [{typeExpression}]
@ptype name Unsupported. No usage.
@readonly@readonly
@removed, @removed 2.0 Some description... Unsupported. One usage in SemanticMediaWiki and otherwise only used by ExtJSBase.
@requires ClassName@requires <someModuleName>
@return {Type}, @return {Type} return.subproperty@return [{type}] [description]
@scss-mixin Unsupported. No usage.
@since Ext JS 4.0 beta@since <versionDescription>
@singleton What to do? Custom tag?
@static@static
@template@template
@throws, @throws {Type}@throws {<type>} [free-form description]
@type {Type}, @type Type@type {<type>} [free-form description]
@uses ClassName@see <namepath> or @see <text>. Mapping isn't one-to-one.
@var, @var $some-name, @var {Type} $some-name, @var {Type} [$some-name="default value"]@member [<type>] [<name>]
@xtype Unsupported. One usage in ExtTab and otherwise only uses by ExtJSBase.
{@link Class#member link text}{@link namepathOrURL}, [link text]{@link namepathOrURL}, {@link namepathOrURL%7clink text}, {@link namepathOrURL link text (after the first space)}
{@img path/to/image.png alt text}Usupported. Only used by ExtJSBase.
{@video vimeo 465123 Some description here...} Unsupported. No usage.

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

This is done for reading web extensions from what I can see.

If I'm not mistaken ContentTranslation and VisualEditor and core are remaining.

Since we're likely to need to touch core code as part of desktop refresh this seems in scope for that project.

Change 588454 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/core@master] Add jsdoc to core

https://gerrit.wikimedia.org/r/588454

Change 588457 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/core@master] Replace jsduck with jsdoc

https://gerrit.wikimedia.org/r/588457

If I'm not mistaken ContentTranslation and VisualEditor and core are remaining.

For what it is worth: Wikibase is also still in the jsduck era.

It is definitely not in scope of any Web team's project, but mentioning it here for completeness (given the pretty non-standard of the repository of Wikibase it might be easy to miss).

Change 588457 abandoned by Jdlrobson:
Replace jsduck with jsdoc

https://gerrit.wikimedia.org/r/588457

Change 590628 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] docs: Remove use of 'jqXHR' and 'mediaWiki' jsduck aliases

https://gerrit.wikimedia.org/r/590628

Change 590628 merged by jenkins-bot:
[mediawiki/core@master] docs: Remove use of 'jqXHR' and 'mediaWiki' jsduck aliases

https://gerrit.wikimedia.org/r/590628

JTannerWMF added a subscriber: JTannerWMF.

Is the Editing Team blocking anyone with this?

Is the Editing Team blocking anyone with this?

Technically, yes, we in Release Engineering would like to drop jsduck support from CI, so all work to migrate off jsduck is blocking us, but that's aspirational with no particular timeline.

Change 588454 abandoned by Jdlrobson:
[mediawiki/core@master] Add jsdoc to core, replace jsduck with jsdoc

Reason:
Not working on this now but honestly I think every day we leave not switching to jsduck we're making it harder to migrate. Please can somebody articulate on ticket what "broken" means as I am not seeing any problems with the jsdoc output with this patch and that would help move this conversation along.

https://gerrit.wikimedia.org/r/588454

Change 626428 had a related patch set uploaded (by DannyS712; owner: DannyS712):
[mediawiki/extensions/GlobalWatchlist@master] Switch documentation to use jsdoc

https://gerrit.wikimedia.org/r/626428

Change 626428 merged by jenkins-bot:
[mediawiki/extensions/GlobalWatchlist@master] Switch documentation to use jsdoc

https://gerrit.wikimedia.org/r/626428

Change 644289 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/TemplateData@master] Migrate from jsduck to jsdoc

https://gerrit.wikimedia.org/r/644289

Change 644289 merged by jenkins-bot:
[mediawiki/extensions/TemplateData@master] Migrate from jsduck to jsdoc

https://gerrit.wikimedia.org/r/644289

T138401#6315952

Just wanted to note, I have wasted quite a bit of time making jsdoc compatible code work with jsduck today, so there is a real developer cost to prolonging the use of jsdoc in core.

It's also frustrating as it's still not clear to me what we consider broken in the output of https://gerrit.wikimedia.org/r/588454 - the fact nobody is coming forward suggests that we either don't care about this documentation or there is no problem with it.

In short: by focusing on short term damage of migrating from jsduck to jsdoc, we are adding to long term damage by encouraging neglect of documentation in mediawiki core.

T138401#6315952

Just wanted to note, I have wasted quite a bit of time making jsdoc compatible code work with jsduck today, so there is a real developer cost to prolonging the use of jsdoc in core.

It's also frustrating as it's still not clear to me what we consider broken in the output of https://gerrit.wikimedia.org/r/588454 - the fact nobody is coming forward suggests that we either don't care about this documentation or there is no problem with it.

In short: by focusing on short term damage of migrating from jsduck to jsdoc, we are adding to long term damage by encouraging neglect of documentation in mediawiki core.

Maybe bringing this to the tech-decision-forum could help clarify a path forward?

Maybe bringing this to the tech-decision-forum could help clarify a path forward?

Maybe, or perhaps something more lightweight with leads in each of the different product teams and anybody else who may be interested?

It's also frustrating as it's still not clear to me what we consider broken in the output of https://gerrit.wikimedia.org/r/588454 […]

When run on MW, the output is broken in pretty much every way possible. The work by Prateek, CScott, and Volker got it to a state where it kind of worked for OOjs. I say "kind of", because it hosts only a few classes where most deficiencies we found from the Parsoid pilot, didn't apply, or were tolerable to OOjs as being a low-stakes project where a simple README suffices to understand most of its API. I accepted it anyway in OOjs since it allowed us to showcase the problems from a public URL, to then work on in a way that's visible and somewhat rewarding; instead of invisible and never released. It is, however, by no means ready. And for the first pilot, Parsoid, the output never made it to a point where it was correct. (Parsoid has since been ported to PHP/Doxygen)

The purpose of these documentation page is to allow new extension developers (incl staff), and gadget authors, to explore and understand the public APIs of our code base - in a way that must have a lower entry barrier and have added value, compared to finding and browsing the code from Git.

The MW output generated an unusable mess of HTML pages with no cohesion or navigability of any kind. No UI to access its categories, modules, or namespaces. Most methods are incorrectly associated or otherwise corrupted. It only just about passed the compiler without errors, but the output wasn't correct by means, let alone useful.

See also:

  • T138401#3054806.
  • And T262403: Search is broken.
  • And T207383: Some inherited methods are missing. Inheritance is one of the primary values doc output provides since view-source only shows you the implementation details of what the last mile added. A documentation page instead should tell you what you can do with an object, regardless of how where its code comes from.
  • And T262406: The grouping mechanisms don't appear to work (e.g. namespaces of classes, or modules). E.g. try to find mw.rcfilters and see which static methods and classes it houses, or try to find the mw.notification.convertmessagebox module and see what parameters the convertmessagebox function takes, or find "mediawiki.base" (not sure if a namespace, module, or category) and see which classes and methods it provides.
  • Plus various other issues explained earlier in this task by @Jhernandez and others. I haven't yet seen those addressed in any of the jsdoc-based outputs I checked on doc.wikimedia.org, including MobileFrontend which still uses the jsdoc default.

Those are just the issues I initially found on mwhen looking at the output for OOjs and Parsoid. We should expect to find more corruption bugs and UX defects as we try it out on larger and more diverse code bases. I'm sure we can handle those issue when we face them, none of this has been advanced or complicated so far. It just needs resourcing.

Hearing complaints for 5+ years, but little to no investments in a solution, has not been particilarly reassuring of us actually moving to a stack that isn't (also) abandoned.

I too want to see this move along. I think, if we want to see progress here, we need to get meaningful and on-going investment, to start addressing some of the most obvious defects. Ideally, with a team taking stewardship commitment to maintain the theme and triage incoming tasks. (The content will come; porting jsduck comments and/or writing new ones isn't our bottleneck).

Krinkle added a subscriber: bd808.

@Aklapper @bd808 This might be in scope for some of the work we've been talking about recently.

In a nut shell:

  • This task is about finding a replacement for the JSDuck tool, which currently generates MediaWiki core JS Documentation, VisualEditor Documentation, and most other library and MW extension docs for JavaScript, as served from doc.wikimedia.org.
  • JSDuck is a Ruby tool with an interpreter, generator, and page theme; all-in-one. But, it has been abandoned by its vendor many years ago, and while it had some minor bugs that we worked around, the most pressing issue is that it doesn't support ES 6 syntax, which in the upcoming year we expect to slowly start adopting natively in MediaWiki extensions and eventually core. (currently blocked on IE11 support).
  • The industry has largely embraced JSDoc3, which is an actively maintained open-source project with a healty ecosystem of plugins and adapters.
  • The issue is, however, that it is mostly a specification, with a popular parser. The part that users actually see is DIY. The default theme it bundles is fairly useless for something other than a single file project. There are some good themes out there probably. But we're leaning towards developing our own WMF-style theme (see live example of Wikimedia OOjs docs). This was a volunteer effort in 2017-2019 by CScott, Prateek, and Volker. It needs someone to finish it, and ideally also to give it some amount of on-going maintenance.

The purpose of these documentation page is to allow new extension developers (incl staff), and gadget authors, to explore and understand the public APIs of our code base - in a way that must have a lower entry barrier and have added value, compared to finding and browsing the code from Git.

I guess this captures our disconnect. The documentation output for me is probably the least important part of me as a developer (by a long way) and the documentation in current form is pretty useless and plays no part in my team's onbording. I am curious what kind of page views it gets.

JSDoc empowers developer productivity in that it gives typing hints in Code Editor, support for TypeScript (even without an extension) as well as more expression as a developer writing inline developer. I value all these things more highly as I use those to navigate around codebases and write good code.

It sounds like getting a shared understanding of the problem is a good first step.. I have no idea what "broken" means in practice for docs.wikimedia.org as I do not use them and even within Joaquin's analysis it's never been clarified which of those are showstoppers by the engineers that use that documentation. Who is using this documentation? Please speak up!

If we can't make progress here, it sounds like involved parties are coming from different places hence the lack of progress. I think the suggestion from @kostajh is a really solid one here and probably the best next step here if we want to move things forward T138401#6915662.

@phuedx had some thoughts when I brought this up. Care to share?

I had the use case of adding JSDoc to a "new" repository (mediawiki/tools/api-testing - T236915) and could not find documentation about it. We have a jsduck wikipage that redirects to https://www.mediawiki.org/wiki/Manual:Coding_conventions/JavaScript#Documentation which still refers to jsduck with a note left mentioning it is pending the switch of mediawiki/core

Can we potentially have a JSDoc page for the new repositories / non mediawiki ones? Looks like it might be fairly simple based on a recent commit ( https://gerrit.wikimedia.org/r/c/jquery-client/+/626136 ), but I am afraid I don't understand everything.

@Aklapper @bd808 This might be in scope for some of the work we've been talking about recently.

I unfortunately honestly don't remember why this was tagged with Developer-Advocacy. :(

I'm boldly going to add tech-decision-forum per T138401#6915662.

A quick (pretty incomplete) local grep shows

./integration/config/zuul/layout.yaml:  - { name: operations/debs/ruby-jsduck,
./mediawiki-core/.eslintrc.json:		"wikimedia/jsduck"
./mediawiki-extensions/ArticleCreationWorkflow/package.json:		"doc": "jsduck"
./mediawiki-extensions/Citoid/.eslintrc.json:		"wikimedia/jsduck"
./mediawiki-extensions/CollaborationKit/jsduck.json:		"jsduck.external.js",
./mediawiki-extensions/ContentTranslation/CONTRIBUTING.md:### Installing jsduck
./mediawiki-extensions/DonationInterface/package.json:		"doc": "jsduck",
./mediawiki-extensions/Echo/jsduck.json:		"jsduck.external.js",
./mediawiki-extensions/EventLogging/package.json:		"doc": "jsduck"
./mediawiki-extensions/FormWizard/composer.json:			"jsduck"
./mediawiki-extensions/Flow/jsduck.json:		"jsduck.external.js",
./mediawiki-extensions/GuidedTour/jsduck.json:		".jsduck/external.js",
./mediawiki-extensions/ImportArticles/package.json:		"doc": "jsduck"
./mediawiki-extensions/Kartographer/package.json:		"doc": "jsduck",
./mediawiki-extensions/MultimediaViewer/jsduck.json:		"jsduck.external.js",
./mediawiki-extensions/OOJsUIAjaxLogin/jsduck.json:	"--categories": "./jsduck.categories.json",
./mediawiki-extensions/PageTriage/jsduck.json:		"jsduck.external.js",
./mediawiki-extensions/Translate/resources/.eslintrc.json:		"wikimedia/jsduck"
./mediawiki-extensions/UploadWizard/docs/generate:jsduck \
./mediawiki-extensions/VisualEditor/lib/ve/jsduck.json:	"--categories": ".jsduck/categories.json",
./mediawiki-extensions/Wikibase/package.json:		"doc:jsduck": "jsduck",
./mediawiki-extensions/Wikispeech/composer.json:			"jsduck"
./mediawiki-extensions/Wikispeech/jsduck.json:	"--tags": "../../maintenance/jsduck/custom_tags.rb",
./mediawiki-skins/Metrolook/package.json:		"doc": "jsduck"
./mediawiki-vagrant/puppet/modules/role/manifests/jsduck.pp:    include ::jsduck
./phabricator-extensions/src/gerrit/GerritProjectMap.php:    'operations/debs/ruby-jsduck' => 'ODCN',

For the records, it seems that an access-restricted (why?) Google Doc has been created at https://docs.google.com/document/d/1RRU2vo8mPMM_Fh3F4pcsHJXr76tiFsVtokOv2nHdCz0/edit in parallel to this ticket. Mentioning here for the sake/lack of transparency.

@Aklapper
When you added this to Tech Decision Forum with your comment, it initiated to start the formal review process outlined https://www.mediawiki.org/wiki/Technical_decision_making
Google docs are used to collaborate with the chairs of the decision making process and will be made public once finalized. Please reach out to me directly on Slack if you have additional questions. Thanks!

@Aklapper @bd808 This might be in scope for some of the work we've been talking about recently.

I unfortunately honestly don't remember why this was tagged with Developer-Advocacy. :(

I'm boldly going to add tech-decision-forum per T138401#6915662.

A quick (pretty incomplete) local grep shows

./integration/config/zuul/layout.yaml:  - { name: operations/debs/ruby-jsduck,
./mediawiki-core/.eslintrc.json:		"wikimedia/jsduck"
./mediawiki-extensions/ArticleCreationWorkflow/package.json:		"doc": "jsduck"
./mediawiki-extensions/Citoid/.eslintrc.json:		"wikimedia/jsduck"
./mediawiki-extensions/CollaborationKit/jsduck.json:		"jsduck.external.js",
./mediawiki-extensions/ContentTranslation/CONTRIBUTING.md:### Installing jsduck
./mediawiki-extensions/DonationInterface/package.json:		"doc": "jsduck",
./mediawiki-extensions/Echo/jsduck.json:		"jsduck.external.js",
./mediawiki-extensions/EventLogging/package.json:		"doc": "jsduck"
./mediawiki-extensions/FormWizard/composer.json:			"jsduck"
./mediawiki-extensions/Flow/jsduck.json:		"jsduck.external.js",
./mediawiki-extensions/GuidedTour/jsduck.json:		".jsduck/external.js",
./mediawiki-extensions/ImportArticles/package.json:		"doc": "jsduck"
./mediawiki-extensions/Kartographer/package.json:		"doc": "jsduck",
./mediawiki-extensions/MultimediaViewer/jsduck.json:		"jsduck.external.js",
./mediawiki-extensions/OOJsUIAjaxLogin/jsduck.json:	"--categories": "./jsduck.categories.json",
./mediawiki-extensions/PageTriage/jsduck.json:		"jsduck.external.js",
./mediawiki-extensions/Translate/resources/.eslintrc.json:		"wikimedia/jsduck"
./mediawiki-extensions/UploadWizard/docs/generate:jsduck \
./mediawiki-extensions/VisualEditor/lib/ve/jsduck.json:	"--categories": ".jsduck/categories.json",
./mediawiki-extensions/Wikibase/package.json:		"doc:jsduck": "jsduck",
./mediawiki-extensions/Wikispeech/composer.json:			"jsduck"
./mediawiki-extensions/Wikispeech/jsduck.json:	"--tags": "../../maintenance/jsduck/custom_tags.rb",
./mediawiki-skins/Metrolook/package.json:		"doc": "jsduck"
./mediawiki-vagrant/puppet/modules/role/manifests/jsduck.pp:    include ::jsduck
./phabricator-extensions/src/gerrit/GerritProjectMap.php:    'operations/debs/ruby-jsduck' => 'ODCN',

@Aklapper
When you added this to Tech Decision Forum with your comment, it initiated to start the formal review process outlined https://www.mediawiki.org/wiki/Technical_decision_making
Google docs are used to collaborate with the chairs of the decision making process and will be made public once finalized. Please reach out to me directly on Slack if you have additional questions. Thanks!

Who are the chairs? This page mentions this:

Forum chairs DepartmentChair
ProductTBD
TechnologyMoriel Schottlender

and it also says:

Chairs are appointed by the Wikimedia Foundation CTO and CPO respectively and rotated on a quarterly basis.

So if that's correct:

  • We never had a chair for product
  • We never rotated the chair for technology.

Is that correct?

@Ladsgroup
No that is incorrect. Tech has rotated chairs and Moriel is the new chair. Product has had a chair which has successfully rotated and the last chair left and the new chairs have just been appointed on Friday. I am in the process of updating the page today to reflect the new product chairs.

Thanks. I somehow missed it. I now looked at history and it seems the technology department had one chair for really really long time and didn't get rotated until recently. But better late than never.

I had a really bad experience setting up JSDuck, and I had to use docker image to have a stable setup and a lot of bash hacks to pipe my commands to it, and still I cannot run it through npm, I have to run it in terminal. Also other findings:

  • It is an unsupported archived tool
  • It has a lot of cross language dependencies (ruby + node)
  • It requires to have old versions of ruby that is hard to setup on most modern machines

the solution I had to do:
$ docker pull poum/jsduck
Add Alias to your .bashrc or .zshrc or .profile
$ alias jsduck="docker run --rm -v $(pwd):/code --workdir /code poum/jsduck --config=jsduck.json"