Page MenuHomePhabricator

Replace jsduck with JSDoc3 across all Wikimedia code bases
Open, HighPublic

Description

JSDuck is formally unmaintained and seems abandonware by now. It is hurting our ability to properly document our code.

  1. Examples of problems:
    • No support for @interface
    • No support for namespaced events (change:title is invalid)
    • No support for ES6 (T156469)
  2. Links that show it is unmaintained:
  3. Proposed alternatives:

Requirements for alternative

What are blockers for adopting JSDoc3 and/or a custom theme? In other words, compared to viewing code comments directly, what added value do we want our generated documentation to have for consumers?

TDB, see T187672: Create Wikimedia theme for JSDoc3 (version 1)

AC

  • Usage of jsduck is dropped in favor of jsdoc in MobileFrontend
  • Usage of jsduck is dropped in favor of jsdoc in Minerva
  • Usage of jsduck is dropped in favor of jsdoc in core
  • Usage of jsduck is dropped in favor of jsdoc in VisualEditor
  • Usage of jsduck is dropped in favor of jsdoc in ContentTranslation
  • Usage of jsduck is dropped in favor of jsdoc in Wikibase

Checklist for each migration

  • Code documentation works fine, fix any problems
  • Docs get properly published to docs.wikimedia.org e.g. https://doc.wikimedia.org/MobileFrontend/master/js/
  • CI lints documentation properly with jenkins-bot on patches
  • Documentation about how to run documentation in mw.org, README, etc is updated to point to the new instructions if any change

Migration guide

JSDoc configuration:

  • Filename is jsdoc.json.
  • Don't forget the tabs.
  • Update destination to wherever the autogenerated JavaScript documentation output is. This should be ignored in a .gitignore.
  • Try hard to enable pedantic.
  • If you have a JS-specific readme, update the readme option.
  • If you do not have many externals, try to enable the Wikimedia template. See T250022 if this is blocking adoption. It's better to have pedantic than a theme.
  • Update source.include as needed.
  • Output should most probably be available on doc.wikimedia.org since you've taken the trouble to write it.
jsdoc.json
{
	"opts": {
		"destination": "docs/js",
		"package": "package.json",
		"pedantic": true,
		"readme": "README.md",
		"recurse": true,
		"template": "node_modules/jsdoc-wmf-theme"
	},
	"plugins": [
		"plugins/markdown"
	],
	"source": {
		"include": [ "resources", "src" ]
	},
	"templates": {
		"cleverLinks": true,
		"default": {
			"useLongnameInNav": true
		}
	}
}

NPM configuration:

  • If a separate NPM script is added, the script should be called jsdoc to avoid confusion with other languages like PHP. Invocation can be inline though too.
  • Invocation looks like jsdoc -c jsdoc.json.
  • Script should be invoked as part of test like ... && npm -s run jsdoc && ....
  • Add the latest JSDoc release (v3.6.4 at time of writing) and jsdoc-wmf-theme (v0.0.3, at time of writing) to devDependencies.
  • All jsduck references should be clean (e.g., rg jsduck).
Example package.json
{
  ...,
  "scripts": {
    ...,
    "test": "npm -s run build && npm -s run doc",
    "doc": "jsdoc -c jsdoc.json && npm run build-storybook",
    ...,
  },
  "devDependencies": {
    ...,
    "jsdoc": "3.6.4",
    "jsdoc-wmf-theme": "0.0.3",
    ...,
  }
}

JSDuck to JSDoc mapping:

JSDuck tagJSDoc tag and notes
@abstract@abstract
@accessor Unsupported. Only used by ExtJSBase.
@alias prefix.name@alias <aliasNamepath>
@alternateClassName OtherClassName What to do? @alias <aliasNamepath>? @typedef [<type>] <namepath>? Sparse usage.
@aside <name> Unsupported. No usage.
@author Some name...@author <name> [<emailAddress>]
@cfg, @cfg name, @cfg {Type} name, @cfg {Type} [name="default value"], @cfg {Type} name.subproperty What to do?
@chainable What to do? Custom tag?
@class, @class ClassName@class [<type> <name>]
@constructor@return [{type}] [description].
@deprecated@deprecated.
@docauthor Some name... Unsupported and unused.
@enum, @enum {Type}, @enum {Type} EnumName, @enum [EnumName=alias.*]@enum [<type>]
@event, @event name@event <className>#[event:]<eventName>.
@evented Unsupported. Only used by ExtJSBase.
@example@example.
@experimental, @experimental 2.0 Some description...@experimental
@extends ParentClassName@extends <namepath>
@fires eventName (in 5.x beta)@fires <className>#[event:]<eventName>
@ftype name Unsupported. No usage.
@hide Unsupported. Only used by ExtJSBase.
@ignore What to do? @ignore but move to end of block?
@inheritable What to do? Delete?
@inheritdoc, @inheritdoc ClassName, @inheritdoc #memberName, @inheritdoc ClassName#memberName, @inheritdoc ClassName#static-type-memberNameWhat to do? If no value, @inheritdoc. If value, @extends <value>?
@localdoc This documentation is only visible inside this class/member. (in 5.x beta) What to do? Delete?
@markdown Unsupported. No usage.
@member ClassName What to do? @member [<type>] [<name>], @memberof <parentNamepath> @memberof! <parentNamepath>, or something else?
@method, @method name@method [<FunctionName>]
@mixins ClassName@mixin [<MixinName>]
@new Unsupported. No usage.
@override OverriddenClassName@override
@param name, @param {Type} name, @param {Type} [name], @param {Type} [name="default-value"], @param {Type} name.subproperty@param [<type>] [<name>] [<description>
@preventable Unsupported. Only used by ExtJSBase.
@private@private [{typeExpression}]
@property, @property name, @property {Type} name, @property {Type} [name="default value"], @property {Type} name.subproperty@property [<type>] [<name>] [<description>
@protected@protected [{typeExpression}]
@ptype name Unsupported. No usage.
@readonly@readonly
@removed, @removed 2.0 Some description... Unsupported. One usage in SemanticMediaWiki and otherwise only used by ExtJSBase.
@requires ClassName@requires <someModuleName>
@return {Type}, @return {Type} return.subproperty@return [{type}] [description]
@scss-mixin Unsupported. No usage.
@since Ext JS 4.0 beta@since <versionDescription>
@singleton What to do? Custom tag?
@static@static
@template@template
@throws, @throws {Type}@throws {<type>} [free-form description]
@type {Type}, @type Type@type {<type>} [free-form description]
@uses ClassName@see <namepath> or @see <text>. Mapping isn't one-to-one.
@var, @var $some-name, @var {Type} $some-name, @var {Type} [$some-name="default value"]@member [<type>] [<name>]
@xtype Unsupported. One usage in ExtTab and otherwise only uses by ExtJSBase.
{@link Class#member link text}{@link namepathOrURL}, [link text]{@link namepathOrURL}, {@link namepathOrURL%7clink text}, {@link namepathOrURL link text (after the first space)}
{@img path/to/image.png alt text}Usupported. Only used by ExtJSBase.
{@video vimeo 465123 Some description here...} Unsupported. No usage.

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

I did my best to summarize the information at hand when I wrote https://www.mediawiki.org/wiki/User:JHernandez_(WMF)/JS_documentation_tools#Which_documentation_format_to_use_(spec_on_comment_syntax_and_tags_for_documenting_code) and https://www.mediawiki.org/wiki/User:JHernandez_(WMF)/JS_documentation_tools#Which_tool_and/or_template_to_use_to_generate_readable_HTML_docs

It seems pretty clear that we should migrate to JSDoc comment syntax, as it has a spec, and competing implementations.

On the other hand, docs to website generators seem to have fallen out of
fashion and there aren't good solid open source options.

TLDR:

  • JSDoc3: doesn't seem terribly well maintained, and templates aren't very fully featured
  • Documentation: Has a nice feature set, I discarded it because it didn't seem to have many contributors. May need to be re-evaluated

So, either we settle with a simple JSDoc template, or we fork and improve one, or we use Documentation.js (if someone can have a look at it and asses it, it would be great)

Either way, if we migrate our comments to the standard JSDoc syntax, we can switch web generators as needed in the future.

@Jhernandez, after reading this blog post by documentationjs' maintainer I am not sure about its roadmap. I have also dabbled with changing the documentationjs template style, and did not find it straightforward (might be my own limitations).

I definitely lean towards your recommendation of using JsDoc3.

In today's Front-end group meeting we've been picking up this topic again and JSDoc 3 or at least JSDoc syntax is the agreed-way to go.
@Jdlrobson has been asking for a starting motivation where we concluded that ES6 syntax is such.
@Krinkle has brought up a new though, that although JSDuck has been discontinued, we haven't had any maintenance issue due to it's being Ruby based for a long time – something that might change to the negative with a Node.js-based solution.
We still want to go that way with all it's advantages altogether.

@cscott would you be willing to try JSDoc syntax to render its needed changes on CI visible before we're stepping onto a bigger project like MediaWiki core?

From the FrontEndStandardsGroup etherpad:

@Krinkle: Start with a smaller project like OOjs core or unicode.js

I think that makes sense. I've been meaning to test this with OOUI (not as familiar with OOjs core), so I finally did. I made a few code changes according to the new syntax:

  1. Use @mixes instead of @mixins
  2. Mark each mixin with @mixin
  3. There is no support for @cfg, so, using @param {Type} [config.option] instead.
  4. I haven't fully understood how @event works. Is everything in the global namespace by default? I tried to namespace the events, for [[ https://prtksxna.github.io/oojs-ui/OO.ui.SelectWidget.html#event:add | SelectWidget ]] and they show up correctly. But I would like to hear from someone who understands this better.

There are some features that I found missing. Most advise I could find said that you could write a template that supports all of this, but it's not given out of the box, or in the default theme.

  1. Custom Table of Contents. At the minimum, for OOUI we need, Widgets, Dialogs, Layouts, etc.
  2. No built-in support for search.
  3. No support to show/hide protected/private/inherited methods. You can hide/show private methods by passing a config flag to the CLI.

You can see the OOUI documentation in the default theme on https://prtksxna.github.io/oojs-ui/ . For OOUI we'll definitely need our own theme, I was thinking we could do something similar to the style guide (something @Volker_E and Design has been polishing). That way all our documentation could have a similar design. Creating custom layouts for JSDoc seems pretty straightforward too: https://rebel-sisters.surge.sh/ (WIP). Thanks for introducing me to surge.sh @Jhernandez

Nice work! The events seem a bit screwed up as you say (https://prtksxna.github.io/oojs-ui/global.html#event:changeAchangeeventisemittedwhentheon/offstateofthetogglechanges. etc.) but this is almost good enough to migrate to right now (and re-skin in time).

There are some features that I found missing. [..]

You can see the OOUI documentation in the default theme on https://prtksxna.github.io/oojs-ui/

Nice work! The events seem a bit screwed up [..]

Indeed! Looks pretty good overall. Noticed one other issue, which is that @property seems to be meant for sub-sub-properties somehow. Documenting /** @property {jQuery} */ this.$overlay = $(); results in the system documenting a member named $overlay that has a sub-property named jQuery. Something didn't go right there.

but this is almost good enough to migrate to right now (and re-skin in time).

Could we articulate what needs fixing? @Prtksxna would be willing to submit a patch of what you've done so far? (No matter if it's a little raw :))

but this is almost good enough to migrate to right now (and re-skin in time).

Could we articulate what needs fixing? @Prtksxna would be willing to submit a patch of what you've done so far? (No matter if it's a little raw :))

The @event naming that I mentioned and the @property documentation that Timo mentioned are I think it.

Jdforrester-WMF renamed this task from Replace jsduck in favor of a better maintained alternative to Replace jsduck with JSDoc3 across all Wikimedia code bases.Jan 20 2018, 8:37 PM
Jdforrester-WMF added a project: Epic.
Jdforrester-WMF updated the task description. (Show Details)
  1. Mark each mixin with @mixin
  2. There is no support for @cfg, so, using @param {Type} [config.option] instead.

That seems to work, for @cfg you can use @property, see http://usejsdoc.org/tags-property.html and an example of output here http://mobilefrontend-jsdoc-test.surge.sh/Overlay.html#defaults

  1. I haven't fully understood how @event works. Is everything in the global namespace by default? I tried to namespace the events, for [[ https://prtksxna.github.io/oojs-ui/OO.ui.SelectWidget.html#event:add | SelectWidget ]] and they show up correctly. But I would like to hear from someone who understands this better.

Yes! As you mention, events are global by default (namespaced under event:), you can use that notation @event <className>#[event:]<eventName>. I think you did it correctly

(for reference: http://usejsdoc.org/tags-event.html)

There are some features that I found missing. Most advise I could find said that you could write a template that supports all of this, but it's not given out of the box, or in the default theme.

  1. Custom Table of Contents. At the minimum, for OOUI we need, Widgets, Dialogs, Layouts, etc.
  2. No built-in support for search.
  3. No support to show/hide protected/private/inherited methods. You can hide/show private methods by passing a config flag to the CLI.

Same issues I found. Something like docstrap has search and some other features but I didn't find any one that had everything we needed out of the box.

Here's a few from 1 to try out and maybe fork:


Nice try @Prtksxna

Indeed! Looks pretty good overall. Noticed one other issue, which is that @property seems to be meant for sub-sub-properties somehow. Documenting /** @property {jQuery} */ this.$overlay = $(); results in the system documenting a member named $overlay that has a sub-property named jQuery. Something didn't go right there.

Class properties/fields should be automatically picked up, and they can be forced with memberof, but should not be annotated with property, as you mentioned, which is for object (sub)properties and not class fields.


The events stuff seems to be fine if you follow the convention for the event names Class#eventName, but if you don't specify a class then they will be put in the global namespace, so it is definitely something to look out for when migrating.

cscott added a comment.Feb 5 2018, 6:48 PM

but this is almost good enough to migrate to right now (and re-skin in time).

Could we articulate what needs fixing? @Prtksxna would be willing to submit a patch of what you've done so far? (No matter if it's a little raw :))

The @event naming that I mentioned and the @property documentation that Timo mentioned are I think it.

Custom Table of Contents would be good to have. Parsoid also uses this to separate out the "public" parsoid API from the plethora of internal implementation classes.

Change 408455 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/services/parsoid@master] WIP: Switch from jsduck to jsdoc3 for documentation

https://gerrit.wikimedia.org/r/408455

Here's a few from 1 to try out and maybe fork:

I did try these, and several more (see the added packages in https://gerrit.wikimedia.org/r/#/c/405543/7/package.json) but did not like the results of any. Rather, none of them had everything.

cscott added a comment.EditedFeb 7 2018, 10:03 PM

You can see the OOUI documentation in the default theme on https://prtksxna.github.io/oojs-ui/ . For OOUI we'll definitely need our own theme, I was thinking we could do something similar to the style guide (something @Volker_E and Design has been polishing). That way all our documentation could have a similar design. Creating custom layouts for JSDoc seems pretty straightforward too: https://rebel-sisters.surge.sh/ (WIP). Thanks for introducing me to surge.sh @Jhernandez

@Prtksxna Could you push the WIP theme that you used for rebel-sisters.surge.sh somewhere? I've got a WIP patch to replace jsduck with jsdoc3 for Parsoid, but the default theme is *ugly*. And I'd like to try hacking up your custom theme to allow custom Table of Content / grouping categories, since that seems like the thing the default themes are missing.

Change 409422 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/services/parsoid@master] Use new jsdoc-wmf-theme for documentation

https://gerrit.wikimedia.org/r/409422

Change 408455 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Switch from jsduck to jsdoc3 for documentation

https://gerrit.wikimedia.org/r/408455

Change 409422 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Use new jsdoc-wmf-theme for documentation

https://gerrit.wikimedia.org/r/409422

https://github.com/jsdoc3/jsdoc/issues/483 is a problem with jsdoc -- it currently doesn't resolve {@link Foo} or {@link #bar} -- you have to spell out the full name of the class, including module name and full class name, every time if you want it to be linked. It may be possible to resolve this in the jsdoc 'theme'.

My jsdoc-wmf-theme does manage to resolve a number of these @link shortcuts.

Prtksxna added a comment.EditedFeb 19 2018, 3:52 AM

@Prtksxna Could you push the WIP theme that you used for rebel-sisters.surge.sh somewhere? I've got a WIP patch to replace jsduck with jsdoc3 for Parsoid, but the default theme is *ugly*. And I'd like to try hacking up your custom theme to allow custom Table of Content / grouping categories, since that seems like the thing the default themes are missing.

Sorry @cscott! For rebel-sisters I had created the theme as a hack in node_modules, and didn't back it up before doing a routine rm -rf node_modules && npm install. Not sure how to get it from surge.sh either! Your theme definitely looks much better anyway

Also, see T187672: Create Wikimedia theme for JSDoc3 (version 1)

cscott added a comment.EditedFeb 20 2018, 9:53 PM

My theme broke today because I was hotlinking to the WikimediaUI-Style-Guide github repo. But it should be fixed as soon as https://gerrit.wikimedia.org/r/410208 is merged.

(The fix was https://github.com/cscott/jsdoc-wmf-theme/commit/58b153aac76fbc6de73fe4cc50705532017eb5a4 )

This is what it should look like: http://tidy-club.surge.sh/

Change 419846 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/skins/MinervaNeue@master] Use jsdoc for generating Minerva documentation

https://gerrit.wikimedia.org/r/419846

Change 419846 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] Use jsdoc for generating Minerva documentation

https://gerrit.wikimedia.org/r/419846

Change 419941 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[integration/config@master] Use npm run docs not jsduck

https://gerrit.wikimedia.org/r/419941

^^ So we have adopted jsdoc inside Minerva. It looks like Minerva uses a Jenkins job called mwext-jsduck-publish to publish documentation which is now failing.

I thought we generalised documentation generation to be made when npm run doc was run, but looking at the logs ? Is there another job we should be using @hashar and @zeljkofilipin ?

For CI:

The npm and npm-stretch containers do have jsduck included. So we can change all repositories to add a "npm run-script doc" which would simply shell out to jsduck. Then developers can later change that 'doc' run-script to use jsdoc.

An example is mediawiki/core which triggers:

JobContainerCommand
mediawiki-core-jsduck-dockerreleng/jsduck containerjsduck
mediawiki-core-npm-node-6-dockerreleng/npmnpm test

When jsduck support is dropped from mediawiki/core we would instruct CI to stop triggering jsduck for the master branch (and we can keep it on the old REL branches).

Essentially, the change would make npm run-script doc the default entry point. Up to developers to use jsduck/jsdoc or whatever else.

Change 419941 abandoned by Jdlrobson:
Use npm run docs not jsduck

https://gerrit.wikimedia.org/r/419941

So should I open a new ticket?

[..]

Essentially, the change would make npm run-script doc the default entry point. Up to developers to use jsduck/jsdoc or whatever else.

+1. Note that this was previously planned in 2015. Looks like we did it for some of the publish jobs, and for VisualEditor, but not others.

Krinkle updated the task description. (Show Details)Mar 16 2018, 9:37 PM
Volker_E updated the task description. (Show Details)Jan 7 2019, 10:48 PM

Change 496882 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] Update mwext-EventLogging postmerge from jsduck to generic node10

https://gerrit.wikimedia.org/r/496882

Change 496882 merged by jenkins-bot:
[integration/config@master] Update mwext-EventLogging postmerge from jsduck to generic node10

https://gerrit.wikimedia.org/r/496882

phuedx removed a subscriber: phuedx.Mar 18 2019, 9:21 AM
Niedzielski updated the task description. (Show Details)Mar 21 2020, 8:30 PM

This is done for reading web extensions from what I can see.

If I'm not mistaken ContentTranslation and VisualEditor and core are remaining.

Since we're likely to need to touch core code as part of desktop refresh this seems in scope for that project.

Restricted Application added a project: Growth-Team. · View Herald TranscriptApr 13 2020, 6:23 PM

Change 588454 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/core@master] Add jsdoc to core

https://gerrit.wikimedia.org/r/588454

Change 588457 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/core@master] Replace jsduck with jsdoc

https://gerrit.wikimedia.org/r/588457

If I'm not mistaken ContentTranslation and VisualEditor and core are remaining.

For what it is worth: Wikibase is also still in the jsduck era.

It is definitely not in scope of any Web team's project, but mentioning it here for completeness (given the pretty non-standard of the repository of Wikibase it might be easy to miss).

Jdlrobson updated the task description. (Show Details)Apr 14 2020, 8:45 PM
Restricted Application added a subscriber: Masumrezarock100. · View Herald TranscriptApr 14 2020, 8:45 PM

Change 588457 abandoned by Jdlrobson:
Replace jsduck with jsdoc

https://gerrit.wikimedia.org/r/588457

Niedzielski updated the task description. (Show Details)Apr 19 2020, 7:19 PM
Niedzielski updated the task description. (Show Details)Apr 19 2020, 7:34 PM

Change 590628 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] docs: Remove use of 'jqXHR' and 'mediaWiki' jsduck aliases

https://gerrit.wikimedia.org/r/590628

Change 590628 merged by jenkins-bot:
[mediawiki/core@master] docs: Remove use of 'jqXHR' and 'mediaWiki' jsduck aliases

https://gerrit.wikimedia.org/r/590628

JTannerWMF moved this task from To Triage to Triaged on the VisualEditor board.Apr 21 2020, 3:32 PM
JTannerWMF added a subscriber: JTannerWMF.

Is the Editing Team blocking anyone with this?

Is the Editing Team blocking anyone with this?

Technically, yes, we in Release Engineering would like to drop jsduck support from CI, so all work to migrate off jsduck is blocking us, but that's aspirational with no particular timeline.