Page MenuHomePhabricator

Replace jsduck with JSDoc3 across all Wikimedia code bases
Open, HighPublic

Description

JSDuck is formally unmaintained and seems abandonware by now. It is hurting our ability to properly document our code.

  1. Examples of problems:
    • No support for @interface
    • No support for namespaced events (change:title is invalid)
    • No support for ES6 (T156469)
  2. Links that show it is unmaintained:
  3. Proposed alternatives:

Requirements for alternative

What are blockers for adopting JSDoc3 and/or a custom theme? In other words, compared to viewing code comments directly, what added value do we want our generated documentation to have for consumers?

TDB, see T187672: Create Wikimedia theme for JSDoc3 (version 1)

AC

  • Usage of jsduck is dropped in favor of jsdoc in MobileFrontend
  • Usage of jsduck is dropped in favor of jsdoc in Minerva
  • Usage of jsduck is dropped in favor of jsdoc in core
  • Usage of jsduck is dropped in favor of jsdoc in VisualEditor
  • Usage of jsduck is dropped in favor of jsdoc in ContentTranslation
  • Usage of jsduck is dropped in favor of jsdoc in Wikibase

Checklist for each migration

  • Code documentation works fine, fix any problems
  • Docs get properly published to docs.wikimedia.org e.g. https://doc.wikimedia.org/MobileFrontend/master/js/
  • CI lints documentation properly with jenkins-bot on patches
  • Documentation about how to run documentation in mw.org, README, etc is updated to point to the new instructions if any change

Migration guide

JSDoc configuration:

  • Filename is jsdoc.json.
  • Don't forget the tabs.
  • Update destination to wherever the autogenerated JavaScript documentation output is. This should be ignored in a .gitignore.
  • Try hard to enable pedantic.
  • If you have a JS-specific readme, update the readme option.
  • If you do not have many externals, try to enable the Wikimedia template. See T250022 if this is blocking adoption. It's better to have pedantic than a theme.
  • Update source.include as needed.
  • Output should most probably be available on doc.wikimedia.org since you've taken the trouble to write it.
jsdoc.json
{
	"opts": {
		"destination": "docs/js",
		"package": "package.json",
		"pedantic": true,
		"readme": "README.md",
		"recurse": true,
		"template": "node_modules/jsdoc-wmf-theme"
	},
	"plugins": [
		"plugins/markdown"
	],
	"source": {
		"include": [ "resources", "src" ]
	},
	"templates": {
		"cleverLinks": true,
		"default": {
			"useLongnameInNav": true
		}
	}
}

NPM configuration:

  • If a separate NPM script is added, the script should be called jsdoc to avoid confusion with other languages like PHP. Invocation can be inline though too.
  • Invocation looks like jsdoc -c jsdoc.json.
  • Script should be invoked as part of test like ... && npm -s run jsdoc && ....
  • Add the latest JSDoc release (v3.6.4 at time of writing) and jsdoc-wmf-theme (v0.0.3, at time of writing) to devDependencies.
  • All jsduck references should be clean (e.g., rg jsduck).
Example package.json
{
  ...,
  "scripts": {
    ...,
    "test": "npm -s run build && npm -s run doc",
    "doc": "jsdoc -c jsdoc.json && npm run build-storybook",
    ...,
  },
  "devDependencies": {
    ...,
    "jsdoc": "3.6.4",
    "jsdoc-wmf-theme": "0.0.3",
    ...,
  }
}

JSDuck to JSDoc mapping:

JSDuck tagJSDoc tag and notes
@abstract@abstract
@accessor Unsupported. Only used by ExtJSBase.
@alias prefix.name@alias <aliasNamepath>
@alternateClassName OtherClassName What to do? @alias <aliasNamepath>? @typedef [<type>] <namepath>? Sparse usage.
@aside <name> Unsupported. No usage.
@author Some name...@author <name> [<emailAddress>]
@cfg, @cfg name, @cfg {Type} name, @cfg {Type} [name="default value"], @cfg {Type} name.subproperty What to do?
@chainable What to do? Custom tag?
@class, @class ClassName@class [<type> <name>]
@constructor@return [{type}] [description].
@deprecated@deprecated.
@docauthor Some name... Unsupported and unused.
@enum, @enum {Type}, @enum {Type} EnumName, @enum [EnumName=alias.*]@enum [<type>]
@event, @event name@event <className>#[event:]<eventName>.
@evented Unsupported. Only used by ExtJSBase.
@example@example.
@experimental, @experimental 2.0 Some description...@experimental
@extends ParentClassName@extends <namepath>
@fires eventName (in 5.x beta)@fires <className>#[event:]<eventName>
@ftype name Unsupported. No usage.
@hide Unsupported. Only used by ExtJSBase.
@ignore What to do? @ignore but move to end of block?
@inheritable What to do? Delete?
@inheritdoc, @inheritdoc ClassName, @inheritdoc #memberName, @inheritdoc ClassName#memberName, @inheritdoc ClassName#static-type-memberNameWhat to do? If no value, @inheritdoc. If value, @extends <value>?
@localdoc This documentation is only visible inside this class/member. (in 5.x beta) What to do? Delete?
@markdown Unsupported. No usage.
@member ClassName What to do? @member [<type>] [<name>], @memberof <parentNamepath> @memberof! <parentNamepath>, or something else?
@method, @method name@method [<FunctionName>]
@mixins ClassName@mixin [<MixinName>]
@new Unsupported. No usage.
@override OverriddenClassName@override
@param name, @param {Type} name, @param {Type} [name], @param {Type} [name="default-value"], @param {Type} name.subproperty@param [<type>] [<name>] [<description>
@preventable Unsupported. Only used by ExtJSBase.
@private@private [{typeExpression}]
@property, @property name, @property {Type} name, @property {Type} [name="default value"], @property {Type} name.subproperty@property [<type>] [<name>] [<description>
@protected@protected [{typeExpression}]
@ptype name Unsupported. No usage.
@readonly@readonly
@removed, @removed 2.0 Some description... Unsupported. One usage in SemanticMediaWiki and otherwise only used by ExtJSBase.
@requires ClassName@requires <someModuleName>
@return {Type}, @return {Type} return.subproperty@return [{type}] [description]
@scss-mixin Unsupported. No usage.
@since Ext JS 4.0 beta@since <versionDescription>
@singleton What to do? Custom tag?
@static@static
@template@template
@throws, @throws {Type}@throws {<type>} [free-form description]
@type {Type}, @type Type@type {<type>} [free-form description]
@uses ClassName@see <namepath> or @see <text>. Mapping isn't one-to-one.
@var, @var $some-name, @var {Type} $some-name, @var {Type} [$some-name="default value"]@member [<type>] [<name>]
@xtype Unsupported. One usage in ExtTab and otherwise only uses by ExtJSBase.
{@link Class#member link text}{@link namepathOrURL}, [link text]{@link namepathOrURL}, {@link namepathOrURL%7clink text}, {@link namepathOrURL link text (after the first space)}
{@img path/to/image.png alt text}Usupported. Only used by ExtJSBase.
{@video vimeo 465123 Some description here...} Unsupported. No usage.

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

For CI:

The npm and npm-stretch containers do have jsduck included. So we can change all repositories to add a "npm run-script doc" which would simply shell out to jsduck. Then developers can later change that 'doc' run-script to use jsdoc.

An example is mediawiki/core which triggers:

JobContainerCommand
mediawiki-core-jsduck-dockerreleng/jsduck containerjsduck
mediawiki-core-npm-node-6-dockerreleng/npmnpm test

When jsduck support is dropped from mediawiki/core we would instruct CI to stop triggering jsduck for the master branch (and we can keep it on the old REL branches).

Essentially, the change would make npm run-script doc the default entry point. Up to developers to use jsduck/jsdoc or whatever else.

Change 419941 abandoned by Jdlrobson:
Use npm run docs not jsduck

https://gerrit.wikimedia.org/r/419941

[..]

Essentially, the change would make npm run-script doc the default entry point. Up to developers to use jsduck/jsdoc or whatever else.

+1. Note that this was previously planned in 2015. Looks like we did it for some of the publish jobs, and for VisualEditor, but not others.

Change 496882 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] Update mwext-EventLogging postmerge from jsduck to generic node10

https://gerrit.wikimedia.org/r/496882

Change 496882 merged by jenkins-bot:
[integration/config@master] Update mwext-EventLogging postmerge from jsduck to generic node10

https://gerrit.wikimedia.org/r/496882

This is done for reading web extensions from what I can see.

If I'm not mistaken ContentTranslation and VisualEditor and core are remaining.

Since we're likely to need to touch core code as part of desktop refresh this seems in scope for that project.

Change 588454 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/core@master] Add jsdoc to core

https://gerrit.wikimedia.org/r/588454

Change 588457 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/core@master] Replace jsduck with jsdoc

https://gerrit.wikimedia.org/r/588457

If I'm not mistaken ContentTranslation and VisualEditor and core are remaining.

For what it is worth: Wikibase is also still in the jsduck era.

It is definitely not in scope of any Web team's project, but mentioning it here for completeness (given the pretty non-standard of the repository of Wikibase it might be easy to miss).

Change 588457 abandoned by Jdlrobson:
Replace jsduck with jsdoc

https://gerrit.wikimedia.org/r/588457

Change 590628 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] docs: Remove use of 'jqXHR' and 'mediaWiki' jsduck aliases

https://gerrit.wikimedia.org/r/590628

Change 590628 merged by jenkins-bot:
[mediawiki/core@master] docs: Remove use of 'jqXHR' and 'mediaWiki' jsduck aliases

https://gerrit.wikimedia.org/r/590628

JTannerWMF added a subscriber: JTannerWMF.

Is the Editing Team blocking anyone with this?

Is the Editing Team blocking anyone with this?

Technically, yes, we in Release Engineering would like to drop jsduck support from CI, so all work to migrate off jsduck is blocking us, but that's aspirational with no particular timeline.

Change 588454 abandoned by Jdlrobson:
[mediawiki/core@master] Add jsdoc to core, replace jsduck with jsdoc

Reason:
Not working on this now but honestly I think every day we leave not switching to jsduck we're making it harder to migrate. Please can somebody articulate on ticket what "broken" means as I am not seeing any problems with the jsdoc output with this patch and that would help move this conversation along.

https://gerrit.wikimedia.org/r/588454

Change 626428 had a related patch set uploaded (by DannyS712; owner: DannyS712):
[mediawiki/extensions/GlobalWatchlist@master] Switch documentation to use jsdoc

https://gerrit.wikimedia.org/r/626428

Change 626428 merged by jenkins-bot:
[mediawiki/extensions/GlobalWatchlist@master] Switch documentation to use jsdoc

https://gerrit.wikimedia.org/r/626428

Change 644289 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/TemplateData@master] Migrate from jsduck to jsdoc

https://gerrit.wikimedia.org/r/644289

Change 644289 merged by jenkins-bot:
[mediawiki/extensions/TemplateData@master] Migrate from jsduck to jsdoc

https://gerrit.wikimedia.org/r/644289

T138401#6315952

Just wanted to note, I have wasted quite a bit of time making jsdoc compatible code work with jsduck today, so there is a real developer cost to prolonging the use of jsdoc in core.

It's also frustrating as it's still not clear to me what we consider broken in the output of https://gerrit.wikimedia.org/r/588454 - the fact nobody is coming forward suggests that we either don't care about this documentation or there is no problem with it.

In short: by focusing on short term damage of migrating from jsduck to jsdoc, we are adding to long term damage by encouraging neglect of documentation in mediawiki core.

T138401#6315952

Just wanted to note, I have wasted quite a bit of time making jsdoc compatible code work with jsduck today, so there is a real developer cost to prolonging the use of jsdoc in core.

It's also frustrating as it's still not clear to me what we consider broken in the output of https://gerrit.wikimedia.org/r/588454 - the fact nobody is coming forward suggests that we either don't care about this documentation or there is no problem with it.

In short: by focusing on short term damage of migrating from jsduck to jsdoc, we are adding to long term damage by encouraging neglect of documentation in mediawiki core.

Maybe bringing this to the tech-decision-forum could help clarify a path forward?

Maybe bringing this to the tech-decision-forum could help clarify a path forward?

Maybe, or perhaps something more lightweight with leads in each of the different product teams and anybody else who may be interested?

It's also frustrating as it's still not clear to me what we consider broken in the output of https://gerrit.wikimedia.org/r/588454 […]

When run on MW, the output is broken in pretty much every way possible. The work by Prateek, CScott, and Volker got it to a state where it kind of worked for OOjs. I say "kind of", because it hosts only a few classes where most deficiencies we found from the Parsoid pilot, didn't apply, or were tolerable to OOjs as being a low-stakes project where a simple README suffices to understand most of its API. I accepted it anyway in OOjs since it allowed us to showcase the problems from a public URL, to then work on in a way that's visible and somewhat rewarding; instead of invisible and never released. It is, however, by no means ready. And for the first pilot, Parsoid, the output never made it to a point where it was correct. (Parsoid has since been ported to PHP/Doxygen)

The purpose of these documentation page is to allow new extension developers (incl staff), and gadget authors, to explore and understand the public APIs of our code base - in a way that must have a lower entry barrier and have added value, compared to finding and browsing the code from Git.

The MW output generated an unusable mess of HTML pages with no cohesion or navigability of any kind. No UI to access its categories, modules, or namespaces. Most methods are incorrectly associated or otherwise corrupted. It only just about passed the compiler without errors, but the output wasn't correct by means, let alone useful.

See also:

  • T138401#3054806.
  • And T262403: Search is broken.
  • And T207383: Some inherited methods are missing. Inheritance is one of the primary values doc output provides since view-source only shows you the implementation details of what the last mile added. A documentation page instead should tell you what you can do with an object, regardless of how where its code comes from.
  • And T262406: The grouping mechanisms don't appear to work (e.g. namespaces of classes, or modules). E.g. try to find mw.rcfilters and see which static methods and classes it houses, or try to find the mw.notification.convertmessagebox module and see what parameters the convertmessagebox function takes, or find "mediawiki.base" (not sure if a namespace, module, or category) and see which classes and methods it provides.
  • Plus various other issues explained earlier in this task by @Jhernandez and others. I haven't yet seen those addressed in any of the jsdoc-based outputs I checked on doc.wikimedia.org, including MobileFrontend which still uses the jsdoc default.

Those are just the issues I initially found on mwhen looking at the output for OOjs and Parsoid. We should expect to find more corruption bugs and UX defects as we try it out on larger and more diverse code bases. I'm sure we can handle those issue when we face them, none of this has been advanced or complicated so far. It just needs resourcing.

Hearing complaints for 5+ years, but little to no investments in a solution, has not been particilarly reassuring of us actually moving to a stack that isn't (also) abandoned.

I too want to see this move along. I think, if we want to see progress here, we need to get meaningful and on-going investment, to start addressing some of the most obvious defects. Ideally, with a team taking stewardship commitment to maintain the theme and triage incoming tasks. (The content will come; porting jsduck comments and/or writing new ones isn't our bottleneck).

Krinkle added a subscriber: bd808.

@Aklapper @bd808 This might be in scope for some of the work we've been talking about recently.

In a nut shell:

  • This task is about finding a replacement for the JSDuck tool, which currently generates MediaWiki core JS Documentation, VisualEditor Documentation, and most other library and MW extension docs for JavaScript, as served from doc.wikimedia.org.
  • JSDuck is a Ruby tool with an interpreter, generator, and page theme; all-in-one. But, it has been abandoned by its vendor many years ago, and while it had some minor bugs that we worked around, the most pressing issue is that it doesn't support ES 6 syntax, which in the upcoming year we expect to slowly start adopting natively in MediaWiki extensions and eventually core. (currently blocked on IE11 support).
  • The industry has largely embraced JSDoc3, which is an actively maintained open-source project with a healty ecosystem of plugins and adapters.
  • The issue is, however, that it is mostly a specification, with a popular parser. The part that users actually see is DIY. The default theme it bundles is fairly useless for something other than a single file project. There are some good themes out there probably. But we're leaning towards developing our own WMF-style theme (see live example of Wikimedia OOjs docs). This was a volunteer effort in 2017-2019 by CScott, Prateek, and Volker. It needs someone to finish it, and ideally also to give it some amount of on-going maintenance.

The purpose of these documentation page is to allow new extension developers (incl staff), and gadget authors, to explore and understand the public APIs of our code base - in a way that must have a lower entry barrier and have added value, compared to finding and browsing the code from Git.

I guess this captures our disconnect. The documentation output for me is probably the least important part of me as a developer (by a long way) and the documentation in current form is pretty useless and plays no part in my team's onbording. I am curious what kind of page views it gets.

JSDoc empowers developer productivity in that it gives typing hints in Code Editor, support for TypeScript (even without an extension) as well as more expression as a developer writing inline developer. I value all these things more highly as I use those to navigate around codebases and write good code.

It sounds like getting a shared understanding of the problem is a good first step.. I have no idea what "broken" means in practice for docs.wikimedia.org as I do not use them and even within Joaquin's analysis it's never been clarified which of those are showstoppers by the engineers that use that documentation. Who is using this documentation? Please speak up!

If we can't make progress here, it sounds like involved parties are coming from different places hence the lack of progress. I think the suggestion from @kostajh is a really solid one here and probably the best next step here if we want to move things forward T138401#6915662.

@phuedx had some thoughts when I brought this up. Care to share?