Page MenuHomePhabricator

Replace jsduck with JSDoc3 across all Wikimedia code bases
Open, HighPublic

Description

JSDuck seems abandonware by now, and it is hurting our ability to properly document our code.

  1. Examples of problems:
    • No support for @interface
    • No support for namespaced events (change:title is invalid)
  2. Links that show it is unmaintained:
  3. Proposed alternatives:

Requirements for alternative

What are blockers for adopting JSDoc3 and/or a custom theme? In other words, compared to viewing code comments directly, what added value do we want our generated documentation to have for consumers?

TDB, see T187672: Create Wikimedia theme for JSDoc3 (version 1)

AC

  • All usage of jsduck changed to jsdoc from npm
  • Code documentation works fine, fix any problems
  • Docs get properly published to https://doc.wikimedia.org/MobileFrontend/master/js/
  • CI lints documentation properly with jenkins-bot on patches
  • Documentation about how to run documentation in mw.org, README, etc is updated to point to the new instructions if any change
  • All the above ^ but for other extensions we maintain and that use jsduck

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Krinkle added a comment.EditedFeb 25 2017, 3:43 AM

Here's a summary of some of JSDuck's features that I believe made it our choice originally:

  • User-friendly interface.
  • Search feature. (Client-side, serverless)
  • Stable permalinks to classes and methods.
  • Support Markdown for all text content.
  • Easy discovery through a browsable class hierarchy. (e.g. Sidebar shows "mw.Title" class nested in "mw" singleton)
  • Support for class inheritance and mixins, and pages include inherited members by default.
  • Support for singletons.
  • Support for private members.
  • Support for static members.
  • Support for events.
  • Support for documenting which events are emitted by a method. (@fires)
  • Support for private classes.

These less-essential features are also quite nice:

  • Interface allows toggling private members and private classes (and hidden by default).
  • Interface shows in which file(s) a class is defined.
  • Each class page show class hierarchy and mixins, in both directions.
  • Live examples using @example that make for interactive demos in the context of documentation. Especially useful to help understand how UI library works and which widgets to use.
  • Classes organised by group, on the homepage. ("Categories")
  • Easy to extend with additional tags. (@context, @see, @since, etc.)
  • Intelligent linking to mentioned classes and methods. ("Foo bar #baz")
  • Support for documenting properties of parameters that are objects.
  • Support for documenting properties of returned objects.
  • Support for documenting callback parameters.
  • Support for documenting upstream code with pure comments. ("jsduck.external.js")
  • Support for class aliases.
  • Type validation (unknown param or return type produces an error.)

Of course, we've also come across various limitations over time:

  • Event names cannot contain dots. jsduck #497 (wikipage.content documented as wikipage_content)
  • There cannot be a class and singleton with the same case-insensitive name. jsduck #304 (mw.notification and mw.Notification documented as mw.Notification_)
  • No concept of "modules", which would allow grouping classes together. As well as allowing methods to be documented as being part of a module without having a global class (e.g. module.exports). jsduck #553
  • No longer actively maintained.
  • Written for Ruby instead of Node.js.
  • No support for ES6 syntax. T156469. jsduck #630

Thanks for the list @Krinkle, I've updated the description of the spike T146917 to see if jsdoc supports all needed or what is it missing.

Are there any other documentation generators for jsdoc syntax in JS that we should be checking out?

Not sure if it satisfies all needs, but could consider documentationjs (supports jsdoc) as an alternative too.

Thanks @Prtksxna, I've had a look and it doesn't seem to depend on jsdoc the package so it is a different alternative. I'll update the description.

Jhernandez renamed this task from Replace jsduck in favor of a better maintained alternative (jsdoc) to Replace jsduck in favor of a better maintained alternative.Mar 9 2017, 10:51 AM
Jhernandez updated the task description. (Show Details)

Front-end-Standards-Group just to confirm - did we officially decide to move from jsduck to jsdoc or is this still being discussed (I see it's moved to done on that board)?

Let's resume this conversation. @Jhernandez has written an excellent comparison of jsduck vs jsdoc here: https://www.mediawiki.org/wiki/User:JHernandez_(WMF)/JS_documentation_tools

Let's resume this conversation. @Jhernandez has written an excellent comparison of jsduck vs jsdoc here: https://www.mediawiki.org/wiki/User:JHernandez_(WMF)/JS_documentation_tools

Really, excellent comparison indeed. Thanks @Jhernandez!

Volker_E changed the task status from Stalled to Open.Jun 14 2017, 6:17 PM
Volker_E removed Volker_E as the assignee of this task.
Volker_E added a subscriber: Volker_E.
Krinkle updated the task description. (Show Details)Jun 29 2017, 8:50 PM
Jdlrobson moved this task from Backlog to Tracking on the MobileFrontend board.Jul 13 2017, 5:37 PM

How do we decide to accept @Jhernandez's recommendation? Seems good to me…

cscott added a subscriber: cscott.Dec 8 2017, 10:07 PM

Just ran into this for Parsoid -- we want to start using ES6 class syntax in our codebase but (surprise) jsduck doesn't support it, as @Krinkle mentioned above (T156469). Is @Jhernandez's suggestion JsDoc or documentation.js? I note that documentation.js seems to have commits from this month, while jsdoc hasn't had a commit to its repo since September.... but I agree that the jsdoc output looks nicer.

I did my best to summarize the information at hand when I wrote https://www.mediawiki.org/wiki/User:JHernandez_(WMF)/JS_documentation_tools#Which_documentation_format_to_use_(spec_on_comment_syntax_and_tags_for_documenting_code) and https://www.mediawiki.org/wiki/User:JHernandez_(WMF)/JS_documentation_tools#Which_tool_and/or_template_to_use_to_generate_readable_HTML_docs

It seems pretty clear that we should migrate to JSDoc comment syntax, as it has a spec, and competing implementations.

On the other hand, docs to website generators seem to have fallen out of
fashion and there aren't good solid open source options.

TLDR:

  • JSDoc3: doesn't seem terribly well maintained, and templates aren't very fully featured
  • Documentation: Has a nice feature set, I discarded it because it didn't seem to have many contributors. May need to be re-evaluated

So, either we settle with a simple JSDoc template, or we fork and improve one, or we use Documentation.js (if someone can have a look at it and asses it, it would be great)

Either way, if we migrate our comments to the standard JSDoc syntax, we can switch web generators as needed in the future.

@Jhernandez, after reading this blog post by documentationjs' maintainer I am not sure about its roadmap. I have also dabbled with changing the documentationjs template style, and did not find it straightforward (might be my own limitations).

I definitely lean towards your recommendation of using JsDoc3.

In today's Front-end group meeting we've been picking up this topic again and JSDoc 3 or at least JSDoc syntax is the agreed-way to go.
@Jdlrobson has been asking for a starting motivation where we concluded that ES6 syntax is such.
@Krinkle has brought up a new though, that although JSDuck has been discontinued, we haven't had any maintenance issue due to it's being Ruby based for a long time – something that might change to the negative with a Node.js-based solution.
We still want to go that way with all it's advantages altogether.

@cscott would you be willing to try JSDoc syntax to render its needed changes on CI visible before we're stepping onto a bigger project like MediaWiki core?

From the FrontEndStandardsGroup etherpad:

@Krinkle: Start with a smaller project like OOjs core or unicode.js

I think that makes sense. I've been meaning to test this with OOUI (not as familiar with OOjs core), so I finally did. I made a few code changes according to the new syntax:

  1. Use @mixes instead of @mixins
  2. Mark each mixin with @mixin
  3. There is no support for @cfg, so, using @param {Type} [config.option] instead.
  4. I haven't fully understood how @event works. Is everything in the global namespace by default? I tried to namespace the events, for SelectWidget and they show up correctly. But I would like to hear from someone who understands this better.

There are some features that I found missing. Most advise I could find said that you could write a template that supports all of this, but it's not given out of the box, or in the default theme.

  1. Custom Table of Contents. At the minimum, for OOUI we need, Widgets, Dialogs, Layouts, etc.
  2. No built-in support for search.
  3. No support to show/hide protected/private/inherited methods. You can hide/show private methods by passing a config flag to the CLI.

You can see the OOUI documentation in the default theme on https://prtksxna.github.io/oojs-ui/ . For OOUI we'll definitely need our own theme, I was thinking we could do something similar to the style guide (something @Volker_E and Design has been polishing). That way all our documentation could have a similar design. Creating custom layouts for JSDoc seems pretty straightforward too: https://rebel-sisters.surge.sh/ (WIP). Thanks for introducing me to surge.sh @Jhernandez

Nice work! The events seem a bit screwed up as you say (https://prtksxna.github.io/oojs-ui/global.html#event:changeAchangeeventisemittedwhentheon/offstateofthetogglechanges. etc.) but this is almost good enough to migrate to right now (and re-skin in time).

There are some features that I found missing. [..]
You can see the OOUI documentation in the default theme on https://prtksxna.github.io/oojs-ui/

Nice work! The events seem a bit screwed up [..]

Indeed! Looks pretty good overall. Noticed one other issue, which is that @property seems to be meant for sub-sub-properties somehow. Documenting /** @property {jQuery} */ this.$overlay = $(); results in the system documenting a member named $overlay that has a sub-property named jQuery. Something didn't go right there.

but this is almost good enough to migrate to right now (and re-skin in time).

Could we articulate what needs fixing? @Prtksxna would be willing to submit a patch of what you've done so far? (No matter if it's a little raw :))

but this is almost good enough to migrate to right now (and re-skin in time).

Could we articulate what needs fixing? @Prtksxna would be willing to submit a patch of what you've done so far? (No matter if it's a little raw :))

The @event naming that I mentioned and the @property documentation that Timo mentioned are I think it.

Jdforrester-WMF renamed this task from Replace jsduck in favor of a better maintained alternative to Replace jsduck with JSDoc3 across all Wikimedia code bases.Jan 20 2018, 8:37 PM
Jdforrester-WMF added a project: Epic.
Jdforrester-WMF updated the task description. (Show Details)
  1. Mark each mixin with @mixin
  2. There is no support for @cfg, so, using @param {Type} [config.option] instead.

That seems to work, for @cfg you can use @property, see http://usejsdoc.org/tags-property.html and an example of output here http://mobilefrontend-jsdoc-test.surge.sh/Overlay.html#defaults

  1. I haven't fully understood how @event works. Is everything in the global namespace by default? I tried to namespace the events, for SelectWidget and they show up correctly. But I would like to hear from someone who understands this better.

Yes! As you mention, events are global by default (namespaced under event:), you can use that notation @event <className>#[event:]<eventName>. I think you did it correctly

(for reference: http://usejsdoc.org/tags-event.html)

There are some features that I found missing. Most advise I could find said that you could write a template that supports all of this, but it's not given out of the box, or in the default theme.

  1. Custom Table of Contents. At the minimum, for OOUI we need, Widgets, Dialogs, Layouts, etc.
  2. No built-in support for search.
  3. No support to show/hide protected/private/inherited methods. You can hide/show private methods by passing a config flag to the CLI.

Same issues I found. Something like docstrap has search and some other features but I didn't find any one that had everything we needed out of the box.

Here's a few from 1 to try out and maybe fork:


Nice try @Prtksxna

Indeed! Looks pretty good overall. Noticed one other issue, which is that @property seems to be meant for sub-sub-properties somehow. Documenting /** @property {jQuery} */ this.$overlay = $(); results in the system documenting a member named $overlay that has a sub-property named jQuery. Something didn't go right there.

Class properties/fields should be automatically picked up, and they can be forced with memberof, but should not be annotated with property, as you mentioned, which is for object (sub)properties and not class fields.


The events stuff seems to be fine if you follow the convention for the event names Class#eventName, but if you don't specify a class then they will be put in the global namespace, so it is definitely something to look out for when migrating.

cscott added a comment.Feb 5 2018, 6:48 PM

but this is almost good enough to migrate to right now (and re-skin in time).

Could we articulate what needs fixing? @Prtksxna would be willing to submit a patch of what you've done so far? (No matter if it's a little raw :))

The @event naming that I mentioned and the @property documentation that Timo mentioned are I think it.

Custom Table of Contents would be good to have. Parsoid also uses this to separate out the "public" parsoid API from the plethora of internal implementation classes.

Change 408455 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/services/parsoid@master] WIP: Switch from jsduck to jsdoc3 for documentation

https://gerrit.wikimedia.org/r/408455

Here's a few from 1 to try out and maybe fork:

I did try these, and several more (see the added packages in https://gerrit.wikimedia.org/r/#/c/405543/7/package.json) but did not like the results of any. Rather, none of them had everything.

cscott added a comment.EditedFeb 7 2018, 10:03 PM

You can see the OOUI documentation in the default theme on https://prtksxna.github.io/oojs-ui/ . For OOUI we'll definitely need our own theme, I was thinking we could do something similar to the style guide (something @Volker_E and Design has been polishing). That way all our documentation could have a similar design. Creating custom layouts for JSDoc seems pretty straightforward too: https://rebel-sisters.surge.sh/ (WIP). Thanks for introducing me to surge.sh @Jhernandez

@Prtksxna Could you push the WIP theme that you used for rebel-sisters.surge.sh somewhere? I've got a WIP patch to replace jsduck with jsdoc3 for Parsoid, but the default theme is *ugly*. And I'd like to try hacking up your custom theme to allow custom Table of Content / grouping categories, since that seems like the thing the default themes are missing.

Change 409422 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/services/parsoid@master] Use new jsdoc-wmf-theme for documentation

https://gerrit.wikimedia.org/r/409422

Change 408455 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Switch from jsduck to jsdoc3 for documentation

https://gerrit.wikimedia.org/r/408455

Change 409422 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Use new jsdoc-wmf-theme for documentation

https://gerrit.wikimedia.org/r/409422

https://github.com/jsdoc3/jsdoc/issues/483 is a problem with jsdoc -- it currently doesn't resolve {@link Foo} or {@link #bar} -- you have to spell out the full name of the class, including module name and full class name, every time if you want it to be linked. It may be possible to resolve this in the jsdoc 'theme'.

My jsdoc-wmf-theme does manage to resolve a number of these @link shortcuts.

Prtksxna added a comment.EditedFeb 19 2018, 3:52 AM

@Prtksxna Could you push the WIP theme that you used for rebel-sisters.surge.sh somewhere? I've got a WIP patch to replace jsduck with jsdoc3 for Parsoid, but the default theme is *ugly*. And I'd like to try hacking up your custom theme to allow custom Table of Content / grouping categories, since that seems like the thing the default themes are missing.

Sorry @cscott! For rebel-sisters I had created the theme as a hack in node_modules, and didn't back it up before doing a routine rm -rf node_modules && npm install. Not sure how to get it from surge.sh either! Your theme definitely looks much better anyway

Also, see T187672: Create Wikimedia theme for JSDoc3 (version 1)

cscott added a comment.EditedFeb 20 2018, 9:53 PM

My theme broke today because I was hotlinking to the WikimediaUI-Style-Guide github repo. But it should be fixed as soon as https://gerrit.wikimedia.org/r/410208 is merged.

(The fix was https://github.com/cscott/jsdoc-wmf-theme/commit/58b153aac76fbc6de73fe4cc50705532017eb5a4 )

This is what it should look like: http://tidy-club.surge.sh/

Change 419846 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/skins/MinervaNeue@master] Use jsdoc for generating Minerva documentation

https://gerrit.wikimedia.org/r/419846

Change 419846 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] Use jsdoc for generating Minerva documentation

https://gerrit.wikimedia.org/r/419846

Change 419941 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[integration/config@master] Use npm run docs not jsduck

https://gerrit.wikimedia.org/r/419941

^^ So we have adopted jsdoc inside Minerva. It looks like Minerva uses a Jenkins job called mwext-jsduck-publish to publish documentation which is now failing.

I thought we generalised documentation generation to be made when npm run doc was run, but looking at the logs ? Is there another job we should be using @hashar and @zeljkofilipin ?

For CI:

The npm and npm-stretch containers do have jsduck included. So we can change all repositories to add a "npm run-script doc" which would simply shell out to jsduck. Then developers can later change that 'doc' run-script to use jsdoc.

An example is mediawiki/core which triggers:

JobContainerCommand
mediawiki-core-jsduck-dockerreleng/jsduck containerjsduck
mediawiki-core-npm-node-6-dockerreleng/npmnpm test

When jsduck support is dropped from mediawiki/core we would instruct CI to stop triggering jsduck for the master branch (and we can keep it on the old REL branches).

Essentially, the change would make npm run-script doc the default entry point. Up to developers to use jsduck/jsdoc or whatever else.

Change 419941 abandoned by Jdlrobson:
Use npm run docs not jsduck

https://gerrit.wikimedia.org/r/419941

So should I open a new ticket?

[..]
Essentially, the change would make npm run-script doc the default entry point. Up to developers to use jsduck/jsdoc or whatever else.

+1. Note that this was previously planned in 2015. Looks like we did it for some of the publish jobs, and for VisualEditor, but not others.

Krinkle updated the task description. (Show Details)Mar 16 2018, 9:37 PM
Volker_E updated the task description. (Show Details)Jan 7 2019, 10:48 PM

Change 496882 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] Update mwext-EventLogging postmerge from jsduck to generic node10

https://gerrit.wikimedia.org/r/496882

Change 496882 merged by jenkins-bot:
[integration/config@master] Update mwext-EventLogging postmerge from jsduck to generic node10

https://gerrit.wikimedia.org/r/496882

phuedx removed a subscriber: phuedx.Mar 18 2019, 9:21 AM