Page MenuHomePhabricator

Replace jsduck with JSDoc3 across all Wikimedia code bases
Open, HighPublic

Assigned To
Authored By
Jhernandez
Jun 22 2016, 2:12 PM
Referenced Files
F41522432: Screenshot 2023-11-20 at 5.42.55 PM.png
Nov 21 2023, 1:43 AM
F12518296: Screen Shot 2018-01-12 at 22.58.24.png
Jan 12 2018, 10:58 PM
F12518294: Screen Shot 2018-01-12 at 22.58.10.png
Jan 12 2018, 10:58 PM
Tokens
"Love" token, awarded by Bmueller."Like" token, awarded by SCherukuwada."Orange Medal" token, awarded by Aklapper."100" token, awarded by MusikAnimal."Barnstar" token, awarded by Sportzpikachu."100" token, awarded by Niedzielski."Orange Medal" token, awarded by Krinkle.

Description

JSDuck is formally unmaintained and seems abandonware by now. It is hurting our ability to properly document our code.

  1. Examples of problems:
    • No support for @interface
    • No support for namespaced events (change:title is invalid)
    • No support for ES6 (T156469)
  2. Links that show it is unmaintained:
  3. Proposed alternatives:

Migration plan

  1. Research: Start researching who uses the JS HTML docs and what features are important to them, including:
  1. Scope improvements to the WMF theme: Informed by user feedback and documentation best practices, compile a list of critical usability fixes in the form of Phabricator tasks with clear descriptions and acceptance criteria. See T187672: Release jsdoc-wmf-theme 1.0 (Wikimedia theme for JSDoc3) and T138401#9079509.
  1. Consider temporary mitigations: To address concerns about the impact to documentation usability while work on the theme is in progress, we looked into options to help mitigate that impact in the short term. For MediaWiki core, we decided to use the MediaWiki version 1.41 JSDuck docs as a backup while the migration is in progress with the goal of completing the migration in time for version 1.42. In addition, after looking into a few other open-source theme options, we decided to move forward with the WMF theme.

Codebases to port to JSDoc

Checklist for each migration

  • Code documentation works fine, fix any problems
  • Docs get properly published to docs.wikimedia.org e.g. https://doc.wikimedia.org/MobileFrontend/master/js/
  • CI lints documentation properly with jenkins-bot on patches
  • Documentation about how to run documentation in mw.org, README, etc is updated to point to the new instructions if any change

Migration guide

For how to set up JSDoc, see https://www.mediawiki.org/wiki/JSDoc.

JSDuck to JSDoc mapping

JSDuck tagJSDoc tag and notes
@abstract@abstract
@accessor Unsupported. Only used by ExtJSBase.
@alias prefix.name@alias aliasNamepath
@alternateClassName OtherClassName What to do? @alias aliasNamepath? @typedef {type} namepath? Sparse usage.
@aside name Unsupported. No usage.
@author Some name...@author name <emailAddress>
@cfg, @cfg name, @cfg {Type} name, @cfg {Type} [name="default value"], @cfg {Type} name.subpropertyUse @param, or use`@typedef` for shared types
@chainableUse @return {Type} where {Type} refers to the memberof class of the function instead.
@class, @class ClassName@class name
@constructor@return {type} description
@deprecated@deprecated
@docauthor Some name... Unsupported and unused.
@enum, @enum {Type}, @enum {Type} EnumName, @enum [EnumName=alias.*]@enum {type}
@event, @event name@event ~'className.eventName' (example: ~'util.addPortlet')
@evented Unsupported. Only used by ExtJSBase.
@example@example
@experimental, @experimental 2.0 Some description...@experimental
@extends ParentClassName@extends namepath
@fires eventName (in 5.x beta)For hooks: @fires Hooks~'className.eventName' (example:Hooks~'util.addPortletLink')
@ftype name Unsupported. No usage.
@hide Unsupported. Only used by ExtJSBase.
@ignore@ignore
@inheritable What to do? Delete?
@inheritdoc, @inheritdoc ClassName, @inheritdoc #memberName, @inheritdoc ClassName#memberName, @inheritdoc ClassName#static-type-memberNameWhat to do? If no value, @inheritdoc. If value, @extends value?
@localdoc This documentation is only visible inside this class/member. (in 5.x beta) What to do? Delete?
@markdown Unsupported. No usage.
@member ClassName What to do? @member {type} name, @memberof parentNamepath @memberof! parentNamepath, or something else?
@method, @method name@method FunctionName
@mixins ClassName@mixes OtherObjectPath
@new Unsupported. No usage.
@override OverriddenClassName@override
@param name, @param {Type} name, @param {Type} [name], @param {Type} [name="default-value"], @param {Type} name.subproperty@param {type} name description. For an optional parameter: @param {type} [name] description or @param {type} [name=default value] description. See jsdoc.app.
@preventable Unsupported. Only used by ExtJSBase.
@private@private [{typeExpression}]
@property, @property name, @property {Type} name, @property {Type} [name="default value"], @property {Type} name.subproperty@property {type} name description. For an optional property: @property {type} [name] description
@protected@protected [{typeExpression}]
@ptype name Unsupported. No usage.
@readonly@readonly
@removed, @removed 2.0 Some description... Unsupported. One usage in SemanticMediaWiki and otherwise only used by ExtJSBase.
@requires ClassName@requires someModuleName
@return {Type}, @return {Type} return.subproperty (Multiple @return tags )Replace with single @return {Type} description and use @typedef {Object} Type to document the type.
@scss-mixin Unsupported. No usage.
@since Ext JS 4.0 beta@since versionDescription
@singleton What to do? Custom tag?
@static@static
@template@template
@throws, @throws {Type}@throws {type} free-form description
@type {Type}, @type Type@type {type} free-form description
@uses ClassName@see namepath or @see text. Mapping isn't one-to-one.
@var, @var $some-name, @var {Type} $some-name, @var {Type} [$some-name="default value"]@member {type} name
@xtype Unsupported. One usage in ExtTab and otherwise only uses by ExtJSBase.
{@link Class#member link text}{@link namepathOrURL}, [link text]{@link namepathOrURL}, {@link namepathOrURL%7clink text}, {@link namepathOrURL link text (after the first space)}
{@img path/to/image.png alt text}Usupported. Only used by ExtJSBase.
{@video vimeo 465123 Some description here...} Unsupported. No usage.

Related Objects

StatusSubtypeAssignedTask
OpenFeatureNone
Resolvedhashar
OpenKBach
Resolved Jhernandez
DuplicateNone
ResolvedKrinkle
Openapaskulin
OpenNone
ResolvedEsanders
ResolvedEsanders
ResolvedBUG REPORTKrinkle
ResolvedJdlrobson
DuplicateNone
DuplicateNone
DuplicateNone
Resolved nray
OpenNone
Resolvedmatmarex
DeclinedNone
Resolvedcscott
ResolvedTheDJ
ResolvedSamwilson
ResolvedNone
OpenNone
DeclinedNone
Resolvedapaskulin
ResolvedKrinkle
Resolvedmatmarex
Resolvedapaskulin
DeclinedBUG REPORTJdlrobson
Resolvedapaskulin
Resolvedegardner

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Today web team spent an hour debugging an issue with getting our code compatible with jsduck (again the problem was ES6 support - T138401#8817748) . This is getting more confusing as we adopt ES6 in more places and to me is becoming more urgent to fix than the docs. I have suggested this as a topic for the next meeting of the frontend standards group.

I've also found JSDuck to be increasingly troublesome recently. For example, an attempt to use a tiny bit of ES6 features in OOUI had to be reverted (see https://gerrit.wikimedia.org/r/c/mediawiki/core/+/935793/comments/c78f4d1a_14d9f167, https://gerrit.wikimedia.org/r/c/oojs/ui/+/935861).

If we can't get JSDoc to match the quality of its output… I think we need to lower our standards.

This is on the agenda for the frontend standards group meeting in 40 mins time.

I was getting the following error today when trying to run jsduck (5.3.4): jsduck.json is not a valid JSON file. The file wasn't the problem, I think my version of Ruby or some Gem was causing it — the actual error turned out to be:

gems/json-1.8.6/lib/json/common.rb:155:in `initialize': wrong number of arguments (given 2, expected 1) (ArgumentError)
        from /home/sam/gems/gems/json-1.8.6/lib/json/common.rb:155:in `new'
        from /home/sam/gems/gems/json-1.8.6/lib/json/common.rb:155:in `parse'
        from /home/sam/gems/gems/jsduck-5.3.4/lib/jsduck/util/json.rb:46:in `read'

which I solved by hacking json-1.8.6/lib/json/common.rb:155 to remove opts from Parser.new(source, opts).parse.

Then I could get back to debugging the actual ES6 issues (never use let, a rule that is helpfully expressed as "parser.y:606:in `_reduce_273': compile error (RKelly::SyntaxError)").

(Sorry, I know I should probably avoid the above detail, but this is just me adding my +1 to switching to JSDoc, and hopefully helping anyone running into the same issue I was.)

Change 942004 had a related patch set uploaded (by Thiemo Kreuz (WMDE); author: Thiemo Kreuz (WMDE)):

[mediawiki/extensions/Kartographer@master] Revert "Revert "Port jsduck -> jsdoc""

https://gerrit.wikimedia.org/r/942004

@Jdlrobson Did the frontend standards group meeting come to a conclusion, or did it bring up any new insights not already shared in this ticket?

Summarizing:

We agree that we need to replace JSDuck with Jsdoc

  • JSDuck is getting in the way of of adopting ES6 in MediaWiki core
  • JSDuck is no longer maintained.
  • There seemed no concerns with doing it. Just that it takes time.

Hypothesis: documentation is important to 3rd parties

  • Documentation pages at docs.wikimedia.org are viewed as important to our gadget developers/3rd parties
  • However they are not owned by any team.
  • We don't know how many page views they get
  • We don't know how the consumer uses it as we don't hear from people that use the documentation
    • perhaps they build them locally or use docs.wikimedia.org
  • It's not clear if in current form they are useful - they document every function rather than functions that are core APIs

We don't want to degrade documentation pages and the change is not trivial

  • Switching from JSDuck to jsdoc initially would reduce the value of the documentation. A lot of this can be fixed easily by familiarizing ourselves with the differences in syntax but this would be a big time suck and lots of manually string replacements.
  • A concern was raised that the default jsdoc theme is not as good to use as the currently used JSDuck theme. We couldn't decide amongst the small group discussing whether this was a blocker.
  • There seemed to be no clear definition of what "good enough" looks like. For example it wasn't clear if the theme was a blocker
  • It would be useful to have a team owning this work and making product decisions about what is needed as the work is non-trivial

Next steps

  • Ideally a team should own docs.wikimedia.org. @Krinkle suggested tech advocacy team might be a good fit and was going to explore if that was feasible.
  • Work seems doable it just seems like a focused effort rather than 2 people working on it part time while juggling whatever their current project is. I think it's 1-3 weeks work for 2 or 3 engineers ideally with product support accountable for any decisions that need to be made.
  • If we can't spin up a small, I did talk to @Catrope about how it might be a good inspiration week project (but not sure exactly when that would be)

I figured I'd document my personal gripes about this jsdoc skin.

Things that could be improved about the theme:

  • Make private/protected/inherited checkboxes work
    • It seems that this only applies to methods, not to members/properties, This is confusing, as the UI is presented at the top of the page, not at the methods level. We should move that control down, or make it also apply to member/properties
  • The parameters table is hard to read This was actually solved in 0.0.6 version of the plugin. Though it should be noted that the props table should probably styled as well.
    • The indentation is not aligning
    • The descriptions of parameters gets very narrow due to the reduced width
    • The documentation of the properties of an object is even harder to read.
    • The headings of the paramaters table could be left aligned
    • The values of the parameter cells should be valign-top
  • Not really clear how to document RL modules
    • One of the problems here is that @module's can't really have periods in their name, because that creates namespaces. @borrows only creates an alias, and doesn't document the name very well. In TMH I used - instead of . for the module names. Another possibility is using namespaces perhaps.
  • Not really clear how to document events and triggers
  • Methods etc are collapsed, which makes the documentation less scannable if there are no summaries provided.
    • Could be fixable with autosummaries
    • The font size is also pretty large, requiring significant scrolling
    • And there is no symbol ToC list to scan as mentioned below either
  • No property/method table of contents/symbols T354763
  • No link back to the other documentation T353778
  • When you enter, you land on Home, and the other sections (module, classes etc) are collapsed. This reduces scan-ability of the page and I suggest we uncollapse by default.
  • The elements on the Class page are all collapsed, and you need to click the triangle to disclose them. Suggest we make the entire line clickable to open up the description
  • The flags for private, protected, abstract, etc are all text and in front of the definition. I think we can improve the readability of these design wise.
  • The way to navigate to the source takes up a lot of space. Suggest making it smaller and aligning to the right or something
  • Not working search
    • Actually fixed in 0.0.6
  • Classes do not show inherited/super properties/methods (nice to have)
    • This possibly has to do with how some of our classes that we are working with are this ES5 era extends mess.
  • When there is no @summary, the skin generates ... for a method when collapsed, which just makes things less readable in my opinion.
  • We should add a auto summary option by default, like I have in TMH
  • The contribute link in the footer seems broken T353502

I've started https://www.mediawiki.org/wiki/JSDoc with the intent that at some point it can replace the JSDuck instructions.

Folks, here's a plan of action.

Talking to various people involved here, there seem to be two problems we need to solve here:
A) the bit around porting code documentation from JSDuck->JSDoc, unlocking ES6, and flipping the switch and
B) the bit around the impact of this change on doc.wikimedia.org, some decisions around how to decide what it needs to look like, what the JSDoc theme needs to do, etc.

The Tech Docs team have kindly offered to take on ownership of doc.wikimedia.org. In other words, they will provide guidance around (B) and decide on the priorities and requirements for public documentation.

The next steps could be the following:

  • Some folks from the Frontend Standards Group will meet with the Tech Docs team and come up with a timeline and list of tasks for switching over to JSDoc.
  • They'll list what needs to be done prior to the switchover, what will break after the switchover and what needs to be fixed over time (a lot of this has already been discussed here) and report back on this ticket. It will also be useful to know what needs to be done in one shot as a sprint and what can be done piecemeal over time.

Assigning this task to myself as I'll be working on this from the Tech Docs team's side. I'll be reaching out to different individuals involved in this task shortly.

In JSDoc issue #669, someone brings up a project that uses @returns {<class name>} this, chainable (where <class name> is the name of the class the function returns) to represent chainable functions
There's also PR#1162 that would add it, which has been under review for 7 years.

Hi all, following up on @Jdlrobson's comment in T138401#9078948, here's what the Tech Docs team would like to propose to move this forward:

  • We don't know how many page views they get
  • We don't know how the consumer uses it as we don't hear from people that use the documentation
  1. Research: Start researching who uses the JS HTML docs and what features are important to them, including:
    • Attempt to query pageview data for JS HTML docs from the webrequest tables in Hive
    • Put out a call for on-wiki feedback through relevant channels

A concern was raised that the default jsdoc theme is not as good to use as the currently used JSDuck theme.

I agree that, in its current state, the WMF JSDoc theme has some usability issues (such as T262406 and T207383) that prevent it from being a functional replacement for the JSDuck docs. (I don't mean this as a call out of the people who have worked on the theme by any means. There are a lot of really good things about the theme as well.)

  1. Scope improvements to the WMF theme: Informed by user feedback and documentation best practices, compile a list of critical usability fixes in the form of Phabricator tasks with clear descriptions and acceptance criteria. There's already been a lot of good work done on this, most recently TheDJ's comment in T138401#9079509. The expectation would be that some WMF engineering time will be made available to work on these fixes within a reasonable timeframe (such as the 2024 calendar year or the 2024-25 fiscal year) either in the form of development time or Outreachy mentorship time. @SCherukuwada would be responsible for signing off on engineering time commitment relative to the scope of the fixes.

There seemed to be no clear definition of what "good enough" looks like. For example it wasn't clear if the theme was a blocker

It seems like delaying JSDoc migration is causing issues for some projects. (See T138401#8045282, T138401#9009746, and T138401#9030517.) As long as we can get a commitment to dedicating time to work on the theme, I think we can give projects currently using JSDuck the go-ahead to migrate to JSDoc.

  1. Consider temporary mitigations: To address concerns about the impact to documentation usability while work on the theme is in progress, look into options to help mitigate that impact in the short term. For example:
    • Temporarily use another open-source JSDoc theme, such as clean-jsdoc-theme or better-docs
    • Serve static snapshots of the JSDuck docs alongside the JSDoc docs. (Although I'm not sure of the technical feasibility of this, and it does introduce extra risk, complexity, and effort.)

Feedback on these plans is welcome! Particularly if anyone has ideas about the technical feasibility of temporarily serving static snapshots of the JSDuck docs. The immediate next steps would be to open subtasks to capture the work described here.

  1. Really looking forward to that actually !
  2. I think some of the most critical issues are already dealt with since my original comment, but I've also discovered some new issues. I'm just going to add those to my previous comment, to keep stuff together.
  3. I think it will be difficult to serve both. At most, you could keep a copy of old state at a separate path ?
  4. Besides the 3 mentioned points I think you should add an effort to expand on https://www.mediawiki.org/wiki/JSDoc
  5. Perhaps do some social media outreach on ES6 documentation, to tickle some outsiders and get some stories of 3rd parties experiences with setting up HTML output of JSDoc for products ?

I do think that a lot has to do with giving people good examples of how to solve more complex situations. We can already see how having multiple extensions converted has helped people along.

The biggest lessons I have discovered so far are:

  1. JSdoc is not that good with Typescript (it can't infer some of the structure that TS adds)
  2. JSDoc is not good with ES5, as you need to provide annotations for literally everything
  3. Do not over annotate ES6 with annotations that can be inferred from the structure.

@hashar could you help us preserve the current version (e.g. static cp of the directory) of https://doc.wikimedia.org/mediawiki-core/master/js/ (is that possible?) so that we can point to it in the JSDoc version and begin this migration?
e.g. something like that content published to https://doc.wikimedia.org/archive/mediawiki-core/master/js/ ?

Change 975921 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/core@master] WIP: Add jsdoc with jsdoc-wmf-theme

https://gerrit.wikimedia.org/r/975921

@TheDJ @Catrope @AnneT I started a skeleton for what a first patch could look like: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/975921
I limited the documentation to mediawiki.util module since it's widely used, complex and hopefully well understood.

Hopefully the output is self explanatory (please note outstanding TODOs in the commit message)

Screenshot 2023-11-20 at 5.42.55 PM.png (703×1 px, 116 KB)

I've merged Jon's patch. The next steps for jsdoc in core are:

  • Archive the old jsduck docs, per T351764#9353954, and link to that archive in resources/README.md
    • Right now this link points to the REL1_41 docs; we could also decide that that's good enough, and decline T351764
  • Actually remove jsduck from core. Jon's patch deactivates it, but doesn't remove its configuration files or various other things
  • Convert all the JS files in core from jsduck to jsdoc one by one. Right now there's a giant list of exclusions in jsdoc.js; we'll need to review each of the excluded files/directories, fix whatever needs to be fixed in them, and remove them from the exclude list

Change 975921 merged by jenkins-bot:

[mediawiki/core@master] Add jsdoc with jsdoc-wmf-theme

https://gerrit.wikimedia.org/r/975921

You will also need to inherit wikimedia/jsdoc instead of wikimedia/jsduck in eslintrc.

Change 978711 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/core@master] Remove further references to jsduck

https://gerrit.wikimedia.org/r/978711

Change 978711 merged by jenkins-bot:

[mediawiki/core@master] docs: Remove further references to jsduck

https://gerrit.wikimedia.org/r/978711

That conversion guide is pretty nice. Perhaps it should be moved to MediaWiki at some point. That would probably give it a wider audience, and also reduce Phab emails :)

Change 988013 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/extensions/Flow@master] Disable JSDuck documentation

https://gerrit.wikimedia.org/r/988013

Change 988080 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[integration/config@master] Flow should no longer generate JSDuck documentation

https://gerrit.wikimedia.org/r/988080

Change 988080 merged by jenkins-bot:

[integration/config@master] Flow should no longer generate JSDuck documentation

https://gerrit.wikimedia.org/r/988080

Change 988013 merged by jenkins-bot:

[mediawiki/extensions/Flow@master] Disable JSDuck documentation

https://gerrit.wikimedia.org/r/988013