Page MenuHomePhabricator

Replace jsduck with JSDoc3 across all Wikimedia code bases
Open, HighPublic

Description

JSDuck is formally unmaintained and seems abandonware by now. It is hurting our ability to properly document our code.

  1. Examples of problems:
    • No support for @interface
    • No support for namespaced events (change:title is invalid)
    • No support for ES6 (T156469)
  2. Links that show it is unmaintained:
  3. Proposed alternatives:

Migration plan

  1. Research: Start researching who uses the JS HTML docs and what features are important to them, including:
  1. Scope improvements to the WMF theme: Informed by user feedback and documentation best practices, compile a list of critical usability fixes in the form of Phabricator tasks with clear descriptions and acceptance criteria. See T187672: Release jsdoc-wmf-theme 1.0 (Wikimedia theme for JSDoc3) and T138401#9079509.
  1. Consider temporary mitigations: To address concerns about the impact to documentation usability while work on the theme is in progress, we looked into options to help mitigate that impact in the short term. For MediaWiki core, we decided to use the MediaWiki version 1.41 JSDuck docs as a backup while the migration is in progress with the goal of completing the migration in time for version 1.42. In addition, after looking into a few other open-source theme options, we decided to move forward with the WMF theme.

Codebases to port to JSDoc

  • MediaWiki core - T352308
  • mediawiki/extensions/ArticleCreationWorkflow
  • mediawiki/extensions/Citoid
  • mediawiki/extensions/ContentTranslation
  • mediawiki/extensions/DonationInterface
  • mediawiki/extensions/Echo
  • mediawiki/extensions/EventLogging - T357444
  • mediawiki/extensions/Flow
  • mediawiki/extensions/Graph
  • mediawiki/extensions/GuidedTour
  • mediawiki/extensions/Kartographer - T342864
  • mediawiki/extensions/MobileFrontend - T188261
  • mediawiki/extensions/SyntaxHighlight_GeSHi
  • mediawiki/extensions/Translate
  • mediawiki/extensions/UploadWizard
  • mediawiki/extensions/VisualEditor - T250843
  • mediawiki/extensions/Wikibase - T342905
  • mediawiki/services/cxserver
  • mediawiki/skins/Minerva
  • OOUI - T185396
  • OOjs Router

Checklist for each migration

  • Code documentation works fine, fix any problems
  • Docs get properly published to docs.wikimedia.org e.g. https://doc.wikimedia.org/MobileFrontend/master/js/
  • CI lints documentation properly with jenkins-bot on patches
  • Documentation about how to run documentation in mw.org, README, etc is updated to point to the new instructions if any change

Migration guide

For how to set up JSDoc, see https://www.mediawiki.org/wiki/JSDoc.

JSDuck to JSDoc mapping

JSDuck tagJSDoc tag and notes
@abstract@abstract
@accessor Unsupported. Only used by ExtJSBase.
@alias prefix.name@alias aliasNamepath
@alternateClassName OtherClassName What to do? @alias aliasNamepath? @typedef {type} namepath? Sparse usage.
@aside name Unsupported. No usage.
@author Some name...@author name <emailAddress>
@cfg, @cfg name, @cfg {Type} name, @cfg {Type} [name="default value"], @cfg {Type} name.subpropertyUse @param, or use`@typedef` for shared types
@chainableUse @return {Type} where {Type} refers to the memberof class of the function instead.
@class, @class ClassName@class name
@constructor@return {type} description
@deprecated@deprecated
@docauthor Some name... Unsupported and unused.
@enum, @enum {Type}, @enum {Type} EnumName, @enum [EnumName=alias.*]@enum {type}
@event, @event name@event ~'className.eventName' (example: ~'util.addPortlet')
@evented Unsupported. Only used by ExtJSBase.
@example@example
@experimental, @experimental 2.0 Some description...@experimental
@extends ParentClassName@extends namepath
@fires eventName (in 5.x beta)For hooks: @fires Hooks~'className.eventName' (example:Hooks~'util.addPortletLink')
@ftype name Unsupported. No usage.
@hide Unsupported. Only used by ExtJSBase.
@ignore@ignore
@inheritable What to do? Delete?
@inheritdoc, @inheritdoc ClassName, @inheritdoc #memberName, @inheritdoc ClassName#memberName, @inheritdoc ClassName#static-type-memberNameWhat to do? If no value, @inheritdoc. If value, @extends value?
@localdoc This documentation is only visible inside this class/member. (in 5.x beta) What to do? Delete?
@markdown Unsupported. No usage.
@member ClassName What to do? @member {type} name, @memberof parentNamepath @memberof! parentNamepath, or something else?
@method, @method name@method FunctionName
@mixins ClassName@mixes OtherObjectPath
@new Unsupported. No usage.
@override OverriddenClassName@override
@param name, @param {Type} name, @param {Type} [name], @param {Type} [name="default-value"], @param {Type} name.subproperty@param {type} name description. For an optional parameter: @param {type} [name] description or @param {type} [name=default value] description. See jsdoc.app.
@preventable Unsupported. Only used by ExtJSBase.
@private@private [{typeExpression}]
@property, @property name, @property {Type} name, @property {Type} [name="default value"], @property {Type} name.subproperty@property {type} name description. For an optional property: @property {type} [name] description
@protected@protected [{typeExpression}]
@ptype name Unsupported. No usage.
@readonly@readonly
@removed, @removed 2.0 Some description... Unsupported. One usage in SemanticMediaWiki and otherwise only used by ExtJSBase.
@requires ClassName@requires someModuleName
@return {Type}, @return {Type} return.subproperty (Multiple @return tags )Replace with single @return {Type} description and use @typedef {Object} Type to document the type.
@scss-mixin Unsupported. No usage.
@since Ext JS 4.0 beta@since versionDescription
@singleton What to do? Custom tag?
@static@static
@template@template
@throws, @throws {Type}@throws {type} free-form description
@type {Type}, @type Type@type {type} free-form description
@uses ClassName@see namepath or @see text. Mapping isn't one-to-one.
@var, @var $some-name, @var {Type} $some-name, @var {Type} [$some-name="default value"]@member {type} name
@xtype Unsupported. One usage in ExtTab and otherwise only uses by ExtJSBase.
{@link Class#member link text}{@link namepathOrURL}, [link text]{@link namepathOrURL}, {@link namepathOrURL%7clink text}, {@link namepathOrURL link text (after the first space)}
{@img path/to/image.png alt text}Usupported. Only used by ExtJSBase.
{@video vimeo 465123 Some description here...} Unsupported. No usage.

Details

SubjectRepoBranchLines +/-
integration/configmaster+5 -0
mediawiki/extensions/UploadWizardmaster+189 -114
mediawiki/extensions/UploadWizardmaster+519 -22
mediawiki/extensions/Translatemaster+21 -14
mediawiki/extensions/Citoidmaster+1 -1
mediawiki/extensions/Flowmaster+2 -2
mediawiki/extensions/Graphmaster+1 -1
mediawiki/extensions/SyntaxHighlight_GeSHimaster+1 -1
mediawiki/coremaster+2 -3 K
mediawiki/extensions/Flowmaster+1 -225
integration/configmaster+0 -1
mediawiki/coremaster+3 -450
mediawiki/coremaster+638 -51
mediawiki/extensions/Kartographermaster+12 -450
mediawiki/extensions/Kartographermaster+455 -17
mediawiki/extensions/TemplateDatamaster+186 -37
mediawiki/extensions/GlobalWatchlistmaster+870 -616
mediawiki/coremaster+172 -483
mediawiki/coremaster+3 -4
mediawiki/coremaster+0 -0
integration/configmaster+4 -5
integration/configmaster+1 -1
mediawiki/skins/MinervaNeuemaster+57 -68
mediawiki/services/parsoidmaster+13 -1
mediawiki/services/parsoidmaster+367 -404
Show related patches Customize query in gerrit

Related Objects

StatusSubtypeAssignedTask
OpenFeatureNone
Resolvedhashar
OpenNone
OpenNone
OpenKBach
Resolved Jhernandez
DuplicateNone
ResolvedKrinkle
Resolvedapaskulin
OpenNone
ResolvedEsanders
ResolvedEsanders
ResolvedBUG REPORTKrinkle
ResolvedJdlrobson
DuplicateNone
DuplicateNone
DuplicateNone
Resolved nray
ResolvedJdforrester-WMF
Resolvedmatmarex
DeclinedNone
Resolvedcscott
ResolvedTheDJ
ResolvedSamwilson
ResolvedNone
OpenNone
DeclinedNone
Resolvedapaskulin
ResolvedKrinkle
Resolvedmatmarex
Resolvedapaskulin
DeclinedBUG REPORTJdlrobson
Resolvedapaskulin
Openegardner
OpenNone

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

In JSDoc issue #669, someone brings up a project that uses @returns {<class name>} this, chainable (where <class name> is the name of the class the function returns) to represent chainable functions
There's also PR#1162 that would add it, which has been under review for 7 years.

Hi all, following up on @Jdlrobson's comment in T138401#9078948, here's what the Tech Docs team would like to propose to move this forward:

  • We don't know how many page views they get
  • We don't know how the consumer uses it as we don't hear from people that use the documentation
  1. Research: Start researching who uses the JS HTML docs and what features are important to them, including:
    • Attempt to query pageview data for JS HTML docs from the webrequest tables in Hive
    • Put out a call for on-wiki feedback through relevant channels

A concern was raised that the default jsdoc theme is not as good to use as the currently used JSDuck theme.

I agree that, in its current state, the WMF JSDoc theme has some usability issues (such as T262406 and T207383) that prevent it from being a functional replacement for the JSDuck docs. (I don't mean this as a call out of the people who have worked on the theme by any means. There are a lot of really good things about the theme as well.)

  1. Scope improvements to the WMF theme: Informed by user feedback and documentation best practices, compile a list of critical usability fixes in the form of Phabricator tasks with clear descriptions and acceptance criteria. There's already been a lot of good work done on this, most recently TheDJ's comment in T138401#9079509. The expectation would be that some WMF engineering time will be made available to work on these fixes within a reasonable timeframe (such as the 2024 calendar year or the 2024-25 fiscal year) either in the form of development time or Outreachy mentorship time. @SCherukuwada would be responsible for signing off on engineering time commitment relative to the scope of the fixes.

There seemed to be no clear definition of what "good enough" looks like. For example it wasn't clear if the theme was a blocker

It seems like delaying JSDoc migration is causing issues for some projects. (See T138401#8045282, T138401#9009746, and T138401#9030517.) As long as we can get a commitment to dedicating time to work on the theme, I think we can give projects currently using JSDuck the go-ahead to migrate to JSDoc.

  1. Consider temporary mitigations: To address concerns about the impact to documentation usability while work on the theme is in progress, look into options to help mitigate that impact in the short term. For example:
    • Temporarily use another open-source JSDoc theme, such as clean-jsdoc-theme or better-docs
    • Serve static snapshots of the JSDuck docs alongside the JSDoc docs. (Although I'm not sure of the technical feasibility of this, and it does introduce extra risk, complexity, and effort.)

Feedback on these plans is welcome! Particularly if anyone has ideas about the technical feasibility of temporarily serving static snapshots of the JSDuck docs. The immediate next steps would be to open subtasks to capture the work described here.

  1. Really looking forward to that actually !
  2. I think some of the most critical issues are already dealt with since my original comment, but I've also discovered some new issues. I'm just going to add those to my previous comment, to keep stuff together.
  3. I think it will be difficult to serve both. At most, you could keep a copy of old state at a separate path ?
  4. Besides the 3 mentioned points I think you should add an effort to expand on https://www.mediawiki.org/wiki/JSDoc
  5. Perhaps do some social media outreach on ES6 documentation, to tickle some outsiders and get some stories of 3rd parties experiences with setting up HTML output of JSDoc for products ?

I do think that a lot has to do with giving people good examples of how to solve more complex situations. We can already see how having multiple extensions converted has helped people along.

The biggest lessons I have discovered so far are:

  1. JSdoc is not that good with Typescript (it can't infer some of the structure that TS adds)
  2. JSDoc is not good with ES5, as you need to provide annotations for literally everything
  3. Do not over annotate ES6 with annotations that can be inferred from the structure.

@hashar could you help us preserve the current version (e.g. static cp of the directory) of https://doc.wikimedia.org/mediawiki-core/master/js/ (is that possible?) so that we can point to it in the JSDoc version and begin this migration?
e.g. something like that content published to https://doc.wikimedia.org/archive/mediawiki-core/master/js/ ?

Change 975921 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/core@master] WIP: Add jsdoc with jsdoc-wmf-theme

https://gerrit.wikimedia.org/r/975921

@TheDJ @Catrope @AnneT I started a skeleton for what a first patch could look like: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/975921
I limited the documentation to mediawiki.util module since it's widely used, complex and hopefully well understood.

Hopefully the output is self explanatory (please note outstanding TODOs in the commit message)

Screenshot 2023-11-20 at 5.42.55 PM.png (703×1 px, 116 KB)

I've merged Jon's patch. The next steps for jsdoc in core are:

  • Archive the old jsduck docs, per T351764#9353954, and link to that archive in resources/README.md
    • Right now this link points to the REL1_41 docs; we could also decide that that's good enough, and decline T351764
  • Actually remove jsduck from core. Jon's patch deactivates it, but doesn't remove its configuration files or various other things
  • Convert all the JS files in core from jsduck to jsdoc one by one. Right now there's a giant list of exclusions in jsdoc.js; we'll need to review each of the excluded files/directories, fix whatever needs to be fixed in them, and remove them from the exclude list

Change 975921 merged by jenkins-bot:

[mediawiki/core@master] Add jsdoc with jsdoc-wmf-theme

https://gerrit.wikimedia.org/r/975921

You will also need to inherit wikimedia/jsdoc instead of wikimedia/jsduck in eslintrc.

Change 978711 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/core@master] Remove further references to jsduck

https://gerrit.wikimedia.org/r/978711

Change 978711 merged by jenkins-bot:

[mediawiki/core@master] docs: Remove further references to jsduck

https://gerrit.wikimedia.org/r/978711

That conversion guide is pretty nice. Perhaps it should be moved to MediaWiki at some point. That would probably give it a wider audience, and also reduce Phab emails :)

Change 988013 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/extensions/Flow@master] Disable JSDuck documentation

https://gerrit.wikimedia.org/r/988013

Change 988080 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[integration/config@master] Flow should no longer generate JSDuck documentation

https://gerrit.wikimedia.org/r/988080

Change 988080 merged by jenkins-bot:

[integration/config@master] Flow should no longer generate JSDuck documentation

https://gerrit.wikimedia.org/r/988080

Change 988013 merged by jenkins-bot:

[mediawiki/extensions/Flow@master] Disable JSDuck documentation

https://gerrit.wikimedia.org/r/988013

Change #1026023 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/core@master] Drop last references to JSDuck

https://gerrit.wikimedia.org/r/1026023

Change #1026025 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/Graph@master] eslint: Configure to lint against JSDoc, not old JSDuck

https://gerrit.wikimedia.org/r/1026025

Change #1026046 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/Citoid@master] eslint: Configure to lint against JSDoc, not old JSDuck

https://gerrit.wikimedia.org/r/1026046

Change #1026047 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/Flow@master] eslint: Configure to lint against JSDoc, not old JSDuck

https://gerrit.wikimedia.org/r/1026047

Change #1026048 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/SyntaxHighlight_GeSHi@master] eslint: Configure to lint against JSDoc, not old JSDuck

https://gerrit.wikimedia.org/r/1026048

Change #1026023 merged by jenkins-bot:

[mediawiki/core@master] Drop last references to JSDuck

https://gerrit.wikimedia.org/r/1026023

Change #1026048 merged by jenkins-bot:

[mediawiki/extensions/SyntaxHighlight_GeSHi@master] eslint: Configure to lint against JSDoc, not old JSDuck

https://gerrit.wikimedia.org/r/1026048

Change #1026025 merged by jenkins-bot:

[mediawiki/extensions/Graph@master] eslint: Configure to lint against JSDoc, not old JSDuck

https://gerrit.wikimedia.org/r/1026025

Change #1026047 merged by jenkins-bot:

[mediawiki/extensions/Flow@master] eslint: Configure to lint against JSDoc, not old JSDuck

https://gerrit.wikimedia.org/r/1026047

Change #1026046 merged by jenkins-bot:

[mediawiki/extensions/Citoid@master] eslint: Configure to lint against JSDoc, not old JSDuck

https://gerrit.wikimedia.org/r/1026046

Change #1030944 had a related patch set uploaded (by Abijeet Patro; author: Esanders):

[mediawiki/extensions/Translate@master] Remove wikimedia/jsduck config and fix resulting warnings

https://gerrit.wikimedia.org/r/1030944

Change #1030944 merged by jenkins-bot:

[mediawiki/extensions/Translate@master] Remove wikimedia/jsduck config and fix resulting warnings

https://gerrit.wikimedia.org/r/1030944

Change #1032015 had a related patch set uploaded (by Novem Linguae; author: Novem Linguae):

[mediawiki/extensions/UploadWizard@master] [WIP] Port jsduck -> jsdoc

https://gerrit.wikimedia.org/r/1032015

Change #1032015 merged by jenkins-bot:

[mediawiki/extensions/UploadWizard@master] Port jsduck -> jsdoc

https://gerrit.wikimedia.org/r/1032015

Change #1032098 had a related patch set uploaded (by Novem Linguae; author: Novem Linguae):

[mediawiki/extensions/UploadWizard@master] Upgrade from JSDoc 3 to JSDoc 4

https://gerrit.wikimedia.org/r/1032098

Change #1032098 merged by jenkins-bot:

[mediawiki/extensions/UploadWizard@master] Upgrade from JSDoc 3 to JSDoc 4

https://gerrit.wikimedia.org/r/1032098

@aliu. I went ahead and unticked extension/UploadWizard. My conversion of extension/UploadWizard to JSDoc is not complete. Lots of eslint warnings, lots of npm run doc console warnings, entire classes not showing up, clicking on events gives 404 pages. I understand the basics like @param, @return, but will probably need to pepper someone with questions to learn the advanced stuff and finish the extension/UploadWizard conversion. Is anyone willing to be my mentor and DM with me a bit on Discord or IRC? Once I get taught how to do this I'll bet I can make really good progress on converting multiple repos.

Change #1032408 had a related patch set uploaded (by Novem Linguae; author: Novem Linguae):

[mediawiki/extensions/UploadWizard@master] JSDoc: add missing classes

https://gerrit.wikimedia.org/r/1032408

Change #1032475 had a related patch set uploaded (by Novem Linguae; author: Novem Linguae):

[integration/config@master] JSDoc: add 5 extensions/skins

https://gerrit.wikimedia.org/r/1032475

Change #1032475 merged by jenkins-bot:

[integration/config@master] JSDoc: add 5 extensions/skins

https://gerrit.wikimedia.org/r/1032475