I'm asking this question in the frame of mwoffliner scraper from Kiwix, creating offline versions of Wikimedia wikis but also from "random" Mediawiki instances.
Sorry for the long text, but I'm trying to be a bit exhaustive / precise.
Currently, we use following one of the following endpoints / methods to retrieve article HTML:
- Wikimedia instances:
- (1) either "desktop" REST API: https://hu.wikipedia.org/api/rest_v1/page/html/Brian_May
- (2) or "mobile" REST API: https://hu.wikipedia.org/api/rest_v1/page/mobile-html/Brian_May
- "Random" Mediawiki instances:
We also have some future plans. At some point in time, we plan to investigate:
- (5) use ActionParse: https://hu.wikipedia.org/w/api.php?action=parse&format=json&prop=modules|jsconfigvars|headhtml|text&parsoid=1&page=Brian_May
- (6) use Wikimedia Enterprise (dumps probably, or API maybe)
- (7) use new(ish) REST API: https://api.wikimedia.org/core/v1/wikipedia/hu/page/Brian_May/with_html
Side-note: should we prefer ActionParse endpoints
With all but (5) we need to make a separate call to ActionParse retrieve the list of JS/CSS modules which are needed. This means two HTTP queries for every article to retrieve (more on that later).
This also means that we use endpoints which work either only on Wikimedia instances, or on standalone Mediawiki instances. Obviously, this means branching logic is the codebase which can only leads to some trouble / more maintenance effort.
Solution (5) seems to have the promise that it will work on all situations with only one HTTP query. Quite compelling.
Is there any drawback I have to keep in mind?
Experiments shown below shows that there is more JS embedded, and this JS is doing some nasty jquery side-loading, not working when offline. Am I wrong?
Render properly (on mobile screens)
Other struggle we have is that we need an HTML which renders properly, especially on mobile screens. And ideally well on desktop / tablets as well. Our primary target is definitely mobile, but I find it clearly sad to not be at a point where we can have responsive CSS working quite good at all resolutions in 2025 (meaning at least not too much degradation when you move out of your primary target, I still like the mobile-first approach).
To achieve this goal, we probably have multiple approach possible. I won't go into details of how mwoffliner currently solves this problem for now because it is still way too blurry for me, and it might well be more the consequence of multiple years of maintenance than something totally correctly architected.
I will rather go into details of what I've found in every endpoints. See Tests done below.
Note that I've use only one sample page for all endpoints, meaning it might not represent the complexities of all possible situations.
The conclusion to these tests is that mostly only Wikimedia mobile API returns sufficient JS and CSS to render properly, especially on mobile screen. But even desktop is kinda ugly on all other APIs.
So here is my second question: what am I doing wrong? Is there any "global" CSS / JS which must be loaded on all pages and can be retrieved through APIs as well?
How are we supposed to do it well on "Random" Mediawiki instances where we do not have any REST mobile API?
What about templates?
What about skins?
What about gadgets? Do you confirm they are never present in the HTML sent by these APIs?
Retrieve JS/CSS modules for a given page
As can be seen in the tests below, it looks like in most cases we have "broad" information about the JS/CSS modules needed for given page in HTML <head> section. However, when calling Mediawiki ActionParse API, the list of modules is bigger, and it includes a distinction between modules and modulescripts (what is that for?) and the additional jsconfigvars. Does it means that "broad" information of HTML <head> section in insufficient in many cases?$
mwoffliner currently systematimatically retrieves the list of modules from ActionParse API, hence the two HTTP queries I mentionned before. If this is confirmed to be necessary, should we at least pass parsoid=1 to make the call a bit faster / more optimized or will it have other side effect because the HTML is not retrieved from Parsoid?
Tests done
You can find all the HTML/JS/CSS I've built manually to test these pages here:
Wikimedia desktop API
https://hu.wikipedia.org/api/rest_v1/page/html/Brian_May
In this API, the rendered HTML contains these interesting properties in <head>:
<meta property="mw:generalModules" content="ext.cite.ux-enhancements|mediawiki.page.media|ext.tmh.player"> <meta property="mw:moduleStyles" content="ext.cite.parsoid.styles|ext.cite.styles|ext.tmh.player.styles"> <link rel="stylesheet" href="/w/load.php?lang=hu&modules=ext.cite.parsoid.styles%7Cext.cite.styles%7Cext.tmh.player.styles%7Cmediawiki.skinning.content.parsoid%7Cmediawiki.skinning.interface%7Csite.styles&only=styles&skin=vector" />
Rendering all this locally causes JS issues due missing mw object + not responsive at all + many things including quick-facts are not properly positioned at all.
Wikimedia mobile API
https://hu.wikipedia.org/api/rest_v1/page/mobile-html/Brian_May
In this API, the rendered HTML contains these interesting properties in <head>:
<meta property="mw:generalModules" content="ext.cite.ux-enhancements|mediawiki.page.media|ext.tmh.player" /> <meta property="mw:moduleStyles" content="ext.cite.parsoid.styles|ext.cite.styles|ext.tmh.player.styles" /> <link rel="stylesheet" href="//meta.wikimedia.org/api/rest_v1/data/css/mobile/base" /> <link rel="stylesheet" href="//hu.wikipedia.org/api/rest_v1/data/css/mobile/site" /> <link rel="stylesheet" href="//meta.wikimedia.org/api/rest_v1/data/css/mobile/pcs" /> <script src="//meta.wikimedia.org/api/rest_v1/data/javascript/mobile/pcs"></script>
Rendering all this locally works pretty well.
Mediawiki Visual Editor
In this API, the rendered HTML contains these interesting properties in <head>:
<meta property="mw:generalModules" content="ext.smw.styles"> <meta property="mw:moduleStyles" content="mediawiki.page.gallery.styles"> <link rel="stylesheet" href="/w/load.php?lang=en&modules=mediawiki.page.gallery.styles%7Cmediawiki.skinning.content.parsoid%7Cmediawiki.skinning.interface%7Csite.styles&only=styles&skin=vector">
Rendering all this locally causes JS issues due missing mw object + not responsive at all + many things including quick-facts are not properly positioned at all.
Mediawiki REST API
https://www.appropedia.org/w/rest.php/v1/page/Main_Page_for_Kiwix/html
In this API, the rendered HTML contains these interesting properties in <head>:
<meta property="mw:generalModules" content="ext.smw.styles"> <meta property="mw:moduleStyles" content="mediawiki.page.gallery.styles"> <link rel="stylesheet" href="/w/load.php?lang=en&modules=mediawiki.page.gallery.styles%7Cmediawiki.skinning.content.parsoid%7Cmediawiki.skinning.interface%7Csite.styles&only=styles&skin=vector">
Rendering all this locally causes JS issues due missing mw object + not responsive at all + many things including quick-facts are not properly positioned at all.
Mediawiki ActionParse API
In this API, the rendered HTML contains these interesting properties in <head>:
<script>document.documentElement.className="client-js vector-feature-language-in-header-enabled vector-feature-language-in-main-page-header-disabled vector-feature-page-tools-pinned-enabled vector-feature-toc-pinned-clientpref-1 vector-feature-main-menu-pinned-enabled vector-feature-limited-width-clientpref-1 vector-feature-limited-width-content-enabled vector-feature-custom-font-size-clientpref-1 vector-feature-appearance-pinned-clientpref-1 vector-feature-night-mode-enabled skin-theme-clientpref-day vector-sticky-header-enabled";RLCONF={"wgBreakFrames":true,"wgSeparatorTransformTable":[",\\t."," \\t,"],"wgDigitTransformTable":["",""],"wgDefaultDateFormat":"ymd","wgMonthNames":["","január","február","március","április","május","június","július","augusztus","szeptember","október","november","december"],"wgRequestId":"35b37723-fe3b-4a0b-8aae-afd7d19ed2a9","wgCanonicalNamespace":"","wgCanonicalSpecialPageName":false,"wgNamespaceNumber":0,"wgPageName":"Brian_May","wgTitle":"Brian May","wgCurRevisionId":27765432,"wgRevisionId":0,"wgArticleId":19638,"wgIsArticle":true,"wgIsRedirect":false,"wgAction":"view","wgUserName":"B-root","wgUserGroups":["*","user","autoconfirmed"],"wgCategories":["Mágikus ISBN-linkeket használó lapok","Wikipédia-szócikkek VIAF-azonosítóval","Wikipédia-szócikkek LCCN-azonosítóval","Wikipédia-szócikkek ISNI-azonosítóval","Wikipédia-szócikkek ORCID-azonosítóval","Wikipédia-szócikkek GND-azonosítóval","Wikipédia-szócikkek BNF-azonosítóval","Wikipédia-szócikkek SBN-azonosítóval","Wikipédia-szócikkek KKT-azonosítóval","Wikipédia-szócikkek BIBSYS-azonosítóval","1947-ben született személyek","Londoniak","Élő személyek","Queen","Brit rockgitárosok","Brit zeneszerzők","A Brit Birodalom Rendjével kitüntetett személyek","Minden idők 100 legjobb gitárosa (Rolling Stone magazin)"],"wgPageViewLanguage":"hu","wgPageContentLanguage":"hu","wgPageContentModel":"wikitext","wgRelevantPageName":"Brian_May","wgRelevantArticleId":19638,"wgUserId":587166,"wgUserIsTemp":false,"wgUserEditCount":0,"wgUserRegistration":1740993008000,"wgUserFirstRegistration":1426538644000,"wgIsProbablyEditable":true,"wgRelevantPageIsProbablyEditable":true,"wgRestrictionEdit":[],"wgRestrictionMove":[],"wgNoticeProject":"wikipedia","wgNoticeUserData":{"registration":"20250303091008"},"wgCiteReferencePreviewsActive":true,"wgFlaggedRevsParams":{"tags":{"accuracy":{"levels":2}}},"wgStableRevisionId":27765432,"wgGlobalGroups":[],"wgMediaViewerOnClick":true,"wgMediaViewerEnabledByDefault":true,"wgPopupsFlags":0,"wgVisualEditor":{"pageLanguageCode":"hu","pageLanguageDir":"ltr","pageVariantFallbacks":"hu"},"wgMFDisplayWikibaseDescriptions":{"search":true,"watchlist":true,"tagline":true,"nearby":true},"wgWMESchemaEditAttemptStepOversample":false,"wgWMEPageLength":70000,"wgULSAcceptLanguageList":[],"wgULSBabelLanguages":[]};\nRLSTATE={"site.styles":"ready","user.styles":"ready","user":"ready","user.options":"loading","ext.cite.parsoid.styles":"ready","ext.cite.styles":"ready","ext.tmh.player.styles":"ready","mediawiki.skinning.content.parsoid":"ready"};RLPAGEMODULES=["ext.cite.ux-enhancements","mediawiki.page.media","ext.tmh.player"];</script>
<script>(RLQ=window.RLQ||[]).push(function(){mw.loader.impl(function(){return["user.options@12s5i",function($,jQuery,require,module){mw.user.tokens.set({"patrolToken":"1c9f22ae41c4db237009124605551f6167cffed7+\\\\","watchToken":"6cb81619d736e98b9541dac4f7e59b6367cffed7+\\\\","csrfToken":"f32aa4ef3c021aa4b0af3f2ea7a1671e67cffed7+\\\\"});mw.user.options.set({"discussiontools-autotopicsub":1,"echo-subscriptions-email-edit-user-talk":1,"echo-subscriptions-web-reverted":false,"echo-subscriptions-email-article-linked":true,"echo-subscriptions-web-article-linked":true,"echo-subscriptions-email-mention":true,"echo-subscriptions-email-dt-subscription":true});\n}];});});</script>
<link rel="stylesheet" href="/w/load.php?lang=hu&modules=ext.cite.parsoid.styles%7Cext.cite.styles%7Cext.tmh.player.styles%7Cmediawiki.skinning.content.parsoid&only=styles&skin=vector-2022" />
<script async="" src="/w/load.php?lang=hu&modules=startup&only=scripts&raw=1&skin=vector-2022"></script>
<link rel="stylesheet" href="/w/load.php?lang=hu&modules=site.styles&only=styles&skin=vector-2022" />And following JSON details:
"modules": [
"ext.cite.ux-enhancements",
"mediawiki.page.media",
"ext.tmh.player",
"user",
"user.options"
],
"modulescripts": [],
"modulestyles": [
"ext.cite.parsoid.styles",
"ext.cite.styles",
"ext.tmh.player.styles",
"mediawiki.skinning.content.parsoid",
"site.styles",
"noscript",
"user.styles"
],
"jsconfigvars": {}Rendering all this locally causes JS issues due to dynamic loading of jquery module + missing mw object + not responsive at all + many things including quick-facts are not properly positioned at all.
Wikimedia Enterprise Dumps
In the dump of March 2025 (20250301), the 'article_body.html' property containing rendered HTML has these interesting properties in HTML <head>:
<meta property="mw:generalModules" content="ext.cite.ux-enhancements|mediawiki.page.media|ext.tmh.player" /> <meta property="mw:moduleStyles" content="ext.cite.parsoid.styles|ext.cite.styles|ext.tmh.player.styles" /> <link rel="stylesheet" href="/w/load.php?lang=hu&modules=ext.cite.parsoid.styles%7Cext.cite.styles%7Cext.tmh.player.styles%7Cmediawiki.skinning.content.parsoid%7Cmediawiki.skinning.interface%7Csite.styles&only=styles&skin=vector" />
Rendering all this locally causes JS issues due missing mw object + not responsive at all + many things including quick-facts are not properly positioned at all.
Wikimedia new(ish) REST API
https://api.wikimedia.org/core/v1/wikipedia/hu/page/Brian_May/with_html
In this API, the rendered HTML contains these interesting properties in <head>:
<meta property="mw:generalModules" content="ext.cite.ux-enhancements|mediawiki.page.media|ext.tmh.player" /> <meta property="mw:moduleStyles" content="ext.cite.parsoid.styles|ext.cite.styles|ext.tmh.player.styles" /> <link rel="stylesheet" href="/w/load.php?lang=hu&modules=ext.cite.parsoid.styles%7Cext.cite.styles%7Cext.tmh.player.styles%7Cmediawiki.skinning.content.parsoid%7Cmediawiki.skinning.interface%7Csite.styles&only=styles&skin=vector" />
Rendering all this locally causes JS issues due missing mw object + not responsive at all + many things including quick-facts are not properly positioned at all.