Possibly consider scanning all templates to identify templates missing itemType or title fields and notify interested users.

Published in w2c-server v1.5.0.

Published in w2c-server v1.5.0.

Published in w2c-server v1.5.0.

Deployed to production and using in the following domain main configurations:

• https://meta.wikimedia.org/w/index.php?title=Web2Cit/data/fr/leparisien/www/templates.json&diff=prev&oldid=30936366
• https://meta.wikimedia.org/w/index.php?title=Web2Cit/data/nz/co/thepress/www/templates.json&diff=prev&oldid=30936370
• https://meta.wikimedia.org/w/index.php?title=Web2Cit/data/uk/co/telegraph/www/templates.json&diff=prev&oldid=30936382

Other domain template configurations where this has been used so far:

• https://meta.wikimedia.org/wiki/Web2Cit/data/com/straitstimes/www/templates.json
• https://meta.wikimedia.org/wiki/Web2Cit/data/com/channelnewsasia/www/templates.json
• https://meta.wikimedia.org/wiki/Web2Cit/data/ar/com/pagina12/www/tests.json

Solved by adding a 10-second timeout to fetch calls in Web2Cit-Core with 3ecef4d0.

Increased timeout to 15 seconds with 34633dd4 as it was otherwise failing for https://collections.slq.qld.gov.au/guide/3282/details.

Thank you, @abi_. The merge request was created successfully, and set to be merged into production as I had requested. Great!

Thanks @abi_. So from your message I understand that the commit was pushed to the i18n branch successfully but that due to the bug a merge request into the production branch was not created automatically.

Should I submit a separate patch to add that line and reference the old change id in the commit message?

As per discussion on #wikimedia-operations, submitted a new patch here: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1322012

Should I submit a separate patch to add that line and reference the old change id in the commit message?

Pinging @dancy who commented about this on the patch on Gerrit a while ago.

Thank you so much, @hashar! Unfortunately I just tested it on the beta cluster and there is an error:

Thanks @matmarex! Ok, done. I'll wait for your new +1 now then, and once done I'll ping @hashar asking if he could give +2 for merging. Thank you!!

Thanks @matmarex! Your suggestion sounds good to me too. So the change would look like:

@@ -558,7 +558,8 @@ if ( $wmgRealm == 'labs' ) {
                $wgHooks['ContentHandlerDefaultModelFor'][] = static function ( Title $title, &$model ) {
                        if (
                                $title->getNamespace() === NS_MAIN &&
-                               preg_match( '/^Web2Cit\/data\/.+\.json$/', $title->getText() )
+                               str_starts_with( $title->getText() , 'Web2Cit/data/') &&
+                               str_ends_with( $title->getText() , '.json')
                        ) {
                                $model = CONTENT_MODEL_JSON;
                        }

Should I follow the instructions here to update the patch on Gerrit before merging? Would that reset your +1 vote?

Deployed to w2c-beta for testing.

The editor interface part should be relatively trivial.

This is pending. If JSONEditor.default.language is used, make sure error_messages overrides from JSON schemas continue working as expected.

Spanish translations for JSON schemas provided with 3c7f8903.

On the other hand, because most text on the JSON editing form comes from the JSON schema file used to generate it, a way to translate these labels must be found. See T305904 for a proposal to use separate schema files based on one main version.

Internationalized JSON schemas in Web2Cit-Core with 50923c69.

Data from non-applicable templates is then used to return a result for a non-applicable fallback template.

Made an additional change in Web2Cit-Server to not return field scores in these cases where field outputs have been ignored: 727096fb.

With T408865 resolved, asked to have the translation project reinstated: T433290.

Pending update Docs/Server.

Updated: https://meta.wikimedia.org/w/index.php?title=Web2Cit%2FDocs%2FServer&diff=30859609&oldid=30825040

Re-deployed currently deployed v1.4.0 release in production like so:

• Create production branch locally from v1.4.0 tag: git checkout -b production v1.4.0
• Cherry-pick the two commits from main implementing (and fixing) automatic deployments.
• Push production branch remotely.

To support automatic deployments while also supporting immediate integration of translatewiki translations (T408866) I decided to adopt a Gitlab-Flow-like workflow:

• Continue using the main branch for development, with feature branches as needed.
• Create a staging branch for release stabilization. This branch to be deployed automatically to our beta server.
• Create a production branch for the latest stable release. This branch to be deployed automatically to our production server.
• Translatewiki to pull and push from the production branch for immediate deployment of translation updates (T408866).

If there is a list of actions that can be made to verify, the list can be added to this task description and mentioned on the change (so the deployer have access to it easily). We usually expect the patch author to be available to do the test themselves, but I guess a deployer can do that as well.

I submitted the patch here: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1314816

Thank you @hashar for your detailed explanation! I think I should be ready to submit my first gerrit patch now.

The reason why we have paths defined as com/example/www/ is probably because we wanted Web2Cit to use configuration files hierarchically (i.e., use com.example.com.sub, if not available use com.example.com, else use com.example). But then plans changed.

Note this may still be desired, as mentioned in T320771#12060424

Started working in branch T309869 of Web2Cit-Core:

• implement a QID transformation step that resolves an input QID into corresponding [[sitelink]] or label value from Wikidata, using a list of one or more language codes as config value to decide which sitelink/label to return.

Once deployed, use for www.telegraph.co.uk as well.

Solved by adding a 10-second timeout to fetch calls in Web2Cit-Core with 3ecef4d0.

Likewise, I'm getting internal server error for www.animenewsnetwork.com URLs, with Web2Cit but not with Citoid.

This was T322875.

T403799#11151450 explains why source may have shown as citoid instead of zotero:

the response is coming from citoid, not zotero. Possibly the nytimes is temporarily blocking our zotero UA but not our citoid one,

But why would that result in Web2Cit fallback template failing?

Can't reproduce anymore. It seems to be working now.

We may consider moving these messages from schemas in Web2Cit-Core to some complementary or overriding logic in Web2Cit-Server for more straightforward translation once T408866 has been solved.

Implemented in Web2Cit-Core with e8b91273.

This selection step may be useful in cases where we can't get the webpage because we have been blocked. As suggested in T305168, we could use such a URL selection step to guess a title.

Web2Cit/Docs/Templates documentation updated on meta-wiki.

Began implementation on a separate Web2Cit-Core branch: https://gitlab.wikimedia.org/diegodlh/w2c-core/-/tree/T304326

The configuration value may mimic the browsers' URL object. For example, href for the full URL, hash for the fragment identifier, etc.

• hash: alternatively, fragment. yes; maybe we could drop the leading #? Consider decoding.
• hostname: alternatively, domain. yes; e.g., www.example.com.
• href: yes; the whole URL.
• pathname: alternatively, path. yes; including leading /.
• protocol: alternatively, scheme. yes; e.g., https:, but won't be useful until T322601 is addressed. Consider dropping the trailing :.
• port: yes; e.g., 8080, but won't be useful until T315020 is addressed.
• host: no; it's just hostname + port.
• origin: no; it's just protocol + hostname + port.
• search: alternatively, query. yes; e.g., ?a=b&c=d&e=f.
  • Consider dropping leading ? or even returning an array of parameters: `[ "a=b", "c=d", "e=f" ].
  • Evaluate how the output chosen could be processed by a later transformation step (e.g, Match transformation) to get value or presence of a given parameter, considering parameter order can vary.
  • Can be decode considering we have = as delimiter between parameter name and value?
  • Consider some edge cases. For example ?a=b=c&d=f would be valid, with a parameter having b=c as value.
• username: no?
• ~~password: no?~

1. https://rok.catholic.net.au/our-diocese/our-bishop/#
2. https://schoolsdirectory.eq.edu.au/Details/A713#
3. https://www.abs.gov.au/ausstats/abs@.nsf/products/8DBA196EBE5D5C2BCA2575E70019CD88?OpenDocument#

Removed unnecessary empty fragments from corresponding templates.json files.

Update on-wiki documentation...

• https://meta.wikimedia.org/w/index.php?title=Web2Cit/Docs/Templates&diff=prev&oldid=30775928
• https://meta.wikimedia.org/w/index.php?title=Web2Cit/Docs/Server&diff=prev&oldid=30775932
• https://meta.wikimedia.org/w/index.php?title=Web2Cit/Docs/Patterns&diff=prev&oldid=30775934

Likewise, Web2Cit-Server homepage will be updated to also encode # as %23 when clicking on "Extract", but won't be instructed to drop trailing # for now.

Implemented in the homepage and also when common translate endpoint request are redirected to shortcut URLs: e3251ae5.

• We will likely want to implement this in Web2Cit-Core utils' normalizeUrlPath function.
• Make sure unit tests are added or updated accordingly.
• Make sure we encode URL fragment correctly when passing it on to Citoid.
• Consider whether we should ignore empty URL fragments.
• ...
• Consider how this would affect URL path sorting into URL path patterns.

Apparently, Citoid selection is working but XPath and JSON-LD selection aren't. Maybe they block Web2Cit but don't block Citoid?

Maybe this is what happening with www.bloomberg.com? https://meta.wikimedia.org/w/index.php?title=Web2Cit/monitor/com/bloomberg/www/results&oldid=30602042

Noting that an alternative to the proposed URL selection step may be using Citoid selection with url config value. However, this would not be useful in cases where we have been blocked, which has been mentioned above as one of the benefits of having this URL selection step.

Consider whether we should ignore empty URL fragments.

Updated Web2Cit-Monitor Web2Cit/results module (30761466) and Web2Cit-Gadget (3f95a655) to encode fragment delimiter # when creating https://web2cit.toolforge.org/<url> shortcut URLs.

Then, it would make no sense to have this hash option in the URL selection step.

A Jupyter notebook was created on PAWS to identify template paths for which no test had been defined, and the users who had contributed to the corresponding templates configuration files: https://public-paws.wmcloud.org/User:Diegodlh/Web2Cit/templates.ipynb

Zotero translation server (used by Citoid) does use cookies, precisely to "make websites with cookie forgetting browser/scrapper protection happy". See https://github.com/search?q=repo%3Azotero%2Ftranslation-server+cookie&type=code:

According to internal Web2Cit server logs, getting similar errors with:

• https://www.tdg.ch/la-transition-energetique-a-lepreuve-des-urnes-807325473741
• https://www.sudouest.fr/sante/moustique-tigre-en-gironde-on-ne-l-eradiquera-pas-12490339.php

Consider https://www.npmjs.com/package/fetch-cookie wrapper.

I first thought this may be related to T408874. Tried this with curl and got a redirect. Even when trying with the User-Agent header specified here: https://radar.cloudflare.com/bots/directory/wikimedia-citoid

This would be particularly useful in the common case were www.some.domain.com is an alias of some.domain.com. For cases like this, we may consider defaulting to use config files for some.domain.com if config files for p www.some.domain.com` have not been set.

we could use such a URL selection step to guess a title.

Another example where this could be useful: https://web2cit.toolforge.org/https://www.thepress.co.nz/a/nz-news/350060204/series-inexplicable-coincidences-murder-case-against-david-benbow

Found it. In w2c-server we were getting data from local Citoid cache twice if options.citoid=true. In one of them we were using a try...catch block, whereas in the other we weren't (https://gitlab.wikimedia.org/diegodlh/w2c-server/-/blob/6eb85d83f378f0e44d6a1cb66027c89fc211d871/src/app.ts#L549):

Tried a few more things but couldn't reproduce this bug.

Find a URL for which Citoid is failing, for example https://teara.govt.nz/en/biographies/4g2/galbraith-alexander (thanks @DrThneed!)

Pending documentation update at https://meta.wikimedia.org/wiki/Web2Cit/Docs/Templates#Range_transformation.

Done: https://meta.wikimedia.org/w/index.php?diff=30697399

Deployed with w2c-server v1.4.0

Deployed in w2c-server v1.4.0

Deployed with w2c-server v1.4.0

Deployed with w2c-server v1.4.0

Deployed with w2c-server 1.4.0

Deployed with w2c-server v1.4.0

Oh sure, no worries. I can perfectly wait. Closing this then. Thanks!

Deployed with w2c-server v1.4.0

Citing the following from an old email to @DrThneed:

Deployed with w2c-server v1.4.0

Added Case transformation to templates schema with c8665bed.

Added Replacement transformation to templates schema with 7b9a4f91.

Some errors are already caught, therefore not preventing translation from proceeding, but it would still be useful to log them to help diagnose problems:

Maybe this could be fixed if there was a way to not automatically choose one of the enum values?

In addition, JSON editor was updated to provide a custom validation error in cases where a path not matching one of the test paths has already been defined.

Note that on Toolforge, the webservice toolforge logs output shows encoding issues and gets truncated, as opposed to the kubectl logs output. Reported in T429028.

Also lowered logging level of some common events in w2c-core as they would otherwise flood the server logs: 859ca801.

Added support for setting custom logger in Web2Cit-Core with 2105091d. In the future, consider T302588.

Jobs logging in Toolforge is moving to Loki (log aggregation system): https://wikitech.wikimedia.org/wiki/News/2025_New_Toolforge_log_storage. But it's not clear if this is also the case for web services.

Done in ca75959e.

To identify requests made by the Web2Cit-Gadget we may consider whether the user script should use the Api-User-Agent header.

See T390791

This would also let us monitor pageviews of these pages with Massviews.

https://pageviews.wmcloud.org/massviews/?platform=all-access&agent=user&source=category&start=2025-05-28&end=2026-05-28&subjectpage=0&subcategories=0&sort=views&direction=1&view=chart&target=https://meta.wikimedia.org/wiki/Category:Web2Cit_Monitor_Results

In the meantime we may offer the list of recent changes to any of our Web2Cit/monitor/*/*/results pages.

Add errors and coverage parameters to Web2Cit/log/row template. For now, adding them at the end of the row, to support old log and results pages with hardcoded log header:

Monitor changes deployed and log and result pages updated. Changed errors and coverage column order in log and header templates:

• https://meta.wikimedia.org/w/index.php?title=Template:Web2Cit/log/row&diff=prev&oldid=30606074
• https://meta.wikimedia.org/w/index.php?title=Template:Web2Cit/log/header&diff=prev&oldid=30606079