Zotero dev here. I'm not clear on when all the above was written, but a few clarifications.

• translation-server currently runs on Firefox 52 ESR, so as noted above the underlying Mozilla framework is supported until August 2018.
• The current version available on Docker Hub is based on Zotero 4.0, but a version based on Zotero 5.0 has been available on GitHub since June 2017, and that's what we'd recommend using. We'll push that to Docker Hub soon.
• The recommended configuration is to run translation-server in Docker, so admin requirements should be minimal. There've been memory leaks in the past, which may or may not still exist — we've ignored them by just restarting it in a cron job twice a day.
• The Zotero project itself is extremely active and is certainly not in a limbo state. We still plan to port the Zotero client to Electron on some timeline (depending on whether we're able to get Zotero running on Firefox 60 ESR), but that's largely unrelated to translation-server, which can likely be ported to Node or Electron much more easily. That's something we plan to work on over the next couple months. (We initially implemented translation-server in a very early version of Node.js, and switched to XULRunner for performance reasons due to the limited JS DOM options available in Node at the time. Between Electron and improved DOM options in Node, I suspect this is no longer an issue.)
• translation-server is an important part of our infrastructure and is in no way abandoned. translation-server is used by our website and API, and we have a major new feature based around it that's launching soon. We recently added a /search endpoint that can translate DOIs, ISBNs, and PubMed IDs, and will soon be adding the ability to save any URL, not just ones with translator coverage, so it's under active development.

The extent of translation-server use in Wikimedia projects has never been entirely clear to me, since we mostly just see the occasional translator error report from @Mvolz, but we're happy to work more closely if desired. Questions can always be posted to GitHub or the zotero-dev mailing list.

Anyway, hope this helps! Let me know if you have any questions.

In T187194#3968832, @danstillman wrote:

Zotero dev here. I'm not clear on when all the above was written, but a few clarifications.

Hello @danstillman, great to have you here, your input is greatly appreciated. This was written just 4 hours before your comment, so your very fast :-). As a clarification this task is not about zotero translation-server THE software, but rather the installation in WMF.

• translation-server currently runs on Firefox 52 ESR, so as noted above the underlying Mozilla framework is supported until August 2018.
• The current version available on Docker Hub is based on Zotero 4.0, but a version based on Zotero 5.0 has been available on GitHub since June 2017, and that's what we'd recommend using. We'll push that to Docker Hub soon.

That would be great! A couple of notes, and they relevant to our installation alone, and hence this task, but not zotero translation-server as a software.

• We do not currently have an infrastructure for running docker containers. We have been however creating a kubernetes infrastructure and plan to have our first service deployed this quarter so this is going to change really soon.
• In the above said infrastructure we have taken an approach of "we only trust containers we build ourselves" for security reasons. Yes I know docker supports notaries and docker images can be signed these days and yes the entire discussion of trusting things off pypi/npm/rubygems does not inherently differ much but we will feel that running images directly off Docker Hub is not currently worth the risk.

• The recommended configuration is to run translation-server in Docker, so admin requirements should be minimal. There've been memory leaks in the past, which may or may not still exist — we've ignored them by just restarting it in a cron job twice a day.

Indeed deployment requirements do become less of problem in a container environment, hence our move to a kubernetes infrastructure, but please operational issues still stand (and can be anything but minimal at a large scale)

Also, Docker supports memory limits and they are bound to be more responsive (albeit perhaps a bit less flexible) than a cronjob so perhaps the recommended configuration should include a --memory=<x>g recommendation ? I am guessing you are the best people around to provide a value for <x>

• The Zotero project itself is extremely active and is certainly not in a limbo state. We still plan to port the Zotero client to Electron on some timeline (depending on whether we're able to get Zotero running on Firefox 60 ESR), but that's largely unrelated to translation-server, which can likely be ported to Node or Electron much more easily. That's something we plan to work on over the next couple months. (We initially implemented translation-server in a very early version of Node.js, and switched to XULRunner for performance reasons due to the limited JS DOM options available in Node at the time. Between Electron and improved DOM options in Node, I suspect this is no longer an issue.)

That's great to hear! As you note the client itself is not really relevant to us, it's translation-server we care about. From your answer I understand this process hasn't even started, so as WMF we should probably wait for a production ready release before we even begin contemplating our next move.

• translation-server is an important part of our infrastructure and is in no way abandoned. translation-server is used by our website and API, and we have a major new feature based around it that's launching soon. We recently added a /search endpoint that can translate DOIs, ISBNs, and PubMed IDs, and will soon be adding the ability to save any URL, not just ones with translator coverage, so it's under active development.

You mention an API. Got a link handy ? I can't help but wonder if it would make sense for us to use that API instead of hosting our own infrastructure ?

The extent of translation-server use in Wikimedia projects has never been entirely clear to me, since we mostly just see the occasional translator error report from @Mvolz, but we're happy to work more closely if desired. Questions can always be posted to GitHub or the zotero-dev mailing list.

TL;DR is that VisualEditor uses citoid to generate citations that can be directly embedded in wiki articles. Citoid uses zotero translation-server for much of that functionality. A starting link for Citoid would be https://www.mediawiki.org/wiki/Citoid. If you need/want more in-depth information, please ask.

Anyway, hope this helps! Let me know if you have any questions.

It sure does, thanks for weighing in!

So, this task has been open for a couple of months now, with the underlying issues have been present for far longer than that. In case it wasn't clear from the lengthy and detailed task description, there are currently two deadlines here:

• Firefox 52 ESR (which this is indirectly based on) EOLs in August 2018.
• Ubuntu 14.04 trusty EOLs in April 2019.

These are externally set, and affect security support among other things, so they're unfortunately hard deadlines.

The first of these is coming up in just 4 months, which isn't much time if we e.g. want to reengineer this. @Jrbranaa do you have any updates from the code stewardship angle perhaps? Thanks!

there are currently two deadlines here:

Firefox 52 ESR (which this is indirectly based on) EOLs in August 2018

This won't be a problem. While a Node.js port is still the long-term plan, we've gotten Zotero running on Firefox 60, which is the next ESR being released in a couple weeks, and we'll have translation-server running on that around the same time. That will buy us at least another year of a supported underlying platform.

(And while the server will remain up-to-date security-wise, it's worth noting that the attack surface here is much more constrained than Firefox itself. translation-server runs on xpcshell, the XPCOM-integrated JS shell that's part of the Mozilla codebase, so much of Firefox isn't involved, and all requests are processed with XHR and DOMParser, so it's not loading a browser at all, not parsing remote JS, etc. )

Ubuntu 14.04 trusty EOLs in April 2019

This one has nothing to do with translation-server, which runs fine both in Docker and on current Linux distros. We provide instructions for both.

Just as an update, in addition to working on Firefox 60 support, we've made a number of improvements to translation-server since February that might be of interest:

• It will now save basic details from any URL, even without translator coverage. (Previously it required at least some meta tags, but now it will save title/URL/access date from anywhere.)
• You can now pass an arXiv ID to /search in addition to DOI/ISBN/PMID. (Since Citoid is running on an ancient version of translation-server without /search, it's presumably still doing its own identifier lookups. Now that translation-server supports identifiers, and this is core Zotero functionality, you could consider just using /search directly. I'm actually not sure whether there's any functionality Citoid provides that's not now available in translation-server itself (maybe PMCID lookup in addition to PMID, but that would be easy to port).)
• It's now possible to configure translation-server to return CORS headers for use from JavaScript.
• We've added the beginnings of support for generic text search. We're still working on improving the results for this one, but in theory it could allow people to add items just by searching for title/author/year.

Finally, to follow up on @akosiaris's question above:

You mention an API. Got a link handy ? I can't help but wonder if it would make sense for us to use that API instead of hosting our own infrastructure ?

We use translation-server to power various public services, with more coming soon, but I'm afraid we don't provide a public endpoint. Given that the server makes external requests to arbitrary websites, it's not really something we're comfortable hosting for others, since it would be fairly easy for it to get used/abused in a way that caused website operators to block access. Because of this, we think it makes sense for projects to host their own versions (with rate-limiting tailored to the use case). Fortunately, running this is pretty easy, and trivial in a container setup.

@danstillman this is very useful information (and good news!), thank you for the detailed updated! It still seems like the options are either running a Docker image which embeds custom builds of Firefox and Node.js though, which comes with certain maintenance challenges.

Also, note that this is an internal task talking about the ownership of Wikimedia's service, not the software itself (but your updates about where the software is going are very useful to this conversation!). Someone would need to investigate and work on testing and deploying this new FF60-based version to our infrastructure, and this task is about identifying the who/how/what/when :)

"The translators repository has had 6 commits, 5 of them updating from upstream" - note that the recommended way to write/update translators is to do it upstream so that the Zotero and Wikimedia communities both benefit. I don't know to what extent this has happened, but the number of direct commits is not really a useful engagement metric.

In T187194#4224971, @Tgr wrote:

"The translators repository has had 6 commits, 5 of them updating from upstream" - note that the recommended way to write/update translators is to do it upstream so that the Zotero and Wikimedia communities both benefit. I don't know to what extent this has happened, but the number of direct commits is not really a useful engagement metric.

Yes, hence the reason for pointing out 5 out of 6 are updates from upstream, while still adhering to the requested rubric outlined in https://www.mediawiki.org/wiki/Code_stewardship_reviews#Rubric

That number doesn't really mean anything though. The latest mediawiki/services/zotero/translators commit for example bundles 213 upstream commits (and there is no easy way to tell what fraction of those have been written with an intent of improving Citoid).

I'd like to point out that this is about the Zotero translation server. We want to be able to preserve the benefits that the translators give us no matter what.

In T187194#4226047, @Tgr wrote:

[...] (and there is no easy way to tell what fraction of those have been written with an intent of improving Citoid).

Most of the translators that have the v flag set are intended to be used by Citoid actually.

In T187194#4226047, @Tgr wrote:

That number doesn't really mean anything though.

I respectfully disagree. That number caries information about the rate of updates to the zotero translators in WMF infrastructure. I guess that, from that number and others, conjectures can be made about the level of support the service enjoys in our infrastructure.

In T187194#4226548, @mobrovac wrote:

I'd like to point out that this is about the Zotero translation server.

Agreed. Just repeating myself, I am stressing once more that we are talking purely about the WMF infrastructure, not the upstream software(s).

Just a quick update from our end.

We're now using translation-server for ZoteroBib (https://zbib.org), a new web-based service we just launched. And while this is using the latest Firefox 52 ESR–based version, and headless Firefox 60 ESR remains the fallback plan, we've decided to push up work on a Node version after all, in the hopes that we can just switch to that by August. A Node version would presumably make things much easier for whoever takes over stewardship of this for WMF, particularly if containers aren't an easy option, so I just wanted to let you know before someone spends too much time getting the Firefox version running.

I'll update again once we've made some progress with that.

@danstillman thanks for the additional info regarding your developments. As timing is of the essence with this, please keep us apprised of any new developments.

So we need to do something in a very short amount of time (~two months) ­-- does anyone have a game plan? @Jrbranaa what's the latest?

We've gotten translation internals working in Node. We'll be hooking those up to the existing translation-server endpoints and should have a working Node server that you can use within a few weeks.

(FWIW, while Zotero runs a Firefox/xpcshell 52 ESR version of translation-server and will be switching to the Node-based solution by August, the initial post above makes it sound like WMF is running a version based on XULRunner 24.8.1esr-2~deb7u1, so it's not clear to me that the 52 ESR EOL is particularly relevant for you anyway.)

@faidon it looks like there's a plan in place to address the pending EOL, in short Marielle and Marko are working on a NodeJS based solution. Now it's more a question of timing. What happens in August in terms of our infrastructure? Are we able to keep it going for a little while longer or does it get turned off no matter what?

More details about the work/project will be known next week.

Not sure what you're planning, but the initial version of our Node port is up:

https://github.com/zotero/translation-server-v2

Basic single-page web translation is in place, though there might be some holes in coverage. We'll be porting the remaining functionality (multi-page saves, identifier resolution, import/export) over the next couple weeks. We're also planning to add (optional) instantaneous translator updates, as in Zotero, so that sites are fixed as soon as we fix them.

This should be ready for use well in advance of August and should be easy to deploy. Follow that repo for updates, and feel free to create issues there for any problems you run into.

In T187194#4271773, @danstillman wrote:

Not sure what you're planning, but the initial version of our Node port is up:

https://github.com/zotero/translation-server-v2

Basic single-page web translation is in place, though there might be some holes in coverage. We'll be porting the remaining functionality (multi-page saves, identifier resolution, import/export) over the next couple weeks. We're also planning to add (optional) instantaneous translator updates, as in Zotero, so that sites are fixed as soon as we fix them.

This should be ready for use well in advance of August and should be easy to deploy. Follow that repo for updates, and feel free to create issues there for any problems you run into.

YAY you have NO idea how happy this makes us! I had a poke around, looking good! I'll be playing around with it this week for sure.

In T187194#4271773, @danstillman wrote:

Not sure what you're planning, but the initial version of our Node port is up:

https://github.com/zotero/translation-server-v2

Basic single-page web translation is in place, though there might be some holes in coverage. We'll be porting the remaining functionality (multi-page saves, identifier resolution, import/export) over the next couple weeks. We're also planning to add (optional) instantaneous translator updates, as in Zotero, so that sites are fixed as soon as we fix them.

This should be ready for use well in advance of August and should be easy to deploy. Follow that repo for updates, and feel free to create issues there for any problems you run into.

This is great news. Thanks for the update.

We're looking to have Audiences->Contributors->Editing be the Code Stewards for this moving forward.

Added entry to developers/maintainers page. Please augment with more accurate description and list of maintainers.

In T187194#4256587, @faidon wrote:

So we need to do something in a very short amount of time (~two months) ­-- does anyone have a game plan? @Jrbranaa what's the latest?

Hello @faidon, looks like we're on track to have something in place by August. However, if things get delayed, what are our options knowing that we are working towards a solution? More directly, do things go offline or can we keep them active until the solution is ready and deployed?

I've created a separate task to track transition from translation-server to translation-server-v2 (although it is not yet ready for deploy): T197242