zotero translation server: code stewardship request
Closed, ResolvedPublic

Description

The zotero translation-server service was introduced in the infrastructure a few years ago as a supporting service for the citoid service. It's a effectively a firefox extension running in the context of a xulrunner (now defunct software) instance that accepts HTTP requests and reaches to outside sites parsing them for citation information returning that information structured. The service has gone through a number of partial and full outages.

  • Current maintainer:

Services ?
Operations introduced the service after a nonfunctional introduction of it failed to work. T76308 was the task that started it all. Operations back then poured resources into it to get it working correctly since it was a VisualEditor 2014/15 Q3 blockers, but it never adopted the service. Services has inherited the service, at least to my understanding. Unfortunately updating the core software still requires Operations involvement and that has been minimal. T140539 is quite telling. The lack of documentation of a clear process for the service updating is evident in T91646. At the same time a lot of work has gone from the community to enrich the experience. One such outcome is in https://www.mediawiki.org/wiki/Citoid/Creating_Zotero_translators

  • Number, severity, and age of known and confirmed security issues

Unclear. The xulrunner software the powers the service is probably vulnerable to anything listed in https://www.mozilla.org/en-US/security/known-vulnerabilities/firefox/ for versions 25+. We currently run 24.8.1esr-2~deb7u1. Most are probably non exploitable in the zotero environment but no investigation has ever taken place.

  • Was it a cause of production outages or incidents? List them.

While it has never really caused production wide outages, the service itself has suffered from out of memory conditions multiple times with members in the Operations team receiving pages. The tracking task is at T121992, but up to now the action taken has always been to just restart it.

  • Does it have sufficient hardware resources for now and the near future (to take into account expected usage growth)?

Yes.

  • Is it a frequent cause of monitoring alerts that need action, and are they addressed timely and appropriately?

Used to but not so much anymore. The most frequent problem ops were notified about was resolved by turning off monitoring in e0bdcfb5074cf1e13168, as it was non-actionable. Also 2617d4f0a71 and subsequently 57cd2eb7076 have mitigations that kill and restart the service when misbehaving memory wise and that has increased the availability. When an issue does indeed arise, the usual action is a restart of the software

  • When it was first deployed to Wikimedia production

2014

  • Usage statistics based on audience(s) served

Zotero powers part of the citation functionality of VE. Exact numbers of editors using that functionality should be provided.

  • Changes committed in last 1, 3, 6, and 12 months

The service is comprised of 2 repos. The service itself and the translators. The service itself has 1 commits (https://github.com/wikimedia/mediawiki-services-zotero-translation-server/commits/master) in the aforementioned timeperiods. The translators repository has had 6 commits, 5 of them updating from upstream https://github.com/wikimedia/mediawiki-services-zotero-translators/commits/master

  • Reliance on outdated platforms (e.g. operating systems)

OS it runs on is trusty which is going to be EOLed by April 2019 so a solution must be found by then at the latest. The software powering it is xulrunner version 24, which is defunct software. The functionality that allows zotero to work has been removed from firefox 57 as part of the old extensions deprecation and removal. The zotero firefox extension has been restructured to work with zotero standalone but it's unclear if the translation-server will ever manage to update to it (and greater versions). It currently explicitly uses firefox 52 ESR which is already a product in extended support that ends on Aug 1st 2018. It's unclear to us what this means for zotero translation-server.

As a side note, mediawiki-vagrant project had to remove the zotero translation-server service from the list of services it provides due to the above. See 3ee02710c69fb5b97d38ac03b1cc134584510948

  • Number of developers who committed code in the last 1, 3, 6, and 12 months

1

  • Number and age of open patches

0 locally and reporting upstream bugs probably makes no sense

  • Number and age of open bugs

9, dating up to 3 years ago.

  • Number of known dependencies?

citoid partially depends on zotero and VE depends on citoid for the Cite functionality

  • Is there a replacement/alternative for the feature? Is there a plan for a replacement?

Plans have been drawn and discussed at multiple points in time. The most indicative ones are documented in T93579 where the then head of the Services team calls the software a major liability for security, stability and maintainability and in my humble opinion, rightfully so.
Per T93579#2154994, the idea that citoid could directly use the translators instead of the translation-server and provide nearly equal functionality was pitched and mildly supported but never implemented. The only thing that has been made possible is to allow citoid to run without zotero support but at the cost of greatly reduced functionality (https://gerrit.wikimedia.org/r/317168)

In the meantime, the zotero project itself went through a limbo state were plans were put forward (but seemingly never realized). Those are documented in https://groups.google.com/forum/#!searchin/zotero-dev/xulrunner%7Csort:relevance/zotero-dev/yy4-q_ZUA4M/1YjAx5SODAAJ, with a saying of We will be porting Zotero to Electron. As far as I can tell this has never happened.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 13 2018, 2:05 PM

Zotero dev here. I'm not clear on when all the above was written, but a few clarifications.

  • translation-server currently runs on Firefox 52 ESR, so as noted above the underlying Mozilla framework is supported until August 2018.
  • The current version available on Docker Hub is based on Zotero 4.0, but a version based on Zotero 5.0 has been available on GitHub since June 2017, and that's what we'd recommend using. We'll push that to Docker Hub soon.
  • The recommended configuration is to run translation-server in Docker, so admin requirements should be minimal. There've been memory leaks in the past, which may or may not still exist — we've ignored them by just restarting it in a cron job twice a day.
  • The Zotero project itself is extremely active and is certainly not in a limbo state. We still plan to port the Zotero client to Electron on some timeline (depending on whether we're able to get Zotero running on Firefox 60 ESR), but that's largely unrelated to translation-server, which can likely be ported to Node or Electron much more easily. That's something we plan to work on over the next couple months. (We initially implemented translation-server in a very early version of Node.js, and switched to XULRunner for performance reasons due to the limited JS DOM options available in Node at the time. Between Electron and improved DOM options in Node, I suspect this is no longer an issue.)
  • translation-server is an important part of our infrastructure and is in no way abandoned. translation-server is used by our website and API, and we have a major new feature based around it that's launching soon. We recently added a /search endpoint that can translate DOIs, ISBNs, and PubMed IDs, and will soon be adding the ability to save any URL, not just ones with translator coverage, so it's under active development.

The extent of translation-server use in Wikimedia projects has never been entirely clear to me, since we mostly just see the occasional translator error report from @Mvolz, but we're happy to work more closely if desired. Questions can always be posted to GitHub or the zotero-dev mailing list.

Anyway, hope this helps! Let me know if you have any questions.

Mvolz added a subscriber: DarTar.Feb 14 2018, 8:09 AM
fgiunchedi triaged this task as Normal priority.Feb 14 2018, 9:14 AM

Zotero dev here. I'm not clear on when all the above was written, but a few clarifications.

Hello @danstillman, great to have you here, your input is greatly appreciated. This was written just 4 hours before your comment, so your very fast :-). As a clarification this task is not about zotero translation-server THE software, but rather the installation in WMF.

  • translation-server currently runs on Firefox 52 ESR, so as noted above the underlying Mozilla framework is supported until August 2018.
  • The current version available on Docker Hub is based on Zotero 4.0, but a version based on Zotero 5.0 has been available on GitHub since June 2017, and that's what we'd recommend using. We'll push that to Docker Hub soon.

That would be great! A couple of notes, and they relevant to our installation alone, and hence this task, but not zotero translation-server as a software.

  • We do not currently have an infrastructure for running docker containers. We have been however creating a kubernetes infrastructure and plan to have our first service deployed this quarter so this is going to change really soon.
  • In the above said infrastructure we have taken an approach of "we only trust containers we build ourselves" for security reasons. Yes I know docker supports notaries and docker images can be signed these days and yes the entire discussion of trusting things off pypi/npm/rubygems does not inherently differ much but we will feel that running images directly off Docker Hub is not currently worth the risk.
  • The recommended configuration is to run translation-server in Docker, so admin requirements should be minimal. There've been memory leaks in the past, which may or may not still exist — we've ignored them by just restarting it in a cron job twice a day.

Indeed deployment requirements do become less of problem in a container environment, hence our move to a kubernetes infrastructure, but please operational issues still stand (and can be anything but minimal at a large scale)

Also, Docker supports memory limits and they are bound to be more responsive (albeit perhaps a bit less flexible) than a cronjob so perhaps the recommended configuration should include a --memory=<x>g recommendation ? I am guessing you are the best people around to provide a value for <x>

  • The Zotero project itself is extremely active and is certainly not in a limbo state. We still plan to port the Zotero client to Electron on some timeline (depending on whether we're able to get Zotero running on Firefox 60 ESR), but that's largely unrelated to translation-server, which can likely be ported to Node or Electron much more easily. That's something we plan to work on over the next couple months. (We initially implemented translation-server in a very early version of Node.js, and switched to XULRunner for performance reasons due to the limited JS DOM options available in Node at the time. Between Electron and improved DOM options in Node, I suspect this is no longer an issue.)

That's great to hear! As you note the client itself is not really relevant to us, it's translation-server we care about. From your answer I understand this process hasn't even started, so as WMF we should probably wait for a production ready release before we even begin contemplating our next move.

  • translation-server is an important part of our infrastructure and is in no way abandoned. translation-server is used by our website and API, and we have a major new feature based around it that's launching soon. We recently added a /search endpoint that can translate DOIs, ISBNs, and PubMed IDs, and will soon be adding the ability to save any URL, not just ones with translator coverage, so it's under active development.

You mention an API. Got a link handy ? I can't help but wonder if it would make sense for us to use that API instead of hosting our own infrastructure ?

The extent of translation-server use in Wikimedia projects has never been entirely clear to me, since we mostly just see the occasional translator error report from @Mvolz, but we're happy to work more closely if desired. Questions can always be posted to GitHub or the zotero-dev mailing list.

TL;DR is that VisualEditor uses citoid to generate citations that can be directly embedded in wiki articles. Citoid uses zotero translation-server for much of that functionality. A starting link for Citoid would be https://www.mediawiki.org/wiki/Citoid. If you need/want more in-depth information, please ask.

Anyway, hope this helps! Let me know if you have any questions.

It sure does, thanks for weighing in!

So, this task has been open for a couple of months now, with the underlying issues have been present for far longer than that. In case it wasn't clear from the lengthy and detailed task description, there are currently two deadlines here:

  • Firefox 52 ESR (which this is indirectly based on) EOLs in August 2018.
  • Ubuntu 14.04 trusty EOLs in April 2019.

These are externally set, and affect security support among other things, so they're unfortunately hard deadlines.

The first of these is coming up in just 4 months, which isn't much time if we e.g. want to reengineer this. @Jrbranaa do you have any updates from the code stewardship angle perhaps? Thanks!

there are currently two deadlines here:

Firefox 52 ESR (which this is indirectly based on) EOLs in August 2018

This won't be a problem. While a Node.js port is still the long-term plan, we've gotten Zotero running on Firefox 60, which is the next ESR being released in a couple weeks, and we'll have translation-server running on that around the same time. That will buy us at least another year of a supported underlying platform.

(And while the server will remain up-to-date security-wise, it's worth noting that the attack surface here is much more constrained than Firefox itself. translation-server runs on xpcshell, the XPCOM-integrated JS shell that's part of the Mozilla codebase, so much of Firefox isn't involved, and all requests are processed with XHR and DOMParser, so it's not loading a browser at all, not parsing remote JS, etc. )

Ubuntu 14.04 trusty EOLs in April 2019

This one has nothing to do with translation-server, which runs fine both in Docker and on current Linux distros. We provide instructions for both.

Just as an update, in addition to working on Firefox 60 support, we've made a number of improvements to translation-server since February that might be of interest:

  • It will now save basic details from any URL, even without translator coverage. (Previously it required at least some meta tags, but now it will save title/URL/access date from anywhere.)
  • You can now pass an arXiv ID to /search in addition to DOI/ISBN/PMID. (Since Citoid is running on an ancient version of translation-server without /search, it's presumably still doing its own identifier lookups. Now that translation-server supports identifiers, and this is core Zotero functionality, you could consider just using /search directly. I'm actually not sure whether there's any functionality Citoid provides that's not now available in translation-server itself (maybe PMCID lookup in addition to PMID, but that would be easy to port).)
  • It's now possible to configure translation-server to return CORS headers for use from JavaScript.
  • We've added the beginnings of support for generic text search. We're still working on improving the results for this one, but in theory it could allow people to add items just by searching for title/author/year.

Finally, to follow up on @akosiaris's question above:

You mention an API. Got a link handy ? I can't help but wonder if it would make sense for us to use that API instead of hosting our own infrastructure ?

We use translation-server to power various public services, with more coming soon, but I'm afraid we don't provide a public endpoint. Given that the server makes external requests to arbitrary websites, it's not really something we're comfortable hosting for others, since it would be fairly easy for it to get used/abused in a way that caused website operators to block access. Because of this, we think it makes sense for projects to host their own versions (with rate-limiting tailored to the use case). Fortunately, running this is pretty easy, and trivial in a container setup.

@danstillman this is very useful information (and good news!), thank you for the detailed updated! It still seems like the options are either running a Docker image which embeds custom builds of Firefox and Node.js though, which comes with certain maintenance challenges.

Also, note that this is an internal task talking about the ownership of Wikimedia's service, not the software itself (but your updates about where the software is going are very useful to this conversation!). Someone would need to investigate and work on testing and deploying this new FF60-based version to our infrastructure, and this task is about identifying the who/how/what/when :)

Tgr added a subscriber: Tgr.May 23 2018, 12:18 PM

"The translators repository has had 6 commits, 5 of them updating from upstream" - note that the recommended way to write/update translators is to do it upstream so that the Zotero and Wikimedia communities both benefit. I don't know to what extent this has happened, but the number of direct commits is not really a useful engagement metric.

"The translators repository has had 6 commits, 5 of them updating from upstream" - note that the recommended way to write/update translators is to do it upstream so that the Zotero and Wikimedia communities both benefit. I don't know to what extent this has happened, but the number of direct commits is not really a useful engagement metric.

Yes, hence the reason for pointing out 5 out of 6 are updates from upstream, while still adhering to the requested rubric outlined in https://www.mediawiki.org/wiki/Code_stewardship_reviews#Rubric

Tgr added a comment.May 23 2018, 4:33 PM

That number doesn't really mean anything though. The latest mediawiki/services/zotero/translators commit for example bundles 213 upstream commits (and there is no easy way to tell what fraction of those have been written with an intent of improving Citoid).

I'd like to point out that this is about the Zotero translation server. We want to be able to preserve the benefits that the translators give us no matter what.

[...] (and there is no easy way to tell what fraction of those have been written with an intent of improving Citoid).

Most of the translators that have the v flag set are intended to be used by Citoid actually.

That number doesn't really mean anything though.

I respectfully disagree. That number caries information about the rate of updates to the zotero translators in WMF infrastructure. I guess that, from that number and others, conjectures can be made about the level of support the service enjoys in our infrastructure.

I'd like to point out that this is about the Zotero translation server.

Agreed. Just repeating myself, I am stressing once more that we are talking purely about the WMF infrastructure, not the upstream software(s).

Just a quick update from our end.

We're now using translation-server for ZoteroBib (https://zbib.org), a new web-based service we just launched. And while this is using the latest Firefox 52 ESR–based version, and headless Firefox 60 ESR remains the fallback plan, we've decided to push up work on a Node version after all, in the hopes that we can just switch to that by August. A Node version would presumably make things much easier for whoever takes over stewardship of this for WMF, particularly if containers aren't an easy option, so I just wanted to let you know before someone spends too much time getting the Firefox version running.

I'll update again once we've made some progress with that.

Restricted Application added a project: VisualEditor. · View Herald TranscriptMay 29 2018, 9:23 PM

@danstillman thanks for the additional info regarding your developments. As timing is of the essence with this, please keep us apprised of any new developments.

Mvolz moved this task from Backlog to Zotero & External on the Citoid board.Jun 2 2018, 9:56 AM

So we need to do something in a very short amount of time (~two months) ­-- does anyone have a game plan? @Jrbranaa what's the latest?

We've gotten translation internals working in Node. We'll be hooking those up to the existing translation-server endpoints and should have a working Node server that you can use within a few weeks.

(FWIW, while Zotero runs a Firefox/xpcshell 52 ESR version of translation-server and will be switching to the Node-based solution by August, the initial post above makes it sound like WMF is running a version based on XULRunner 24.8.1esr-2~deb7u1, so it's not clear to me that the 52 ESR EOL is particularly relevant for you anyway.)

@faidon it looks like there's a plan in place to address the pending EOL, in short Marielle and Marko are working on a NodeJS based solution. Now it's more a question of timing. What happens in August in terms of our infrastructure? Are we able to keep it going for a little while longer or does it get turned off no matter what?

More details about the work/project will be known next week.

Not sure what you're planning, but the initial version of our Node port is up:

https://github.com/zotero/translation-server-v2

Basic single-page web translation is in place, though there might be some holes in coverage. We'll be porting the remaining functionality (multi-page saves, identifier resolution, import/export) over the next couple weeks. We're also planning to add (optional) instantaneous translator updates, as in Zotero, so that sites are fixed as soon as we fix them.

This should be ready for use well in advance of August and should be easy to deploy. Follow that repo for updates, and feel free to create issues there for any problems you run into.

Not sure what you're planning, but the initial version of our Node port is up:

https://github.com/zotero/translation-server-v2

Basic single-page web translation is in place, though there might be some holes in coverage. We'll be porting the remaining functionality (multi-page saves, identifier resolution, import/export) over the next couple weeks. We're also planning to add (optional) instantaneous translator updates, as in Zotero, so that sites are fixed as soon as we fix them.

This should be ready for use well in advance of August and should be easy to deploy. Follow that repo for updates, and feel free to create issues there for any problems you run into.

YAY you have NO idea how happy this makes us! I had a poke around, looking good! I'll be playing around with it this week for sure.

Macro wikilove: this is the best

Not sure what you're planning, but the initial version of our Node port is up:

https://github.com/zotero/translation-server-v2

Basic single-page web translation is in place, though there might be some holes in coverage. We'll be porting the remaining functionality (multi-page saves, identifier resolution, import/export) over the next couple weeks. We're also planning to add (optional) instantaneous translator updates, as in Zotero, so that sites are fixed as soon as we fix them.

This should be ready for use well in advance of August and should be easy to deploy. Follow that repo for updates, and feel free to create issues there for any problems you run into.

This is great news. Thanks for the update.

We're looking to have Audiences->Contributors->Editing be the Code Stewards for this moving forward.

Added entry to developers/maintainers page. Please augment with more accurate description and list of maintainers.

So we need to do something in a very short amount of time (~two months) ­-- does anyone have a game plan? @Jrbranaa what's the latest?

Hello @faidon, looks like we're on track to have something in place by August. However, if things get delayed, what are our options knowing that we are working towards a solution? More directly, do things go offline or can we keep them active until the solution is ready and deployed?

Mvolz added a comment.Jun 14 2018, 3:40 PM

I've created a separate task to track transition from translation-server to translation-server-v2 (although it is not yet ready for deploy): T197242

Jrbranaa closed this task as Resolved.Jul 9 2018, 7:27 PM
Jrbranaa claimed this task.

Closing this as resolved as Audiences->Contributors has been identified as Code Stewards moving forward and plan is in place and being worked on actively.

Restricted Application added a project: User-Ryasmeen. · View Herald TranscriptJul 9 2018, 7:27 PM