Deploy translation-server-v2
Closed, ResolvedPublic

Description

Deploy https://github.com/zotero/translation-server (old translation-server now at https://github.com/zotero/translation-server-old)

Some challenges:

"The new zotero translation server uses async/await which are only available in Node 8+, and we still use Node 6 in production. There is a possibility of putting the new server into Kubernetes (which would allows us to automagically upgrade to Node 8), but that has some drawbacks currently."

Mvolz created this task.Aug 9 2018, 1:08 PM
Mvolz triaged this task as High priority.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 9 2018, 1:08 PM
Restricted Application edited projects, added Services; removed Services (next). · View Herald TranscriptAug 9 2018, 3:08 PM
Mvolz moved this task from Backlog to Zotero & External on the Citoid board.Sep 12 2018, 1:02 PM

Change 461627 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/docker-images/production-images@master] Add nodejs10 docker production image

https://gerrit.wikimedia.org/r/461627

Change 461627 merged by Alexandros Kosiaris:
[operations/docker-images/production-images@master] Add nodejs10 docker production image

https://gerrit.wikimedia.org/r/461627

@thcipriani, @dduvall, nodejs 10 image built and uploaded. It's the "slim" variant, I was wondering if it is enough or if we also require the "devel" variant

@thcipriani, @dduvall, nodejs 10 image built and uploaded. It's the "slim" variant, I was wondering if it is enough or if we also require the "devel" variant

We'll need npm to install the test dependencies and run the tests in the pipeline, so we'll need a -devel variant for that.

Mvolz updated the task description. (Show Details)Sep 24 2018, 12:10 PM

Change 462503 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/docker-images/production-images@master] Add nodejs10-devel docker production image

https://gerrit.wikimedia.org/r/462503

Change 462503 merged by Alexandros Kosiaris:
[operations/docker-images/production-images@master] Add nodejs10-devel docker production image

https://gerrit.wikimedia.org/r/462503

@thcipriani, @dduvall, nodejs 10 image built and uploaded. It's the "slim" variant, I was wondering if it is enough or if we also require the "devel" variant

We'll need npm to install the test dependencies and run the tests in the pipeline, so we'll need a -devel variant for that.

Done. nodejs10-devel image built and uploaded

@Mvolz, SRE has a question about this migration. Assuming this gets deployed successfully next quarter, this will allow us to migrate off the current zotero infrastructure and thus remove it from WMF, right ? Does this sound plausible ?

Mvolz added a comment.Sep 24 2018, 7:14 PM

@Mvolz, SRE has a question about this migration. Assuming this gets deployed successfully next quarter, this will allow us to migrate off the current zotero infrastructure and thus remove it from WMF, right ? Does this sound plausible ?

Yes, this is a complete replacement for the current zotero architecture.

@thcipriani, @dduvall, nodejs 10 image built and uploaded. It's the "slim" variant, I was wondering if it is enough or if we also require the "devel" variant

We'll need npm to install the test dependencies and run the tests in the pipeline, so we'll need a -devel variant for that.

Done. nodejs10-devel image built and uploaded

Seems to work! Do we have a repository for this on gerrit yet?

Once we have a repo in gerrit I have the start of a .pipeline/blubber.yaml this will get tests running in CI at least:

---
version: v3
base: docker-registry.wikimedia.org/nodejs10-slim:0.0.1

variants:
  build:
    base: docker-registry.wikimedia.org/nodejs10-devel:0.0.1
    apt: { packages: [git] }
    node:
      requirements: [package.json, package-lock.json]

  test:
    includes: [build]
    runs:
      insecurely: true
    entrypoint: [npm, test]

building via blubber results with: 9 passing (967ms)

https://gerrit.wikimedia.org/g/mediawiki/services/zotero/+/refs/heads/master would be the repository @Mvolz I just created it and gave it the same permissions as the zotero/translators repo in gerrit. Let me know if anything else is required.

Mvolz added a comment.EditedSep 28 2018, 10:35 AM

https://gerrit.wikimedia.org/g/mediawiki/services/zotero/+/refs/heads/master would be the repository @Mvolz I just created it and gave it the same permissions as the zotero/translators repo in gerrit. Let me know if anything else is required.

Great, I'll test it today/Monday with citoid and hopefully get a patch up soon. That repo points to and uses two submodules hosted on github - does anything need to be done for that? My inclination is to leave it if that's okay with SRE.

https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/services/zotero/+/refs/heads/master/.gitmodules

Great, I'll test it today/Monday with citoid and hopefully get a patch up soon. That repo points to and uses two submodules hosted on gerrit - does anything need to be done for that? My inclination is to leave it if that's okay with SRE.

https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/services/zotero/+/refs/heads/master/.gitmodules

I think we can leave them as-are for now. If the need arises to modify the translators (hopefully we won't need to be doing that any more), we can revert back to our fork.

Mvolz added a comment.Oct 2 2018, 6:39 AM

This version is now tested the compatible patch for citoid is here: https://gerrit.wikimedia.org/r/#/c/mediawiki/services/citoid/+/463713/

Change 464041 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[mediawiki/services/zotero@master] Add .pipeline files for zotero

https://gerrit.wikimedia.org/r/464041

Change 464048 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[integration/config@master] Add mediawiki/servies/zuul to the CD Pipeline

https://gerrit.wikimedia.org/r/464048

Change 464048 merged by jenkins-bot:
[integration/config@master] Add mediawiki/services/zotero to the CD Pipeline

https://gerrit.wikimedia.org/r/464048

Added test .pipeline files for zotero.

Now running through CI and tests are passing: https://gerrit.wikimedia.org/r/c/mediawiki/services/zotero/+/464041/

Change 466287 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] First draft of a zotero helm chart

https://gerrit.wikimedia.org/r/466287

Change 464041 merged by jenkins-bot:
[mediawiki/services/zotero@master] Add .pipeline files for zotero

https://gerrit.wikimedia.org/r/464041

Change 468605 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[integration/config@master] Temporarily remove helm test for zotero

https://gerrit.wikimedia.org/r/468605

Change 468605 abandoned by Thcipriani:
Temporarily remove helm test for zotero

https://gerrit.wikimedia.org/r/468605

I was able to generate the image docker-registry.wikimedia.org/wikimedia/mediawiki-services-zotero:20181019165254-production which runs, and listens on port 1969. Not sure how to exercise the image. Also, we'll need to add a .pipeline/helm.yaml chart to point it to the correct helm chart whenever that is finalized. For now, I've triggered a pipeline build that skips helm test to create the above image.

Mvolz added a comment.Nov 5 2018, 10:15 AM

Not sure how to exercise the image.

Do you need sample requests?

What's need to finish this up?

Mentioned in SAL (#wikimedia-operations) [2018-11-06T15:37:42Z] <akosiaris> create zotero namespace in eqiad, codfw, staging cluster T201611

Change 471992 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Add zotero kubernetes hiera stanzas

https://gerrit.wikimedia.org/r/471992

Change 471992 merged by Alexandros Kosiaris:
[operations/puppet@production] Add zotero kubernetes hiera stanzas

https://gerrit.wikimedia.org/r/471992

Change 473727 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/dns@master] Introduce zoterov2 LVS IPs

https://gerrit.wikimedia.org/r/473727

Change 473733 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Introduce zoterov2 LVS endpoint

https://gerrit.wikimedia.org/r/473733

mobrovac moved this task from Backlog to watching on the Services board.Nov 16 2018, 3:24 PM
mobrovac edited projects, added Services (watching); removed Services.

This has now been deployed to the kubernetes staging cluster.

akosiaris@deploy1001:~$ curl -d 'http://www.nytimes.com/2018/06/11/technology/net-neutrality-repeal.html' -H 'Content-Type: text/plain' kubestage1002.eqiad.wmnet:26568/web
[{"key":"R5JQVA9J","version":0,"itemType":"newspaperArticle","creators":[{"firstName":"Keith","lastName":"Collins","creatorType":"author"}],"tags":[{"tag":"Net Neutrality","type":1},{"tag":"Pai, Ajit","type":1},{"tag":"Federal Communications Commission","type":1},{"tag":"Regulation and Deregulation of Industry","type":1},{"tag":"Computers and the Internet","type":1}],"title":"Net Neutrality Has Officially Been Repealed. Here’s How That Could Affect You.","section":"Technology","url":"https://www.nytimes.com/2018/06/11/technology/net-neutrality-repeal.html","abstractNote":"Net Neutrality rules that required internet service providers to offer equal access to all web content are no longer in effect as of Monday.","language":"en-US","libraryCatalog":"NYTimes.com","accessDate":"2018-11-17T16:26:52Z","date":"2018-06-11","publicationTitle":"The New York Times","ISSN":"0362-4331"}]

Moving on to production next week

Mvolz added a comment.EditedSun, Nov 18, 8:32 AM

This has now been deployed to the kubernetes staging cluster.

akosiaris@deploy1001:~$ curl -d 'http://www.nytimes.com/2018/06/11/technology/net-neutrality-repeal.html' -H 'Content-Type: text/plain' kubestage1002.eqiad.wmnet:26568/web
[{"key":"R5JQVA9J","version":0,"itemType":"newspaperArticle","creators":[{"firstName":"Keith","lastName":"Collins","creatorType":"author"}],"tags":[{"tag":"Net Neutrality","type":1},{"tag":"Pai, Ajit","type":1},{"tag":"Federal Communications Commission","type":1},{"tag":"Regulation and Deregulation of Industry","type":1},{"tag":"Computers and the Internet","type":1}],"title":"Net Neutrality Has Officially Been Repealed. Here’s How That Could Affect You.","section":"Technology","url":"https://www.nytimes.com/2018/06/11/technology/net-neutrality-repeal.html","abstractNote":"Net Neutrality rules that required internet service providers to offer equal access to all web content are no longer in effect as of Monday.","language":"en-US","libraryCatalog":"NYTimes.com","accessDate":"2018-11-17T16:26:52Z","date":"2018-06-11","publicationTitle":"The New York Times","ISSN":"0362-4331"}]

Moving on to production next week

Yay, thank you so much :).
Macro itshappening:

Change 466287 merged by Alexandros Kosiaris:
[operations/deployment-charts@master] First draft of a zotero helm chart

https://gerrit.wikimedia.org/r/466287

Change 473727 merged by Alexandros Kosiaris:
[operations/dns@master] Introduce zoterov2 LVS IPs

https://gerrit.wikimedia.org/r/473727

Change 475303 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Introduce zoterov2 LVS conftool data

https://gerrit.wikimedia.org/r/475303

Change 475304 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Add zoterov2 LVS ip block and realserver config

https://gerrit.wikimedia.org/r/475304

Change 475303 merged by Alexandros Kosiaris:
[operations/puppet@production] Introduce zoterov2 LVS conftool data

https://gerrit.wikimedia.org/r/475303

Change 475304 merged by Alexandros Kosiaris:
[operations/puppet@production] Add zoterov2 LVS ip block and realserver config

https://gerrit.wikimedia.org/r/475304

Change 473733 merged by Alexandros Kosiaris:
[operations/puppet@production] Introduce zoterov2 LVS endpoint

https://gerrit.wikimedia.org/r/473733

akosiaris closed this task as Resolved.EditedFri, Nov 23, 9:07 AM

Finally deployed to production

Things to note:

  • zotero is configured to use the same outoing proxy (url-downloader) as all other services. It does honor the NO_PROXY variables and it is set to the following domains wikipedia.org,wiktionary.org,wikiquote.org,wikibooks.org,wikiquote.org,wikinews.org,wikisource.org,wikiversity.org,wikivoyage.org,www.wikidata.org,meta.wikimedia.org,commons.wikimedia.org,www.mediawiki.org as access to these should not happen via the proxy. The list is taken from https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/CommonSettings.php#L147 (and my take is that until we have a better way to document/implement this, all services should honor this as the canonical list)
  • There are 2 LVS services (1 per DC), zoterov2.svc.eqiad.wmnet and zoterov2.svc.codfw.wmnet. Those are TEMPORARY and are meant to facilitate rolling back citoid to the previous working translation-server installation in case all hell breaks loose. When the citoid translation to zotero v2 is done, the old LVS LVS IPs/DNS will point to the new infrastructure and the zoterov2 ones WILL BE removed.
  • There is no discovery DNS (like zoterov2.discovery.wmnet) for these as they are TEMPORARY and will be removed. However when the old LVS IPs/DNS are moved to the new infrastructure per above, zotero.discovery.wmnet will be working fine.
  • There is no uniform and standardized logging unfortunately as zotero translation-server logs everything in stdout in a very weird format, so no logstash. However getting the logs is possible by something like.
akosiaris@deploy1001$ KUBECONFIG=/etc/kubernetes/zotero-eqiad.config kubectl -n zotero logs zotero-production-b68dbb465-22wbt

The zotero-production-b68dbb465-22wbt above is a pod name. Getting the pod names is done by

akosiaris@deploy1001:~$ KUBECONFIG=/etc/kubernetes/zotero-eqiad.config kubectl -n zotero get pods

This was true before as well, the minor improvement is that now logs are accessible to devs devs. This should be documented (I 'll do so)

  • There are no metrics as zotero translation-server does not expose metrics in any way. This was true before as well.
  • Health monitoring is minimal as zotero translation-server does not expose any easily monitored interface. So we are doing just TCP health checks per pod and a global HTTP POST check at the LVS level. This was true before as well. That one looks easy to fix by adding swagger specs to the upstream software. I 'll try and have a PR next week.

An example curl looks like

akosiaris@deploy1001:~$ curl -d 'http://www.nytimes.com/2018/06/11/technology/net-neutrality-repeal.html' -H 'Content-Type: text/plain' zoterov2.svc.eqiad.wmnet:1968/web
[{"key":"WK9MLRT7","version":0,"itemType":"newspaperArticle","creators":[{"firstName":"Keith","lastName":"Collins","creatorType":"author"}],"tags":[{"tag":"Net Neutrality","type":1},{"tag":"Pai, Ajit","type":1},{"tag":"Federal Communications Commission","type":1},{"tag":"Regulation and Deregulation of Industry","type":1},{"tag":"Computers and the Internet","type":1}],"title":"Net Neutrality Has Officially Been Repealed. Here’s How That Could Affect You.","section":"Technology","url":"https://www.nytimes.com/2018/06/11/technology/net-neutrality-repeal.html","abstractNote":"Net Neutrality rules that required internet service providers to offer equal access to all web content are no longer in effect as of Monday.","language":"en-US","libraryCatalog":"NYTimes.com","accessDate":"2018-11-23T08:41:17Z","date":"2018-06-11","publicationTitle":"The New York Times","ISSN":"0362-4331"}]

And for codfw

akosiaris@deploy1001:~$ curl -d 'http://www.nytimes.com/2018/06/11/technology/net-neutrality-rov2.svc.codfw.wmnet:1968/web: text/plain' zotertov2.svc.codfw.wmnet:1968/web
[{"key":"RFRC8EHV","version":0,"itemType":"newspaperArticle","creators":[{"firstName":"Keith","lastName":"Collins","creatorType":"author"}],"tags":[{"tag":"Net Neutrality","type":1},{"tag":"Pai, Ajit","type":1},{"tag":"Federal Communications Commission","type":1},{"tag":"Regulation and Deregulation of Industry","type":1},{"tag":"Computers and the Internet","type":1}],"title":"Net Neutrality Has Officially Been Repealed. Here’s How That Could Affect You.","section":"Technology","url":"https://www.nytimes.com/2018/06/11/technology/net-neutrality-repeal.html","abstractNote":"Net Neutrality rules that required internet service providers to offer equal access to all web content are no longer in effect as of Monday.","language":"en-US","libraryCatalog":"NYTimes.com","accessDate":"2018-11-23T08:41:28Z","date":"2018-06-11","publicationTitle":"The New York Times","ISSN":"0362-4331"}]

I 'll resolve this, feel free to reopen

@akosiaris: I think your last paste is slightly broken (most of the URL got muddled with the data)

Does this mean that there should be a zotero v2 instance made in deployment-prep?

@akosiaris: I think your last paste is slightly broken (most of the URL got muddled with the data)

Fixed.

Does this mean that there should be a zotero v2 instance made in deployment-prep?

I guess if we are to maintain functionality and status quo, yes. That being said, it's not yet at all clear when, how and where exactly this will happen

marcella added a subscriber: Ryasmeen.

@Mvolz can you work with @Ryasmeen to identify whether there are any specific QA needs we should resolve?

Restricted Application added a project: User-Ryasmeen. · View Herald TranscriptMon, Nov 26, 3:03 PM