Deploy https://github.com/zotero/translation-server (old translation-server now at https://github.com/zotero/translation-server-old)
Some challenges:
"The new zotero translation server uses async/await which are only available in Node 8+, and we still use Node 6 in production. There is a possibility of putting the new server into Kubernetes (which would allows us to automagically upgrade to Node 8), but that has some drawbacks currently."
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Resolved | Sebastian_Berlin-WMSE | T187023 Create Zotero translator for sverigesradio.se | ||
| Resolved | Krenair | T204500 cloudvps: deployment-prep project trusty deprecation | ||
| Resolved | None | T204467 Update Citoid with translators for Swedish news sites | ||
| Resolved | Mvolz | T197242 Transition citoid to use Zotero's translation-server-v2 | ||
| Open | None | T198901 Migrate production services to kubernetes using the pipeline | ||
| Resolved | thcipriani | T205919 TEC3:O3:O3.1:Q2 Goal - Move Blubberoid, ZoteroV2, and Graphoid through the production CD Pipeline | ||
| Resolved | mobrovac | T201611 Deploy translation-server-v2 |
Change 461627 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/docker-images/production-images@master] Add nodejs10 docker production image
Change 461627 merged by Alexandros Kosiaris:
[operations/docker-images/production-images@master] Add nodejs10 docker production image
@thcipriani, @dduvall, nodejs 10 image built and uploaded. It's the "slim" variant, I was wondering if it is enough or if we also require the "devel" variant
We'll need npm to install the test dependencies and run the tests in the pipeline, so we'll need a -devel variant for that.
Change 462503 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/docker-images/production-images@master] Add nodejs10-devel docker production image
Change 462503 merged by Alexandros Kosiaris:
[operations/docker-images/production-images@master] Add nodejs10-devel docker production image
@Mvolz, SRE has a question about this migration. Assuming this gets deployed successfully next quarter, this will allow us to migrate off the current zotero infrastructure and thus remove it from WMF, right ? Does this sound plausible ?
Seems to work! Do we have a repository for this on gerrit yet?
Once we have a repo in gerrit I have the start of a .pipeline/blubber.yaml this will get tests running in CI at least:
---
version: v3
base: docker-registry.wikimedia.org/nodejs10-slim:0.0.1
variants:
build:
base: docker-registry.wikimedia.org/nodejs10-devel:0.0.1
apt: { packages: [git] }
node:
requirements: [package.json, package-lock.json]
test:
includes: [build]
runs:
insecurely: true
entrypoint: [npm, test]building via blubber results with: 9 passing (967ms)
https://gerrit.wikimedia.org/g/mediawiki/services/zotero/+/refs/heads/master would be the repository @Mvolz I just created it and gave it the same permissions as the zotero/translators repo in gerrit. Let me know if anything else is required.
Great, I'll test it today/Monday with citoid and hopefully get a patch up soon. That repo points to and uses two submodules hosted on github - does anything need to be done for that? My inclination is to leave it if that's okay with SRE.
I think we can leave them as-are for now. If the need arises to modify the translators (hopefully we won't need to be doing that any more), we can revert back to our fork.
This version is now tested the compatible patch for citoid is here: https://gerrit.wikimedia.org/r/#/c/mediawiki/services/citoid/+/463713/
Change 464041 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[mediawiki/services/zotero@master] Add .pipeline files for zotero
Change 464048 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[integration/config@master] Add mediawiki/servies/zuul to the CD Pipeline
Change 464048 merged by jenkins-bot:
[integration/config@master] Add mediawiki/services/zotero to the CD Pipeline
Added test .pipeline files for zotero.
Now running through CI and tests are passing: https://gerrit.wikimedia.org/r/c/mediawiki/services/zotero/+/464041/
Change 466287 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] First draft of a zotero helm chart
Change 464041 merged by jenkins-bot:
[mediawiki/services/zotero@master] Add .pipeline files for zotero
Change 468605 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[integration/config@master] Temporarily remove helm test for zotero
I was able to generate the image docker-registry.wikimedia.org/wikimedia/mediawiki-services-zotero:20181019165254-production which runs, and listens on port 1969. Not sure how to exercise the image. Also, we'll need to add a .pipeline/helm.yaml chart to point it to the correct helm chart whenever that is finalized. For now, I've triggered a pipeline build that skips helm test to create the above image.
Mentioned in SAL (#wikimedia-operations) [2018-11-06T15:37:42Z] <akosiaris> create zotero namespace in eqiad, codfw, staging cluster T201611
Change 471992 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Add zotero kubernetes hiera stanzas
Change 471992 merged by Alexandros Kosiaris:
[operations/puppet@production] Add zotero kubernetes hiera stanzas
Change 473727 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/dns@master] Introduce zoterov2 LVS IPs
Change 473733 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Introduce zoterov2 LVS endpoint
This has now been deployed to the kubernetes staging cluster.
akosiaris@deploy1001:~$ curl -d 'http://www.nytimes.com/2018/06/11/technology/net-neutrality-repeal.html' -H 'Content-Type: text/plain' kubestage1002.eqiad.wmnet:26568/web
[{"key":"R5JQVA9J","version":0,"itemType":"newspaperArticle","creators":[{"firstName":"Keith","lastName":"Collins","creatorType":"author"}],"tags":[{"tag":"Net Neutrality","type":1},{"tag":"Pai, Ajit","type":1},{"tag":"Federal Communications Commission","type":1},{"tag":"Regulation and Deregulation of Industry","type":1},{"tag":"Computers and the Internet","type":1}],"title":"Net Neutrality Has Officially Been Repealed. Here’s How That Could Affect You.","section":"Technology","url":"https://www.nytimes.com/2018/06/11/technology/net-neutrality-repeal.html","abstractNote":"Net Neutrality rules that required internet service providers to offer equal access to all web content are no longer in effect as of Monday.","language":"en-US","libraryCatalog":"NYTimes.com","accessDate":"2018-11-17T16:26:52Z","date":"2018-06-11","publicationTitle":"The New York Times","ISSN":"0362-4331"}]Moving on to production next week
Change 466287 merged by Alexandros Kosiaris:
[operations/deployment-charts@master] First draft of a zotero helm chart
Change 473727 merged by Alexandros Kosiaris:
[operations/dns@master] Introduce zoterov2 LVS IPs
Change 475303 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Introduce zoterov2 LVS conftool data
Change 475304 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Add zoterov2 LVS ip block and realserver config
Change 475303 merged by Alexandros Kosiaris:
[operations/puppet@production] Introduce zoterov2 LVS conftool data
Change 475304 merged by Alexandros Kosiaris:
[operations/puppet@production] Add zoterov2 LVS ip block and realserver config
Change 473733 merged by Alexandros Kosiaris:
[operations/puppet@production] Introduce zoterov2 LVS endpoint
Finally deployed to production
Things to note:
akosiaris@deploy1001$ KUBECONFIG=/etc/kubernetes/zotero-eqiad.config kubectl -n zotero logs zotero-production-b68dbb465-22wbt
The zotero-production-b68dbb465-22wbt above is a pod name. Getting the pod names is done by
akosiaris@deploy1001:~$ KUBECONFIG=/etc/kubernetes/zotero-eqiad.config kubectl -n zotero get pods
This was true before as well, the minor improvement is that now logs are accessible to devs devs. This should be documented (I 'll do so)
An example curl looks like
akosiaris@deploy1001:~$ curl -d 'http://www.nytimes.com/2018/06/11/technology/net-neutrality-repeal.html' -H 'Content-Type: text/plain' zoterov2.svc.eqiad.wmnet:1968/web
[{"key":"WK9MLRT7","version":0,"itemType":"newspaperArticle","creators":[{"firstName":"Keith","lastName":"Collins","creatorType":"author"}],"tags":[{"tag":"Net Neutrality","type":1},{"tag":"Pai, Ajit","type":1},{"tag":"Federal Communications Commission","type":1},{"tag":"Regulation and Deregulation of Industry","type":1},{"tag":"Computers and the Internet","type":1}],"title":"Net Neutrality Has Officially Been Repealed. Here’s How That Could Affect You.","section":"Technology","url":"https://www.nytimes.com/2018/06/11/technology/net-neutrality-repeal.html","abstractNote":"Net Neutrality rules that required internet service providers to offer equal access to all web content are no longer in effect as of Monday.","language":"en-US","libraryCatalog":"NYTimes.com","accessDate":"2018-11-23T08:41:17Z","date":"2018-06-11","publicationTitle":"The New York Times","ISSN":"0362-4331"}]And for codfw
akosiaris@deploy1001:~$ curl -d 'http://www.nytimes.com/2018/06/11/technology/net-neutrality-rov2.svc.codfw.wmnet:1968/web: text/plain' zotertov2.svc.codfw.wmnet:1968/web
[{"key":"RFRC8EHV","version":0,"itemType":"newspaperArticle","creators":[{"firstName":"Keith","lastName":"Collins","creatorType":"author"}],"tags":[{"tag":"Net Neutrality","type":1},{"tag":"Pai, Ajit","type":1},{"tag":"Federal Communications Commission","type":1},{"tag":"Regulation and Deregulation of Industry","type":1},{"tag":"Computers and the Internet","type":1}],"title":"Net Neutrality Has Officially Been Repealed. Here’s How That Could Affect You.","section":"Technology","url":"https://www.nytimes.com/2018/06/11/technology/net-neutrality-repeal.html","abstractNote":"Net Neutrality rules that required internet service providers to offer equal access to all web content are no longer in effect as of Monday.","language":"en-US","libraryCatalog":"NYTimes.com","accessDate":"2018-11-23T08:41:28Z","date":"2018-06-11","publicationTitle":"The New York Times","ISSN":"0362-4331"}]I 'll resolve this, feel free to reopen
@akosiaris: I think your last paste is slightly broken (most of the URL got muddled with the data)
Does this mean that there should be a zotero v2 instance made in deployment-prep?
Fixed.
Does this mean that there should be a zotero v2 instance made in deployment-prep?
I guess if we are to maintain functionality and status quo, yes. That being said, it's not yet at all clear when, how and where exactly this will happen