Page MenuHomePhabricator

Public schema.wikimedia.org endpoint for schema.svc
Closed, ResolvedPublic5 Estimated Story Points

Description

We should expose our schema repositories for read-only usage and for browseability.

schema.svc.*.wmnet is deployed internally. We expose this service publicly.

  • Name - schema.wikimedia.org
  • Description - Serves multiple git repositories of JSONSchemas as static files over HTTP.
  • Timeline - Q2 2019-2019.
  • Point person - Andrew Otto
  • Technologies - This is a simple nginx instance serving files from checked out git repositories. This service is also available in beta at https://schema-beta.wmflabs.org.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Ottomata triaged this task as Medium priority.

Ping @ema and/or @BBlack. Should I make some patches?

@Ottomata, I 'd say so. I would ask for review from Traffic or serviceops, but unless it was under the goals of either Traffic or serviceops I don't see how those team would end up driving it.

Change 549105 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/dns@master] Add schema.discovery.wmnet

https://gerrit.wikimedia.org/r/549105

Change 549106 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Set up schema.discovery.wmnet

https://gerrit.wikimedia.org/r/549106

Ok @ema @akosiaris patches for schema.discovery.wmnet ^. Not sure if I need this but I think it would be good for https://github.com/wikimedia/operations-deployment-charts/blob/master/helmfile.d/services/eqiad/eventgate-analytics/values.yaml#L19 and possibly other future uses.

I'll also need to set up the public DNS and routing for schema.wikimedia.org. Should I route to schema.discovery.wmnet?

Ottomata moved this task from Next Up to In Progress on the Event-Platform board.
Ottomata added a project: Analytics-Kanban.
Ottomata set the point value for this task to 5.
Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.

Change 549173 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/dns@master] Add schema.wikimedia.org

https://gerrit.wikimedia.org/r/549173

Change 549177 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Set up cache routing for schema.wikimedia.org

https://gerrit.wikimedia.org/r/549177

Change 549105 abandoned by Ottomata:
Add schema.discovery.wmnet

Reason:
Don't need discovery, LVS is fine.

https://gerrit.wikimedia.org/r/549105

Change 549106 abandoned by Ottomata:
Set up schema.discovery.wmnet

https://gerrit.wikimedia.org/r/549106

Change 549173 merged by Ottomata:
[operations/dns@master] Add schema.wikimedia.org

https://gerrit.wikimedia.org/r/549173

@ema, @Joe informs me the that the nginx server serving schema.svc should terminate TLS for the 'encrypt all the things' goal. I have Qs!

  • How does this work with LVS?
  • Will Do I need to provision my own SSL certs?
  • Will schema.wikimedia.org also be TLS terminated at the backend nginx server, or will that be done by frontend nginx or ATS?
  • It might be simpler for the schema service if we weren't locked into nginx; does it make sense to use profile::tlsproxy::envoy to do TLS in front of my non-HTTPS nginx virtualhost?

Change 549105 restored by Ottomata:
Add schema.discovery.wmnet

https://gerrit.wikimedia.org/r/549105

Change 549106 restored by Ottomata:
Set up schema.discovery.wmnet

https://gerrit.wikimedia.org/r/549106

Change 549106 merged by Ottomata:
[operations/puppet@production] Set up schema.discovery.wmnet

https://gerrit.wikimedia.org/r/549106

Change 549105 merged by Ottomata:
[operations/dns@master] Add schema.discovery.wmnet

https://gerrit.wikimedia.org/r/549105

Change 550493 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Fix indentation in discovery.yaml for schema

https://gerrit.wikimedia.org/r/550493

Change 550493 merged by Ottomata:
[operations/puppet@production] Fix indentation in discovery.yaml for schema

https://gerrit.wikimedia.org/r/550493

Ok, just heard from @ema. To proceed we need to set up an HTTPS internal endpoint for schema.svc. We can do this in a few ways, but as it is right now we'd need an extra LVS entry for the HTTPS svc. If we wait until after end of Q2 for T227432: Replace Varnish backends with ATS on cache text nodes, we can make the main LVS schema.svc entries be HTTPS only, and avoid the extra svc.

Setting up public schema.wikimedia.org isn't strictly blocking anything this quarter, so I'm inclined to wait.

FYI for my own reference, an example of enabling envoyproxy in puppet: https://gerrit.wikimedia.org/r/c/operations/puppet/+/534597

Change 558660 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Enable envoyproxy tls for schema.svc

https://gerrit.wikimedia.org/r/558660

Change 558660 merged by Ottomata:
[operations/puppet@production] Enable envoyproxy tls for schema.svc

https://gerrit.wikimedia.org/r/558660

Change 561700 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add public cert for schema.discovery.wmnet

https://gerrit.wikimedia.org/r/561700

Change 561700 merged by Ottomata:
[operations/puppet@production] Add public cert for schema.discovery.wmnet

https://gerrit.wikimedia.org/r/561700

Change 549177 merged by Alexandros Kosiaris:
[operations/puppet@production] Set up cache routing for schema.wikimedia.org

https://gerrit.wikimedia.org/r/549177

Change 562541 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use https://schema.discovery.wmnet for refine mediawiki_events job

https://gerrit.wikimedia.org/r/562541

Change 562541 merged by Ottomata:
[operations/puppet@production] Use https://schema.discovery.wmnet for refine mediawiki_events job

https://gerrit.wikimedia.org/r/562541

Change 562624 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use relative nginx redirects for schema.wikimedia.org

https://gerrit.wikimedia.org/r/562624

Change 562624 merged by Ottomata:
[operations/puppet@production] Use relative nginx redirects for schema.wikimedia.org

https://gerrit.wikimedia.org/r/562624