Page MenuHomePhabricator

Public schema.wikimedia.org endpoint for schema.svc
Open, MediumPublic5 Estimate Story Points

Description

We should expose our schema repositories for read-only usage and for browseability.

schema.svc.*.wmnet is deployed internally. We expose this service publicly.

  • Name - schema.wikimedia.org
  • Description - Serves multiple git repositories of JSONSchemas as static files over HTTP.
  • Timeline - Q2 2019-2019.
  • Point person - Andrew Otto
  • Technologies - This is a simple nginx instance serving files from checked out git repositories. This service is also available in beta at https://schema-beta.wmflabs.org.

Details

Related Gerrit Patches:
operations/puppet : productionUse relative nginx redirects for schema.wikimedia.org
operations/puppet : productionUse https://schema.discovery.wmnet for refine mediawiki_events job
operations/puppet : productionSet up cache routing for schema.wikimedia.org
operations/puppet : productionAdd public cert for schema.discovery.wmnet
operations/puppet : productionEnable envoyproxy tls for schema.svc
operations/puppet : productionFix indentation in discovery.yaml for schema
operations/dns : masterAdd schema.discovery.wmnet
operations/puppet : productionSet up schema.discovery.wmnet
operations/dns : masterAdd schema.wikimedia.org

Event Timeline

Ottomata created this task.Sep 23 2019, 2:15 PM
Restricted Application edited projects, added Operations, Services; removed Services (watching). · View Herald TranscriptSep 23 2019, 2:15 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Ottomata moved this task from Backlog to Next Up on the Event-Platform board.Sep 23 2019, 2:58 PM
Ottomata moved this task from Incoming to Event Platform on the Analytics board.Sep 23 2019, 3:21 PM
Ottomata removed Ottomata as the assignee of this task.Sep 23 2019, 3:26 PM
Ottomata triaged this task as Medium priority.

Ping @ema and/or @BBlack. Should I make some patches?

@Ottomata, I 'd say so. I would ask for review from Traffic or serviceops, but unless it was under the goals of either Traffic or serviceops I don't see how those team would end up driving it.

Ok, I will make patches then.

Change 549105 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/dns@master] Add schema.discovery.wmnet

https://gerrit.wikimedia.org/r/549105

Change 549106 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Set up schema.discovery.wmnet

https://gerrit.wikimedia.org/r/549106

Ok @ema @akosiaris patches for schema.discovery.wmnet ^. Not sure if I need this but I think it would be good for https://github.com/wikimedia/operations-deployment-charts/blob/master/helmfile.d/services/eqiad/eventgate-analytics/values.yaml#L19 and possibly other future uses.

I'll also need to set up the public DNS and routing for schema.wikimedia.org. Should I route to schema.discovery.wmnet?

Ottomata claimed this task.Nov 6 2019, 6:56 PM
Ottomata moved this task from Next Up to In Progress on the Event-Platform board.
Ottomata added a project: Analytics-Kanban.
Ottomata set the point value for this task to 5.
Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.

Change 549173 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/dns@master] Add schema.wikimedia.org

https://gerrit.wikimedia.org/r/549173

Change 549177 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Set up cache routing for schema.wikimedia.org

https://gerrit.wikimedia.org/r/549177

Change 549105 abandoned by Ottomata:
Add schema.discovery.wmnet

Reason:
Don't need discovery, LVS is fine.

https://gerrit.wikimedia.org/r/549105

Change 549106 abandoned by Ottomata:
Set up schema.discovery.wmnet

https://gerrit.wikimedia.org/r/549106

Change 549173 merged by Ottomata:
[operations/dns@master] Add schema.wikimedia.org

https://gerrit.wikimedia.org/r/549173

Ottomata added a subscriber: Joe.Nov 7 2019, 4:46 PM

@ema, @Joe informs me the that the nginx server serving schema.svc should terminate TLS for the 'encrypt all the things' goal. I have Qs!

  • How does this work with LVS?
  • Will Do I need to provision my own SSL certs?
  • Will schema.wikimedia.org also be TLS terminated at the backend nginx server, or will that be done by frontend nginx or ATS?
  • It might be simpler for the schema service if we weren't locked into nginx; does it make sense to use profile::tlsproxy::envoy to do TLS in front of my non-HTTPS nginx virtualhost?

Change 549105 restored by Ottomata:
Add schema.discovery.wmnet

https://gerrit.wikimedia.org/r/549105

Change 549106 restored by Ottomata:
Set up schema.discovery.wmnet

https://gerrit.wikimedia.org/r/549106

Change 549106 merged by Ottomata:
[operations/puppet@production] Set up schema.discovery.wmnet

https://gerrit.wikimedia.org/r/549106

Change 549105 merged by Ottomata:
[operations/dns@master] Add schema.discovery.wmnet

https://gerrit.wikimedia.org/r/549105

Change 550493 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Fix indentation in discovery.yaml for schema

https://gerrit.wikimedia.org/r/550493

Change 550493 merged by Ottomata:
[operations/puppet@production] Fix indentation in discovery.yaml for schema

https://gerrit.wikimedia.org/r/550493

Ok, just heard from @ema. To proceed we need to set up an HTTPS internal endpoint for schema.svc. We can do this in a few ways, but as it is right now we'd need an extra LVS entry for the HTTPS svc. If we wait until after end of Q2 for T227432: Replace Varnish backends with ATS on cache text nodes, we can make the main LVS schema.svc entries be HTTPS only, and avoid the extra svc.

Setting up public schema.wikimedia.org isn't strictly blocking anything this quarter, so I'm inclined to wait.

Ottomata moved this task from In Progress to Paused on the Analytics-Kanban board.Nov 13 2019, 8:17 PM

FYI for my own reference, an example of enabling envoyproxy in puppet: https://gerrit.wikimedia.org/r/c/operations/puppet/+/534597

Change 558660 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Enable envoyproxy tls for schema.svc

https://gerrit.wikimedia.org/r/558660

Change 558660 merged by Ottomata:
[operations/puppet@production] Enable envoyproxy tls for schema.svc

https://gerrit.wikimedia.org/r/558660

Change 561700 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add public cert for schema.discovery.wmnet

https://gerrit.wikimedia.org/r/561700

Change 561700 merged by Ottomata:
[operations/puppet@production] Add public cert for schema.discovery.wmnet

https://gerrit.wikimedia.org/r/561700

Change 549177 merged by Alexandros Kosiaris:
[operations/puppet@production] Set up cache routing for schema.wikimedia.org

https://gerrit.wikimedia.org/r/549177

Change 562541 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use https://schema.discovery.wmnet for refine mediawiki_events job

https://gerrit.wikimedia.org/r/562541

Change 562541 merged by Ottomata:
[operations/puppet@production] Use https://schema.discovery.wmnet for refine mediawiki_events job

https://gerrit.wikimedia.org/r/562541

Ottomata moved this task from In Progress to Done on the Event-Platform board.Jan 7 2020, 8:42 PM

Change 562624 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use relative nginx redirects for schema.wikimedia.org

https://gerrit.wikimedia.org/r/562624

Change 562624 merged by Ottomata:
[operations/puppet@production] Use relative nginx redirects for schema.wikimedia.org

https://gerrit.wikimedia.org/r/562624