Page MenuHomePhabricator

Schema Registry HTTP Service
Closed, ResolvedPublic3 Story Points

Description

This will simply be a readonly HTTP fileserver, with checked out and up to date clones of event schema repositories.

The schemas in the repos will be accessible by URL in the file hiearchary, something like:

https://schema.svc.eqiad.wmnet/mediawiki/api/request/0.0.1
https://schema.svc.eqiad.wmnet/analytics/page/preview/0.0.2

etc.

This service will host all of the (N=2?) git schema repos we use. I'm not yet sure the best way to present the multiple repositories together. Should they appear as one? Should they namespaced? Etc.

I think the easiest thing to do would be to get a couple of ganeti hosts (in both DCs?) and puppetize them to clone and serve the repos via either Apache or Nginx.

Event Timeline

Ottomata created this task.Mar 28 2019, 8:07 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 28 2019, 8:07 PM

I'd like to set this up sooner rather than later (Q4 hopefully) so that I can more easily use remotely hosted schemas for ingestion into Hive in T215442: Make Refine use JSONSchemas of event data to support Map types and proper types for integers vs decimals. I don't strictly need a remote repo do this, but otherwise I have to automate deployment of the mediawiki/event-schemas reposistory into HDFS.

Pchelolo moved this task from Backlog to watching on the Services board.Mar 29 2019, 12:06 PM
Pchelolo edited projects, added Services (watching); removed Services.
Milimetric triaged this task as High priority.Apr 1 2019, 3:39 PM
Milimetric moved this task from Incoming to Modern Event Platform on the Analytics board.

Change 504787 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Puppetize schema.wikimedia.org and refactor eventschemas module

https://gerrit.wikimedia.org/r/504787

Change 504787 merged by Ottomata:
[operations/puppet@production] Puppetize event schema http service and refactor eventschemas module

https://gerrit.wikimedia.org/r/504787

Change 505244 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use Optional[String] for eventschema::service server_alias

https://gerrit.wikimedia.org/r/505244

Change 505244 merged by Ottomata:
[operations/puppet@production] Use Optional[String] for eventschema::service server_alias

https://gerrit.wikimedia.org/r/505244

Change 505245 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Fix source url for eventschemas service document_root

https://gerrit.wikimedia.org/r/505245

Change 505246 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Fix reference to subscribe eventschemas::mediawiki

https://gerrit.wikimedia.org/r/505246

Change 505245 merged by Ottomata:
[operations/puppet@production] Fix source url for eventschemas service document_root

https://gerrit.wikimedia.org/r/505245

Change 505246 merged by Ottomata:
[operations/puppet@production] Fix reference to subscribe eventschemas::mediawiki

https://gerrit.wikimedia.org/r/505246

Change 505247 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] eventschemas service fixes

https://gerrit.wikimedia.org/r/505247

Change 505247 merged by Ottomata:
[operations/puppet@production] eventschemas service fixes

https://gerrit.wikimedia.org/r/505247

Change 505250 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/dns@master] Add schema.svc IPs for eqiad and codfw

https://gerrit.wikimedia.org/r/505250

Change 505250 merged by Ottomata:
[operations/dns@master] Add schema.svc IPs for eqiad and codfw

https://gerrit.wikimedia.org/r/505250

Change 505254 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] LVS for schema.svc

https://gerrit.wikimedia.org/r/505254

Change 505254 merged by Ottomata:
[operations/puppet@production] LVS for schema.svc

https://gerrit.wikimedia.org/r/505254

Change 505258 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Fix eventschemas name in services.yaml

https://gerrit.wikimedia.org/r/505258

Change 505258 merged by Ottomata:
[operations/puppet@production] Fix eventschemas name in services.yaml and service.yaml

https://gerrit.wikimedia.org/r/505258

Change 505261 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use profile::lvs::realserver::pools in eventschemas service.yaml

https://gerrit.wikimedia.org/r/505261

Change 505261 merged by Ottomata:
[operations/puppet@production] Use profile::lvs::realserver::pools in eventschemas service.yaml

https://gerrit.wikimedia.org/r/505261

Ottomata moved this task from Next Up to Done on the Analytics-Kanban board.
Ottomata moved this task from Backlog to Done on the EventBus board.Apr 23 2019, 3:13 PM

Change 507326 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] eventschemas::service - gzip minified static css/js files

https://gerrit.wikimedia.org/r/507326

Change 507326 merged by Ottomata:
[operations/puppet@production] eventschemas::service - gzip minified static css/js files

https://gerrit.wikimedia.org/r/507326

Change 507338 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] eventschemas::service - use relative path to ./repositories

https://gerrit.wikimedia.org/r/507338

Change 507338 merged by Ottomata:
[operations/puppet@production] eventschemas::service - use relative path to ./repositories

https://gerrit.wikimedia.org/r/507338

Change 507341 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] eventschemas::service - Don't render config.js.erb, it is in files/ now

https://gerrit.wikimedia.org/r/507341

Change 507341 merged by Ottomata:
[operations/puppet@production] eventschemas::service - Don't render config.js.erb, it is in files/ now

https://gerrit.wikimedia.org/r/507341

Change 507364 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add eventschemas clusters in monitoring.yaml

https://gerrit.wikimedia.org/r/507364

Change 507364 merged by Ottomata:
[operations/puppet@production] Add eventschemas clusters in monitoring.yaml

https://gerrit.wikimedia.org/r/507364

Nuria closed this task as Resolved.May 14 2019, 8:47 PM
Nuria set the point value for this task to 3.