Page MenuHomePhabricator
Create Task
Maniphest T219552

Schema Registry HTTP Service
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

This will simply be a readonly HTTP fileserver, with checked out and up to date clones of event schema repositories.

The schemas in the repos will be accessible by URL in the file hiearchary, something like:

https://schema.svc.eqiad.wmnet/mediawiki/api/request/0.0.1
https://schema.svc.eqiad.wmnet/analytics/page/preview/0.0.2

etc.

This service will host all of the (N=2?) git schema repos we use. I'm not yet sure the best way to present the multiple repositories together. Should they appear as one? Should they namespaced? Etc.

I think the easiest thing to do would be to get a couple of ganeti hosts (in both DCs?) and puppetize them to clone and serve the repos via either Apache or Nginx.

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Ottomata created this task.Mar 28 2019, 8:07 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 28 2019, 8:07 PM
Ottomata added a comment.Mar 28 2019, 8:07 PM

I'd like to set this up sooner rather than later (Q4 hopefully) so that I can more easily use remotely hosted schemas for ingestion into Hive in T215442: Make Refine use JSONSchemas of event data to support Map types and proper types for integers vs decimals. I don't strictly need a remote repo do this, but otherwise I have to automate deployment of the mediawiki/event-schemas reposistory into HDFS.

Ottomata added a project: Services.Mar 28 2019, 8:08 PM
• Pchelolo moved this task from Backlog to watching on the Services board.Mar 29 2019, 12:06 PM
• Pchelolo edited projects, added Services (watching); removed Services.
• mobrovac added a project: Platform Engineering (Modern Event Platform (TEC2)).Mar 29 2019, 7:35 PM
Milimetric triaged this task as High priority.Apr 1 2019, 3:39 PM
Milimetric moved this task from Incoming to Event Platform on the Analytics board.
gerritbot added a comment.Apr 17 2019, 9:11 PM

Change 504787 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Puppetize schema.wikimedia.org and refactor eventschemas module

https://gerrit.wikimedia.org/r/504787

gerritbot added a project: Patch-For-Review.Apr 17 2019, 9:11 PM
gerritbot added a comment.Apr 19 2019, 1:49 PM

Change 504787 merged by Ottomata:
[operations/puppet@production] Puppetize event schema http service and refactor eventschemas module

https://gerrit.wikimedia.org/r/504787

gerritbot added a comment.Apr 19 2019, 1:53 PM

Change 505244 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use Optional[String] for eventschema::service server_alias

https://gerrit.wikimedia.org/r/505244

gerritbot added a comment.Apr 19 2019, 1:54 PM

Change 505244 merged by Ottomata:
[operations/puppet@production] Use Optional[String] for eventschema::service server_alias

https://gerrit.wikimedia.org/r/505244

gerritbot added a comment.Apr 19 2019, 2:01 PM

Change 505245 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Fix source url for eventschemas service document_root

https://gerrit.wikimedia.org/r/505245

gerritbot added a comment.Apr 19 2019, 2:01 PM

Change 505246 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Fix reference to subscribe eventschemas::mediawiki

https://gerrit.wikimedia.org/r/505246

gerritbot added a comment.Apr 19 2019, 2:05 PM

Change 505245 merged by Ottomata:
[operations/puppet@production] Fix source url for eventschemas service document_root

https://gerrit.wikimedia.org/r/505245

gerritbot added a comment.Apr 19 2019, 2:06 PM

Change 505246 merged by Ottomata:
[operations/puppet@production] Fix reference to subscribe eventschemas::mediawiki

https://gerrit.wikimedia.org/r/505246

gerritbot added a comment.Apr 19 2019, 2:12 PM

Change 505247 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] eventschemas service fixes

https://gerrit.wikimedia.org/r/505247

gerritbot added a comment.Apr 19 2019, 2:13 PM

Change 505247 merged by Ottomata:
[operations/puppet@production] eventschemas service fixes

https://gerrit.wikimedia.org/r/505247

gerritbot added a comment.Apr 19 2019, 2:20 PM

Change 505250 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/dns@master] Add schema.svc IPs for eqiad and codfw

https://gerrit.wikimedia.org/r/505250

gerritbot added a comment.Apr 19 2019, 2:25 PM

Change 505250 merged by Ottomata:
[operations/dns@master] Add schema.svc IPs for eqiad and codfw

https://gerrit.wikimedia.org/r/505250

gerritbot added a comment.Apr 19 2019, 2:49 PM

Change 505254 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] LVS for schema.svc

https://gerrit.wikimedia.org/r/505254

gerritbot added a comment.Apr 19 2019, 3:02 PM

Change 505254 merged by Ottomata:
[operations/puppet@production] LVS for schema.svc

https://gerrit.wikimedia.org/r/505254

gerritbot added a comment.Apr 19 2019, 3:08 PM

Change 505258 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Fix eventschemas name in services.yaml

https://gerrit.wikimedia.org/r/505258

gerritbot added a comment.Apr 19 2019, 3:11 PM

Change 505258 merged by Ottomata:
[operations/puppet@production] Fix eventschemas name in services.yaml and service.yaml

https://gerrit.wikimedia.org/r/505258

gerritbot added a comment.Apr 19 2019, 3:16 PM

Change 505261 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use profile::lvs::realserver::pools in eventschemas service.yaml

https://gerrit.wikimedia.org/r/505261

gerritbot added a comment.Apr 19 2019, 3:19 PM

Change 505261 merged by Ottomata:
[operations/puppet@production] Use profile::lvs::realserver::pools in eventschemas service.yaml

https://gerrit.wikimedia.org/r/505261

Ottomata added a project: Analytics-Kanban.Apr 19 2019, 4:01 PM
Ottomata moved this task from Next Up to Done on the Analytics-Kanban board.
Ottomata moved this task from Backlog to To be Estimated/To be discussed on the Event-Platform board.Apr 23 2019, 3:13 PM
Ottomata mentioned this in T160748: Create /v2/schema/:schema_uri endpoint for eventstreams that proxies schemas from eventbus.Apr 24 2019, 2:56 PM
ayounsi closed subtask T221690: Allow analytics VLAN to reach schema.svc.$site.wmnet as Resolved.Apr 25 2019, 5:02 PM
ayounsi mentioned this in T221690: Allow analytics VLAN to reach schema.svc.$site.wmnet.
Ottomata reopened subtask T221690: Allow analytics VLAN to reach schema.svc.$site.wmnet as Open.Apr 25 2019, 6:36 PM
elukey closed subtask T221690: Allow analytics VLAN to reach schema.svc.$site.wmnet as Resolved.Apr 26 2019, 7:09 AM
gerritbot added a comment.Apr 30 2019, 2:18 PM

Change 507326 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] eventschemas::service - gzip minified static css/js files

https://gerrit.wikimedia.org/r/507326

gerritbot added a comment.Apr 30 2019, 2:20 PM

Change 507326 merged by Ottomata:
[operations/puppet@production] eventschemas::service - gzip minified static css/js files

https://gerrit.wikimedia.org/r/507326

gerritbot added a comment.Apr 30 2019, 2:50 PM

Change 507338 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] eventschemas::service - use relative path to ./repositories

https://gerrit.wikimedia.org/r/507338

gerritbot added a comment.Apr 30 2019, 2:55 PM

Change 507338 merged by Ottomata:
[operations/puppet@production] eventschemas::service - use relative path to ./repositories

https://gerrit.wikimedia.org/r/507338

gerritbot added a comment.Apr 30 2019, 2:56 PM

Change 507341 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] eventschemas::service - Don't render config.js.erb, it is in files/ now

https://gerrit.wikimedia.org/r/507341

gerritbot added a comment.Apr 30 2019, 2:57 PM

Change 507341 merged by Ottomata:
[operations/puppet@production] eventschemas::service - Don't render config.js.erb, it is in files/ now

https://gerrit.wikimedia.org/r/507341

gerritbot added a comment.Apr 30 2019, 4:32 PM

Change 507364 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add eventschemas clusters in monitoring.yaml

https://gerrit.wikimedia.org/r/507364

gerritbot added a comment.Apr 30 2019, 5:25 PM

Change 507364 merged by Ottomata:
[operations/puppet@production] Add eventschemas clusters in monitoring.yaml

https://gerrit.wikimedia.org/r/507364

• Nuria closed this task as Resolved.May 14 2019, 8:47 PM
• Nuria set the point value for this task to 3.
• Nuria closed subtask T219556: Create schema[12]00[12] (schema.svc.{eqiad,codfw}.wmnet) as Resolved.May 14 2019, 9:13 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits