Page MenuHomePhabricator

EventGate (in k8s) takes a long time to load new schemas
Closed, ResolvedPublic5 Story Points

Description

When testing EventGate in kubernetes, I've noticed that each instance will take a long time (many seconds) to load a new schema, and seems to block during this time. I haven't noticed this slow down when testing locally. This might be a problem with EventGate (and AJV?), but it also might be a k8s related problem. I need to investigate more.

In either case I don't think EventGate should block in the way it does when loading a new schema. This could potentially cause request timeouts to a particular pod while the service is loading some schema it hasn't seen before. For schemas that are available locally, we could implement some pre-caching, but this would still be a problem in the future when we start using a remote schema registry to replace EventLogging analytics use cases.

Event Timeline

Ottomata created this task.Mar 5 2019, 2:50 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptMar 5 2019, 2:50 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

BTW, just checked, and I believe the 'blocking' only happens for requests that use a schema that is in the process of being compiled by AJV. This is good news and makes sense, as those requests can't be served until the schema is compiled anyway. This is bad news for the prod/main service restart case, in that every new instance will block the first requests it receives as it compile the schemas. I'm going to implement pre-caching for a provided list of schema URIs.

fdans moved this task from Incoming to Modern Event Platform on the Analytics board.
fdans triaged this task as High priority.

Change 495904 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventgate-analytics - add schema_precache_uris configuration

https://gerrit.wikimedia.org/r/495904

Ottomata moved this task from Backlog to Done on the EventBus board.Mar 12 2019, 3:14 PM
Ottomata moved this task from Next Up to Done on the Analytics-Kanban board.

Change 495904 merged by Ottomata:
[operations/deployment-charts@master] eventgate-analytics - add schema_precache_uris configuration

https://gerrit.wikimedia.org/r/495904

Nuria set the point value for this task to 5.Mar 18 2019, 4:15 PM
Nuria closed this task as Resolved.