Requests to MW API is supposed to be proxied by envoy. To utilise envoy from the flink application, our HTTP client must route through it. Right now we can only map full matches of target hostnames, this is not flexible enough, however.
AC
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| use pattern matching for HTTP routes | repos/search-platform/cirrus-streaming-updater!14 | pfischer | http-routes-pattern-matching | main |
Ideally we can also provide the routes from a properties file in a way that composes well with helm. So for example the following, with multiple values packed into a single property, is harder to use:
http-routes: schema.wikimedia.org:http://localhost:6023,*:http://localhost:6500
Something like the following might be better, but unsure how the dotted domain name interacts with parsing:
http-routes.schema.wikimedia.org: http://localhost:6023 http-routes.*: http://localhost:6500
This will allow definining the routes for schema.wikimedia.org in the generic values files while putting the env specific routes (elasticsearch, etc.) in env specific values files. ( Maybe not strictly necessary since everything in envoy has a unique port, we could probably define all routes in one place). I'm also of the opinion it will be easier to review and parse (as a human) the definition with one route per line.
The routes will also need to take ports into account, for example:
https://search.svc.eqiad.wmnet:9243/ -> http://localhost:6102 https://search.svc.eqiad.wmnet:9443/ -> http://localhost:6103
pfischer opened https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/14
use pattern matching for HTTP routes
dcausse merged https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/14
use pattern matching for HTTP routes
pfischer opened https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/24
use pattern matching for HTTP routes