Currently, the api gateway is configured to be bypass the edge cache.
Given we want to start serving traffic with it, we need to start using the edge cache, and allow to invalidate properties properly when e.g. an edit happens, but (probably?) also when the wikitext parsed output changes (because of say a template change).
Invalidating the cache amounts to posting a json payload that follows https://schema.wikimedia.org/repositories//primary/jsonschema/resource_change/current.yaml to eventgate-main.
Basically, what we want is to have a daemon, one per service and per datacenter, that listens to $dc.resource_change, only select urls that don't start with /api/rest_v1 (so, mediawiki properties), and:
- Emit a corresponding event for the URL in our service to $dc.resource_change to notify anyone of the change (optional; frankly, I don't think this is strictly needed but I'll let someone from DE comment on this - maybe @Ottomata )
- Emit a corresponding event for the URL in our service to $dc.resource-purge
- If the service has a cache, emit a purge request for the service. This last thing is currently not necessary anywhere.
A few considerations:
- While I think there should be a way to allow a simple stream processor to look at the changes coming from mediawiki and emit purges based on rules defined centrally, I don't think that's the best idea here.
- while we can't really use varnish's Xkey because trafficserver doesn't have the same or a similar concept, we can still reduce at least the traffic on the kafka queue if we moved some logic to purged. Basically add a configuration that allows to figure out article_url => derived urls as a mapping, and eliminate all the need for additional configuratione everywhere. This though can only work if for services that have no caching of their own
- The other services would be responsible to send out the purges after invalidating their own cache - basically what restbase does today