Page MenuHomePhabricator

Re-enable Superset metadata caching
Closed, ResolvedPublic

Description

We recently switched off Superset's data cache because it didn't respect access permissions. See {T273850} for details.

However, in the change where this was effected. we seem also to have disabled metadata caching, which I don't think was necessary. If we could re-enable metadata caching then I think that this might have a beneficial effect on Superset performance for users.

From this page: https://superset.apache.org/docs/installation/cache

For security reasons, there are two separate cache configs for Superset's own metadata (CACHE_CONFIG) and charting data queried from connected datasources (DATA_CACHE_CONFIG)

Here is where the two types of cache are configured in the superset configuration:
https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/superset/templates/superset_config.py.erb$77-91

The patch to disable caching undefined profile::superset::cache_uri and this had the effect of removing CACHE_CONFIG as well as DATA_CACHE_CONFIG.

My understanding, from reading {T273850} is that the DATA_CACHE_CONFIG is security sensitive and shouldn't currently be cached, but that CACHE_CONFIG is not security sensitive and we should be OK to cache it.

This ticket also relates to T294046: Write document about "Fast Enough Superset".

Event Timeline

We can cache metadata using CACHE_CONFIG; however I don't expect this will yield much of a performance boost. Worth testing it out though.

Yes, I know that it's not going to make a difference to the queries or the dashboard themselves, but it should help Superset feel more snappy to navigate.
My point really was that we used to have it enabled, and now it's disabled when it doesn't need to be.

Change 743386 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Refactor superset caching to enable dual caches

https://gerrit.wikimedia.org/r/743386

Change 743386 merged by Btullis:

[operations/puppet@production] Refactor superset caching to enable dual caches

https://gerrit.wikimedia.org/r/743386

This is now merged and deployed. I also restarted memcached to ensure that the cache was fresh.
It feels more responsive to me, but it's not easy to quantify it.

BTullis renamed this task from Re-enable Superset **metadata** caching to Re-enable Superset metadata caching.Dec 7 2021, 11:56 AM