Page MenuHomePhabricator

Upgrade to Superset 0.37.x
Closed, ResolvedPublic8 Estimated Story Points

Description

Superset 0.37.0 was released a couple of weeks ago, but 0.37.1 is under voting (bugfix release) and 0.37.2 will follow. Let's wait for either 0.1 or 0.2 to be out and then upgrade.

This is a call for the vote to release Apache Superset (incubating) version 0.37.1. This is a bugfix release to address a shortcoming in the jinja2 functionality of superset. The majority of originally intended patches for the 0.37.1 version have been deferred to 0.37.2, which will be voted on once 0.37.1 has been released.

Event Timeline

Milimetric triaged this task as Medium priority.Sep 10 2020, 4:23 PM
Milimetric edited projects, added Analytics-Clusters; removed Analytics.

I deployed the new version and got a dashboard broken (500 from superset):

Sep 16 09:30:14 an-tool1005 superset[4191]: ERROR:superset.app:Exception on /superset/dashboard/141/ [GET]
Sep 16 09:30:14 an-tool1005 superset[4191]: Traceback (most recent call last):
Sep 16 09:30:14 an-tool1005 superset[4191]:   File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
Sep 16 09:30:14 an-tool1005 superset[4191]:     response = self.full_dispatch_request()
Sep 16 09:30:14 an-tool1005 superset[4191]:   File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
Sep 16 09:30:14 an-tool1005 superset[4191]:     rv = self.handle_user_exception(e)
Sep 16 09:30:14 an-tool1005 superset[4191]:   File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
Sep 16 09:30:14 an-tool1005 superset[4191]:     reraise(exc_type, exc_value, tb)
Sep 16 09:30:14 an-tool1005 superset[4191]:   File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
Sep 16 09:30:14 an-tool1005 superset[4191]:     raise value
Sep 16 09:30:14 an-tool1005 superset[4191]:   File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
Sep 16 09:30:14 an-tool1005 superset[4191]:     rv = self.dispatch_request()
Sep 16 09:30:14 an-tool1005 superset[4191]:   File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
Sep 16 09:30:14 an-tool1005 superset[4191]:     return self.view_functions[rule.endpoint](**req.view_args)
Sep 16 09:30:14 an-tool1005 superset[4191]:   File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/flask_appbuilder/security/decorators.py", line 109, in wraps
Sep 16 09:30:14 an-tool1005 superset[4191]:     return f(self, *args, **kwargs)
Sep 16 09:30:14 an-tool1005 superset[4191]:   File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/superset/views/core.py", line 1616, in dashboard
Sep 16 09:30:14 an-tool1005 superset[4191]:     for datasource, slices in datasources.items()
Sep 16 09:30:14 an-tool1005 superset[4191]:   File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/superset/views/core.py", line 1616, in <dictcomp>
Sep 16 09:30:14 an-tool1005 superset[4191]:     for datasource, slices in datasources.items()
Sep 16 09:30:14 an-tool1005 superset[4191]:   File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/superset/connectors/base/models.py", line 287, in data_for_slices
Sep 16 09:30:14 an-tool1005 superset[4191]:     metric_names.add(utils.get_metric_name(metric))
Sep 16 09:30:14 an-tool1005 superset[4191]: TypeError: unhashable type: 'dict'

http://localhost:9080/superset/dashboard/141/

elukey added subscribers: Dzahn, Ottomata.

I am also seeing that http://localhost:9080/superset/dashboard/7/ is broken as well.

Sep 17 07:53:34 an-tool1005 superset[5963]: Traceback (most recent call last):
Sep 17 07:53:34 an-tool1005 superset[5963]:   File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/superset/viz.py", line 472, in get_df_payload
Sep 17 07:53:34 an-tool1005 superset[5963]:     df = self.get_df(query_obj)
Sep 17 07:53:34 an-tool1005 superset[5963]:   File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/superset/viz.py", line 250, in get_df
Sep 17 07:53:34 an-tool1005 superset[5963]:     self.results = self.datasource.query(query_obj)
Sep 17 07:53:34 an-tool1005 superset[5963]:   File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/superset/connectors/druid/models.py", line 1440, in query
Sep 17 07:53:34 an-tool1005 superset[5963]:     query_str = self.get_query_str(client=client, query_obj=query_obj, phase=2)
Sep 17 07:53:34 an-tool1005 superset[5963]:   File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/superset/connectors/druid/models.py", line 1017, in get_query_str
Sep 17 07:53:34 an-tool1005 superset[5963]:     return self.run_query(client=client, phase=phase, **query_obj)
Sep 17 07:53:34 an-tool1005 superset[5963]:   File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/superset/connectors/druid/models.py", line 1399, in run_query
Sep 17 07:53:34 an-tool1005 superset[5963]:     df, pre_qry["dimensions"], qry["filter"]
Sep 17 07:53:34 an-tool1005 superset[5963]: KeyError: 'filter'
Sep 17 07:53:34 an-tool1005 superset[5963]: ERROR:superset.viz:'filter'
Sep 17 07:53:34 an-tool1005 superset[5963]: Traceback (most recent call last):
Sep 17 07:53:34 an-tool1005 superset[5963]:   File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/superset/viz.py", line 472, in get_df_payload
Sep 17 07:53:34 an-tool1005 superset[5963]:     df = self.get_df(query_obj)
Sep 17 07:53:34 an-tool1005 superset[5963]:   File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/superset/viz.py", line 250, in get_df
Sep 17 07:53:34 an-tool1005 superset[5963]:     self.results = self.datasource.query(query_obj)
Sep 17 07:53:34 an-tool1005 superset[5963]:   File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/superset/connectors/druid/models.py", line 1440, in query
Sep 17 07:53:34 an-tool1005 superset[5963]:     query_str = self.get_query_str(client=client, query_obj=query_obj, phase=2)
Sep 17 07:53:34 an-tool1005 superset[5963]:   File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/superset/connectors/druid/models.py", line 1017, in get_query_str
Sep 17 07:53:34 an-tool1005 superset[5963]:     return self.run_query(client=client, phase=phase, **query_obj)
Sep 17 07:53:34 an-tool1005 superset[5963]:   File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/superset/connectors/druid/models.py", line 1399, in run_query
Sep 17 07:53:34 an-tool1005 superset[5963]:     df, pre_qry["dimensions"], qry["filter"]
Sep 17 07:53:34 an-tool1005 superset[5963]: KeyError: 'filter'

Opened https://github.com/apache/incubator-superset/issues/10928

Upstream is voting on 0.37.2 at the moment, the release contains a lot of bug fixes, so I am inclined to wait few days and re-do the testing to see if anything improves.

I followed up with upstream on slack and they are about to merge a little change that should fix the KeyError: 'filter'. Since it is related to the "legacy" druid connector, as opposed to the SQL Alchemy one, the bug will not lead to another RC for the upcoming 0.37.2 (so it will be merged in master and present in future releases only). We can follow up with the owner of the dashboard after the upgrade to move it to the sql alchemy table.

I'll have another round of testing/debugging with upstream for the first issue tomorrow morning my time.

Upstream fixed the two problems and cancelled the current 0.37.2rc1 vote, in theory a new version with fixes should come out soon to get voted/tested again (and possibly released). This will take some days for sure.

New version is out: 0.37.2

The new code is currently deployed on an-tool1005 (staging instance) for further testing. As soon as the Analytics team is done we'll also upgrade the production instance.

Change 627738 merged by Elukey:
[analytics/superset/deploy@master] Update to Superset 0.37.2

https://gerrit.wikimedia.org/r/627738

Mentioned in SAL (#wikimedia-analytics) [2020-09-30T07:04:38Z] <elukey> superset upgraded to 0.37.2 on analytics-tool1004 - T262162

Mentioned in SAL (#wikimedia-analytics) [2020-09-30T07:29:53Z] <elukey> execute "alter table superset_production.alerts drop key ix_alerts_active;" on db1108's analytics-meta instance to fix replication after Superset upgrade - T262162

New version is out: 0.37.2

The new code is currently deployed on an-tool1005 (staging instance) for further testing. As soon as the Analytics team is done we'll also upgrade the production instance.

JFTR; this fixes CVE-2020-13952 (https://www.openwall.com/lists/oss-security/2020/09/29/3)

elukey set the point value for this task to 8.
elukey moved this task from Next Up to Done on the Analytics-Kanban board.