Page MenuHomePhabricator

Bug: can't make a YoY time series chart in Superset
Closed, ResolvedPublic

Description

So @kzimmerman and I are trying to make a chart in Superset that would show daily (or weekly or monthly) pageviews that would let us compare trends across years. It looks like "Time Series - Periodicity Pivot" may be the one to use but it's not working:

Screen Shot 2018-11-28 at 4.49.34 PM.png (1×1 px, 170 KB)

Are we doing it wrong or is there a problem with Superset? This type of chart is very important for us to be able to make using this tool, so help would be greatly appreciated! Thanks!

Event Timeline

For illustration: It might look like this chart (that I'm currently generating by hand in Google Sheets).

Errors in log:

12]: Traceback (most recent call last):
Nov 30 07:23:12 analytics-tool1003 superset[7912]: File "/srv/deployment/analytics/superset/venv/lib/python3.5/site-packages/superset/views/core.py", line 1104, in generate_json
Nov 30 07:23:12 analytics-tool1003 superset[7912]: payload = viz_obj.get_payload()
Nov 30 07:23:12 analytics-tool1003 superset[7912]: File "/srv/deployment/analytics/superset/venv/lib/python3.5/site-packages/superset/viz.py", line 361, in get_payload
Nov 30 07:23:12 analytics-tool1003 superset[7912]: payload['data'] = self.get_data(df)
Nov 30 07:23:12 analytics-tool1003 superset[7912]: File "/srv/deployment/analytics/superset/venv/lib/python3.5/site-packages/superset/viz.py", line 1380, in get_data
Nov 30 07:23:12 analytics-tool1003 superset[7912]: df = self.process_data(df)
Nov 30 07:23:12 analytics-tool1003 superset[7912]: File "/srv/deployment/analytics/superset/venv/lib/python3.5/site-packages/superset/viz.py", line 1146, in process_data
Nov 30 07:23:12 analytics-tool1003 superset[7912]: values=utils.get_metric_names(fd.get('metrics')))
Nov 30 07:23:12 analytics-tool1003 superset[7912]: File "/srv/deployment/analytics/superset/venv/lib/python3.5/site-packages/superset/utils.py", line 839, in get_metric_names
Nov 30 07:23:12 analytics-tool1003 superset[7912]: return [get_metric_name(metric) for metric in metrics]
Nov 30 07:23:12 analytics-tool1003 superset[7912]: TypeError: 'NoneType' object is not iterable

Need to dig further and probably file as bug to upstream

For some reason @JKatzWMF was more recently able to get Superset to make a YoY chart here: https://goo.gl/Vg88uv

Yeah, I've only gotten it to work well by looking at 1 year and the previous year overlaid. I tried but have not been able to create a "periodicity pivot" chart (or learn what that means), but it sounds cool.

Assigning to fdans, please take a look at logs and file bug with upstream , tool is deployed to analytics-tool1003.eqiad.wmnet

After digging a lil bit, I'm pretty sure this issue was fixed with this change:

https://github.com/apache/incubator-superset/pull/5931

Current version we have is 0.26.3, we should upgrade to 0.28.1.

Per our conversation @fdans to coordinate with @elukey to test whether we can run the 28 version on python 2.7 which is what we are running on debian strech.

Ansible change that seems to be enabling both 2.7 and 3.6, right?
https://github.com/onaio/ansible-superset/commit/5d390b9c35736f087dfd037f5cd3db32453777a2

Things to look for:

  1. there seems to be issues with upgrade

https://github.com/apache/incubator-superset/issues/6243

  1. From 0.27 onwards dashboard db schema has changed, upgrade to 0.27 requires a db migration. See changelog: https://github.com/apache/incubator-superset/blob/master/UPDATING.md

Me and @fdans deployed 0.28.1 this morning, and we had to apply a hot fix for an outstanding upstream bug (see T211605#4820128 for more juicy info). Please check if everything is ok, we did a quick check and didn't notice anything significant (except what referenced before of course :)).

I've added a PR in our github fork that fixes this particular problem:

https://github.com/wikimedia/incubator-superset/pull/1

This is now fixed in superset production, periodicity pivots are working correctly.

See graph For YoY pageviews: https://bit.ly/2QRRiCw in superset cc @JKatzWMF

Closing ticket

For some reason, that link only shows me data up to March - even after changing the metric to SUM(view_count) (which I guess was intended) and hitting "Run query" again (new link):

Superset pageviews YoY all projects, user, 2016-2018.png (863×1 px, 101 KB)

Just try again, teh tiem granularity needs to be set to "day"

OK, that works, thanks (here is the fixed link for others who might like to try it out too).