Page MenuHomePhabricator

Superset throwing up performance errors
Closed, ResolvedPublicBUG REPORT

Description

Steps to Reproduce:
Go to this URL: See error.

Actual Results:

Error:

HTTP Error 500: Internal Server Error Druid Error: {'error': 'Resource limit exceeded', 'errorMessage': 'query[12126dcd-f65c-479e-a346-1b09e3638b02] url[http://druid1001.eqiad.wmnet:8083/druid/v2/] timed out or max bytes limit reached.', 'errorClass': 'io.druid.query.ResourceLimitExceededException', 'host': 'druid1001.eqiad.wmnet:8083'} Query is: { "aggregations": [ { "fieldName": "view_count", "name": "sum__view_count", "type": "doubleSum" } ], "dataSource": "pageviews_daily", "dimension": "ua_browser_family", "filter": { "fields": [ { "fields": [ { "dimension": "ua_browser_family", "type": "selector", "value": "Chrome" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Chrome Mobile" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Mobile Safari" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Firefox" }, { "dimension": "ua_browser_family", "type": "selector", "value": "IE" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Other" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Safari" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Android" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Edge" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Chrome Mobile iOS" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Opera" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Samsung Internet" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Opera Mini" }, { "dimension": "ua_browser_family", "type": "selector", "value": "UC Browser" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Yandex Browser" }, { "dimension": "ua_browser_family", "type": "selector", "value": "IE Mobile" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Firefox Mobile" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Chrome Mobile WebView" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Opera Mobile" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Mobile Safari UIWebView" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Amazon Silk" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Facebook" }, { "dimension": "ua_browser_family", "type": "selector", "value": "CFNetwork" }, { "dimension": "ua_browser_family", "type": "selector", "value": "AppleMail" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Edge Mobile" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Chromium" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Mobile Safari UI/WKWebView" }, { "dimension": "ua_browser_family", "type": "selector", "value": "BlackBerry WebKit" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Apple Mail" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Vivaldi" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Firefox iOS" }, { "dimension": "ua_browser_family", "type": "selector", "value": "NetFront NX" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Maxthon" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Mail.ru Chromium Browser" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Crosswalk" }, { "dimension": "ua_browser_family", "type": "selector", "value": "YandexSearch" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Sleipnir" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Thunderbird" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Pale Moon (Firefox Variant)" }, { "dimension": "ua_browser_family", "type": "selector", "value": "NetFront" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Coc Coc" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Nokia Browser" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Nokia Services (WAP) Browser" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Firefox Alpha" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Puffin" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Sogou Explorer" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Iceweasel" }, { "dimension": "ua_browser_family", "type": "selector", "value": "QQ Browser Mobile" }, { "dimension": "ua_browser_family", "type": "selector", "value": "Chrome Frame" }, { "dimension": "ua_browser_family", "type": "selector", "value": "SeaMonkey" } ], "type": "or" }, { "dimension": "agent_type", "type": "selector", "value": "user" } ], "type": "and" }, "granularity": { "period": "P1M", "timeZone": "UTC", "type": "period" }, "intervals": "2015-08-29T00:00:00+00:00/2019-08-29T00:00:00+00:00", "metric": "sum__view_count", "postAggregations": [], "queryType": "topN", "threshold": 1000 }

Expected Results:

A chart that looks something like this:

Notes: Doesn't seem to impact other metrics or other charts that I tried.

Event Timeline

Restricted Application added subscribers: jeblad, Danmichaelo, jhsoby and 3 others. · View Herald TranscriptAug 29 2019, 10:52 PM
Nuria added a subscriber: Nuria.Aug 31 2019, 8:20 AM

I I see that chart is trying to access pageview_hourly for 4 years split by browser, reducing the time range would help.

fdans moved this task from Incoming to Ops Week on the Analytics board.Sep 2 2019, 3:32 PM
JKatzWMF added a comment.EditedSep 4 2019, 12:15 AM

@Nuria it's not working on even 1 year. I should add that it used to work.

Nuria added a comment.Sep 4 2019, 8:01 AM

Link provided does not work , can you please run it through a shorterner and repost?

@Nuria Apparently it's too long for a shortener (hah!). The name of the chart is "Pageviews by browser family" and I am the creator. You should be able to find it from that, but let me know.

Nuria added a comment.Sep 6 2019, 8:15 PM

1 year works, i wonder if there is a bug with time selection and thus not taking effect as UI indicates

Nuria added a comment.Sep 6 2019, 8:24 PM

Now, for 4 years might not work likely cause cardinality of OSFamily on that time is huge, there would be probably over thousand different values. You can filter by OS (that reduces cardinality) and thus get a bigger time interval:

Also, you can take a look at:
https://analytics.wikimedia.org/dashboards/browsers/#all-sites-by-browser

Nuria added a comment.EditedSep 6 2019, 8:28 PM

See the dashboard I created "Pageviews by browser (with filters)"

Nuria added a comment.Sep 10 2019, 7:44 PM

Closing, rather than a bug is a performance limitation, data is too large to be split by browser family for all families for 4 years. Filtering browsing families or limiting by timespan of pageviews should help.

Nuria closed this task as Resolved.Sep 10 2019, 7:44 PM
JKatzWMF reopened this task as Open.EditedOct 4 2019, 12:08 AM

Reopening. This is happening for other metrics now @Nuria

Much of my original reading dashboard...which used to work, is throwing up performance errors. You can look at each chart to try and figure out why. Unlike the browser family issue, where there are hundreds of potential values, for referers there are < 10.
https://superset.wikimedia.org/superset/dashboard/13/

Nuria added a comment.EditedOct 4 2019, 10:12 PM

I am closing this in favour of this ticket: T234684: Superset not able to load a reading dashboard

The problem in the dashboard here had to do with cardinality and number of segments (i checked some UA fields have cardinality of 10.000 in couple days) . The number of segments (daily) in a query that spans several years is closer to the thousands so i can see how those numbers would cause problems. Now the problem in the reading dashboard seems to be a different one.

Nuria closed this task as Resolved.Oct 4 2019, 10:12 PM