| BTullis | |
| Jun 22 2023, 4:50 PM |
| F41734269: Screenshot 2024-01-31 at 10.18.36.png | |
| Jan 31 2024, 9:57 AM |
| F41734267: Screenshot 2024-01-31 at 10.18.46.png | |
| Jan 31 2024, 9:57 AM |
| F41731590: image.png | |
| Jan 30 2024, 3:30 PM |
We have received a request to enable the PRESTO_EXPAND_DATA feature flag in Superset, so that nested types in Presto will be expanded.
It is an experimental feature, so we may with to consider enabling it on superset-next before the production deployment.
Update: evaluation complete - the feature is not yet stable enough to use
Update (2024/01/31): the feature stills displays issues in Superset 3.1.0 but we've been able to address them upstream
Change 932298 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Enable the PRESTO_EXPAND_DATA feature flag in Superset
Change 932298 merged by Btullis:
[operations/puppet@production] Enable the PRESTO_EXPAND_DATA feature flag in Superset
@JAllemandou - @Milimetric
I've rolled out this experimental feature to superset-next.wikimedia.org - but in order to do so I've temporarily disabled puppet on an-tool1010.
What's your feeling on whether we need more time to test this feature, or whether we roll back the change?
I can update puppet so that the feature-flag is enabled via hiera, if you think it's worth it, then keep it available for testing on superset-next.
Alternatively, if you think it's harmless, then I can just roll out this feature to production.
Thanks for doing his @BTullis - While super useful when it works, the feature is not stable enough to roller-out to production. You can disable it on both clusters for now on, we'll re-investigate with new versions of superset. Many thanks
Change 932644 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Revert "Enable the PRESTO_EXPAND_DATA feature flag in Superset"
Change 932644 merged by Btullis:
[operations/puppet@production] Revert "Enable the PRESTO_EXPAND_DATA feature flag in Superset"
While super useful when it works, the feature is not stable enough to roller-out to production
@JAllemandou, can you explain what was unstable / didn't work?
When looking at relatively small schema (not many columns and sub-columns), the feature worked ok. When looking at complex schema (the ones we have in production) the UI was not showing results and was not responding.
Another test of this feature flag has been requested by @JAllemandou and @lbowmaker (here: T335356#9486565 and here ) and if it is now stable enough to offer to our users, then it would help us to move forward with T333011: [Maintenance] Define Migration/Deprecation Plan for Hue
Change 994168 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Enable the presto_nested_data feature on superset-next
Change 994168 merged by Btullis:
[operations/puppet@production] Enable the presto_nested_data feature on superset-next
I have just enabled this for superset-next and announced it to our users.
btullis@an-tool1005:~$ sudo run-puppet-agent Info: Using environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts Info: Caching catalog for an-tool1005.eqiad.wmnet Info: Unable to serialize catalog to json, retrying with pson Info: Applying configuration version '(9b7b8a36be) Ben Tullis - Enable the presto_nested_data feature on superset-next' Notice: /Stage[main]/Superset/File[/etc/superset/superset_config.py]/content: --- /etc/superset/superset_config.py 2024-01-22 18:18:54.772578871 +0000 +++ /tmp/puppet-file20240130-801033-c4vvo9 2024-01-30 13:14:26.027521041 +0000 @@ -85,4 +85,5 @@ "ENABLE_TEMPLATE_PROCESSING": True, "DASHBOARD_NATIVE_FILTERS": True, "ENABLE_FILTER_BOX_MIGRATION": True, + "PRESTO_EXPAND_DATA": True, } Notice: /Stage[main]/Superset/File[/etc/superset/superset_config.py]/content: content changed '{sha256}2881bb3d03257907312abeafb4367acd6fdf5649e41db0093655f7354adb0f9b' to '{sha256}3205adba126f9cbcff4a72d2c601ece11dcb972534fe086c0359141218a18b90' Notice: Applied catalog in 28.56 seconds btullis@an-tool1005:~$ sudo systemctl restart superset btullis@an-tool1005:~$
I'll set the ticket to waiting while we evaluate it.
Reassigning this ticket to @brouberol as he has identified a bug in the Superset codebase relating to this display of nested columns.
The error relates to the column definition, whis uses the row() function and some of the nested columns have the type map(varchar, varchar), as shown in the event.navigationtiming.http column below.
Hopefully, we will be able to fix this parsing error and upstream the code change to the Superset project.
We have sent a patch upstream for this bug. In the meantime, we can pull in the patch into our own Superset build, to avoid being blocked. @JAllemandou reported that Superset worked as intended with the patch locally applied.
Change 994213 had a related patch set uploaded (by Btullis; author: Btullis):
[analytics/superset/deploy@master] Fix an issue with displaying nested columns from presto
I've also raised the suggestion to expand nested columns as key/value mappings upstream. In the meantime, we have a patch applied that allows us to display
Change 994213 merged by Btullis:
[analytics/superset/deploy@master] Improve the display of nested columns from presto
Change 994995 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Enable the presto nested data feature by default on superset
Change 994995 merged by Btullis:
[operations/puppet@production] Enable the presto nested data feature by default on superset