The container image for Superset will be built using PipelineLib from the Release Engineering team.
We will create a GitLab project at: https://gitlab.wikimedia.org/repos/data-engineering/superset and the build pipeline will be maintained primarily as a .gitlab-ci.yaml file and a blubber.yaml file.
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| Add /app to the PYTHONPATH for superset | repos/data-engineering/superset!23 | btullis | add_app_to_pythonpath | main | |
| Add our customisation to the WMF superset build | repos/data-engineering/superset!22 | btullis | add_wmf_requirements | main | |
| Configure the npm proxy settings | repos/data-engineering/superset!9 | btullis | config_npm | main | |
| Test the use of the https_proxy environment variable | repos/data-engineering/superset!8 | btullis | test_https_proxy | main | |
| Use a plain node builder | repos/data-engineering/superset!7 | btullis | use_node_install | main | |
| Use a node builder for the npm ci step | repos/data-engineering/superset!6 | btullis | use_node_builder | main | |
| Set the npm proxy on the trusted runners | repos/data-engineering/superset!4 | btullis | fix_npm_proxy | main | |
| Fix the location of the blubber file | repos/data-engineering/superset!3 | btullis | fix_publish_config | main | |
| Add a publish stage to the superset image | repos/data-engineering/superset!2 | btullis | publish_superset | main | |
| Add the data-engineering/superset project to trusted-runners | repos/releng/gitlab-trusted-runner!52 | btullis | add_superset_project | main | |
| Add initial files for building superset | repos/data-engineering/superset!1 | btullis | begin_superset_build | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | brouberol | T353782 Decommission an-tool1010 | |||
| Resolved | brouberol | T347710 Migrate the Analytics Superset instances to our DSE Kubernetes cluster | |||
| Resolved | BTullis | T352165 Create a superset container image using the PipelineLib framework |
btullis opened https://gitlab.wikimedia.org/repos/data-engineering/superset/-/merge_requests/1
Add initial files for building superset
I have made some progress on the Superset image in this MR.
I will check the repository settings and then make a request to gain access to the trusted runners for this project, as per these instructions.
To start with, this is just a vanilla interpretation of the upstream Dockerfile, using blubber and kokkuri.
I will make some further modifications to incorporate our specific requirements and the additional packages that we will need.
btullis opened https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/52
Add the data-engineering/superset project to trusted-runners
dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/52
Add the data-engineering/superset project to trusted-runners
btullis merged https://gitlab.wikimedia.org/repos/data-engineering/superset/-/merge_requests/1
Add initial files for building superset
btullis opened https://gitlab.wikimedia.org/repos/data-engineering/superset/-/merge_requests/2
Add a publish stage to the superset image
btullis merged https://gitlab.wikimedia.org/repos/data-engineering/superset/-/merge_requests/2
Add a publish stage to the superset image
btullis opened https://gitlab.wikimedia.org/repos/data-engineering/superset/-/merge_requests/3
Fix the location of the blubber file
btullis merged https://gitlab.wikimedia.org/repos/data-engineering/superset/-/merge_requests/3
Fix the location of the blubber file
btullis opened https://gitlab.wikimedia.org/repos/data-engineering/superset/-/merge_requests/4
Set the npm proxy on the trusted runners
btullis merged https://gitlab.wikimedia.org/repos/data-engineering/superset/-/merge_requests/4
Set the npm proxy on the trusted runners
btullis opened https://gitlab.wikimedia.org/repos/data-engineering/superset/-/merge_requests/6
Use a node builder for the npm ci step
btullis merged https://gitlab.wikimedia.org/repos/data-engineering/superset/-/merge_requests/6
Use a node builder for the npm ci step
btullis opened https://gitlab.wikimedia.org/repos/data-engineering/superset/-/merge_requests/7
Use a plain node builder
btullis merged https://gitlab.wikimedia.org/repos/data-engineering/superset/-/merge_requests/7
Use a plain node builder
btullis opened https://gitlab.wikimedia.org/repos/data-engineering/superset/-/merge_requests/8
Test the use of the https_proxy environment variable
btullis merged https://gitlab.wikimedia.org/repos/data-engineering/superset/-/merge_requests/8
Test the use of the https_proxy environment variable
btullis opened https://gitlab.wikimedia.org/repos/data-engineering/superset/-/merge_requests/9
Configure the npm proxy settings
btullis merged https://gitlab.wikimedia.org/repos/data-engineering/superset/-/merge_requests/9
Configure the npm proxy settings
It took a while, but I've got past a real blocker that I had with publishing via the trusted runners.
I've now got what I feel is a good vanilla image of superset, that is ready for customisation.
I can do the following:
btullis@marlin:~$ openssl rand -base64 42 E22wJCmT74ZKrPHtks0D/uT9qdx3llrP33HYI/nG3bWEiKFExJC0FSrR btullis@marlin:~$ docker run --env SUPERSET_SECRET_KEY=E22wJCmT74ZKrPHtks0D/uT9qdx3llrP33HYI/nG3bWEiKFExJC0FSrR -it docker-registry.wikimedia.org/repos/data-engineering/superset:latest [2023-12-13 02:18:21 +0000] [7] [INFO] Starting gunicorn 20.1.0 [2023-12-13 02:18:21 +0000] [7] [INFO] Listening at: http://0.0.0.0:8088 (7) [2023-12-13 02:18:21 +0000] [7] [INFO] Using worker: gthread [2023-12-13 02:18:21 +0000] [8] [INFO] Booting worker with pid: 8 logging was configured successfully 2023-12-13 02:18:22,793:INFO:superset.utils.logging_configurator:logging was configured successfully 2023-12-13 02:18:22,797:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'> /home/superset/.local/lib/python3.9/site-packages/flask_limiter/extension.py:293: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend. warnings.warn(
The next step is to incorporate our specific python packages from here.
btullis opened https://gitlab.wikimedia.org/repos/data-engineering/superset/-/merge_requests/22
Add our customisation to the WMF superset build
I've made a good start with this, adding kerberos and memcached support to the superset container. Marking as ready for review.
btullis merged https://gitlab.wikimedia.org/repos/data-engineering/superset/-/merge_requests/22
Add our customisation to the WMF superset build
I think we can call this done now. There will likely be some iteration on the image once we start testing it, but for now I'm happy that it's got both the debian packages and python libraries that we will need to begin working with it on dse-k8s.
btullis opened https://gitlab.wikimedia.org/repos/data-engineering/superset/-/merge_requests/23
Add /app to the PYTHONPATH for superset
btullis merged https://gitlab.wikimedia.org/repos/data-engineering/superset/-/merge_requests/23
Add /app to the PYTHONPATH for superset