Page MenuHomePhabricator

Upgrade Fundraising Superset to 0.37.x
Closed, ResolvedPublic

Description

A new Superset release is available: https://github.com/apache/incubator-superset/releases

When we are able, this task is to upgrade our instance of Superset to 0.37.

Ref. T262162 for progress on the deployment project.

Related Objects

Event Timeline

EYener created this task.Sep 9 2020, 4:08 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 9 2020, 4:08 PM
Jgreen moved this task from Triage to Backlog on the fundraising-tech-ops board.Sep 10 2020, 3:54 PM
Jgreen renamed this task from Upgrade for Superset to Upgrade Fundraising Superset to 0.37.x.Sep 10 2020, 4:34 PM
Jgreen updated the task description. (Show Details)
DStrine moved this task from Triage to FR-Ops on the Fundraising-Backlog board.Sep 14 2020, 2:47 PM
Jgreen moved this task from Backlog to Up Next on the fundraising-tech-ops board.Oct 21 2020, 4:01 PM
Dwisehaupt added a subscriber: Dwisehaupt.

Too far into Q2 to address this. Pushing it until Q3 to avoid interfering with Q2 fundraising.

elukey added a subscriber: elukey.Nov 24 2020, 7:16 AM

Hi everybody! Can I ask how is this instance working? Is it reading from Druid's analytics cluster? I am asking since I have some work planned in the backlog related to Superset (like removing old datasources not in sql alchemy, etc..) and I think it would be probably better to sync with you as the work proceeds. Moreover I can share more details about what changes (config wise etc..) with you between versions, so we can have the same baseline.

@elukey Howdy. This instance is fully self contained. We are currently reading only from our internal mysql datasources. You should feel free to do any datasource work you need as it won't affect us.

There is some desire to possibly query production data in the future, but that will require significant work on our end and is not currently on our roadmap.

Our deployment uses a significant puppet refactor since our puppet deployment is different. Otherwise, there is little difference in our base setup from production. Any info you want to share so that we keep a similar baseline would be greatly appreciated.

Jgreen moved this task from Backlog to Up Next on the fundraising-tech-ops board.Dec 18 2020, 5:46 PM
Jgreen added a subscriber: Jgreen.EditedDec 18 2020, 6:05 PM

Reviewed the changes and tested the upgrade in virtualbox, it looks straightforward. The Superset install process includes a database update step, it's hard to tell whether what 0.37.2 does is incompatible with 0.36.0. Easy enough to back up the database first. Here's my recommendation for a procedure and rollback.

upgrade

  1. dump superset database
  2. fundraising_code_update -p superset; rsync_blaster fran1001:ALL
  3. fran1001: mv /srv/superset/venv /srv/superset/venv-0.36.0
  4. fran1001: puppet agent -tv
  5. fran1001: systemctl restart superset

rollback

  1. fran1001: systemctl stop superset
  2. fran1001: mv /srv/superset/venv /srv/superset/venv-0.37.2
  3. fran1001: mv /srv/superset/venv-0.36.0 /srv/superset/venv
  4. restore superset db dump
  5. fran1001: systemctl start superset
Jgreen added a comment.Thu, Jan 7, 2:12 PM

This was a minor disaster. I tried upgrade using commit 7bdc41487a3, which is the version that was successful in virtualbox testing.

The upgrade fails during the db upgrade step, when it tries to create table that already exists:

sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (1050, "Table 'rls_filter_tables' already exists")

That was empty table, so I ended up working around the problem by rolling back the database, dropping the table, and starting over.

Jgreen closed this task as Resolved.Thu, Jan 7, 2:28 PM
Jgreen claimed this task.
Jgreen moved this task from Up Next to Done on the fundraising-tech-ops board.