Page MenuHomePhabricator

Make superset more scalable
Closed, ResolvedPublic8 Estimated Story Points

Description

  • Figure out async vs sync worker database locking problem; possibly update to Python 3 to solve this.
  • Set up celery workers for async database queries. This way the web UI doesn't have to block to wait for queries to finish. Sync queries have a gui timeout anyway, I think of around 60 seconds. Perhaps this is good? Perhaps superset shouldn't be used for queries that take longer than 60 seconds?

Related Objects

Event Timeline

Ok! Currently running python3.4 and with async gthread worker mode. Things seem snappier...and MySQL locking isn't happening, sooo GREAT!

Along the way, I ran into some nasty python2 -> python3 database charset encoding issues.

I deleted all existing database and table records from the superset database, and then converted table charsets:

ALTER DATABASE superset CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
ALTER TABLE tablename CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci

 alter table ab_permission           CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table ab_permission_view      CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table ab_permission_view_role CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table ab_register_user        CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table ab_role                 CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table ab_user                 CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table ab_user_role            CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table ab_view_menu            CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table access_request          CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table alembic_version         CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table annotation              CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table annotation_layer        CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table clusters                CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table columns                 CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table css_templates           CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table dashboard_slices        CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table dashboard_user          CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table dashboards              CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table datasources             CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table dbs                     CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table favstar                 CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table keyvalue                CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table logs                    CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table metrics                 CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table query                   CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table saved_query             CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table slice_user              CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table slices                  CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table sql_metrics             CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table table_columns           CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table tables                  CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 alter table url                     CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

Then I ran init_superset.sh again. This helped!

Change 402411 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use python3 async gthread workers for superset

https://gerrit.wikimedia.org/r/402411

Change 402411 merged by Ottomata:
[operations/puppet@production] Use python3 async gthread workers for superset

https://gerrit.wikimedia.org/r/402411

If we do celery workers, it will be as a different task.