Page MenuHomePhabricator

Improve performance of function filter API
Closed, ResolvedPublic

Description

Description

The filters implemented as part of T382140: Provide an API to filter functions by input/output types which can return functions usable by the prototype require complex queries joining the tables:

  • wikilambda_zobject_labels
  • wikilambda_zobject_join
  • page

On a production environment with more than 2000 functions, it becomes necessary to carefully provide the necessary indexes to filter, join and group and order by.

Performance improvements needed:
  • On wikilambda_zobject_labels:
    • Add index on wlzl_zobject_zid: used for JOIN and WHERE
    • Add composite index on wlzl_language and wlzl_label_primary: used for ORDER BY priority
    • Add index on wlzl_type: used for WHERE (already covered by existing indexes)
    • Add index on wlzl_label_normalised: used for LIKE (already covered by existing indexes)
  • On wikilambda_zobject_join:
    • Add index on wlzo_main_zid: used for GRUP BY and FILTER (already covered by existing indexes)
    • Add composite index on wlzo_key and wlzo_related_zobject: used for JOIN and WHERE (already covered by existing indexes)
    • Add index on wlzo_main_type: used for WHERE
  • On page:
    • use page_namespace instead of page_content_model, as it is already indexed and it contains the necessary data for our usecase

Follow schema change process as detailed in https://wikitech.wikimedia.org/wiki/Schema_changes


Completion checklist

Related Objects

StatusSubtypeAssignedTask
OpenNone
ResolvedJdforrester-WMF
ResolvedJdforrester-WMF
ResolvedJdforrester-WMF
ResolvedJdforrester-WMF
ResolvedJdforrester-WMF
ResolvedJdforrester-WMF
ResolvedJdforrester-WMF
DeclinedNone
Resolved Sharvaniharan
ResolvedAAlhazwani-WMF
ResolvedAAlhazwani-WMF
Resolvedgengh
Resolvedgengh
ResolvedMarostegui

Event Timeline

Change #1110396 had a related patch set uploaded (by Genoveva Galarza; author: Genoveva Galarza):

[mediawiki/extensions/WikiLambda@master] Use indexed page columns for join condition in preferred labels query

https://gerrit.wikimedia.org/r/1110396

Change #1110779 had a related patch set uploaded (by Genoveva Galarza; author: Genoveva Galarza):

[mediawiki/extensions/WikiLambda@master] [Schema change] Add indexes to wikilambda_zobject_labels and wikilambda_zobject_join

https://gerrit.wikimedia.org/r/1110779

Change #1110396 merged by jenkins-bot:

[mediawiki/extensions/WikiLambda@master] Use indexed page columns for join condition in preferred labels query

https://gerrit.wikimedia.org/r/1110396

Change #1110779 merged by jenkins-bot:

[mediawiki/extensions/WikiLambda@master] [Schema change] Add indexes to wikilambda_zobject_labels and wikilambda_zobject_join

https://gerrit.wikimedia.org/r/1110779

Next step, for the DBAs:

OK, the production DB schema change has happened; what's left to do at our end?

After https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikiLambda/+/1105019 we needed to run the updateSecondaryTables script for --zType Z4.
I believe that was already done, but just checking.

Nothing else to do, apart from considering @Amire80 suggestions on the "renderable" wording noted in https://phabricator.wikimedia.org/T382140#10450554

The API is working and is quick.

After https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikiLambda/+/1105019 we needed to run the updateSecondaryTables script for --zType Z4.
I believe that was already done, but just checking.

It was, yes.

Nothing else to do, apart from considering @Amire80 suggestions on the "renderable" wording noted in https://phabricator.wikimedia.org/T382140#10450554

Excellent, thank you. Merged.

The API is working and is quick.

Hurrah.