Let's discuss this in the next Engineering meeting.

Change 1005859 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/WikiLambda@master] ZObjectUtils: Provide a makeFunctionFingerprint() method

https://gerrit.wikimedia.org/r/1005859

Based on discussion within the Abstract Wikipedia team, we have the following proposal for one new table, proposed usage plan, and several questions:

Proposed new table: wikilambda_zobject_join, which captures relationships between a main ZObject and various related ZObjects, with columns

• wlzo_id (Unique ID for index purposes)
• wlzo_zobject_zid (ZID of the main ZObject)
• wlzo_zobject_type (Type of the main ZObject)
• wlzo_ref_zid (ZID of the related ZObject)
• wlzo_ref_type (Type of the related ZObject)
• wlzo_key a key (e.g. Z8K1, Z4K5) indicating the relation between wlzf_zobject_zid and wlzo_ref_zid
• wlzo_list_position (If wlzo_key of main ZObject has a list value, the position of wlzo_ref_zid in that list)

(so, similar to zobject_function_join except more general, with new columns wlzo_zobject_type, wlzo_key, wlzo_list_position, and renaming of other columns)

Proposed usage:

• We use the new table for new DB content needs noted recently, including:
  • Which input types are used by each function
    • (function as main object, type as related object, wlzo_key = Z8K1)
  • Which functions are a equality fns, type converters and validators, and which types are they associated with
    • (type as main object, function as related object, wlzo_key indicating status as equality fn, type converter or validator)
• We continue to use zobject_function_join for its current uses
• We continue to use zobject_labels for its current uses, including
  • Return type of each function

Questions:

• Instead of continuing to use zobject_function_join, should we arrange to include all of its content in the new table, and (over time) decommission it?
• Instead of continuing to use zobject_labels for return type info, should we arrange to move that info into the new table?
• Will wlzo_zobject_type be useful (probably, but not clear yet)?
• How can we accommodate types that do not have a ZID?
  • The helper function in the makeFunctionFingerprint patch above offers a way forward.
• Will we need to represent functions that do not have a ZID?
  • In the near term, will it be sufficient to omit those, or indicate them with 'Z0'?

My quick counter-proposal:

[
	{
		"name": "wikilambda_zfunction_types",
		"comment": "Stored mapping from the ZFunction's ID to its input and return types.",
		"columns": [
			{
				"name": "wlzt_id",
				"comment": "Unique ID for index purposes",
				"type": "bigint",
				"options": { "unsigned": true, "notnull": true, "autoincrement": true }
			},
			{
				"name": "wlzt_zid",
				"comment": "The ZFunction's ZID",
				"type": "binary",
				"options": { "length": 32, "notnull": true }
			},
			{
				"name": "wlzt_return_type",
				"comment": "The ZFunction's return type, e.g. Z40 for a function that returns a Boolean",
				"type": "binary",
				"options": { "length": 255, "notnull": true }
			},
			{
				"name": "wlzt_type_fingerprint",
				"comment": "The ZFunction's type fingerprint, e.g. Z6,Z881(Z6):Z40 for a function that takes a Z6 and a list of Z6s and returns true if the Z6 is a member of the list",
				"type": "binary",
				"options": { "length": 255, "notnull": true }
			}
		],
		"indexes": [
			{
				"name": "wlzt_return_type_index",
				"columns": [ "wlzt_zid", "wlzt_return_type" ],
				"unique": true
			},
			{
				"name": "wlzt_type_fingerprint_index",
				"columns": [ "wlzt_type_fingerprint" ],
				"unique": false
			}
		],
		"pk": [ "wlzt_id" ]
	}
]

This would be the smallest possible listing, whilst allowing mapping function calls to types used and finding lists of functions that use or return a given type.

It would (eventually) replace zobject_labels for the return type, but not yet.

Thanks, @Jdforrester-WMF ! Based on our discussion, I understand the appeal of the fingerprint column, and yes, the counterproposal would meet the need to track inventory and usage of types. However, there are some concerns:

• It would not support tracking function "subtypes" serializer, deserializer, renderer, parser, validator (as partially captured in T355923).
• It lacks the generality of the initial proposal, which I believe could neatly support capturing any relationship (between 2 ZObjects) for which a key is defined. (In other words, if a key is defined for the main object, that object could appear in wlzo_zobject_zid, the key in wlzo_key, and the referenced object in wlzo_ref_zid.) I was thinking this generality would likely allow us to capture other things in future for tracking or search purposes, without having to create another table.
  • For example, it could perhaps be used (eventually) to track best-performing implementations (those which we store in the first position of Z8K4).
• Although the counterproposal would perform better on some queries, undoubtedly there are also some on which the initial proposal would perform better. (For example, the number of distinct types in the system; the number of usages of a given type.) Perhaps we should list out which queries we think will be most used, both within Wikifunctions and in analytics.

Now I'm wondering if would make sense to adopt your counterproposal specifically for type tracking, and also adopt the initial proposal (possibly later depending on priorities) to satisfy the first 2 bullets above.

In T357552#9610016, @DMartin-WMF wrote:

Now I'm wondering if would make sense to adopt your counterproposal specifically for type tracking, and also adopt the initial proposal (possibly later depending on priorities) to satisfy the first 2 bullets above.

Yes, that's what I was thinking; a simple table for the immediate needs now, and we can consider the bigger needs later.

@Jdforrester-WMF - Right; but OTOH the initial proposal is also simple, and has the advantages of greater generality and covering more needs as mentioned above.

Also, to my mind it's not clear yet that the use of the fingerprints wins out overall in terms of performance. I understand that it eliminates the need for some join queries, but I'm thinking even those queries could be handled efficiently if we index on wlzo_zobject_zid in the initial proposal (queries such as "find all functions that take a string input argument and return a natural number output argument").

We could also consider a combined approach, which retains the generality of the initial table structure, but eliminates the wlzo_list_position column, and allows for list encodings or fingerprints in the wlzo_ref_zid column (which would be renamed to wlzo_ref_value). I might flesh this out in a subsequent comment.

In any case, let's allow some time to consider which queries will be most used, and of course happy for other folks to weigh in.

A few queries this table could be used for:

Analytics

• Number of distinct types in Wikifunctions
• Number of usages of a given type
  • How many times does it appear as type of an input argument
  • How many times does it appear as return type
• Order types by number of usages
• Number of distinct type converters (or validators, renderers, parsers)
• Number of types having / not having type converters (or validators, parsers, renderers)
• Find types having / not having type converters (or validators, parsers, renderers)

Search

• Find functions that take an input of type String and return type String
• Find functions that take 2 inputs of type String and return type Natural number
• Find functions with an input and return having the same type

Folk should feel free to add other pertinent examples!

Considering the listed goals in the task description:

• track usage of types (as used in functions)
• track the inventory and uses of the following function "subtypes": serializer, deserializer, renderer, parser, validator

I believe the initial proposal accomplishes both of these goals, the migration is simple and the table design is generic and flexible enough that would allow us to focus on the current goals now, but have a platform to implement new searches and metrics in the future.

Some comments on the initial proposal

wlzo_list_position (If wlzo_key of main ZObject has a list value, the position of wlzo_ref_zid in that list)

What need does this column map to? Do we have an immediate need to query or measure the position of arguments? If not, I would drop this from the proposal and stick to the minimum number of columns necessary.

Instead of continuing to use zobject_function_join, should we arrange to include all of its content in the new table, and (over time) decommission it?

Yes, I believe zobject_function_join would be fully replaceable with the new table. However:

• There would be no hurry to make the migration, we can still keep the old table and the new one for a bit.
• The migration between tables and of the APIs would be simple: the old columns have a direct relation with the new ones, we would not need to make transformations on the data or the way it's used.

[David] Instead of continuing to use zobject_labels for return type info, should we arrange to move that info into the new table?

[James] It would (eventually) replace zobject_labels for the return type, but not yet.

I disagree. The zobject_labels table, including the return type, has a very specific goal since its creation, which is to aid performance of the most used UI component, the zobject lookup. Every keystroke in the search fields means one request, and many of these fields need the API to filter by return type. The return type in this table is not intended for metrics or function-related searches, but for lookups, its presence in this table might become redundant, but removing it will make these calls depend on table joins instead of being just simple queries on one table over indexed values.

Some comments on the counter-proposal

I believe this would only accomplish the first goal from the two listed in the task description:

• ✅ We would be able to track usage of types as used in functions, but
• ❌ We would not be able to track the inventory of type-related functions (validators, equality functions, renderers, parsers, serializers and deserializers)

I am also not convinced about the benefits of the function fingerprint column.

• Table size:
  • ✅ With zfunction_types, we would only have one row per function.
  • ❌ With zobject_join, we would have one row per relation: If a function has two inputs, it would mean 3 rows.
• Search capabilities:
  • ❌ With zfunction_types, not only the generation of the fingerprint can become difficult and lengthy, but querying over it becomes problematic, and it makes it impossible to index input types
  • ✅ With zobject_join we can index the zid and rel_type columns to improve the performance of the most common queries

This said, I don't think the difference in Table size is significant. The number of functions is currently < 1000, and generally functions don't have many arguments (maybe the average is around 2? just guessing) Increasing the table by 2x or 3x doesn't make much of a difference, while being able to index values will be crucial.

Yes, my proposal explicitly ignores the second "requirement" because I don't think that's as useful (and it was out of scope of this work when I wrote it). Instead, it helps us address many of the use-of listings that were mentioned in the chat a month ago about this.

Hi @gengh

wlzo_list_position (If wlzo_key of main ZObject has a list value, the position of wlzo_ref_zid in that list)

What need does this column map to? Do we have an immediate need to query or measure the position of arguments? If not, I would drop this from the proposal and stick to the minimum number of columns necessary.

We do not have an immediate need. I added it because it would increase the generality and expressiveness of the table, while the table would still remain straightforward, it doesn't add much to the required storage space, and it might support needs identified in the future. I thought of one possible use case: it will allow us to track, over time, which implementations have been identified as the best-performing implementations (because they are stored in the first position of Z8K4). But that is not a currently identified need. Are there any other places where list position has significance for us, that might benefit from this column?

Also, I was thinking it might help in writing certain queries, or make certain queries more efficient (because it would allow you to explicitly state that 2 rows are different in that column, even though they are the same in all other columns). For example, that would be one way for a query to state that a function has 2 or more input arguments of a given type. However, it's not clear to me whether queries expressed in that way are needed. EDIT: The wlzo_id column could be used in that way, so the argument of this paragraph is weak.

Change #1031026 had a related patch set uploaded (by David Martin; author: David Martin):

[mediawiki/extensions/WikiLambda@master] Create DB table wikilambda_zobject_join

https://gerrit.wikimedia.org/r/1031026

Change #1031026 merged by jenkins-bot:

[mediawiki/extensions/WikiLambda@master] Create DB table wikilambda_zobject_join

https://gerrit.wikimedia.org/r/1031026

Change #1049276 had a related patch set uploaded (by David Martin; author: David Martin):

[mediawiki/extensions/WikiLambda@master] ZObjectStore: Fix findRelatedZObjectsByKey to use wlzo_id for continuation queries

https://gerrit.wikimedia.org/r/1049276

Change #1049276 merged by jenkins-bot:

[mediawiki/extensions/WikiLambda@master] ZObjectStore: Fix findRelatedZObjectsByKey to use wlzo_id for continuation queries

https://gerrit.wikimedia.org/r/1049276