Page MenuHomePhabricator

WikiLambda metrics: collect type usage for run-function requests
Open, LowPublic

Description

Description

For each run-function request (explicit user request using the function evaluation widget), record the argument types and the return type of the selected function. The instrument already exists to record these requests (per T340786); it is likely possible to add this type info to the existing instrument.

Update: for any given run-function request, we can look up the function's types using the new DB table which will result from T363439. It won't be necessary to add that info to each run-function event.

This is motivated in part by our need to answer questions like the following: How frequently are new types used by the community and what are the most common use cases?

Note: This will generate a lot of repeated info (about the argument types and return type associated with each given function), which could be obtained in other ways. (For example, a new DB table that records the argument and return types for each function.) Should we consider these other ways, so as to keep down the metrics pipeline traffic?

Desired behavior/Acceptance criteria

  • The type info is recorded accurately and reliably, and shows up in the Wikifunctions_ui table in the data lake.

Completion checklist

Event Timeline

Quick note re timing: It would be good to have this instrumentation deployed by the time the new types are rolled out if we want to collect data on initial usage.

Per our measurement plan and the PRD for the types work, this instrumentation will be used to monitor progress towards the following success metrics

  • "New functions created with these types are used at least 1000 times "
  • The community will have created 100 new functions using the new types"

May I chip in with a question?

How will we cope with custom-typed objects in an untyped context? I’m thinking mainly of Z1-lists functioning as informal objects. One might have subject, verb, object in a list representing a “sentence” (supposing these to be custom types), but the function’s argument might be Z1-list and its return type might just be monolingual string, or just string, and inflecting the verb could be a simple string function but one that depends on an interpretation of some object as a particular kind of verb, based on its type or its keys (but that might be just a call to Z803/4 with some indeterminate (Z1) object, so we’d need to spot the returned value(s) to infer that some subsequent string argument might be functioning (implicitly) as an instance of a custom type (when I might have been checking that it was *not* a string of that type, or some other type…)

I’m assuming that “explicit user request using the function evaluation widget” will include function calls while editing a test case or implementation. That is the context in which the majority of my function calls have been made (mostly Try this implementation, I would guess, but then it’s hard to distinguish between de-bugging, testing and actual “use”).

DMartin-WMF renamed this task from WikiLambda metrics: collect type info for run-function requests to WikiLambda metrics: collect type usage in run-function requests.Jun 12 2024, 5:26 PM
DMartin-WMF renamed this task from WikiLambda metrics: collect type usage in run-function requests to WikiLambda metrics: collect type usage for run-function requests.
DMartin-WMF updated the task description. (Show Details)