Abstract Descriptions: Engineering Investigation & Sprint Estimation

Topic: Abstract Descriptions - Wikidata function point to Wikifunctions function
Reference: https://www.wikifunctions.org/view/en/Z28803

• New datatype that will point to which function the Item must use on Wikifunctions
• Call functions on Wikifunctions and cache the results
• UI work in the Termbox similar to the mul work to fill empty descriptions with Abstract descriptions

Engineering Work Breakdown

Do We Need a New Extension?

Yes, a new extension is recommmended.

The new datatype and Wikifunctions integration are functionally distinct from WikibaseRepo's core concerns. A dedicated extension (exmple, WikibaseWikifunctions or WikidataAbstractDescriptions) provides:

• Clean separation of concerns.
• Independent deployment and rollback capability.
• Avoids bloating WikibaseRepo with Wikifunctions-specific logic.
• Consistent with how other Wikibase integrations (for example, WikibaseLexeme, WikibaseMediaInfo) are structured.

The datatype registration can hook into Wikibase's existing DataTypeDefinitions (extensions/Wikibase/lib/includes/DataTypeDefinitions.php) mechanism, so no fork of WikibaseRepo is needed.

Engineering Track 1: New Wikifunctions Reference Datatype

A ZID is the identifier Wikifunctions uses for all its objects - Z followed by a number (example, Z28803). Every function on Wikifunctions has one.

The new datatype is created in Wikibase, not Wikifunctions. Wikifunctions has no concept of Wikibase datatypes - it simply exposes functions that can be called. The work is entirely on the Wikibase/Wikidata side: teaching Wikibase that a property can hold a ZID value, how to validate it, how to display it, and how to represent it in RDF. The interaction is one-directional:

Wikidata item
  -> statement with new property (value: Z28803)
    -> Wikibase reads the ZID
      -> calls Wikifunctions API with (QID + language)
        -> gets back a description string

The new datatype lets a Wikidata property store a ZID as its value - a pointer from a Wikidata item to a function on Wikifunctions. The workflow for an editor is:

1. A new Wikidata property is creted using this datatype.
2. An editor adds a statement on an item: <that property> = Z28803.
3. The system reads that ZID, calls the referenced function on Wikifunctions passing the item's QID and requested language, and gets back a generated description.

Scope:

• Register the new datatype in Wikibase: validation (ZID format), parsing, display formatting, and serialization.
• The formatter renders the ZID as a link to the corresponding Wikifunctions page (example., wikifunctions.org/view/en/Z28803).
• Crete the new Wikidata property using this datatype.
• Add unit + integration tests.

Note: Since a ZID is structurally a formatted string, it may be possible to reuse the existing ExternalId datatype rather than building a fully custom one. This is worth investigating early to reduce scope.

Open questions for investigation:

• Can we reuse/extend the existing ExternalId datatype, or does ZID require a fully custom type?

Estimated effort:

Engineering Track 2: Function Evaluation & Caching

This is the most technically complex track.

Scope:

• Service/client layer to call Wikifunctions orchestrator API, passing QID + language code.
• Cache layer: store (QID, language, ZID) -> description string with appropriate TTL.
• Cache invalidation triggers: when item statements change (ZID property updated), when the Wikifunctions function itself changes.
• Fallback/error handling when Wikifunctions is unavailable or returns an error.
• Rate limiting / back-pressure to protect both Wikidata and Wikifunctions.
• Hook into Wikidata's TermLookup / FallbackChain so the generated description is returned transparently when no explicit description exists for a language.

Open questions for investigation:

• What is the Wikifunctions API endpoint for function evaluation? Is it stable/public?
• What caching backend is appropriate? MediaWiki's WANObjectCache (https://doc.wikimedia.org/mediawiki-core/1.40.3/php/classWANObjectCache.html) (Memcached/Redis) vs. a dedicated description store (for example, a dedicated MySQL/MariaDB table for persistence across cache evictions)?
• Should cache be warm (pre-populated) or lazy (on first request)? Pre-population of 100M items is a significant infra operation.
• How do we handle cache invalidation when a Wikifunctions function is updated? Does Wikifunctions expose a change notification mechanism (webhook, EventStream)?
• Should evaluation be synchronous (blocking on pge render) or async (job queue, serve stale while revalidating)?

Estimated effort:

Engineering Track 3: UI - Termbox & Frontend

Scope:

• Display abstract description in Termbox when no explicit description exists for the requested language (similar to existing mul description fallback UI).
• Visual distinction: indicate the description is auto-generated (not manually entered), with a link to the source function on Wikifunctions.
• Loading / skeleton state while description is being fetched (if async).
• Editing affordance: allow users to override with an explicit description.
• Client-side and SSR (server-side rendering via Termbox PHP fallback) support.
• Add E2E tests (Cypress) to cover the new UI flows along with unit tests for any new Vue components.

Open questions for investigation:

• Should the abstract description be rendered server-side or fetched client-side via AJAX/REST?
• Does the Termbox Vue component need to be aware of the new dattype, or is it purely a description-display concern?
• How does this interact with the anonymous / logged-out view vs. the Termbox edit UI?

Estimated effort:

Efforts Summary

The realistic estimation would be 12–16 sprints (~6 – 7 months). This is a significant effort, but it includes building a robust, production-ready integration with caching and error handling. A more minimal MVP (for example, without caching or with synchronous evaluation) could be delivered faster, but would have limitations in scalability and user experience.

Open Questions Requiring Decision Before Sprint 1

1. New extension vs. WikibaseRepo addition? (Recommendation: new extension)
2. ZID datatype: fully custom or extend ExternalId?
3. Cache backend: How would the caching flow work?
4. Evaluation model: synchronous, async, or batch-scheduled?
5. Wikifunctions change notification: is there an EventStream or do we rely on TTL only?
6. Visual UX chnges for the abstract descriptions in Termbox?

Dear @Arian_Bozorg , Please have a look on the estimation report above.
The first 3 points are only covered in the estimation, the remaining points still needs some investigation.
Thank you!

Task Time: Added to board for a peer review of the investigation. Once complete, it can go to PV

Review of investigation / questions

Engineering track 1: New Wikifunctions Reference Datatype

This looks like a sensible implementation plan. There are a couple of things that I think might be missing here:

1. Rendering the values as links to the functions (e.g. "Z28803") seems like quite an unhelpful visualisation - the user will struggle to know what the function does. I would suggest we retrieve the function description from Wikifunctions so that we get something like "short description for album (Z28803)". This would require a little more engineering, but I think it should be possible within the scope of this engineering track.
2. The ticket and plan are not specific, but I assume that we need to support both Desktop and MEX when we introduce a new datatype. Support for the two user interfaces (including design, tests etc) should be explicit in the plan, and might make mean we need more than the 2.5-3.5 sprints estimated.

Engineering Track 2: Function Evaluation & Caching

The plan makes sense in the abstract, but when I look at the existing implementation of WikiFunctions in WikiLambda, I have the sense that a lot of these questions might already have been addressed. For example, WikifunctionsPFragmentHandler.php already has functionality for rendering WikiFunctions inside Wiki pages, and it seems like WikifunctionsClientStore.php already has an implementation of caching for function calls. It would make sense to take a closer look at the implementation here, and see if we can already use some of this implementation or at least copy the decisions made here about caching and rendering (e.g. cache keys, cache store and expiry policy, (a)sync rendering) since the concerns are likely to be similar here between Wiki clients and WikibaseClients.

I would recommend to spend a sprint / spike investigating the existing implementation in the WikiLambda extension and see how many of the questions / challenges we have in Wikidata are already addressed by that code.

Engineering Track 3: UI - Termbox & Frontend

My sense is that this is actually by far the most complex engineering track. A couple of observations:

1. In MEX, we reuse the Mobile Termbox, so we don't need to do any additional work here for MEX. But we do need to support the existing Mobile and Desktop termboxes, which is a complexity I don't see reflected here so far.
2. As with mul, we need to make some decisions about how these items will be rendered in the API - do API clients see function references or the evaluated function results here?
3. The combination of mul and now WikiFunctions labels seems extremely complex in terms of user experience. Given that we spent (as far as I know) multiple years introducing support for mul, the 4-5 sprints here seems like a drastic underestimate.

I would suggest we spend some spikes considering exactly what the interactions will be between mul and WikiFunctions labels, both in editing and display, and for both termboxes and what that means for the UX. This is work that can be done in parallel to the engineering tracks, but I would feel a lot better estimating the engineering effort for this track once we have a more concrete set of designs for the user experience. Estimating the UX implementation without designs seems like an invitation to create a very inaccurate estimate.

Thank you for this estimate and plan! Between that and Arthur's suggestions, I think it covers the work comprehensively.

I am trying to cut scope in order to get a smaller MVP, on top of which then the work I suggest to cut can be re-added as demand for it increases.

Track 1

• In Track 1 we can drop the "Validator, Parser, Formatter" line. In worst case, it is just rendered like a string. Obviously a link would be preferable, and, as Arthur points out, the label of the Z-Function in the right language would be even better, also a function selector instead of just a string would be really helpful, but all of these can be done later.
• The line "API exposure" is unclear - what does that mean?

By dropping these, we can reduce Track 1 to 1-1.5 sprints.

Track 2

Right now I agree with Mahmoud, that a local Wikidata cache is probably the safer option. The Wikifunctions cache is in flux, and it would be easier if Wikidata took care of its own caching. I might be wrong here, and Arthur might be right.

• I would replace the Cache invalidation work. Getting Cache invalidation right here will be extremely difficult (what if a lexeme, which has been used, or a label that has been used in a description gets updated? How do we figure out that this would invalidate the cache?) Instead of having automatic Cache invalidation, allow a logged-in user to invalidate an individual description, and have a generous TTL on all descriptions for eventual updates, would be more robust and easier to develop. This may reduce the current work from 2-3 Sprints to 1 Sprint?

By dropping these we reduce the effort to 5.5 sprints.

Track 3

• The line "Termbox display when description is missing" in this track and "Integration with TermLookup / FallbackChain" in the previous track -- doesn't the latter make the former not needed?
• I would drop "Visual indicator + link to function". Not because I don't like it, but because the function is already provided in the statements, so this could be dropped from an MVP
• "Loading/async state handling" and "Server-side fallback rendering" is unclear to me. If the description is injected through the TermLookup / FallbackChain and cached, then this seems not necessary? I am probably a bit confused here.

Regarding Arthur's points, mul only exists for labels, whereas the functions only exist for descriptions -- so there shouldn't be an interplay between mul and abstract descriptions. Unless I am missing something.

By relying on the Termlookup and Fallbackchain with caching developed in Track 2, this seems to be reducable to 1-2 sprints (mostly for testing that everything is integrated, and ensuring that manual override still works).

Total

That would reduce the overall work to 7.5-9 sprints.

Further improvements (better display on the statement, a better function selector instead of just a string, visual indicators that an abstract description is displayed)

Again, I am not saying that the suggested work wouldn't be worthwhile -- I think those are great ideas and they would improve the product considerably -- I am merely trying to identify an MVP in order to be able to have this work happen and finish sooner, when it would be the most impactful for the overall projects, and where it would be incentivizing the community to create functions and lexemes.

One more place to reduce could be to indeed drop the whole Wikidata-side caching layer and rely entirely on Wikifunctions to cache (as Arthur suggested). That would further reduce the work for Track 2, but I think you have a good plan here whereas it is a risk whether the caching on our side would be up and working well enough when needed. But that's something we can talk about.