Discussed in sprint planning, this task will include creating a PoC for Approach 1 for getting the text (based on paragraph location)

Test wiki created on Patch demo by LWatson-WMF using patch(es) linked to this task:
https://3177710913.catalyst.wmcloud.org/wiki/Main Page

• Set up a meeting to discuss the approach
• Set up follow-up tickets based on the selected approach

We discussed the keyword matching approach in our team refinement meeting today and decided to postpone implementation for now. The steering committee will discuss next steps and clarify designs (if needed).

ImageBrowsing: Keyword Matching System

The prototype demonstrates a keyword-based paragraph matching system for image descriptions using the Lunr.js full-text search library. The system extracts keywords from image metadata and matches them to article paragraphs with fallback logic.

Technical Implementation

• Keyword extraction strategy: Extracts keywords from image metadata (filename, alt text, and captions) and filters out stop words.
• DOM integration: Uses the DOM API to extract article paragraphs and section information client-side. This avoided the complexity of backend MediaWiki API integration.
• Lunr.js integration: Added Lunr.js as a vendored dependency for client-side full-text search.
• Scoring and ranking: Lunr.js provides weighted scoring across multiple keyword matches, returning a ranked list of paragraph candidates. The prototype displays the best match for each item in the VisualTableOfContents.
• Section-aware fallback: The hierarchical fallback system tries the image's section first, then parent sections, and then falls back to the first article paragraph. This ensures images always have associated text. The prototype also uses section information to explain the relationship between the image and selected text.
• Match indicators: Displays match scores and matching keywords to help understand why specific paragraphs were selected.

Key Learnings

• Pros:
  • Avoids the infrastructure requirements of AI/ML solutions
  • Adds relevance logic through keyword matching
  • Provides transparency into matching decisions

• Cons:
  • Cannot understand the semantic meaning of words
  • Image metadata is not always reliable
  • Adds technical complexity compared to the simplest approach

• Specific Limitations Confirmed: Keyword matching struggles with semantic relationships. For example, in the Barbados article, an image of a horse and jockey can't find a paragraph match because the relevant paragraph refers to horse racing as "polo" rather than using terms from the image metadata.
• Potential Improvements:
  • Experiment with section-based score boosting
  • Prioritize section matches over article-wide matches
  • Extract keywords from article page previews
  • Evaluate AI/ML enhancement feasibility and ROI (long-term)
  • Consider multi-language support (Lunr has built-in support)

• There is no "perfect" solution for finding relevant paragraphs. Even the simplest approach (using image metadata with fallback to the section's first paragraph) has significant limitations:
  • Alt text is inconsistently provided due to legacy practices
  • Captions are often too brief or not descriptive
  • First paragraphs may not relate to the image
  • Some sections lack paragraphs entirely
  • Relevant content may be in nested subsections
• This prototype increased my uncertainty about what would make any approach "good enough" for our use case.
• Does the improved paragraph relevance justify the added technical complexity? If our goal is high accuracy, we may need to invest in AI/ML enhancements when resources permit. If acceptable relevance is sufficient, we should evaluate whether this keyword approach provides enough value over the simpler metadata-only option.

It might be worth considering the summary endpoint instead of lead paragraph (https://en.wikipedia.org/api/rest_v1/page/summary/Polar_bear) which will save you a lot of parsing and removal of "junk" text.

Test wiki on Patch demo by LWatson-WMF using patch(es) linked to this task was deleted:

https://3177710913.catalyst.wmcloud.org/w/