Project title: MediaWiki Code2Code Search

Brief summary: MediaWiki Code2Code Search is a semantic search tool that allows developers to find code snippets based on functional similarity rather than just keyword matching. By converting code into mathematical vectors (embeddings), the tool attempts to identify similar logic across 2,400+ repositories, even if the variable names or programming languages differ. The project also integrates with Software Heritage for long-term archival and features a highly multilingual interface to support global developers.

Expected outcomes: The primary goal is to evolve the tool from a standalone prototype into a core part of the Wikimedia developer workflow. This includes transitioning the UI to the Codex design system, implementing dynamic indexing for real-time code updates, and improving the accuracy of the Jina AI-based retrieval model.

Skills preferred:

• Languages: Python, JavaScript/TypeScript, PHP.
• Technologies: React, Three.js (for current UI), FastAPI/Flask, Vector Databases (FAISS), AI/Machine Learning (Embeddings).
• Wikimedia Tech: Codex Design System, Toolforge, Phabricator.
• Phabricator Tags: #MediaWiki-Code2Code-Search, #SWH-Archiving, #Indic-MediaWiki-Developers-User-Group.
• Experience level: Some proficiency (familiarity with Web APIs and/or basic Machine Learning concepts is helpful).

Project leads: @Super_nabla

Starting tasks:

• Integrate with translatewiki
• Migrate UI components to Codex (?)
• Researching ways to trigger and integrate updates for repository re-indexing.

Why are you proposing this project?
Navigating the massive MediaWiki codebase is often difficult for both new and experienced developers, especially when trying to find existing implementations of specific logic to avoid "reinventing the wheel." This project aims to meet the needs of the Indic MediaWiki Developers UG and the broader global community by making code discovery more intuitive and inclusive, thereby reducing the barrier to entry for non-native English developers.

What is the expected impact?
Success looks like a tool that extension developers and gadget authors regularly utilise to trace vulnerabilities, detect logic patterns, and learn from existing code. By moving to the Codex design system, the tool will feel like a native part of the Wikimedia technical ecosystem, increasing trust and adoption.

Any other additional information for contributors:

• Live Tool: https://code2codesearch.toolforge.org/

Raw session notes

MediaWiki Code2Code Search: Semantic Search to Find Code by Under-the-Surface Similarity

Speaker: USER: super_nabla (Francesco Tosconi), with Indic MediaWiki Developers UG

Participants:

• ...
• at least 13 participants
• [[User:Valerio Bozzolan]]
• ...

This work is about the code search engine.

Code2Code search is an improvement on CodeSearch tool https://www.mediawiki.org/wiki/Codesearch

This is an improvement to CodeSearch as it implements semantic search. It's called code2code search, as you search with a piece of code to find other code.

The code is typed into the interface, and the result is code in the repo that has similar similarity with the entered code, regardless of language or style.

Site URL: https://code2codesearch.toolforge.org/

Has support for multiple languages

There's a permalink on the page that goes to Software Heritage (SoftwareHeritage), which is our archive for source forges. The link highlights the exact code snippet on a specific commit.

How it works

• Mathematical transformation of code extracted from repositories into a high-dimensional vector space. The code itself is translated to the vector space.
• The tool leverages a JINA AI bi-encoder model wth 500 million parameters to compute vector representations. https://github.com/jina-ai/wikipedia-vector-demo, https://huggingface.co/jinaai/jina-embeddings-v5-text-small
• Vector search is powered by FAISS (Facebook AI Similarity Search) for rapid retrieval.

Side outcome of this project: All mediawiki reporsitories have been archived in SoftwareHeritage by using Software Heritage ID (SWHIDs).

The vision of SoftwareHeritage (SH) and WMF in reuse and sharing is similar. The property of SWH ID is included in WikiData. The original vision of SH is to preserve software as its important now to expand the public API model. Everything from the architecture to the models are public and exposed on SWH. WM should encourage the exposure of models created on its dataset.

"We didn't set out to do AI but it happened" - Roberto Di Cosmo, Director and Co-FOunder of SWH

Dandelion image of seeds was used because of the representation that OSS can get scattered and lost like Dandelions seeds. Hence, the need for proper archival.

The tool supports 14 Indic languages, plus Italian and French. Thanks to a collaboration with Francesco and the Indic MediaWiki Developers UG.

The User Interface:

The connection to the "Telugu of the West" is that translation ensures thart code production is available to people outside the English community, ensuring diversity in the Software community.

Scale
1.1M+ Code Snippets indexed
83K source files
2.4K Repositories across different forges.
Languages supported:

PHP, Python, JS, Typescript, Scala, Go, Java, Rust, C and C++

The road ahead
Improvement in User Interface, background dots might not be accessible, possibility to transistion to Codex UI.
Pipeline for dynamic updates from code repos
Expanding language support - Improtant to integrate with TranslateWiki, currently languages are imported via JSON files.

Questions:

    How hard is it to update the indexes?
	    We need to redownload the coderepos and Extract them into the vector space. A computer with a GPU is required for this work.

Comment: Using Apache AirFlow for building pipelines and schedule tasks similar to cron jobs.
Q: Curious about the programming language selected and where they are used.

		Programming languages are defined because the library used is Treesitter which requires the languages to parse the code function. The selected languages were selected because they are used in certain projects. Recommendation to add Ruby as its used in Wiki Dashboard

Q: SWH is used for the archival of repositories, wondering how it plays out in the code search process:

		SWH provides a unified model to view across forges. When a suggestion is generated, a permalink is added to the result that links back to the SoftwareHeritage archives.

Github Repo: https://github.com/ftosoni/mediawiki-code2code-search
SWH Repo: https://archive.softwareheritage.org/browse/origin/directory/?origin_url=https://github.com/ftosoni/mediawiki-code2code-search