weekly update:

1. Summary

weekly update:

• continued making progress on the paper and we have a first draft with all the results and the general framing.
• aiming to finalize during next week

weekly update:

• working on the draft of the paper. made substantial progress on most sections so we almost have a bad first draft.
• will spend the next week revising. tentative internal deadline is October 5

weekly update:

• started exploratory analysis using the benchmark datasets. The idea is to compare the embedding distance (to the query) of relevant results vs non-relevant results. We should use opensearch' similarity score (derived from the standard cosine similarity) as that is the score that will be readily accessible in the backend.

weekly updates:

• completed mapping of templates to policies
• completed results for template prediction tasks using two baseline models
• started writing the papers and organizing results

Thanks for this analysis. I read through the notebook. This is my high-level interpretation:

• From the current analysis, we find one statistically significant feature that is strongly predictive of retention: tick_sum, i.e. the more time readers spent on the site, the more likely they are retained.
• Beyond that, the results are inconclusive because none of the considered features are statistically significantly different from 0 (i.e. we dont know if there is an effect or not) as can be seen from the huge error bars of the estimates. My hunch is that this comes from the fact that essentially all features are raw pageviews which usually span many orders of magnitude; and where outliers strongly affect estimates/errorbars.
• In order to address this, my recommendation would be to conduct a more refined statistical analysis in any potential follow-up work to understand factors that affect retention:
  • run all models with statsmodels (instead of pyspark) in order to get estimates of statistical significance. This is crucial for interpreting the coefficients
  • run the regression with log-transformed variables (for pageview-variables) in order to reduce errors from outliers. This should hopefully yield more conclusive estimates for the coefficients.
  • capture the wiki_id as a fixed effect in the regression in order to take into account that observation of users can be grouped based on the language version. This can be easily done in statsmodels via, e.g., logit("y ~ x1 + x2 + C(wiki_id)", data=df).

Looks good to me. Thanks

weekly update:

• ran first approach to map templates to policies/guidelines. We could match ~60% of the maintenance templates to at least one policy/guideline.
• refining the matching logic to increase recall (e.g. considering redirect pages of policies etc.)
• started drafting paper outline

weekly update:

• Running the full template prediction task with the mmBERT model. Results should be ready next week. We decided against running other (larger models) based on insights from literature, e.g. this paper on the citation needed prediction task shows that fine-tuned pre-trained language models remain a competetive baseline.
• We started the template-policy mapping. We first select a (small) set of main policies and guidelines related to content maintenance in English Wikipedia. My first draft is
  • NPOV policy
  • Verifiability policy
  • No original research policy
  • What Wikipedia is not policy
  • BLP policy
  • Image use policy
  • Article titles policy
  • Copyright policy*
  • Non-free content criteria*
  • Notability guideline
  • Editing guideline
  • Naming convention
  • Style guideline

*Policies with legal considerations that are not explicitly listed in content policies but nevertheless might be relevant for content maintenance (edited)
I extracted this from the following resources:

• Content policies https://en.wikipedia.org/wiki/Wikipedia:List_of_policies#Content
• Relevant guideline categories: https://en.wikipedia.org/wiki/Template:Guideline
• What links here to the Template:policy

weekly update:

• we started first experiments on the template prediction task using mmBERT
• we plan to run the template prediction task with at least one other model. currently discussing the trade-offs on which model to select.

weekly update

• Defined the set of variables for analysis of individual media requests based on previous
• Collected a sample dataset of one hour (~100M requests) for exploratory analysis
• Built data-processing pipeline for represeting categorical data as numeric data via one-hot-encoding
• Currently building the clustering model to group individual requests based on their characteristics combining PCA and Kmeans (potentially later to be updated with more robust clustering such as HDBSCAN)

weekly update

• Reading up on existing work relevant for this work
  • Bot threat model: processing pipeline for (media) requests; specifically, annotation with known bots, and how to calculate relevant metrics
    • RDS investigation on scraping incident : provides some intuition about additional metrics that can be used as features (on top of the bot thread model)
    • Media traffic dashboard: relevant details about how to process media requests, e.g., identifying different file types (images vs thumbnails)
• Starting to collect small sample to run exploratory analysis of media requests with summary statistics
• Collecting a set of relevant features for dimensionality reduction/clustering

final update:

• work has concluded, final report in this doc (internal only)

weekly update:

• we are starting the modeling task to predict templates in articles
• defined a subset consisting of positive and negative samples of revisions
• defined the template prediction task: multi-label classification
• defined the set of models for running the template prediction task
• next steps
  • generating the subset of positive/negative samples
  • implementing the models for running task

weekly update:

• working on final report for the project. completed first draft and did major revision after getting feedback.

weekly update:

• finished running processing/analysis pipeline on 10 languages.
• next step: generating subsample for template prediction task

weekly update:

• no update this week

In T400632#12022350, @APizzata-WMF wrote:

@MGerlach if you can confirm the improvement and have no further concerns, I'll go ahead and close the ticket.

weekly update:

• running the full processing pipeline for 10 languages
• starting to collect subset for the modeling task. first step is to parse template-content to distinguish messageboxes from inline-tags

weekly update:

• Conducted additional analysis to test the robustness of the results. Overall, we find that the initial positive results remain valid under different conditions ruling out some potential confounders.
• Different observation period: I re-ran the analysis for translations that were published in a more recent month (April 2026 vs Jan-Mar 2026).
• Pageviews from editors as confounding factor: I re-ran the analysis by only considering pageviews from loggedout readers. This removes the possibility that the increase in pageviews comes editors working on the translation since we are removing pageviews from logged-in readers. For this, I had to build a different data collection pipeline using the webrequest logs. While this provides additional details about the pageviews (e.g. whether they come from loggedin/out readers), they only contain data for the past 90 days. As a result, I was only able to consider a smaller sample of translations published within a single month (April) to still be able to observe pageviews roughly 1 month before and after.
• Pageviews served via Google's translated results (GT pageviews). I counted the number of pageviews to translated articles that are served via Google's automatic translation. I did not observe a statistically significant change in the number of GT pageviews before and after the translation. This would suggest that the availability of manually translated content does not affect the pageviews from Google served via automated translations. However, I do not fully trust the underlying data as there is a substantial mismatch between the numbers I get from our own webrequest vs those reported in the Google Search Console.

weekly update:

• Extended previous analysis to assess which countries drive the increase in pageviews:
  • eswiki: mainly driven by Spain (other countries not statistically significant)
  • ptwiki: mainly driven by Brazil (other countries not statistically significant
• We see a really strong increase for languages where Google's translated results are not available (e.g. Japanese)
• We observed at least one case (Brazil) where we see increase in pageviews, despite the prevalence of translated search results
• Next step: check if we observe changes in pageviews served via Google's translated results

weekly update:

• fixed issues for running the data collection pipeline
• defined a set of 10 languages for re-running the data collection pipeline based on previous multilingual projects (ar, bn, de, en, es, fr, ja, pl, pt, zh)
• started scoping the modeling task
  • define a subsample of templates: only messageboxes, balanced sample of positive/negative pairs, time-balanced (a fraction of say 20% from recent revisions to have data after model training cutoff), templates with a minimum number of occurrences
  • task: template prediction (binary or multilabel) in articles

In T426416#11975950, @Pginer-WMF wrote:

In T426416#11966357, @MGerlach wrote:

• frwiki: 0.69 (= 61% rel increase)

Thanks @MGerlach , for sharing the results. To make sure I'm interpreting this right, does the above examaple mean the following? On French Wikipedia, pageviews are 61% higher thanks to translated articles compared to how they would be if such translated articles would not exist. This being measured by comparing views of both versions of the article from French-speaking countries (experiment group) and other countries speaking the language from which the article was translated (e.g., English-speaking countries).

This is just based on a quick glance of all the materials. But wanted to make sure that a short and accurate-enough take away can be extracted from this. Thanks!

weekly update:

• organized and prioritized list of issues in data collection/processing pipeline
  • e.g. solved in issue from wrong regex leading to missing templates when appearing in nested templates
  • main issues we are working on now:
    • Calculating global stats by aggregating templates across wikis via their QID
    • Multitag templates with argument replace individual maintenance templates. Example from ptwiki
      • https://pt.wikipedia.org/w/index.php?title=Maison_du_Roi&diff=prev&oldid=46909084
      • https://pt.wikipedia.org/wiki/Predefini%C3%A7%C3%A3o:Multitag
  • my expectation is that we should be able to resolve those within 1-2 weeks
• started scoping the task for template prediction with support from @Trokhymovych

weekly update:
[UPDATED: Effect sizes updated after I found a small bug in code - all significant effects remain and increased in magnitude]

• Increased dataset by considering all translations in Jan-Mar 2026 (instead of only a single month Feb 2026): 19,203 translation (before: 6,451). This will also allow us to compare the effect size over different months to check the robustness of the results (to be done in a next step)
• Ran a 2-way fixed effects model to estimate the treatment effect.
  • Treatment effect: 0.35 ([0.269,0.433] is the 95% confidence interval; p-value <0.001, i.e. effect is statistically significant)
  • This corresponds to a relative increase in pageviews of 27.5%
• Expanded the regression model to capture wiki-specific treatment effect (wiki of the translation target) *Considered 22 wikis with more than 100 translations
  • All wikis show neutral or positive average treatment effect
  • 6 wikis with statistically significant treatment effect
    • eswiki: 0.56 (= 47% rel increase)
    • frwiki: 0.69 (= 61% rel increase)
    • hewiki: 0.58 (= 49% rel increase)
    • itwiki: 0.61 (= 53% rel increase)
    • jawiki: 1.28 (= 147% rel increase)
    • ptwiki: 0.40 (= 32% rel increase)
• Notebook with results: https://gitlab.wikimedia.org/repos/research/natural-experiment-translation/-/blob/main/analysis_effect-size_round1.ipynb?ref_type=heads
• Next steps: robustness checks

weekly update:

• after finalizing the high-level analysis around the usage of maintenance templates, we discovered a few issues in the data collection (such as parsing of nested templates or differences across languages)
• we will be looking into those during the next week and try to resolve before continuing to the modeling and policy-mapping

weekly update

@fkaelin Thanks for working on this and preparing the notebook.
I adapted your code and created a revised notebook after identifying some issues (thanks @KCVelaga_WMF for your support). Some of the main issues that needed to be resolved:

• identifying published translations
• identifying translations that create an article (instead of article expansions via section translation)
• identifying the date of the publication of the translation into the main namespace (via the revision_timestamp from the mediawiki_history table)
• removing translations that were deleted
• inconsistency in page-titles using whitespace/underscore leading to data loss when enriching data with QIDs

weekly update:

• Started work this week
• I am looking through the dataset from T424050 to set up the analysis:
  • I only considered territories which have one dominant language with an official status so that it can be unambiguously mapped to a single language (wiki). The heuristic is that the population percentage of speakers for the main language is >=90% and for all other languages (if any) <50%. This yields 38 wikis (languages) and 91 territories
  • I filtered the translation events where source and target language are from the set of 38 languages.
  • Get the pageviews for treatment/control groups before/after the translation
    • treatment group: pageviews to all articles from countries associated with the target language of the translation
    • control group: pageviews to all articles from countries associated with the source language of the translation
    • for both groups, the pageviews are aggregated into before (30 days before the translation) and after (30 days after the translation)
• Identified some missing information in the dataset which need to be resolved (see next steps):
  • Adding country-level information about the pageviews to the corresponding articles
  • Identifying and filtering out translation events which do not create a new article, Example: https://de.wikipedia.org/w/index.php?title=Arte&diff=prev&oldid=264405731

In T400632#11855621, @xcollazo wrote:

In T400632#11697165, @APizzata-WMF wrote:

In T400632#11652060, @xcollazo wrote:

Will discuss this further with @APizzata-WMF .

After discussing with @xcollazo we are pushing out a fix that uses the redirect table of MariaDB, here is the MR.

Do we need to do anything else here @APizzata-WMF? Or can we pass back to @MGerlach for verification?

The work is completed.
Documentation is available on the project page on meta-wiki: https://meta.wikimedia.org/wiki/Research:Offline_Evaluation_of_Wikipedia_Search

weekly update:

• We have made some progress on collecting the results for the two remaining experiments
• We generated the embeddings and the search results of the semantic search model for 5 different languages (en, de, es, fr, id). We should have the evaluation metrics ready next week.
• We have started running experiments with the multilingual-e5-large-instruct model which was used for the first prototype. We are currently working out how to run the other embedding models in our existing pipeline using spark-nlp. This is bringing some challenges but not yet a blocker. If necessary, we might need to adapt the set of embeddings models in the evaluation.

weekly update:

• we have concluded the first set of exploratory analysis
• Next steps are:
  • mapping the maintenance templates to a set of the main policies and guidelines (for English Wikipedia)
  • running models to predict maintenance templates in untagged articles

Final update:

• heavily revised the reader research direction based on feedback (latest version here)
  • mainly re-organized existing content to make it shorter/more compact and align better with objectives for the next FY
  • shared the latest version with Leila; will pick this up in Q4 based on feedback I receive and cover the work in a separate task
• had more detailed discussion about two proposed projects for Q4 on understanding reader retention and a natural experiment for estimating the causal effect: new content leads to additional pageviews
  • my current understanding is that I will start working on those in Q4.
  • the work on these projects will be covered in separate tasks

weekly update:

• We started running the experiments for semantic search with 3 different models and 5 different languages. For each case, we are generating the set of embeddings of all millions of passages. Once the embeddings are generated, we can obtain the top-10 results for each query and calculate the relevance metric. Hopefully, the data is ready early next week.

weekly update:

• we made lots of progress on analyzing statistics of the dataset. for now this is 6 wikis during the exploratory phase, but we will scale this up to more (all) wikis
  • Number of templates
  • Number of articles with a template
  • Distribution of number of templates per article
  • Number of add/remove events
  • Number of currently added templates (backlog)
  • Average time to fix, Fraction fixed in 1st week
  • Time evolution of number of templates added to articles
  • Per template data: Most common templates across languages, Most/least used templates, Templates which take the longest/shortest from being added to being removed (i.e. time to fix)
• Next step is to organize these results better (plots, tables)
• We will start mapping templates to policies. This is a crucial piece for demonstrating the usefulness of this dataset to broader audiences:
  • this paper from wikiworkship presents a dataset for studying policy invocation and enactment. Templates (that are mapped to policies and guidelines) could provide a complementary approach as they can be considered implicit invocations.
  • this paper discusses the importance of data about Wikipedia policies for improving natural language understanding tasks of large language models.

weekly update;

• reached out/discussed with Rita, Maryana, and Suman about relevent research questions on readers. Started revising the reader direction incorporating the feedback.
• got feedback about the project on understanding reader retention. This is a question that it relevant to folks in the readers teams. My recommendation would be to start by looking at existing data being collected in ongoing projects (attribution and comparative reader research) to get some initial insights. Based on the feedback, we can set up a dedicated experiment in test kitchen in the next FY, if needed.

Weekly update:

• We scoped the tasks around expanding the offline evaluation of the semantic search using benchmark data
  • More models: T419397: Get search results for different embedding models from semantic search. Specified the set of embedding models.
  • More languages: T419409: Get search results from semantic search using MIRACL benchmark dataset. Specified the data and languages.
• Next step is generating the embeddings so we can run queries and get search results for evaluation

weekly update:

• went back to the original draft of the reader direction and am starting to revise it heavily based on the feedback I received. No major new items were brought up so far. Thus, I plan to re-organize and shorten the content into at most 5 major research directions. I will then try to get a new round of feedback.
• I wrote up different options for a research project on better understanding retention of readers (googledoc). I shared this with Sherry and Hsuanwei to get feedback on what would be most useful for Product.
• I wrote up a research plan for running an analysis using content translation as a natural experiment (googledoc). Shared this with Debra as a potential project for the reader growth bucket. waiting for feedback on prioritization.

weekly update:

• obtained first results for summary statistics of templates in 6 languages. We will refine the analysis based on the paper On the Evolution of Quality Flaws and the Effectiveness of Cleanup Tags in the English Wikipedia. Main extension will be to calculate statistics beyond English Wikipedia.
  • Number of templates
  • Usage: How often are they used, what are the most commonly used templates
  • Evolution: How is usage increasing/decreasing over time? How is the backlog increasing/decreasing? Compare against the total number of articles.
  • Time to remove templates
• For this we will also want to aggregate templates by template types (e.g. verifiability vs style issues)
• Discussed with several folks that one way to increase the utility of the dataset would be to match maintenance templates with the main content policies and guidelines.

weekly update:

• met and discussed with Debra: the focus should be on identifying causal relationships that affect reader metrics. We think that the retention survey will be restricted to correlational insights. In addition, some of the ongoing work from others (attribution research and comparative reader research) will already provide some insights into retention. We would like to first assess how much these insights could serve questions around retention from product teams before investing more.
• We also discussed an alternative: The hypothesis is that additional content being available to internet users will lead to additional traffic from external search engines. Content translations could serve as individual natural experiments where some group of internet users are able to access new content (e.g. when an article is translated into ptwiki it is now accessible to speakers of Portuguese) whereas some group of internet users is not able to access the new content (e.g. when an article is translated into ptwiki it does not make a difference for non-Portuguese speakers). These two groups can be considered treatment and control groups and we can do a diff-in-diff comparison of pageviews with search-engine referer between the two groups. I will sketch an outline of this analysis next week.

weekly update:

• Collecting feedback from the first round of analysis for digging deeper.
• We identified different models for testing different variations for the semantic search to assess whether observation from the first round are due to the specific underlying model (qwen-3-0.6B) or hold generally for semantic search:
  • Qwen3-8B A larger version of the same qwen-3 as a comparison even if we would not be able to run this in production *Jina-embeddings-v5-text-small recently published state-of-the-art for this model size
  • Multilingual-e5-large-instruct  the model used for the prototype deployed on Greek Wikipdia
• We identified a multilingual benchmark dataset to i) test variation across languages; and ii) compare our current model's performance with results reported in literature
  • MIRACL is available in EN, DE, ES, FR, ID (but not in IT, NL, PT) from the languages that relevant for the current semantic search project
  • it only covers natural language questions (not from actual search-logs so not representative of WP queries), but our first round of results indicated that these are the queries where semantic search offers the most advantage to our current search.

weekly update:

• manual check of 100 samples in 6 languages in this spreadsheet. Overall, the data looks reliable. Though we identified some issues in parsing, such as nested templates, multi-tag templates, and some false positives probably due to reverted edits. We plan to fix those in the next iteration.
• We are starting the analysis of high-level metrics
• We are starting parsing of the content of templates to map them to the corresponding policy or guideline (e.g. identifying links to Wikipedia namespace). The first approach is to capture all links from the page to the Wikipedia-namespace and then do a manual filtering.

weekly updates:

• still figuring out feasibility of work to understand better retention of users. so far, we are considering two threads
  • retention survey: this would require some engineering work on T417185: Migrate QuickSurveys data collection to Test Kitchen. there are ongoing discussions around prioritization of this work
  • retention action: this work would try to identify which reader actions are correlated with high or low reader retention. This question might be possible to address with the data/analysis from the attribution research.
• I started to look in more detail into traffic from external search engines over time. Specifically, I stratified by article topic (aggregating timeseries of all articles in English Wikipedia belonging to the same topic). While very exploratory, there are some interesting aspects: for some topics (such as biographies or films) traffic from external search engines has been stable or even increasing; while for other topics (most notably STEM) the traffic from external search engines has been decreasing. https://gitlab.wikimedia.org/mgerlach/external-traffic/-/blob/main/trend-external-traffic_timeseries-topics.ipynb?ref_type=heads

weekly update:

• Refactored the dataset pipeline into 2 parts: i) getting the full (raw) dataset, ii) then applying the filtering step. With this, we can now also run the pipeline for enwiki. With this we are fairly confident that we can, in principle, run this on all wikis.
• We built a dataset for an initial set of 6 languages (those were selected based on language familiarity to being able to manually check results): bn, de, en, hi, pt, simple. We created a smaller random subsample for manual investigation/verification.
• We started to identify the main summary statistics to report a high-level overview of the dataset (number of templates, number of revisions, number of articles, time between adding/removing a template).

weekly update

• Compiled results for the offline evaluation of semantic search in English Wikipedia using the benchmark dataset and comparing it with our current lexical search.
• Specifically, we evaluate different search models on the new benchmark dataset. We consider Wikipedia search (lexical search) and semantic search (the current model for the MVP) with different variations (e.g. re-ranking results after retrieval). For each model, we get the top-10 search results for each query. We calculate different evaluation metrics to quantify the relevance of the search results using the pytrec_eval package: NDCG, Prevision, Recall, Binary preference (bpref). We evaluate the relevance of the retrieved results on the article and the paragraph level by comparing with the annotations in the benchmark dataset.
• Results can be found in this doc: https://docs.google.com/document/d/1xgdzD0TFIqyAw45mf9uHjzdpeMauugMRefQBlEu8i6I/edit?tab=t.x3x7obtlsqmn

@dcausse Thanks for generation this dataset.
We succesfully used this to run the offline evaluation with the benchmark dataset.

weekly update:

• from discussions in #talk-to-experiment-platform (thread), my understanding is that, in principle, it is possible to combine quicksurveys with test-kitchen. However, it seems that some engineering work still needs to be done before this is ready. Thus, we might not be able to start working on this immediately.
• I started exploring other opportunities. One potential direction is to understand in more detail the traffic from external search; i.e. what are the factors that lead to more/less traffic from external search engines and potentially identifying quasi-causal relationships via natural experiments.

weekly update:

• We refined the set of metrics for evaluation: nDCG@k, precision@k, recall@k, MAP@k, bpref@k for both paragraph and article level
• We collected search results for the 600 queries of the benchmark dataset for Wikipedia search and the semantic search MVP (qwen-model) with different variations (adding re-ranker, additional context) T417242#11636952
• Next step: calculating metrics

weekly updates

• Extracted full dataset for 4 languages: simple, de, bn, hi. en is still pending as we need to figure out settings for resource allocation to avoid memory issues.
• We are looking manually through a small subset of samples to spot-check any processing issues. One potential issue we have identified is that for some edits, one template is removed and at the same time another one (or more) is added. This might indicate that the former issue is not resolved but rather the latter templates provide a more specific characterization of the issue.
• Next step: Starting analysis of basic summary stats (e.g. number of templates, affected articles) over time.

weekly update:

• discussed with Search Team about how to set up evaluation
• Filtering of benchmark dataset: we will remove annotations from inconsistent raters
• Metrics: nDCG, precision, recall
• Granularity: Article- and paragraph level
• Models: Wikipedia search, and prototype model for semantic search T412338: Q2 FY2025-26 Goal: Semantic Search - Embeddings Service for MVP

weekly update:

• expanded code to capture all maintenance templates for a given wiki. also converted notebook to standalone script https://github.com/au-clan/wiki_templates/blob/development/template_dataset/template_dataset_v2_mod.py
• Next step: running pipeline for different wikis and optimizing queries/setup for larger wikis requiring more resources.

Final update:

• We collected the final dataset
• Documentation is available in this google doc (for now internal only)

weekly update:

• Started to focus in more detail on one project around understand better retention.
• One potential approach would be to combine a reader survey via quicksurvey with measurement of retention in test kitchen. First feedback I got was that this is likely possible but my goal next week is to spend some time to figure out in more detail what are the technical options/limitations.

weekly update:

• implemented several refinements for the processing pipeline: resolving redirect names of templates, marking edits with specific template that was added/removed
• next step is to expand the pipeline to include all maintenance templates from a single wiki