This bucket is a bit smaller than the others (Moderation / Surveys) so this umbrella task might feel a bit unnecessary at this point. I'm hoping to add more to this bucket throughout the quarter if we have time. Some examples of potential tasks:

• Knowledge cut-off dates for AI evaluation data: T383090
• Exploring occurrences of AI-generated content on Wikipedia: T383590
• You could also reasonably argue that the support for peacock detection could go under this bucket but I've chosen Moderation for the time being.

weekly update:

• Diego is attending WikiLibCon this week --> no updates
• I am scoping a task around optimizing the performance of the model to generate simple summaries developed as part of WE.3.1.3. The main idea is to explore and implement different strategies for improving the latency of the model, while at the same time finding a balance with the quality of the model output. I am currently discussing with Research engineering to understand better what is feasible from the optimization side as they have been making substantial progress on this front in the past weeks. My aim is to scope out a subtask next week and start putting together a plan based on the tooling we have available.

weekly update:

• Diego completed the final report for results on SDS 1.2.1 B T377159#10476688. no expected updates on this front in the next 2 weeks.
• I didnt make much progress on the AI Evaluation Datasets as I was out sick for most of the week (and giving priority to wiki workshop reviewer recruitment). will pick this up next week and hopefully make some substantial progress.

weekly update:

• finalizing coordination with Research Engineering on task around LLM optimization for the simple summaries model. Will add the task here next week.

weekly update:

• created task for AI Evaluation Datasets around LLM optimization of the simple summaries model T385481
• collecting already available resources for work on T385481 (e.g. available libraries for inference and quality metrics)

weekly update:

• AI Evaluation datasets: coordination with Research Engineering about planned work. Started creating subtasks to capture specific goals in more detail
• T386445: define a benchmark datasets
• T386448: define quality guardrail metrics
• tasks on optimization will be created shortly

weekly update:

• started to consider different options for defining benchmark datasets. Trying to adapt approaches from the article-country model T369120#10106230. I will most likely be starting from the set of ~10K filtered articles used in the initial experiments T375364 as those are carefully selected articles for which simple summaries were deemed appropriate.
• Next step will be to start putting together the subsets: i) random subsample , ii) edge cases (long/short or easy/difficult to read articles), iii) multilingual dataset.

weekly update:

• put together first draft for (4) benchmark datasets for LLM optimization around simple summaries model T386445#10591209
  • a random sample of English articles
  • a sample of edge cases of English articles
  • a multilingual (core) sample of the 23 languages explicitly supported by the simple summaries model
  • a multilingual (extended) sample of all other languages not explicitly supported by the simple summaries model
• implemented two additional quality metrics for the simple summaries model T386448#10591261
  • Using the language identification model on LiftWing, we check whether the model output is in the expected language (that has been shown to be an issue for multilingual models especially when quantized)
  • Using the peacock detection model, we check whether the model output is written in an encyclopedic tone. In the first round of experiments, we found cases where simple summaries used non-encyclopedic language. With this metric, we can automatically detect at least some of these cases (and filter, if needed).

weekly update:

• finalized the benchmark datasets for evaluating the simple summaries model T386445. Ran the simple summaries model on the benchmarks using different prompts https://docs.google.com/spreadsheets/d/1USF7IQhpi7Z8AcSp_44LUv7iN-vFnWyJidGXYp--jgA/edit?gid=0#gid=0
• finalized the set of quality metrics for evaluating the simple summaries model T386448. Next step is to make sure that these metrics work also for (some) languages beyond English. Main work is to: i) implement the multilingual readability model, ii) verify the level of multilingual support of the peacock detection model for tone, and iii) identify a multilingual alternative to the summac-model for meaning preservation.

weekly update:

• implemented multilingual versions of the quality metrics for the simple summaries model T386448#10635738. coordinating with @MunizaA how these metrics will be integrated into the llmperf module to benchmark LLM models.
• put together a runbook-document with detailed instructions to generate and evaluate simple summaries for current set of experiments: googledoc (internal only)
• gave a presentation to the Research Team detailing the past/ongoing/planned work on simple summaries: slides (internal only)
• put together additional language-specific benchmark datasets for each of the 23 languages supported by the Aya-expanse model to allow for future evaluation of the simple summaries model for other languages: Notebook, Example data German

weekly update:

• closed work on defining and implementing a set of suitable quality metrics for simple summaries T386448 . Applied the evaluation metrics for two test cases: i) Optimizing the prompt for generating simple summaries (Results), ii) Evaluating performance of the simple summaries model beyond English (Results)
• This concludes my work around T385481: The two completed subtasks T386445 (benchmark data) and T386448 (evaluation metrics for model quality) enable a systematic evaluation of different approaches for optimizing the simple summaries model. That work is ongoing and led by Research Engineering T387551.

Final summary update

• T385481 AI Evaluation Datasets. Status: Completed.
  • Completed two subtasks T386445 (building benchmark data) and T386448 (developing multilingual evaluation metrics for model quality).
  • This enables systematic evaluation of different approaches for optimizing the simple summaries model. That work is tracked in T387551 and led by Research Engineering. My understanding is that it is still ongoing and will be completed in Q4.
• T382727 AI Use-case share-out. Status: Ongoing.
  • @diego is planning to submit a longer paper in Q4. Thus, the task will require some extra time.
• Wikimedia+Libraries '25: Status: Completed
  • @diego attended the conference, gave a presentation, and shared learnings from the conference with the team.