weekly update:

• I reached out to all hypothesis owners in WE.3.1 individually We also had a joint meeting to support Support WE.3.1. From this, I obtained a much clearer picture about the needed support
• WE.3.1.1: might need light support (consulting) for options to generate recommendations. Substantial support needed for experiments on simplification/summarization. Web Team is starting to think about specifications in more detail. So these are ongoing discussions at the moment.
• WE.3.1.4: No support needed at this point. They will focus on figuring out what work would need to be done for scaling search (e.g. morelike). They also want to start looking into vector search which would likely require some support from Research (e.g. creating vectors/embeddings). However, they are starting from scratch and during Q1 will start to figure out what they want and what support would be needed in the future.
• WE.3.1.5: No support needed at this point. They want to skip the use of orphans for the first round of experiments.

weekly update:

• Android reached out to understand more about recommended content within search. I shared some resources from research on search and articles recommendations (e.g. from list building)
• Web is starting with first experiments on recommendations in search. Providing support for using the article-similarity search from the list-building tool https://list-building.toolforge.org/
• Web is preparing to start thinking about experiments on simplifications which will happen later in the quarter. Ongoing discussions about what simplification would be useful and how to evaluate.

weekly update

• Shared results on section-gists of Wikipedia articles (T369288#10018291) with Web team as one potential approach for experiments on summarization and simplification for readers.

weekly update:

• I put together documentation for article recommendations with different tools from Research for experiments WE.3.1.1 (doc)
• Shared documentation with hypothesis owner for feedback.

weekly update:

• no updates this week as there were no requests for additional support so far

weekly update

• no updates this week as I was attending ACL 2024 conference.

as a heads-up, I will be out on sabbatical starting next week until mid-October, so updates here will likely slow down in the coming weeks.

weekly update:

• adapted to title to reflect that work is continuing in Q2
• spent most of my time to catch up with what are current needs from corresponding hypothesis owners
• the Web Team started small user tests using the model for generating simple summaries of sections (documentation) I had prepared leaving for sabbatical (T374638). Feedback from the small sample of users is very positive (report)
• I have been syncing with Jan about next experiments to generate simple summaries for the lead sections of 10K articles for use in larger experiments in the browser extension (T375364)
• We also started discussions about usage of https://vector-search.wmcloud.org/ endpoint in recommendation experiments (T374669). These are currently small scale-experiments but there are some questions about how this could be scaled when potentially using it in larger experiments.

weekly update:

• no update this week (no immediate asks for support this week)

weekly update:

• Web Team generated simple summaries for a selected list of ~8K articles T375364 using the aya23-35b model from the Cohere API
• I am trying to evaluate the quality of the simple summaries by calculating 3 proxy metrics for simplicity, fluency, and meaning preservation (googledocs sheet). The aim is to identify low-quality summaries that should be filtered. Qualitatively inspecting the score for meaning preservation indicates that we can identify cases where the summary contains information that is not mentioned in the original article (negative scores or low scores close to 0). Plannning to inspect more samples if this approach for filtering makes sense.

weekly updates:

• shared revised set of guardrail metrics for simple summaries with Web Team (googlesheet).
  • most of the summaries are substantially simpler than the original and have relatively few grammatical issues.
  • most importantly, the meaning preservation metric (summaC) seems very useful to filter simple summaries that are not consistent with the original (e.g. error messages or text that was not contained in the original article). the simple summaries with very low scores should be discarded for the first set of experiments as lower recall is not an issue.
• these guardrail metrics thus offer an option to filter out potentially low-quality simple summaries.

weekly update:

• no updates because I was attending the team offsite during this week

weekly update:

• Coordinated with Web Team about filtering low-quality simple summaries for experiments. They applied one of the proposed guard-rail metrics to ensure factual consistency (meaning preservation) between the original article and the simple summary.

weekly update:

• After successfully implementing the Aya-expanse-32b model, I am generating simple summaries for the set of sample articles from Web experiments model the model on our own GPUs on ML-lab servers (related to T380643); code: https://gitlab.wikimedia.org/repos/research/simple-summaries/-/blob/main/simple-summary_aya-expanse_experiment01.ipynb
• Looking at initial results (see spreadsheet). I have been detecting two issues: i) simple summaries are not much simpler in terms of readability score; ii) output sometimes (~20%) in different language. Testing different ways to adapt the prompt to mitigate the issue.

weekly update:

• working on improving the prompt for generating simple summaries to make sure the language of the output is the same as the input. latest results for sample of 100 articles in this spreadsheet