Weekly updates:

• None (at ICWSM during the week)

Weekly update:

• No progress on blogpost but lots of notes from ICWSM about datasets for computational social science. In particular, some clear trends about the importance of having wiki-specific characteristics available for regression modeling (like MI's Wiki Comparison sheet which we could better share to those communities) and enabling more multilingual datasets for certain phenomena of interest such as things related to patrolling/moderation.

Weekly updates:

• Merged Destinie's tutorial notebook MR (!) and assigned the bug as a next step
• Put together an exploration of a different type of model (ordinal logistic regression instead of linear regression). This came out of Destinie's results and pair-plot analyses which were showing that as the number of features grew, the "effect while holding other variables constant" aspect of linear regression was leading to coefficients whose interpretation didn't match reality -- e.g., fewer sources -> higher quality. At first the thought was to go for something like Naive Bayes where the coefficients are learned independently but I could not find a model of that type that did linear regression so I went back to the drawing board to reconsider what a ordinal logistic regression model would look like. I had originally abandoned because the coefficients were less interpretable, it wasn't clear to me that I could effectively convert the class probabilities to a single point prediction between 0 and 1 (the desired output), and I was hoping to avoid the semi-complicated statsmodel dependency on LiftWing. In this revisiting, I figured out how to reproduce the model outputs without the statsmodel dependency and convert the logits generated by the model into a reasonable 0-1 range in a reproducible and not purely arbitrary way. The coefficients still are harder to interpret than the linear model ones but they're pretty straightforward (positive = good; negative = bad) and they match expectations. And an initial eval of model performance suggests that it matches or beats the linear approach. Notebook: https://public-paws.wmcloud.org/User:Isaac%20(WMF)/Quality/html-qual-exploration.ipynb#Ordinal-Logistic-Regression

Just acknowledging -- thanks @ifried for engaging! All of this makes sense to me. Thinking about not spamming high-edit-count editors though I don't have anything that obviously fixes it if reaching some of these relevant highly-active editors is a priority:

• Simplest way is probably to just pass along global edit count to organizers and encourage them to practice care before inviting -- e.g., checking user pages to see if they express an interest in the topic.
• I raised the prospect of filtering out edits with edit tags that suggested that they're semi-automated (autowikibrowser etc.) which could be a small step in filtering out some of this
• Maybe there's a way to use watchlists to help filter this out -- e.g., for high edit-count editors, check if the worklist articles are on their watchlists? The challenge here is one of privacy -- this is private information and you wouldn't want to leak it. So maybe this is instead future functionality that actually privately notifies editors if an upcoming event has a high overlap with their watchlist?
• Maybe for each worklist, you also collect a small random sample of articles (e.g., 20) and filter out anyone who also appears heavily in those?
• Larger than this one project, but we probably will eventually need some sort of opt-out system for these things (invitations but also surveys etc.)? And then it would be just a matter of cross-referencing the invitations with e.g., the user preference table to see who can be contacted

Weekly updates:

• No progress

Weekly updates:

• Provided feedback on Destinie's tutorial notebook for mwparserfromhtml (MR) and was excited that her work surfaced a bug in the library (though it's my code that is buggy). Many eyes!

Weekly updates:

• Put together a task (T366273) and meta page for my Q1 hypothesis of the article-country model. Hypotheses will be officially shared in the next few weeks and the request was put out for Meta pages with additional information for interested community members.
• Provided feedback on Campaign's worklist -> editor invitation candidates scoring approach (T363022)
• Shared some thoughts on topic infrastructure with Inuka team

Thanks for engaging! Sounds like we're on the same page, and again I wanted to stress that I think what you all have built is quite reasonable.

Ok, I got a chance to take a look. Thanks to @Daimona for the very well-documented code and function graphs, @Iflorez for your notes, and @ifried for the excellent Meta overview -- this made my job far simpler! Broad feedback:

• Overall:
  • Generally I assume your approach is working pretty well (though this is mostly based on hunches as we don't have enough data to really evaluate yet). I have some thoughts on smaller tweaks but I suspect that the report is spot on when it notes that the main feature that seems to impact the effectiveness of these recommendations/invitations is the quality of the landing page for the event and clarity of what they're doing. My main actionable recommendation is to consider flipping the edit-count feature so it prioritizes newcomers over experienced editors (more below).
• Modeling choices:
  • When I think about designing an algorithm (or learned ML model) to solve this sort of task of classifying data, I'm generally think about the trade-offs between simplicity and precision. Where on one end, you have a really simple model that anyone can hopefully understand and modify. Maybe its not perfect but that's okay because it's transparent to users and easy to adjust. On the other hand, are much more complex models that can't fully be explained but maybe are far more accurate at the task at hand. While it's tempting to think there's a good compromise solution that has a balance of both, I actually generally find myself either going with maximally simple or maximally performant depending on the task. The middle ground unfortunately usually just means that it's sorta hard to explain to folks/adjust but it still isn't so good that folks will happily accept the outputs.
  • For this model, I think we're clearly in the space of wanting a maximally simple approach because the task is inherently hazy (there is no "right" answer for who to invite). I think you're largely in that space, which is great. I think the features are pretty straightforward at a high-level. The geometric mean might confuse some folks but it still is pretty easy to explain. I'll admit that personally I find some of the more complicated feature normalization functions to be more than is necessary but that's a nitpick and the graphs provided in the comments help a lot.
• Evaluation data:
  • The main challenge is not having a method to evaluate whether the approach is working and potentially train a model to learn what the more empirical balance of feature weights is (right now they're set to 5 recent-activity vs. 4 bytes vs. 1 edit-count). So most of my feedback is based on hunches. With a fair bit of work, I assume you could encode who was recommended vs. who was invited and use that to adjust the model weights -- e.g., via a logistic regression model as Irene raised in her feedback -- but there isn't a lot of data yet so I don't know that that's useful at this stage. If you expand the pilot, I'd recommend tracking this somewhere (user, overallScore, rank, bytesScore, editCountScore, recentActivityScore, selected-for-invite, did-join-event).
  • As an alternative, it's hard to think of reasonable proxies on the wikis for this sort of task that would have more data. You could seek to re-generate past campaign participant lists but these are quite messy in terms of gathering that data and you'd have to find campaigns that recruited experienced contributors and had some sort of topical focus. WikiProjects don't currently have a good way of recording participants or they'd be a natural fit.
• Feature feedback:
  • Bytes:
    • Simple and while certainly not perfect, in practice it does map pretty well to edit difficulty / engagement once bots and reverts are filtered out as you do (quick analysis). This also makes sense -- adding a new reference/sentence/etc. generally requires adding a fair amount of bytes. The big maintenance edits are often via bots -- e.g., IABot as is mentioned somewhere. If you wanted, you could also do things like remove any edits that match a set of tools often used for basic maintenance -- e.g., for enwiki, I've used ['AWB', 'twinkle', 'huggle', 'WPCleaner', 'canned edit summary', 'OAuth CID: 1805', 'RedWarn', 'Ultraviolet'] in the past as edit tags that indicate tool-assisted edits but that's a manual list I put together based on scanning Special:Tags.
    • The main challenge I see with this is that you're not distinguishing between editors who are active in the topic space and editors who are just very active. For example, when I was doing some work on finding similar users to a given editor (as a proxy for potential sockpuppets), I was building a list of editors who most overlapped with a given editor based on edit history. But if you just use this info, User:Ser Amantio di Nicolao appears on everyone's list because of how prolific they are. Instead, you need to normalize for how many edits someone makes in general (akin to tf-idf). This is challenging to do via the APIs but you might actually consider weakly "penalizing" editors for high-edit count instead of rewarding them. Where the logic is that while high-edit-count editors might be good contributors, they also are more likely to have just incidentally edited these pages and not have an actual topical interest and we want to reduce spam to them too (them ending up on everyone's invitation list).
  • Edit count:
    • Building further on the above about "penalizing" high-edit-count editors. Given the nature of geometric mean (tends towards the lowest value in the set), this means that if you have an editor who signed up and maybe made a few edits via Newcomer Homepage about a topic of interest to them but isn't sure what to do next, they're going to have a really small value for the edit-count feature and also probably for bytes-changed (assuming they made simple edits). This is going to put them always at the bottom of the list when I feel like maybe they're the most important people to invite (as they need that nudge to continue and structure of a campaign and they're least likely to discover the campaign themselves). As I said above, I feel like you might want to actually take the inverse of someone's edit count for this feature so lower-edit-count folks who overlapped with the worklist are prioritized.
  • Recent activity:
    • Seems reasonable to me!
• Expansions:
  • Honestly I feel like you have a reasonable set of features based on what can be efficiently gathered from the databases. I've done some work on classifying edits by what they changed so you could e.g., isolate editors who added new references to articles and prioritize them, but that's a much more expensive feature to compute and it's not cached anywhere yet unfortunately.
  • We've discussed using the list-building tool (or even just Search's morelike API for something production-ready) to expand the worklist (when they're tiny) to include more articles in the same topical space (and therefore more potential editors). We don't have any way of evaluating how impactful that would be. My feeling would be let's explore it if you're hearing that the invitation lists are too short or overlap too much with who they already were going to invite. But otherwise not necessary yet.

Paper: https://arxiv.org/abs/2404.03428

Just adding another note of where these quality scores could be useful (filtering machine translation candidates): T293648#9816202

Weekly updates:

• Added the next section: https://meta.wikimedia.org/wiki/User:Isaac_(WMF)/AI_Datasets#Dumps
• Just some details around benchmarks and the final "Datasets" section to write

Weekly updates:

• Sources is proving tricky as a feature so I left some ideas for how to "debug" what's going on there in: T364014#9810129
• Documentation work is continuning! Details: T365269

Not too much movement in this space this week on my end though some additional tasks that I signed up for:

• Helping with upcoming Community Growth contract hiring
• Review for T363022

Thanks @DJames-WMF for your first pass (notebook)! Next steps now that we have some initial findings:

• You're extracting message-boxes/infoboxes correctly but you'll update the code to make sure they're retained in the final model as well.
• You can trim out some of the old summary at the top of the notebook to focus on just this expanded model and its coefficients (with a link back to Copy-2 where folks can find more details about the earlier iterations).
• What to do about this mysterious negative sources coefficient?
  • I've been wondering how consistent the different languages are. One way to test this would be to train a separate model for each language and see how similar the coefficients are. This might help identify features that are less stable and worth investigating. Perhaps some insight on the sources feature or maybe even some of the others too. I expect the magnitudes might vary a bit but if any switch from positive to negative (or vice versa), that would be the interesting sign to pay attention to.
  • In these sorts of models, each feature coefficient is the impact of that feature when all other features are held constant. Because of this dependence on other features, the coefficient for a feature like sources doesn't just depend on its relationship to quality but also on its relationship to all the other features. I think this is what is going on. This (quite long) tutorial on linear model coefficients/interpretation/fine-tuning has some useful examples of this and ways to chart things out. In particular, I would love to see similar charts for our data/model to:
    • The pairplots in this section, which will help us see which coefficients are correlated with each other.
    • The coefficient variability plot in this section, which will help us see which model coefficients are unstable.
    • If it turns out that sources is highly correlated with the other features, we might have to take it out. It might also be possible to e.g., switch it to a simpler boolean (>= 5 sources) and that that would help reduce the correlation while still retaining most of the benefits of counting unique sources separately from references.

@ngkountas I think you're right. Let me know if you have a task for making the switch because once you all have completed the transfer and verified it's working for a bit, I'll look into deprecating the current endpoint that you're using (as it's long overdue for being shut down).

Weekly updates:

• Draft of intro: https://meta.wikimedia.org/wiki/User:Isaac_(WMF)/AI_Datasets

Summary of some data analysis I did for evaluating the current topic taxonomy and gathering some thoughts about potential changes (google doc with more data/notes):

• We'll have to do some cleaning up of the WikiProject->Topic mapping as WikiProject names etc. have shifted since it was created in 2020. For example, WikiProject Climate Change used to be a task force of WikiProject Environment (I think) and then became its own project so is not currently tracked in the data. This seems pretty doable though as a one-time manual pass and allocation of larger WikiProjects to specific topics.
• Big changes that I think we are pretty certain about:
  • Shifting of geographic topics to a country-based model. This will allow for more granularity than current regions and incorporate data from Wikidata so build on that community work.
  • Shifting of model-based outputs for people (biography/women topics) to a Wikidata-based output (deterministic based on instance-of:human and gender properties). This will lose some of the hazier, women-related topics that the model-based women topic could surface but be clearer (less likely to provide problematic predictions) and we will see about addressing some of this change with the new topics.
• A number of small changes to the arts/science topics -- e.g., perhaps merge a few categories that get low usage and have low coverage.
• The larger discussion will be around how to handle some of the existing history/society topics and what topics are possible for folks engaged in sustainability and human rights work.
• Expanding the data pipeline to incorporate WikiProjects from other language editions wouldn't have a large effect at the moment (most major wikiprojects with coverage of non-English articles are for geographic/biographical topics and only a few are in areas where we probably do need more diverse data like history/society topics). But this might be useful for certain topics if we do have low data volume/diversity from English and we know there are relevant WikiProjects in other language editions supported by PageAssessments.

@Miriam I don't mind either way but I'll be bold. This is my quarterly goal task so it touches on the topic classification evolution but also other related aspects and I mainly see as a personal tracking task that I intend to close out at the end of this quarter. I think best thing would be to make this a subtask of T343241 (as in I'm playing a supporting role for the taxonomy work) and I'll shift my updates over there when they're about the topic taxonomy.

Weekly updates (adding early while it's fresh):

• I put together some data and thoughts around the next iteration of the topic classification model in discussion with Alex as far as what steps Community Growth should be leading to do the community consultations on making improvements to it (google doc). Summary:
  • We'll have to do some cleaning up of the WikiProject->Topic mapping as WikiProject names etc. have shifted since it was created in 2020. This seems pretty doable though.
  • Big changes we both agree on are the shifting of geographic topics to a country-based model and shifting of model-based outputs for biography/gender to a Wikidata-based output (deterministic).
  • A number of small changes to the arts/science topics -- e.g., perhaps merge a few categories that get low usage and have low coverage.
  • The larger discussion will be around how to handle some of the existing history/society topics and what topics are possible for folks engaged in sustainability and human rights work.
  • Expanding the data pipeline to incorporate WikiProjects from other language editions wouldn't have a large effect at the moment (most major wikiprojects with coverage of non-English articles are for geographic/biographical topics and only a few are in areas where we probably do need more diverse data like history/society topics).

Weekly updates:

• I started the writing process by documenting some of the background around the different types of models at Wikimedia: https://meta.wikimedia.org/wiki/User:Isaac_(WMF)/ML_modeling_at_Wikimedia. It's a first pass so I can update and refer to in a blogpost or just pull the salient pieces actually into the blogpost as needed.

Weekly updates:

• New normalization values generated! code and values.
• Destinie has made good progress on incorporating these into the notebook and I think we're essentially in a place where we can begin to add new features to the model. I put together a task for that (T364014). She also has begun work on improving some of the mwparserfromhtml documentation (MRs) so she'll have a better sense of what the library can do re: potential new features.

Weekly updates:

• Added basic form of link-based inference to the country-article prototype -- e.g., https://wiki-region.wmcloud.org/regions?lang=en&title=Japanese%20iris. That was a final feasibility check for me and I'm going to pause on development for that for now until Q1 begins. The next steps for when I pick back up that work:
  • Evaluation:
    • Offline: probably a large stratified sample by geo + language edition to test link-based logic -- i.e. whether it can reproduce what's already in Wikidata. I think I should be able to easily re-write the API logic to use the cluster instead so it's fast to test/iterate.
    • Human: a small sample of articles with Wikidata properties to just verify that those are indeed accurate and complete when present but I think it's fair to assume ~100% precision/recall for those. Focus then would be on articles lacking Wikidata-based country properties. For those, just have folks go through the corresponding Wikipedia article and tag with any relevant countries. Might need to stratify by continent to make sure even-ish coverage but I want to keep the sample size manageable.
  • Guardrails (how to handle links):
    • Motivating challenge here is something like the biodiversity articles -- e.g., https://en.wikipedia.org/wiki/Limonium_strictissimum. This plant is native to Italy/France, which is mentioned in the article, so ideally those two countries would be predicted. There are actually more links, however, to US/UK because many of the orgs linked to in the Taxonbar at the end of the article who track information about plant species are based in those countries.
    • Why it's not trivial to fix: we use the pagelinks API to get info on links because it can easily be run as a generator so with a single API call we can get all the links and their corresponding Wikidata IDs (for looking up countries associated with each). So we can't e.g., exclude links based on how they're presented in the page. We could in theory maintain a list of links to ignore based on how many articles they're present in -- the list is probably not super long and would be effective and filtering out these sorts of links but it's also an additional layer of complexity.
    • My current approach is two-fold:
      • I do apply a tf-idf transformation (code) to the link proportions for each country so e.g., a few links to Ecuador will be treated as a strong signal than a few links to the US. This helps a bit with the US/UK problem (also dampens France quite a bit).
      • I require a minimum count (3) and minimum proportion (0.25) of links in order to elevate a linked country to a prediction (code). This was aimed pretty directly at the taxonbar issue but I'm sure it could be fine-tuned. The challenge is balancing a requirement of enough support to be "real" without making the bar too high for stub articles to exceed. The minimum proportion part also makes it hard for articles that are relevant to many countries to ever reach the threshold, which I don't love but also might be acceptable behavior. For example, the WWII article is certainly relevant to many countries but isn't necessarily a useful result if you're filtering by country to find content to edit.
    • Another possible guardrail is restricting where we apply the link-based logic. One approach could be only running the code for those articles lacking coordinates / any Wikidata-geo-property? This would reduce the possibility of false positives and probably let us better fine-tune the model to articles in topics lacking Wikidata properties about countries. Might confuse things for the end-user but also better latency and maybe nudges editors to improve Wikidata if they find issues with the predictions.

Weekly update:

• Presented initial ideas to Gap Team (presentation). Both help me process all my thoughts and led to some good feedback and sparking of ideas. Writing should be far easier now.
• Things I'm pondering:
  • Folks appreciated my attempt to describe the different types of AI models at Wikimedia as background to why these datasets are important. This is a lot to fit into a blogpost but maybe I write a separate one or even just a wiki page as a explainer
  • The blogpost is definitely ballooning in size. I probably don't have to decide right now what to include/exclude, but TA made the good suggestion that it could be a series instead of single blogpost. So something like: 1. Intro / Current State; 2. Data Gaps; 3. Benchmarks

Weekly updates:

• I started on loading the HTML dumps into HDFS (code courtesy of Fabian) -- this is working well and I tested with Arabic and was quite happy with how quickly it processed. Though loading in English is taking some time...
• Destinie is working out some kinks in our HTML features

Weekly updates:

• ML Platform and Search Platform indicated that my plans were fine for the article-country hypothesis and they can support deployment. In particular, EB on Search indicated that the broader expansion of tags on Search index for recommender systems shouldn't pose any issues.
• Put together basic API for using just the Wikidata properties: https://wiki-topic.toolforge.org/countries
• Good meeting put together by Miriam in which we charted out that Community Growth could do some outreach to get feedback on the current topic taxonomy and we'd work to make updates based on that but then try to freeze the taxonomy.

@YLiou_WMF here's the task -- please sign L3

Ahh this is great news @kevinbazira ! @KartikMistry is there any reason from the Content Translation side why we can't switch over to the LiftWing endpoint? My read is that the code is quite simple -- e.g., if I go to Content Translation on Spanish Wikipedia, the tool hits this endpoint:
https://recommend.wmflabs.org/types/translation/v1/articles?source=en&target=es&seed=Music%20Modernization%20Act|Felony%20disenfranchisement&search=morelike&application=CX

hey all (not sure who exactly to tag but maybe I'll start with @kevinbazira just because I know you did a lot of good work on this) -- I'm working on some planning for improvements to our recommender systems for next fiscal year around what topic filters we provide to editors. Content Translation is of special interest but Android's SuggestedEdits is important too. The recommendation logic for both of these systems is still hosted on GapFinder as far as I can tell, but deploying any improvements is going to require moving them to a proper service (LiftWing). Does anyone know why this effort to move Content Translation's recommendation API over to LiftWing (along with Android's endpoints T340854) stalled last year?

Weekly update:

• Spoke with Stephanie (Enterprise) who will be attending a symposium on ML benchmarking with Wikimedia data and shared my early thoughts on the subject.
• Spoke with Adam B about his 10% project around a Commons dump and some of the likely challenges / needs associated with that. It left me feeling optimistic though it won't be solved overnight.
• Forgot that I was to present on this this week to our team though and postponed to this upcoming Thursday so will be doing some deep thinking and writing in the meantime

Weekly updates:

• I put forth a draft hypothesis for next year related to a country-level article prediction model: If we build a country-level inference model for Wikipedia articles, we will be able to filter lists of articles to those about a specific region with >70% precision and >50% recall. I had a conversation with Fabian about this too and it'd be easy to pull in the cultural/geographic code that currently exists for inferring countries based on Wikidata properties. To take it a step further and cover articles without Wikidata items or with incomplete items or for geographic aspects that are not really covered in Wikidata -- e.g., geographic extent of flora/fauna -- I'd want to do some inference based on the country topics of the links in an article. Doing this online would be challenging (likely high latency as you'd need to evaluate many articles at once). There are ways to build a cache of predictions for articles and use that for evaluating the links but then you run into challenges with cache invalidation etc. Because the intent is to load the model predictions into the Search index as weighted tags, however, we can actually probably use the Search APIs to gather the country predictions for an article's links (analogous example for articletopic for en:Japanese_iris) and infer from there. This is nice because the Search index will always have up-to-date information and so we won't need to store this source of truth in multiple places.

@DJames-WMF has made progress on converting the wikitext over to HTML features. We're finding that the old normalization values -- e.g., how many references are expected in a top-quality article for a given wiki -- are no longer well-aligned for a few features. This seems to be most relevant for page-length which then affects wikilinks and references as well. I'll need to look into re-generating these normalization values. A few options:

• Use the APIs to fetch HTML for a random sample of articles to re-calibrate the values. Sample size could be a challenge though because we're looking at the 95th percentile so we need a large enough sample for that to be stable.
• Slowly loop through the whole Enterprise HTML dump -- this would take a very long time and in my experience the article ordering is not random so we can't stop early unfortunately without biasing the result.
• Load a snapshot of the HTML dumps for the relevant languages into HDFS and process in parallel -- this is probably the most sensible solution because then we can re-use the data if we ever need to come back and recompute a value.

Confirmed -- thanks @DDeSouza !

Would it be possible to add the theme of the showcase to its subpage title

Good idea. I think it should be doable (just need to move the pages to the new titles). The downside would be it's harder to guess the page title but maybe that's not an issue. One alternative too: when we picked up this task, there was also a question about whether we wanted some sort of "summary" of our archive too. Maybe the listing of pages isn't the place to do that and instead we add a basic table to the page with each month, a link to the full description, and the theme?

@DDeSouza I went ahead and made a merge request for a new paper and some small adjustments to the other papers (seemed easier than trying to explain in this case): https://gitlab.wikimedia.org/repos/sre/miscweb/research-landing-page/-/merge_requests/25

Thanks all for the patience on this -- I have now moved all the past showcases onto monthly archive pages and added the search functionality to the main page: https://www.mediawiki.org/wiki/Wikimedia_Research/Showcase#Archive

Next steps for this notebook based on Destinie's assessment (notebook) of how well-distributed each model feature is after switching to HTML. We have three features that are poorly distributed (values all lumped together) so the model cannot learn much from them. They are:

• Page length: the values are all lumped around 1 because Parsoid HTML (with all of its syntax) is far more verbose that wikitext and by definition a superset of the wikitext. We don't have any perfect way of getting back to the wikitext length but probably a more reasonable assessment of article length is how much text is in it. So instead of len(article_html), let's use the get_plaintext() function and take the length of that. That function has a bunch of settings for it to work appropriately so let's use the approach used by html_to_plaintext() in this notebook with a few small tweaks:
  • Don't exclude List elements (they often have valid content from an article quality standpoint)
  • Take out the if len(paragraph.strip()) > 15 clause for each paragraph (we're just counting up things so I'm okay with the occasional "weird" paragraph)
  • Rather than doing the final if paragraphs: check, just use '\n'.join(paragraphs) for computing length -- this will just be an empty string (length 0) if no paragraphs.
• Media: the reason they're lumping to 1s is probably because many articles have lots of little icons that aren't defined in the wikitext (transcluded via templates). These are inflating our counts of images in the article. I put together one heuristic to filter these out in the test cases and I think we can re-use that pixel-size logic here too (code). This should reduce our media counts back to where they're more evenly distributed.
• Categories: Here the lumping towards 1 values likely is the result of hidden categories (usually transcluded via templates again and not in the wikitext). One way around this is to check each category returned by get_categories() to see if it was transcluded. There's an existing function in the library (example import statement) and then we can do something like len([1 for c in article.wikistew.get_categories() if not is_transcluded(c)]).

Weekly update: no progress

Weekly updates:

• Asked for input from Search on adding in the different topic tags we're considering (countries, quality, wikiprojects): https://etherpad.wikimedia.org/p/recsys-search-tags-future
• Talked with Inuka Team about challenges/opportunities in this space as they consider potential projects to take on
• Part of discussions with EH at Wikimedia Uruguay and others around their new templates for WikiProjects, which automatically find tasks to surface to editors: https://es.wikipedia.org/wiki/Wikiproyecto:Cambio_clim%C3%A1tico. This is an exciting replication of the infrastructure that Growth has worked on for Newcomer Homepage but by community members within the WikiProject context. It's further motivation for adding WikiProject tags to Search as well because without that, it's much harder to use our structured task filters (add-a-link; add-an-image) because there's no single query that filters by Wikiproject and task availability.
• Began exploring feasibility of geography model on LiftWing. Ascertained that there could be key-value store support in the future that might be useful (if we use links to infer countries, we'll need to quickly look up the associated countries with each article link). In the meantime, it should be easy to just grab an item's Wikidata JSON and just check the country-related properties as we do with the culture metrics.

Yep, that work is happening under T360455 and T360572

Added another step for the bug-fixing we're working on right now with 0-values for some of the features. I also unchecked the optional exploration -- that actually is separate from the notebook (it involves updating a README file in a code repository) so we can talk about it in a future meeting and decide whether to pick it up or not.

Thanks! Unlikely to happen soon but when we reach a stage where we are re-training the model, I'll see if we can experiment with nudging the model away from these sorts of responses (because agreed that it's ideal to solve it via model architecture / training as opposed to post-hoc filters if possible). And please continue to share if you see other patterns in incorrect recommendations.

Excellent work @DJames-WMF ! Took a readthrough of your notebook and everything looked good. Closing this as resolved. We didn't pursue the Nepalese Wikipedia extension but that's okay -- we can always come back to it later. For now, I'd like to progress to the HTML work that you've started in T361623.

(Shouldn't this be a factor for machine learning? I mean, if matching the title produced a wrong description as a general rule, wouldn't the machine learning algorithm infer it from the training set?)

Closing this task out. We can re-open or create a new one in case substantial new work is required as a result of COLM etc. I'll still update with an arXiv link when available.

@DJames-WMF can claim and start this task when T360815 is complete.

Human
3 beams:

Ethnic group
Ethnic group of humanes
Ethnic group of humans

Thanks for passing this along @Jack_who_built_the_house! I checked a number of other very high-level topics and didn't find it in Civilization or Primates but did get "Class of plants" for Plants. This sort of error seems most likely with article about very high-level concepts (which often already have article descriptions thankfully) but would still be nice to fix obviously. We might be able to address this sort of tautological output by adding a simple string-matching check to ensure that the output doesn't contain the title itself. Before we implement anything, I'd want to think about what sort of issues this might cause though with e.g., very simple titles where text matching might introduce a bunch of false positives (and therefore not return results).

Weekly updates:

• I'm behind on this in part because my thinking is still pretty wide-ranging but I'm continuing to process what I read and mull over the different angles of this. I've been attending to more urgent aspects too with Annual Planning / mentorship / etc.
• Chris A. put together a nice spreadsheet of a few knowledge integrity tasks that he, Marshall, and Maryana P. put together for benchmarking some LLMs: https://docs.google.com/spreadsheets/d/1b2eG8ZlWVJa5LQDSJMACivWzZ9DCxiYjv6oImucnc20/edit#gid=0

Paper submitted to COLM and we'll hear May 24. I'll link to arXiv paper when posted.

Notebook looks good - thanks for the hard work and patience on this!

Task created -- @isarantopoulos just let me know if any details are missing or anything I can do to help with next steps when you are ready!

Weekly updates:

• Sent thoughts over to SB about wiki benchmark datasets so we'll see if he's available to give feedback or has suggestions for others folk to talk with.
• Bellagio draft points to importance of the ideas I had of writing a history of machine translation at Wikimedia with Eli. Specifically relating to the call for: A review of currently deployed systems, including (where available) quantitative and qualitative evidence of use and impact.
• https://foundation.mozilla.org/en/research/library/generative-ai-training-data/common-crawl/ elicits to me a lot of parallels between Wikimedia data for AI and Common Crawl -- e.g., neither was intended as an AI dataset but are heavily used for that purpose, which can bring some considerations for appropriate usage. The major difference is that while there is no moderation of the internet (Common Crawl), Wikimedia does have excellent moderation and therefore the possibility for harm is far lower. In general, a good blogpost to cite when talking about why it's important to think critically about how content is used for training AI.

Weekly update:

• Paper solidifying -- just style / readability aspects at this point. Should be good to submit to COLM this month. We'll upload the submitted paper to arXiv as well.

Weekly updates:

• Reached out to Sam Bowman who has worked on natural language benchmarks / LLM alignment so I could pick his brain over what it would mean to work on a Wikimedia benchmark and/or instruction-tuning dataset.
• Connecting with Enterprise as they have some potential planned work in next FY around datasets on HuggingFace and benchmarks! They're also talking about the feasibility of a Commons dataset as there was another high-profile request for imagery data.

Weekly update:

• Made Leila's requested fixes including adding a future work section that touched on different dataset filtering processes and ways of representing an edit to language models. We're a bit over 9 pages (the limit) but working on bringing it within requirements.

Very excited to see this gaining some traction (thanks @mpopov and @dr0ptp4kt)! Commenting on the analytics side of things (I don't know enough about Varnish to comment on implementation details):

This is really wonderful news! Thanks @kevinbazira for slogging through this with us and @isarantopoulos for your support as well! Those endpoints were working for me too so I'll let Android indicate what the next steps are.

Thanks @taavi! Indeed, unblocked now

Connecting to another ticket focused on producing these topic snapshots: T351118

Weekly updates:

• iteration on recommender system vision
• began writing some of my thoughts around AI datasets though haven't gotten to benchmark data part, which I think is the piece I most need to think through (the rest is things that I've been thinking about and working on for a while now)
• Read the STORM paper on generating Wikipedia articles from scratch using LLMs. There's a lot in there but in particular relevant to this work, they put together a dataset of newly-created, high-quality articles as a means of benchmarking a large-scale LLM on just new content. Good example for the benchmark datasets. I'd want to take their approach probably but combined with some details on how recent the references are perhaps (would add some complications around extracting that info unfortunately).
• Watched a nice, long overview of tokenizers that would have been useful before we did the mwtokenizer project but also largely validated a lot of our thinking there and gives some more food for thought. Looking back, I think I would have dropped the sentences aspect and stuck to just paragraphs (easy) and words (hard). With increasing computational power, most things that use sentences can instead use paragraphs and that's way less prone to issues. Sentences become very important really only in the context of citations because the citation generally refers to the prior sentence and so it's useful to be able to extract that for determining entailment.

Weekly update:

• Marija moved paper onto new template for COLM so going to do a pass on that next week

I'll let others chime in but that would be my feeling about the correct scope. Going historical indeed adds a lot of complications and I think current snapshots are a huge first step. I'd coordinate with Enterprise obviously just to see if any changes are going to happen with schema etc. but hopefully relatively straightforward.

Moving this to discuss with the team. Seems reasonable to have 1 or 2 versions of this if we source it from the Enterprise dumps.

Thanks @lbowmaker for considering and @mfossati for raising! Just chiming in to add my support that having a current snapshot of Parsoid HTML from Enterprise would be very helpful. We've developed a Python library (mwparserfromhtml) that enables us to extract lots of features (references, infoboxes, plaintext, etc.) easily from the HTML so are in a good position to make use of it. Within Research, we're working on switching more of our models to using it too because the gap between wikitext and HTML is definitely growing (example with references). For example, we have an intern who will be working on converting the quality model used for knowledge gap metrics from using wikitext to HTML for this reason, so having a regular snapshot that could be used for computing article quality for all articles would be very helpful.

Can we investigation reducing the computational need to just the language requested?

The model definitely benefits from some translations so "just the language requested" I would say is not the right approach. Are you suggesting capping it at 5 for example? It no longer seems to be an issue from the pre-processing perspective with Ilias' fix at least so hopefully not a blocker at this point and just a bonus for capping model latency. That said, if there's a desire to further constrain, I can look into it in the next few weeks but please don't let it be a blocker given that Android has said they're comfortable moving forward per T343123#9558740.

we currently have 4 workers consuming tasks for programs and events dashboard, and 3 for wikieducational dashboard. So I guess that the max number of concurrent requests would be 7 (if all the workers are working at the same time).

Yeah, that's quite reasonable! Thanks for looping back about it.