Overview

Data contamination is a major concern for LLM benchmarks and particularly relevant to Wikipedia datasets. When LLMs are trained heavily on Wikipedia content and then see that content in benchmarks or test datasets, we often get an inflated sense of their true performance at a given task. A better estimate of the performance of a LLM if it were deployed (and thus often be evaluating new, unseen content) is to also only evaluate it on Wikipedia content that it was unlikely to have seen during training. Different LLMs have different cut-off dates. Ideally for any Wikipedia-based dataset, we could filter down to just data points that are unlikely to have contamination issues with a given knowledge cut-off. This has already been explored in benchmarks like FreshWiki and NPOV/Peacock detection (T377425#10384488).

Task

The goal of this task is to build a dataset of Wikipedia articles and the date at which content in that article is likely to have been included in LLM training data. This will enable researchers to filter down their Wikipedia-based datasets to just post-cutoff data and determine whether data contamination poses a challenge for their evaluations.

There are several approaches to assigning these dates we could include in this dataset. For all of them, there is a variant that focuses just on the specific language edition and another that takes the earliest date for that article across all language editions. I'll use en:Maria_Trubnikova as an example for each, which is a featured article on English Wikipedia as of January 2025.

• The date at which the article was created: the article for Maria Trubnikova was created in English Wikipedia in July 2016 but the first version of the article was created on Russian Wikipedia in June 2013.
• The date at which the article was substantially written: The original English Wikipedia version (July 2016) was a one-paragraph stub and it was not until September 2023 where it was expanded to something close to its current state with multiple sections. The Russian Wikipedia version was actually pretty substantial when it was first created and thus there is no difference there between creation and substantially-written.
• The data at which any content about the topic was likely available: while many Wikipedia articles are written as current events unfold, other topics might have available online sources but not be written about for a while after they become available. While it would likely require extensive pipelines overtop Common Crawl data to effectively estimate the date at which a topic appeared on the internet, we can get a rough estimate based on the sources used by Wikipedia editors. Even this is admittedly tricky for the Maria Trubnikova article: it has 13 distinct sources but 12 of them are books ranging from 1977 to 2022 in publication dates and the other is a journal article from 1979. It is not clear whether any of these are digitized and part of training data (whereas, if the sources were websites, they're more likely to have been seen by the model). The current Russian Wikipedia article does link to a web source from July 2020, but that is not earlier than the date for the Wikipedia article itself.

This initial task can be a one-off dataset that shows how we might calculate each of these three dates (and then follow-up work can determine how to efficiently update the dataset etc.). Ideas for initial approaches:

• The date at which the article was created: there is no explicit source of truth for this in Mediawiki anywhere but Mediawiki history have an approach for calculating it that we can rely on. Mediawiki history is available as a table (WMF cluster) and public dump.
• The date at which the article was substantially written: for the knowledge gaps metrics work, features that are relevant to article quality are computed monthly and extend back to 2001. We could either define "substantially written" as something similar to the standard_quality metric or come up with another set of heuristics for when an article reaches this point. There is no "right" answer here: mainly I would choose some random articles and go through their history and try to decide when the content feels substantive enough that a model would be "contaminated" about the subject (vs. knowing very little). In theory we could adopt more empirical approaches -- e.g., prompting LLMs to tell about a given topic and assessing their knowledge/confidence -- but I suspect those are quite tricky to assess well. The knowledge gaps features are available on the WMF cluster in the research.article_features table and code for this table can be found on Gitlab. Folks without access to the WMF cluster could generate a small sample of data to work with based on that code.
• The data at which any content about the topic was likely available: being able to include this date depends on being able to effectively extract the dates of publication for citations on Wikipedia (across all languages). This likely depends on someone completing T374554 and thus might be on hold for this task. But one basic approach would be to check each source in an article and find the earliest publication date provided for a source that has an associated URL. Some work will likely be needed to also standardize the dates into the same format. This data does not exist yet but could be extracted either from the Enterprise HTML dumps or Snapshot API. Or if you want to start with wikitext, that's available via standard XML dumps.

For finding corresponding articles across language editions, you can either use the langlinks table (available in the dumps) or wikidata_item_page_link table internally (details).

This task is considered [medium]. In general, it's expected that the task will take a a month or two of consistent work and is a good fit for someone with some prior coding experience and interest in being involved in research. The actual time needed, however, will depend greatly on your level of experience.

Recommended Skills

• Python coding is recommended for collecting the data and running the analyses
• Some understanding of LLMs and knowledge cut-offs is useful for understanding the task and making judgments

Acceptance Criteria

• The output of this task will be a Jupyter notebook showing how to calculate each metric with documentation of what choices were made and any caveats (e.g., expected missing data).
• The notebook can either be fully public and use PAWS (Wikimedia-hosted Jupyter notebooks) or use WMF cluster resources/datasets and be shared via Gitlab.

Process

• This is an open task that anyone is free to work on. Feel free to reach out to @Isaac if you have questions and I'll try to respond but support will depend on what availability I have and depth of support is needed.

Additional Resources

• mwparserfromhtml library for working with article HTML (re: sources)
• Example of working with Mediawiki history dumps on PAWS
• Project that focused on extracting information from citation templates
• mwxml library for working with XML (wikitext) dumps
• mwsql library for working with SQL dumps