Work was finished. Thanks!

Great, thank you for the update and for the question.

Hi @dcausse! I wanted to check in on the progress of extracting the query-result pairs. Could you please share an estimated timeline for when this might be ready?

Great news, thank you for the update.

Hi @dcausse! Thank you for the updates.

Hi @dcausse! Thank you for the results!

Hi @dcausse! Thank you for the update!

Hi @dcausse! I have moved the /mteb folder to /user/dcausse/T419409-miracl, so you should have access now.
Please let me know if it works. Sorry for the initial inconvenience.

I have prepared the corpus and queries for MIRACL dataset for EN, DE, ES, FR, ID.

• Created the task description to use in the study.
• Estimated the study duration at approximately 30 seconds per sample, resulting in about 6 minutes per annotator, accounting for 10 samples per annotation and additional time for instruction review.
• Updated the data preparation pipeline for the study by merging the page title, section name, and paragraph text into a single field, improving the annotator user interface.

Implemented the search results selection logic with the following workflow:

• Collect search results from the search engine at the page level (currently limited to Wikipedia; this should be extended to additional sources to avoid selection bias).
• Compute ranking scores for all paragraphs within the retrieved pages using a cross-encoder reranker model.
• Select the top five ranked paragraphs, with a maximum of two paragraphs per source page.

• Refactored the query selection logic to enable execution of the full pipeline in PySpark, allowing processing of the complete 90-day log corpus. Applied a minimum threshold of 25 identities per query.
• Conducted a manual review of the collected queries to assess the presence of PII; none was identified after applying the 25-identity threshold. The review also informed potential improvements to the selection logic.
• Following the initial review, implemented additional post-processing steps, including near-duplicate removal using Levenshtein distance (e.g., <film name> episode 2 vs. <film name> episode 20) and automated filtering of patterns associated with automated parsing of non-notable companies. While this step removes the majority of such cases, some manual post-filtering may still be required.

Model card can be found here: RevertRisk_Wikidata

• Created a code pipeline to prepare the data for annotation using Prolific's AI task builder tool.
• Created a mockup study on Prolific to test the functionality of the annotation UI.
• Agreed to proceed with the Prolific framework with minor adjustments:
  • Try merging page-title/section into paragraph content -> Rename paragraph content into “result”. (to avoid scrolling).
  • Improve the paragraph splitting logic to incorporate corner cases.
  • Estimate the time of study.

I have reviewed the latest version of the model, and the differences in scores appear to be unacceptable, so changes are required.

• Explored the options for external search engines. Most of the options can't be used because our use case might not comply with the law or Terms of Service.
• Created and initial pipeline to search for candidates using Wikipedia's internal search. The results will be served in the format of:

{
    "title": <"title">,
    "snippet": <"snippet">,
    "pageid": <"pageid">,
}

• Explored the possible logic for paragraph reranking for the selected candidates' pages. I recommend proceeding with pretrained crossencoder models. In particular, I tested the pipeline based on jina-reranker (pipeline notebook with usage example). Potentially, we can improve performance with better models, such as Qwen3-Reranker, but I currently face infrastructure constraints.

• Finalized the logic for query selection based on the search logs (Document that observes the full logic). Added the filter to avoid matching page titles with the query (navigational queries).
• Collected an initial set of queries for the pilot experiment (notebook with query selection logic)
• Performed manual evaluation of a small random subset of queries to confirm selection quality (only ~1-4% of queries to be manually filtered)

Thank you for the update!

I have prepared the model binary for the full model and instructions/tests that might help you with productization. I was unsure where to commit these files, so I have prepared a Google Drive folder containing all the necessary files and a README that explains their contents.

I began examining web request logs to compile a dataset of Wikipedia search queries, collecting all queries over a two-day period and exploring their structure and characteristics (based on notebook). Early analysis confirms that most queries are short (around two words). I have found out that a notable portion includes advanced prefixes (e.g., “insource:”) (~3%), links (~3%), or named entities (~55%).

I am working on collecting a binary of all components of this model (BERT + classifier). I just need a bit more time to thoroughly test everything and ensure nothing is missing, as the project has been inactive for some time. The plan is to have it ready by Monday.

Thank you for sharing details and working on it, @kevinbazira!

1. Regarding the service input/output, I believe the best reference is the Wikipedia revert risk model, with the main difference being that for Wikidata we don’t have a language parameter. You can find the API reference here.

We identify 3 main dimensions for types of queries based on existing literature:

• Query intent (see Subtype in the proposed taxonomy): Directed (Closed) or Undirected (Open)
• Query form: lexical (shorter, focuses on word matches), semantical (focuses on meaning rather than wording, e.g., natural language questions)
• Type of expected result: (e.g., description, numeric, entity, location, person, based on MS Marco), specific types to be defined.

Citation Needed experiments

1. I have updated the final report document by incorporating detailed information about the experiments with LLMs. Specifically, I have provided technical details, code links, results, and summaries for the Peacock tone, NPOV violation, and article categorization tasks.
2. I have been exploring a strategy that could enable the extraction of probability scores using the logprobs of prompt tokens feature of large language models (LLMs), as opposed to the current binary prediction approach.
3. I started working on running full experiments for the Citation needed template using an approach similar to Peacock tone and NPOV violation detection.

1. Experiments Finalization for NPOV and Peacock Templates

1. Full data experiments for NPOV and Peacock templates:

1. Experiments with Aya Model

1. NPOV violation detection (initial experiments)

1. Peacock detection (initial experiments)

1. Experiments with Llama7B and Mixtral

1. Metrics Interpretation:

1. Evaluation of Multiple Configurations Using Together AI

1. Experimentation with Category Reduction Strategies for Prompt Generation

• Analysed the collected data and communicated recommendations for improvement to @Aitolkyn.
• Developed and implemented a process to create a dictionary mapping topics to categories using TF-IDF. This involved filtering out rare categories and selecting the top N categories based on non-zero TF-IDF scores.
• Designed and implemented a draft for the LLMCategorizer class, which is initialized with configuration parameters for content, category, prompt, and model strategies. Initial testing was conducted on an extra small dataset. Improved initial prompts to maximize the Precision.
• I implemented a draft code for the metric calculation to assess categorization performance (Precision (prioritized) and Recall).
• Conducted preliminary estimates for Groq usage. Current limitations include the absence of few-shot learning, which could significantly lower performance estimates, and missing page content due to data unavailability.
  • Key Concern: At this stage, Groq seems to be a bottleneck. Even with all simplifications (no content, no few-shot learning), we are limited to processing approximately 1,100-1,600 samples per day (~1-2% of a full configuration sample) (with speed of 20-30 requests per minute). Approximate cost of $4-5 per 1,000 samples.
  • The key problem is the large set of potential categories (even after reduction). I think we need a more sophisticated (better than just based on Topic + TF IDF filter) first-level model to select a limited set of candidates with high recall before passing to LLM.
  • It is essential to investigate the possibility of increasing Groq's usage limits to overcome the current constraints.
• All the mentioned code is added to separate branch at Gitlab

Status:

• I have created the initial version of the experiment book for Article Categorization, available here. This document includes:
  • Definitions of strategies needed for model input formation, including category selection, content selection, and prompt types (along with corresponding prompt templates).
  • Each experiment will be defined as a combination of strategies for category selection, content selection, prompt type, and models.
• I have tested the Groq platform to determine its limitations concerning the models to be tested and the prompting strategies.
• I have created and tested the prompt templates within the Groq playground to work for both Llama and Mistral models.

Hi @achou, thanks so much for your work! I’ve run the tests and can confirm the scale of your observations. The old model averages 1.07s per item, while the new model averages 2.52s per item on the same data, meaning the new model is indeed about twice as slow. (Absolute numbers may vary depending on CPU and connection speed.) My initial assumption that their performance was "similar" was incorrect. I hope this information is helpful.

Hi @achou! Thanks for working on this. Prediction time should be similar to the previous model. I have checked locally, and it is 2.5s per page on average (with 4s for 95 percentile). However, the model should require more RAM.

1. Model Performance on Historical Holdout Testset*

Updates
I was working on the experimental model using a multilingual language model.
It was evaluated and compared with the ORES model on the time-based hold-out dataset of the revisions from 2023.

Merged. Thank you!

Here is MR: https://gitlab.wikimedia.org/trokhymovych/readability-liftwing/-/merge_requests/4

@isarantopoulos Thank you! I have checked and merged your changes.

I have checked the proposed changes (MR: https://gitlab.wikimedia.org/repos/research/knowledge_integrity/-/merge_requests/17).

I have reviewed the logs with errors from the multilingual model, and it seems to be a problem with resources.

Cleaned up 'trokhymovych'

@jbond
New Public SSH key:
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFCyl+eu4X9cI/XT6nCSvud+X6LJyVV7Rcr1g4MnP2xf trokhymovych.mykola@gmail.com

Previously, we tested the TreeSHAP algorithm for Multilingual model explainability (from here: https://shap.readthedocs.io/en/latest/). It is supported by the tools provided in the task description. The main benefit is that it works with our classifiers and provides local explainability (so we can have an explanation for each specific sample without any other data needed).

• I have prepared the MR with the multilingual model inference implementation: https://gitlab.wikimedia.org/repos/research/knowledge_integrity/-/tree/mykola/multilingual_initial
• The model is temporary saved on google drive (as it is too large to save it in repo): https://drive.google.com/file/d/1ZffeT-tm99CpI9HgydyhV_7xUS02ZiIr/view?usp=share_link
• I have prepared the little (1000 samples) random test sample to perform inference time evaluation. I got the following results: median inference time is 3.72s, 90% percentile is 9.94s, 95% percentile is 12.56s.
  

1. I was working on model results interpretation and prepared a notebook with examples of per-sample SHAP values for final model results. Also, I was investigating the method to interpret each independent text model for better understanding and further improvement.
2. Prepared and held a presentation of intermediate research results.
3. Later, I worked on model validation using one week's data, including data collection, features collection, and building report. Also, I was investigating the package that implements the language agnostic model for further possible usage for my model inference feature engineering. Finalized report for complete hold-out data on the one-week dataset. I performed a sample-wise analysis of the differences between models.
4. Studied the possibility of building the model on top of the language-agnostic, ores, and multilingual models that generalize the knowledge. Evaluated it using hold-out.

This week I was working on a complex model that considers meta-features and text changes. What was done:

1. Finetuned models for text-based feature generation (comments, title semantics, changes, inserts). I evaluated them separately. I later extracted features for the complex model from the last layer before softmax + softmax layer outputs for each model (except title semantics, as it was trained as a regression model).
2. Trained model added all features from texts on the data part that was not used in the text model finetuning to omit leakage. As a result, I got a boost in accuracy score (~70%->74% on balanced test)
3. Started error analysis and results observation.

1. Collected datasets for more languages (pl, de, es) and recollected previous ones to proceed with the time-dependent experiment
2. Experimented with multilingual models training for inserts and changes as preparation for finetuning on multiple languages.
3. I highly rely on the mwedittypes package, which uses mwparserfromhell for wikitext parsing. It was reported that an open issue exists that mwparserfromhell causes a very long to infinite parsing. It can be a possible signal of vandalism that is a signal that we want to detect. I have checked that ~1.0% of revisions had problems parsing the wikitext (pretty much the same for all languages). The revert rate for both parsed and not parsed revisions was the same, so this signal is probably not as strong as expected. I decided to consider it in the final model with binary feature is_parsed.
4. Implemented bootstrap strategy for defining confidence intervals of statistics calculated for regions analysis (it is desired to understand the confidence of results I got while analyzing the revert rate for different regions of user/page)
5. Designed the architecture of an end-to-end model that considers both revision features and text features. Started implementing pipeline for experiments with such architecture.

1. Recollected dataset for anonymous users only and fixed minor bugs in text processing.
2. Checked profanity score package - list of bad words (https://pypi.org/project/profanity-check/) -> not working, very weak signal
3. Parsed page's semantic information for further processing (article categories), added wikidata_id.
4. Attempted to get Wikidata embedding, but not successfully. Pretrained models are either huge or include less than 20% of needed entities.
5. Check topics classification tool and country classification tools: https://wiki-topic.toolforge.org/. I found the country classification tool very insightful. Previously, I found out that the location of anonymous users influences the revert rate. However, combining page location and user location gives even more exciting insights that can be useful for revert event modeling.

• Collect dataset of changes for ruwiki, enwiki, and ukwiki along with text changes (inserts, changes, removes)
• Performed EDA for collected datasets
• Build toxicity features of inserts and changes based on detoxify package, checked the predictive power of those features -> they slightly improve the baseline performance, but not significantly.
• Checked text changes meta-features of inserts and changes extracted using https://pypi.org/project/mwedittypes/, checked the predictive power of those features -> they improve the baseline performance.
• Performed initial analysis of changes in references.
• Checked the hypothesis that user location impacts the model's possibility to detect revisions that would be reverted. As a result, the experiment showed that those have good predictive power, which is comparable with text changes meta-features. In addition, I created a more detailed report for the correlation between anonymous users' locations and revert rates, which can be found in the attachment.

• Performed EDA for wmf.mediawiki_history
  • Got familiar with data
  • Found out insights important for training dataset building (a significant number of self-revert, different rate of reverts for groups of users)
• Manually explored recent changes (text differences) for ukwiki, ruwiki, enwiki
  • Find out the differences in causes of reverts for different languages. It should be taken into account while modeling.
  • Got familiar with the logic of reverts, revert-wars
  • Came out with a logic that can reduce noise from the training dataset by filtering "bad" reverts caused by revision wars.
• Get familiar with the Analytics cluster

@cmooney I have already reviewed and signed the Server Access Responsibilities Document

Can you please delete all my old keys and leave only the one I have provided in this ticket? Thank you!

Public SSH key:
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIB8/dfbAQjsOu3EzPIosLsY0Dxz0LOMtW2dKPndAqDnh trokhymovych.mykola@gmail.com

Updates:

• Experimented with multilingual models
• Developed a methodology to train large multilingual models that does not fit into memory
• Explored existing multilingual NLI datasets

Updates:

• Finished documentation for API.
  • Included brief System architecture observation
  • Included Pointers to the code
  • Included Explanation: how to replicate the API
  • Included Description of the 3 end points, and examples

Updates

• Deployed initial version of WikiCheck API.
• Implemented NLI model endpoint
• Implemented fact checking endpoint
• Experimented with aggregation strategies