Status:

• I have created the initial version of the experiment book for Article Categorization, available here. This document includes:
  • Definitions of strategies needed for model input formation, including category selection, content selection, and prompt types (along with corresponding prompt templates).
  • Each experiment will be defined as a combination of strategies for category selection, content selection, prompt type, and models.
• I have tested the Groq platform to determine its limitations concerning the models to be tested and the prompting strategies.
• I have created and tested the prompt templates within the Groq playground to work for both Llama and Mistral models.

Next Steps:

• Reduce the number of strategies to fit within the project timelines.
• Test the strategies on the pre-collected dataset sample to estimate the cost and time required to perform the experiments.
• Begin the implementation for the experiments setup.

• Analysed the collected data and communicated recommendations for improvement to @Aitolkyn.
• Developed and implemented a process to create a dictionary mapping topics to categories using TF-IDF. This involved filtering out rare categories and selecting the top N categories based on non-zero TF-IDF scores.
• Designed and implemented a draft for the LLMCategorizer class, which is initialized with configuration parameters for content, category, prompt, and model strategies. Initial testing was conducted on an extra small dataset. Improved initial prompts to maximize the Precision.
• I implemented a draft code for the metric calculation to assess categorization performance (Precision (prioritized) and Recall).
• Conducted preliminary estimates for Groq usage. Current limitations include the absence of few-shot learning, which could significantly lower performance estimates, and missing page content due to data unavailability.
  • Key Concern: At this stage, Groq seems to be a bottleneck. Even with all simplifications (no content, no few-shot learning), we are limited to processing approximately 1,100-1,600 samples per day (~1-2% of a full configuration sample) (with speed of 20-30 requests per minute). Approximate cost of $4-5 per 1,000 samples.
  • The key problem is the large set of potential categories (even after reduction). I think we need a more sophisticated (better than just based on Topic + TF IDF filter) first-level model to select a limited set of candidates with high recall before passing to LLM.
  • It is essential to investigate the possibility of increasing Groq's usage limits to overcome the current constraints.
• All the mentioned code is added to separate branch at Gitlab

Experimentation with Category Reduction Strategies for Prompt Generation

Notebook with experiment: Gitlab link

Importance

Reducing the number of candidate categories provided in prompts is crucial because certain topics can have an extensive number of possible categories, averaging 966 categories per topic. This leads to high token consumption, increased costs, and longer generation times, particularly for non-English languages.

Data used

• Testing Set 1: Subset of 50 articles per language.
• Testing Set 2: Subset of 50 articles per language, created after 01.01.2024.

Strategies

• Baseline (Random): Random selection of a subsample of categories related to the article topic.
• TF-IDF Reduced: Selection of the most important N categories from those related to the article topic.
• ANN Reduced: Selection of the top N (fixed at 50) most semantically similar categories to the article title or title + main section text from those related to the article topic.
  • Encoders Used:
    • paraphrase-multilingual-MiniLM-L12-v2
    • LaBSE
    • Alibaba-NLP/gte-multilingual-base

Summary

I don't see a significant difference for all and only new articles (created after 01.01.2024).

I recommend proceeding with the ANN + Alibaba-NLP/gte-multilingual-base N=50 candidates selection strategy (using main section text and title). The benefits of using this approach include:

1. Achieving high recall metrics with a much smaller candidate set, comparable to TF-IDF with N=300.
2. Relatively fast performance with the ANN index.
3. The encoder model can handle up to an 8K token context window, allowing for the processing of relatively long texts.

In case of resource limitations, the more lightweight model "paraphrase-multilingual-MiniLM-L12-v2" can be used as an alternative.

Comparison tables:

Testing set 1 (Random subset (50 per country)):

Testing set 2 (Random subset (50 per country) + created after 01.01.2024):

Evaluation of Multiple Configurations Using Together AI

Estimated Spend

• ~$30

Dataset

• Sample of 50 random articles from each language (1150 samples in total)

Models Tested

• meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo
• meta-llama/Meta-Llama-3-70B-Instruct-Turbo
• meta-llama/Meta-Llama-3-70B-Instruct-Lite

Model Selection Logic

The goal was to run the full pipeline and compare "Turbo" vs. "Lite," and "Llama-3.1" vs. "Llama-3.” Model selection was also constrained by the available models on Together AI.

Content Configurations

• Only article name
• Article name and first paragraph

Category Selection

• ANN Top-50 categories

Prompt Strategies

• Zero-shot
• Zero-shot + structured output + chain-of-thought
• Few-shot (random fixed examples) + structured output + chain-of-thought
• Few-shot (similar examples) + structured output + chain-of-thought

Configurations Tested

• 24 configurations in total

Summary

• Llama-3.1 works better than Llama-3 as expected.
• No significant difference between "Lite" and "Turbo" versions. I would even say that “Llama 3 Lite” performs comparably to “Llama 3.1 Turbo.”
• Llama-3 tends to hallucinate more and had issues with structured output returning (exp-011, exp-012).
• Performance varies greatly by language, with more popular languages like English performing much better, as expected.
• Configuration of few-shot with similar articles provided the best boost in performance across all the models.

Full Results:

Precision per language and experiment:

Notebook with Full Experiment

• Gitlab link

Metrics Interpretation:

In our experiment, I use list-based metrics to evaluate performance: Precision and Recall.

• Precision (main metric): This measures the accuracy of the predictions. It is the rate at which the predicted categories appear in the ground truth.

• Recall: This metric assesses the completeness of the predictions. It measures the rate at which the ground truth categories are correctly predicted.

Note 1: Precision and Recall does not take into account the order of the list
Note 2: If no categories are predicted, it results in 0 for Precision, which is one of the limitations of the metric (as predicting no category is better than predicting an incorrect one)
Note 3: We calculate Precision and Recall for each sample and calculate the average as the final metric for the specific configuration.

Example:

True categories: [Cat1, Cat2, Cat3, Cat4]
Predicted categories: [Cat1, Cat2, Cat5]

Precision: 2/3
Recall: 2/4

Experiments with Llama7B and Mixtral

Experiment setup:

• I slightly changed the experiment setup to work with Mixtral models: I changed the type of message from “system” to “user.”
• Reduced experiment sets, removing obviously worse strategies based on previous observations (exp-001-024). I have removed strategies that are not using the content of the page. I have left only two prompting strategies: “Zero-shot + structured output + chain-of-thought” and “Few-shot (similar examples) + structured output + chain-of-thought”

Result table:

Summary:

1. Both small Llama8B and any Mixtral models work considerably worse than Llama 70B
2. Llama8B and Mixtral models have issues following structured output, resulting in high is_none_rate and low precision and recall.
3. No big difference between Mixtral8x7B and Mistral-7B

Experiments with forcing structured output

Considering the issue with the structured output, it was also decided to test the strategy of forcing structured output using the functionality provided by TogetherAI. It is available only for a limited amount of models, but it still allows for evaluation of its effectiveness.

Result table:

Summary:

1. Forcing structure improved a lot the results for Llama-3.1-8B, but it is still much worse than Llama-3.1-70B
2. Forcing structure for Mixtral-8x7B has changed the behavior but not the metrics. The model started to work very poorly for specific non-latin languages (e.g., jawiki, kowiki, zhwiki).
3. As for Llama-3.1-70B, the result only slightly improved (almost no difference).

Peacock detection (initial experiments)

Task: Binary Classification
Metrics: Accuracy (balanced dataset), Precision, Recall, F1
Data Sample: Initially took a sample of 50 random page_ids from each language (1222 revisions). Limit text to 20K chars (cover >90% of articles)

Models Tested:

• Meta-Llama-3.1-70B-Instruct-Turbo
• Meta-Llama-3.1-70B-Instruct-Lite
• Meta-Llama-3.1-8B-Instruct-Turbo
• Mistral-7B-Instruct-v0.3
• Mixtral-8x7B-Instruct-v0.1

Prompts Tested:

• Simple
• Extended (added explanation for peacock patterns)

Summary:

• Overall performance of all models is quite low.
• Adding additional information about peacock template logic does not improve performance.
• Performance is similar across all languages.
• Meta-Llama-3.1-70B-Instruct-Turbo does not differ significantly from Meta-Llama-3.1-70B-Instruct-Lite. All other models perform much worse.

Code: Gitlab link

Results:

Accuracy per language:

NPOV violation detection (initial experiments)

Task: Binary Classification
Metrics: Accuracy (balanced dataset), Precision, Recall, F1
Data Sample: Initially took a sample of 30 random page_ids from each language (1340 revisions). Limit text to 30K chars (cover >90% of articles)

Models Tested:

• Meta-Llama-3.1-70B-Instruct-Turbo
• Meta-Llama-3.1-70B-Instruct-Lite
• Meta-Llama-3.1-8B-Instruct-Turbo
• Mistral-7B-Instruct-v0.3
• Mixtral-8x7B-Instruct-v0.1

Prompts Tested:

• Simple (only one prompt tested)

Summary:

• Overall performance of all models is quite low (close to random)
• Performance is similar across all languages.
• No difference for models.

Code: Gitlab link

Results:

Accuracy per language:

Experiments with Aya Model

Task: Binary Classification (NPOV, Peacock detection)
Metrics: Accuracy (balanced dataset), Precision, Recall, F1
Data Sample: Same as for exp-100 (-105), exp-200 (-204).

Summary:

• The Aya model struggled significantly in providing structured output, necessitating prompt adaptation. Despite these adjustments, the rate of invalid results remains high.
• The local inference endpoint proved to be reliable and relatively fast.
• The performance of the Aya model is notably lower compared to the LLaMA and Mixtral models under the current setup.
• Model performance is sensitive to prompt minor changes.

Code: Gitlab link

Results Table:

(additionally updated previous tables for consistency)

Full data experiments for NPOV and Peacock templates:

Summary:

• Updated (improved) prompts based on what was learned from the Aya case. Tested on sample to validate that it performs slightly better.
• Performed evaluation on the full testing data for two templates and four models. Meta-Llama-3.1-70B-Instruct-Turbo was skipped for now due to issues with the LLM provider.
• The code to complete experiments (including the specific prompt) was loaded to repo: for npov, for peacock
• Metrics calculation, all plots, and full analysis is presented here: for npov, for peacock
• To reproduce the prediction for a specific model (e.g., meta-llama/Meta-Llama-3-70B-Instruct-Lite in this case), run:

export TOGETHER_AI_KEY=“<key>”

python 09_peacock_script.py --model "meta-llama/Meta-Llama-3-70B-Instruct-Lite" --experiment "llama370B" --input "/srv/home/aitolkyn/ai-use-cases/data/peacock_test.csv.gz"

python 10_npov_script.py --model "meta-llama/Meta-Llama-3-70B-Instruct-Lite" --experiment "llama370B" --input "/srv/home/aitolkyn/ai-use-cases/data/npov_test.csv.gz"

Results for NPOV:

Results for Peacock:

*error rate refers to cases when we could not get the prediction that might happen, for example, because of a response of unexpected format (parsing error) or API error. In that case, the prediction was filled with a “0” label.

Experiments Finalization for NPOV and Peacock Templates

• Conducted full-scale experiments using Meta-Llama-3.1-70B-Instruct-Turbo and integrated the results into the general performance comparison table (see previous comment for full comparison).
• Performed temporal performance analysis to compare the model's effectiveness on data before and after January 1, 2024:
  • for npov;
  • for peacock
• Implemented confidence interval calculations for evaluation metrics and updated the respective notebooks:
  • for npov;
  • for peacock

1. I have updated the final report document by incorporating detailed information about the experiments with LLMs. Specifically, I have provided technical details, code links, results, and summaries for the Peacock tone, NPOV violation, and article categorization tasks.
2. I have been exploring a strategy that could enable the extraction of probability scores using the logprobs of prompt tokens feature of large language models (LLMs), as opposed to the current binary prediction approach.
3. I started working on running full experiments for the Citation needed template using an approach similar to Peacock tone and NPOV violation detection.

Citation Needed experiments

• Build the pipeline that allows the extraction of the probabilities for each class. It needs additional checking for each model due to tokenization differences.
• The results obtained are slightly better than random with an AUC score of ~0.6.
• The results don’t differ too much between the models, with the best model Meta-Llama-3-70B-Instruct-Lite

Final results:

Code: link

Moving to backlog until Diego is back and can confirm that this work has been wrapped up and should be resolved.