Week 29/5/23 - 4/6/23 Update:

• Fixed template parsing to accommodate the use of lang param in template
• Parsed and saved all language wiktionary misspelling.
• Did some analysis, de-duplication, and saved all_wiki combined wiktionary misspellings.
• More errors found in template parsing (named params occur before un-named params causing incorrect parsing)

Week 22/5/23 - 28/5/23 Update:

• Updated MR9 with summary
• Created Issue 14. Changed wiktionary parser script to make it work with all languages. Need to figure out some changes in template params.

Week 15/5/23 - 21/5/23 Update:

• Checked example use of misspelling of templates in all the collected 16 languages. All languages look similar to enwiktionary except trwiktionary (small change) and viwiktionary

Week 8/5/23 - 14/5/23 Update:

• Created Issues 12 and 13. Started working on them: identify misspelling of templates in other languages and find usage of these templates. The templates would be collected from Q50368067, misspelling of named templates in other languages, and their redirects.

Week 1/5/23 - 7/5/23 Update:

• Incorporated feedback and had MR7 merged (refactor repo)
• Analysis done on extracted misspellings, sent MR8. Based on feedback, some more analysis done.

Week 24/4/23 - 30/4/23 Update:

• Incorporate Isaacs feedback for MR5 and 6. All MRs merged after some editing and discussion.
• Created MR7 to Refactored repo
• extracted misspelling from all language wikipedias.
  • Todo: analysis on extracted data

Week 10/4/23 - 16/4/23 Update:

• Add info on language detection (language, confidence, text sent to model). Analyze examples.
• Add proxy tables: tables that were not detected by mwparserfromhell.
• Separate cell data of tables: each cell in table is now a node. Stuck with cell data/paragraph text to send to model.

Week 3/4/23 - 9/4/23 Update:

• Pushed revised code that includes all additional formatting as a list (as discussed).
• Fixed quotations detected. Added fasttext language detection.
• Analysed collected misspellings from context. Some work need to be done to increase precision of detected language.

Week 27/3/23 - 2/4/23 Update:

• Apply additional filter information to extracted misspellings: Capitalization, word length, part of a list item, inside of quotations (in any language)
• Still need to figure out the data's structure and add fasttext detected language information

Week 20/3/23 - 26/3/23 Update:

Week 13/3/23 - 19/3/23 Update:

Week 6/3/23 - 12/3/23 Update:

• Compared collected en and fr misspellings with AutowikiBrowser Typo list. Merge requested. Summary here
• Started working on extracting wikipedia text to find the ratio of misspellings

Week 27/2/23 - 5/3/23 Update:

• Address comments for Issue #5
  • Parse sections line by line, consider templates in # items (numbered list)
  • Count the number of definitions by # count, excluding ## #: #; and #*
  • Also change the data format a bit to make it more readable
• To address Issue 6: get list of misspellings from another Language and compare the collected lists to existing approaches
  • collected bnwiktionary templates. It does not have much Bangla words. Its the same as present in enwiktionary. Will work with existing collected Spanish misspellings instead.
  • for English, compared collected list with enwiki Lists_of_common_misspellings

Week 20/2/23 - 26/2/23 Update:

Week 13/2/23 - 19/2/23 Update:

Week 6/2/23 - 12/2/23 Update:

• Set up jupyter notebook (fix issues with getting spark3)
• Get list of enwiktionary pages that use missplelling_of template using the following tables:
  • mediawiki_templatelinks, mediawiki_linktarget, mediawiki_wikitext_current
• Parsed enwiktionary pages to get heading name (typically POS: Noun, Adj, etc), language of misspelling, and the correct spelling from the template
• Some analysis on parsed wikis to get languauge and heading distribution

Thank you, accessed!

Week 1/2/23 - 5/2/23 Update:

• Caught up on previous work on copy editing both in research team and growth team
• Learned about templates in Wiktionary in different langauges and the possible categories they may be in

In T303831#8063021, @EBernhardson wrote:

In terms of the exact code causing this, spark is terrible at telling us exactly where but trying to infer from the SparkUI output i think it's this join:

def getTopSubgraphItems(topSubgraphs: DataFrame): DataFrame = {
  wikidataTriples
    .filter(s"predicate='<$p31>'")
    .selectExpr("object as subgraph", "subject as item")
    .join(topSubgraphs.select("subgraph"), Seq("subgraph"), "right")

Update:
I tested a few options in the statbox, I am not sure how much this will represent the prod env, but here goes:

In T303831#8058159, @EBernhardson wrote:

the airflow patch is deployed but i only turned on *_init dags and subgraph_mapping_weekly today (ran out of time, will do rest tomorrow).

subgraph_mapping_weekly failed the first time through. I updated executor memory from 8g to 12g but the second execution is still failing. something is quite unbalanced about the topSubgraphItems, of the 8 shards they have inputs varying from 100MB to 450MB giving executions times of ~30s on the small ones and ~8m before the final one fails.

Not specifically related to this patch, but i wonder if we could change up the SparkUtils.saveTables method to somehow take parameters in the path to specify coalesce vs repartition and the number of partitions to save by, so we only have to update the airflow invocation and not the jar as well to test variations there.

The analysis is done here (for Q-ids): Wikidata_Item_ORES_Score_Analysis

In T288262#7629267, @Lydia_Pintscher wrote:

@AKhatun_WMF: You mention on the wiki that some Items don't have an ORES score. All Items should have one 😬 Do you have an example of one that does not?

In T288262#7628599, @MPhamWMF wrote:

@AKhatun_WMF , sorry, it's been a while since I wrote this, but I think what I meant when I wrote the question about "optimal separation" is given some distribution of ORES scores (e.g. a normal distribution), is it clear what the threshold is for what qualifies as a "high" vs "low" score: e.g. anything over .75 is a high score. But that's assuming the scores are continuous. I guess it's moot if they're binary (I don't actually know).

If this isn't a sensible way of thinking about the issue, let me know if there's a better way.

@MPhamWMF Hi, could you please clarify the question Is there an optimal separation between high/low ORES scores?. Separation in what respect? To my mind comes the separation of items with respect to the subgraph it is part of.

@ACraze Indeed! I was confusing the models for revision (item quality) with edits (damaging/good faith). The latest revision is all I will need. Thank you!

Counts of queries and triples for astronomical objects were done here: Wikidata_Subgraph_Query_Analysis, along with the top ~300 large subgraphs.
For the specific case of Astronomical objects (and only astronomical objects), a list of all its subclasses was obtained and manually inspected for relevance to astronomical objects. The subclass list also consists of subclasses of subclasses and so on.

Details can be found here: Wikidata_Subgraph_Query_Analysis

With the completion of T293632 and T293636, this task is complete.

With the completion of all subtasks, this task is complete.

The analysis was completed and documented here: Wikidata_Subgraph_Query_Analysis

Some analysis was done here:

• Property usage across subgraphs: Predicates_across_subgraphs
• Top predicates also used in scholarly articles: Top_properties_used_in_other_subgraphs

The analysis was completed and documented here: https://wikitech.wikimedia.org/wiki/User:AKhatun/Wikidata_Subgraph_Analysis

• T275068
• T293632 Wikidata_Subgraph_Analysis
• T281854 Wikidata_Scholarly_Articles_Subgraph_Analysis
• T293636 Query count analysis for the subgraphs

Basically Wikidata's Properties have a datatype.

Ah, datatype of properties.

I am not seeing that in the analysis you linked but maybe I am overlooking something.

The one I listed is for datatype of objects, so you didn't miss anything.
Thank you for clarifying! It should be fairly easy to find out as well :)

@Lydia_Pintscher
Is this ticket asking for counts of various datatype used in WIkidata? Both URI and literals.
Does wikitech:User:AKhatun/Wikidata_Basic_Analysis#Object help?

Interested in playing with autoencoders.

write a script that will randomly combine these audio files and sample the latent spaces of their combined embeddings to create new machine-generated audio files

Does this entail we train the autoencoder with the dataset we curated from commons and then have it generate a sample audio file from random numbers? Maybe I'm a bit confused about what 'randomly combining' audio files means here.

Astronomical objects are structured hierarchically and so not everything is direct instance of Q6999 (unlike scholarly articles).

Query analysis report for some vertical slices of Wikidata: Wikidata_Vertical_Analysis#Query_Analysis
Summary: Wikidata_Vertical_Analysis#TL;DR

Here is the analysis done on scholarly articles in Wikidata and WDQS queries related to them: https://wikitech.wikimedia.org/wiki/User:AKhatun/Wikidata_Scholarly_Articles_Subgraph_Analysis