Hey @elukey . Thanks for sharing, @Ottomata has talked about the general idea, but I was not aware of that document.

Weekly update:

Week update:

Weekly update:

• No updates this week

weekly update:

In T243430#5910987, @santhosh wrote:

As an experiment, I wrote a program that downloads the entire content translation parallel corpus from https://dumps.wikimedia.org/other/contenttranslation/20200214/ and find all section title translation done so far. It took about 9 hours to parse huge jsons files and at the end we have a 31.7 GB Sqlite database with this information.
Database: https://people.wikimedia.org/~santhosh/cx-section-titles-aligned.db

@santhosh that file is 30Mb, where can I get the full 31.7GB file?

Weekly update:

Weekly update:

Weekly update:

Weekly update

Weekly update:

Weekly update:

Hi @santhosh! Yep, this is super useful, I considering do something similar, to find some ground-truth for my approaches.

Weekly update:

weekly update:

Weekly update:

I'll need around 3 weeks (aprox) to finish this.

weekly update:

weekly update:

• Reviewing Corona Virus related cases.

@elukey I've deleted 120Gb. Moved back to 580G :)

Hey @elukey, for this task I need to download - at least - 50 languages models, each of them is around 8G, so I'll use around 400G. I'll do my best to make this work with that data on HDFS, but for starting I need to have it in a local machine. I'm now using stat1007 for my experiments. Is ok if I store temporarily the models there?

Hi @Theory42
I wont say "employed team only", I'll share all the code I'm creating for this, but currently I can't think in tasks where I need help for this. Please feel free to contribute to the repo previously mentioned, and tell me if I you need my help.

For the records, given that in geoeditors_edits_monthly we store information about countries using ISO 3166-1 alpha-2, we are losing Bonaire and Kosovo. The former has no code at all, and the latter has only Alpha-3.
We might want to consider use the full country name to avoid this kind problems in the future.

Update from last two weeks:

weekly update:

Update from last two weeks:

Btw, there is a way to mount the Wikipedia dumps on those machines?

In the short-term, the solution is to use the code as was designed to work with Spark.

Problem solved (thanks @elukey and @JAllemandou.

Hey @Ottomata @JAllemandou, please can you check why Pyspark kernels are not working? I've been trying for a week, with the differents pyspark kernels on the notebook machines, but the notebook freezes with any command (even is you try no-spark commands), pure python is working ok. Thx

Weekly update: Gather internship proposals within the team, and shared the requirements with @leila

Weekly Update: Preparing the dataset.

Hi @Pginer-WMF,
Have you already have a look in our Section Recommendation demo app? That's currently working for 6 languages. Expand it and specially maintaining for many languages could be complex, however, a simplified version of that system, using a dump approach instead of an API (like we have done with the template parameter alignment) it could be feasible.

@Aklapper done!

Here a summary of the work done: https://meta.wikimedia.org/wiki/Research:Disinformation_Literature_Review

@Nuria the entropy approach looks very cool, thanks for sharing.

We (research) will be supporting @ssingh on his work related to this problem, especially focused in censorship.

Slides

@santhosh , I've never tried that (I understand that docker files are kind of virtual environment, but honestly, I've never used it). We can try, but remember that the person will need access to our spark cluster. Do you know if the docker environment can connect with Yarn?

@diego please review below as well since you worked with GII folks during the past iteration:

Looks ok to me.

You might be interested on this project. T221211

About this:

This has been solved here: https://meta.wikimedia.org/wiki/Research:Expanding_Wikipedia_articles_across_languages/Inter_language_approach#Results
Check an example here: https://secrec.wmflabs.org/API/alignment/en/ja/Work

@Pginer-WMF , I'm going to put this task as resolved from me side, and we can continue the follow-up somewhere else, ok?

Considering the feedback obtained in T225136, we conclude that the prioritization should be adapted to the characteristic of the editor being assisted. We can split editors in two disjoint groups, generating two different types of recommendations:

Hi all,
I think this use-case highlight the need for a canonical (standanrized) cross-lingual topic model, that we could all use as the reference for all the projects within the WMF.

oh! I see, that number is distance, so 0 would be perfect match, 1 is not matching at all. I've already put a upper bound .45, so you will just see values lower than that.

If I understood correctly, you are asking why two exacts strings are not having distance = 0; this is because there is not string matching mechanism in this approach. Every language is trained separately, and then aligned using some words or sentences that we know that are equivalent. This is not necessarily bad, because you will find some words that are written exactly the same, but means different things in each language. However, in the examples that you show, this is just part of the noise introduced by the model.
We could add a second step, for example using Levenshtein distance, that would take advantage of string similarity , but it would work only for languages within the same scripts. If we had some training data, we could learn how to mix these two approaches and how useful would be the latter.

Hi,
I have created and uploaded the full experiments and aligned parameters for these languages:

["es", "en", "fr", "ar", "ru", "uk", "pt", "vi", "zh", "ru", "he", "it", "ta", "id", "fa", "ca"]

Sorry, I've put the wrong link to the experiments in the previous comment, now is updated.

You can find the results of the experiments here.