Weekly Update:

• Created Q2 phab task
• No update this week

Weekly Update:

• Brought blocker (see T348249) to the attention of the Language Inclusion working group
• No other major updates this week.

Weekly Update:

• No major updates this week.

Weekly Update:

• No major updates this week.

Weekly update:

• Incorporated Wikiversity Beta data into wrangling files and output files
• Added Multilingual Wikisource data file to source data folder
• Incorporated Multilingual Wikisource data into wrangling files and output files
• Reran stats in state_of_languages.ipynb

Next week:

• QA/QC

Weekly update:

QA/QC ongoing, which has resulted in the following updates/corrections/revisions:

• With Wikiversity Beta and Multilingual Wikisource incorporated into language.ipynb, the output "current_incubator_projects_list.tsv" was changed to "projects.tsv"
• Incorporated graduation dates, as well as hosting status ("open", "closed") into the projects.tsv file
• Added distinction between test projects that are first-time test projects in the Incubator, Wikiveristy Beta, or Multilingual Wikisource (i.e. "test") vs. test projects that were previously hosted but closed (i.e. "test (closed)").
• Dealt with multiple issues of duplicate languages and or language name changes/inconsitencies
• The state_of_languages.ipynb notebook now reflects these updates; as do all outputs, dependent notebooks, etc.

Highlights:

• state_of_languages.csv: allows the user to view which projects (hosted, test, closed, or none) it has for each language
• state_of_languages_sortable.ipynb: allows user to sort languages by number of projects (all), number of test projects, number of hosted projects
• state_of_languages.ipynb: (scroll down to "How many languages....") provides user with stats about "How many languages have X projects?", "How many languages have a Wikipedia?", etc.

Note: QA/QC still underway for everything linked to above! Please do not use these data yet.

Weekly update: QA continued

Weekly Update:

No major updates this week.

Weekly Update:

No major updates this week.

Weekly updates:

Continued discussion of blocker (T348249) with Language Inclusion working group, and plans for temporary place-holding data

Shared my deliverables (so far) with Language Inclusion working group, mentioned above in T348241#9342085:

• state_of_languages.csv: allows the user to view which projects (hosted, test, closed, or none) it has for each language
• state_of_languages_sortable.ipynb: allows user to sort languages by number of projects (all), number of test projects, number of hosted projects
• state_of_languages.ipynb: (scroll down to "How many languages....") provides user with stats about "How many languages have X projects?", "How many languages have a Wikipedia?", etc.

Note: QA/QC still underway for everything linked to above! Please do not use these data yet.

Weekly updates:

• Continued QA
• Began analysis of top language stats

Weekly update:

Met with Language Inclusion working group; discussed current status of work in relation to the current dependencies.

No additional updates.

Weekly update:

Continued baseline stats for top 20 languages: state_of_languages.ipynb

• See "Top Languages breakdowns" section
• Contains overview of which of the top 20 languages have Wp, Wt, Wb, Ws, Wn, Wq, Wv, and Wy

Next week I will add

• For the top languages, stats about the size of the projects (e.g. number of Wp articles for top 20 languages)
• For the top languages, plot number of speakers vs number of articles

Weekly update:

New state_of_languages-top20.ipynb file, dedicated to analysis and plotting stats related to top 20 most spoken languages

• Added stats related to speaker numbers
• Added stats related to number of Wp articles
• Plotted number of speakers vs number of Wp articles

Weekly update:

• Baseline insights and visualizations added to Insights section of research project page on meta-wiki
• Created color-coded visualization of current state of languages
  • visualization includes all 329 languages that have at least one test, hosted, or closed project
  • screenshot preview below, and full PNG available on GitLab here

{F41754807}

No major updates this week.

Weekly update:

• Worked on updating Incubator analysis notebooks to better work with Incubator data I've compiled and wrangled within the repo
• Next week will finish the updates and rerun the analyses

Weekly update:

1. Continued analysis of Incubator data, looking at length of time in Incubator
   • Moved all Incubator-related analyses to Incubator subfolder
   • (In progress) visualizing current Incubator projects' length of time in Incubator: current_incubator_history_visualizations.ipynb
   • (In progress) visualizing Incubator graduates' length of time in Incubator: incubator_graduate_history_visualizations.ipynb

2. Continued analysis of top 20 languages, incorporating more product data
   • Wikipedia, Wiktionary, and Wikisource "size" (i.e., number of mainspace pages):
     • Page counts query: wiki_page_counts_query.ipynb (2024-03-08 update: removed this separate query file, and incorporated query within the state_of_languages-top20.ipynb notebook itself)
     • Visualizations: state_of_languages-top20.ipynb (see "Top 20 Languages: Wikipedia Size", "Top 20 Languages: Wiktionary Size", and "Top 20 Languages: Wikisource Size")
   • Unique devices:
     • Unique devices query: wiki_unique_devices_query.ipynb (2024-03-08 update: removed this separate query file, and incorporated query within the state_of_languages-top20.ipynb notebook itself)
     • Visualizations: state_of_languages-top20.ipynb (see "Top 20 Languages: Wikipedia desktop readers" and "Top 20 Languages: Wikipedia mobile readers")

Weekly update:

1. Now able to query data lake within notebooks running R kernels! 😄
   • Incorporated spark queries within state_of_languages-top20.ipynb (jupyter notebook running an R kernel) using jsonlite and reticulate R packages, per https://wikitech.wikimedia.org/wiki/Data_Engineering/Systems/Conda#wmfdata-python_through_reticulate
   • No longer need (and thus deleted) "queries" folder and contents, which were used to accomplish the following: spark queries run in notebooks running Python kernel --> write queries output to CSVs --> pull CSVs into notebooks running R kernels

2. Created new exploratory visualizations for looking at time-spent-in-Incubator:
   • New visualizations (thanks to brainstorming with @KCVelaga_WMF!) in both_current_and_grad_incubator_history_visualizations.ipynb notebook showing distributions of years-taken-to-graduate, based on year project started.
     • See ## PLOT 4 and ## PLOT 5 in notebook
     • Darker greens = shorter amount of time spent in Incubator
     • Viz excludes projects started after 2018 as they would not have all had the chance to graduate in 5 years
     • Please do not use this visualizations outside the current project yet, as they are exploratory and still need another round of QA

3. Created new exploratory visualizations for looking at graduation rates for projects with an immediately preceding graduated project of the same language vs. for projects without an immediately preceding graduated project in the same language:
   • New visualizations in in both_current_and_grad_incubator_history_visualizations.ipynb notebook
     • See ## PLOT 8A and ## PLOT 8B in notebook
     • Please do not use this visualizations outside the current project yet, as they are exploratory and still need another round of QA

{F42476395} {F42476401}

Weekly update:

• Began writing and testing queries to incorporate product data (edits, editors, unique devices, page views, etc) into state-of-language analyses and Incubator analyses
• Shared status updates with multiple working groups

Weekly update:

• Improved spark queries for product data needed for language metrics and incubator analyses
• Continued Incubator data exploration; the current_incubator_history_visualizations.ipynb file now has four sections:
  • Part 1: Visualizing project size (i.e. how many pages do the current Incubator projects have?)
  • Part 2: Visualizing first and last edits (i.e., how long have current Incubator projects been in the Incubator?)
  • Part 3: Visualizing project size x start date (i.e., looking at projects' number of pages x how long they've been in the Incubator)
  • Part 4: Visualizing monthly edits (i.e., for each project currently in the Incubator, how many edits were made each month?)
• Lots of new visualizations including the following (Please do not use these visualizations outside the current project yet, as they are exploratory and still need another round of QA)

(Days in Incubator: visualizations show a wide range of days spent in Incubator, similar for each project)

{F43074160}
{F43074383}

(Monthly edits in 2023: visualizations indicate that the majority of projects in Incubator were not actively edited last year)

{F43074548}

Weekly update:

• Added Metrics section to meta-wiki project page, where quantitative metrics related to the state of languages across Wikimedia projects are drafted, specifically:
  • Key term definitions
  • Table: "Metrics: state of languages"
  • Table: "Metrics: state of top languages"
  • Table: "Metrics: state of threatened languages"
  • Table: "Metrics: language-specific"
• Tables include the location of notebook, dataset, and/or visualization where each metric is currently calculated or displayed

Weekly update:

• Continued QA (focused on wrangling code in the load_dataframe_for_analysis.ipynb notebook; moved code from current_incubator_history.ipynb and incubator_graduate_history.ipynb (and then deleted those notebooks after migrating the code)
• Created a "/tsv" folder; outputs from load_dataframe_for_analysis.ipynb notebook will need to be saved as TSVs to be loaded into analysis/viz notebooks running R kernels
• After finishing reorganization of the "/incubator" folder, updated the README and added a README in the "/tsv" folder

Weekly update:

• Continued QA
• Provided overview of new Metrics tables in the meta-wiki project page at the Connecting the Dots on Language Inclusion working group meeting

Weekly updates:

• Continued review/testing of the wrangling scripts
• Discussed peer review options with colleagues
• Working on summary of data transformations that occur in the language.ipynb notebook for peer review

Weekly updates:

• Finished prepping review of data sources and transformations
• Finished prepping first set of QA tasks

Weekly updates:

• No major updates related to the integration of secondary data sources
• Met with Movement Insights to brainstorm ideas about productionizing these data
• For additional updates, see T361640

Weekly updates:

• No major updates related to the integration of secondary data sources
• Regenerated state of languages visualizations with April 2024 snapshot data (see, e.g., state_of_languages.ipynb, state_of_languages-top20_project_size.ipynb)
• Began uploading visualizations to Commons, to replace those on meta-wiki project page (Insights section) next week
• For additional updates, see T361640

Weekly updates:

• Met with KC to discuss review of transformations and data sources
• Presented overview of state of languages at Language Community Meeting (T364999)

Weekly updates:

• @KCVelaga_WMF provided notebook feedback T366516
• Will implement changes next week
• Met with Search team to discuss possible search-related metric(s) to add to the state of languages metrics
• Solicited project feedback via a wikimediaresearch-l thread

Weekly update:

• began implementing notebook feedback T366516
• made small updates to project meta page