Initial analysis: https://public-paws.wmcloud.org/User:MNeisler_(WMF)/citation_data_exp_analysis.ipynb
Data collection script: https://public-paws.wmcloud.org/User:MNeisler_(WMF)/citation_data_collection.ipynb

Methodology Notes

• There is currently no structured database of references and citation data used across Wikis and citations consist of varying formats across languages so querying is complicated. As part of this task, I investigated what available options there were to collect this data, identified current complications, and then explored some trends in references by selected topics and wikis.

• Based on the script created in the citoid coverage research, I reviewed two options (1) Using the Citoid API to obtain citation metadata and (2) extracting citations from article text based on citation templates. I decided to go with option #2 as there's variability in Citoid performance generating automated citations and also because I ran into difficulties converting the json output from Citoid API (I will investigate this issue at a later date)

• I decided to pick a selection of articles across 4 wikis (English, French, Spanish, Portuguese) and 2 topic categories (Music and Continents) to collect a decent and diverse sample to compare different trends. Note: Topic categories are based on applied category tags to the article and their usage and accuracy varies greatly between wikis. As a future improvement to the script, this should be changed to use ORES topic model data.

Findings

• Despite having a slightly higher number of distinct articles, music topic articles have less than half the number of citations than continent topic articles at the wikis reviewed. Continent topic articles have 1749 citations and music topic articles have 538 citations.

• There are 9 distinct citation source types used in the articles reviewed. The majority of citations (65%) across all three languages reviewed were identified as web citations. Book and news citations comprised the second and third most used references across all four reviewed wikis.
• Web citations are also the predominant source used for both topic categories followed by books. News, encyclopedia, and journal references are cited more frequently for the continent topic compared to the music topic category.
• The majority of citations are used only once. About 10% of citations in the continent topic articles as used more than once compared to 6% of citations in music articles.
• English has the highest proportion (9.2%) of citations used multiple times, while Portuguese Wikipedia had the lowest (1.2%; only 1 citation used more than once).

Ideas for improvement:

• Further investigate the use and performance of Citoid API. The Citoid API has more consistent and structured data about citations across languages.
• Revise script to use topic modeling data vs assigned topic categories. Topic categories are not applied consistently across all languages, which impacted the results above.

Thanks for participating in the Hackathon! We hope you had a great time.

• If this task was being worked on and resolved at the Hackathon: Please change the task status to resolved via the Add Action... → Change Status dropdown, and make sure that this task has a link to the public codebase.
• If this task is still valid and should stay open: Please add another active project tag to this task, so others can find this task (as likely nobody in the future will look back at the Hackathon workboard when trying to find something they are interested in).
• In case there is nothing else to do for this task, or nobody plans to work on this task anymore: Please set the task status to declined.

Thank you,
Phabricator housekeeping service