Page MenuHomePhabricator

Update "number of Wikimedia wikis used on" for popular gadgets on meta wiki (September 2022)
Closed, ResolvedPublic

Description

Update the "number of Wikimedia wikis used on" for popular gadgets on meta-wiki: https://meta.wikimedia.org/wiki/Gadgets#Gadgets_popular_on_wikis. As this page will always have a limited number of gadgets chosen based on criteria, manually updating the statistics (every year) seems like a reasonable option.

One can obtain these numbers by running the following python script via PAWS:
https://public.paws.wmcloud.org/User:SSethi_(WMF)/List%20of%20most%20used%20gadgets%20across%20wikimedia%20wikis.ipynb. For more on how to use PAWS, see https://wikitech.wikimedia.org/wiki/PAWS.

This task is due on 2022-09-30. Whoever resolves this task can file another one with a due date for 2023-09-30 unless you could find a better plan :)

Details

Due Date
Sep 30 2022, 7:00 AM

Event Timeline

Aklapper renamed this task from Update "number of Wikimedia wikis used on" for popular gadgets on meta wiki to Update "number of Wikimedia wikis used on" for popular gadgets on meta wiki (September 2022).Nov 4 2021, 1:04 PM
Aklapper changed the task status from Open to Stalled.
Aklapper triaged this task as Lowest priority.

I honestly think that turning this process into a bot running on Toolforge would be a good idea. It could update the counts monthly and we would mostly not need to think about it.

Aklapper changed the task status from Stalled to Open.Sep 22 2022, 4:12 PM

@mseckington: This task is overdue as it remained unassigned for the last quarter.

The 29 month old table on https://meta.wikimedia.org/w/index.php?title=Gadgets&oldid=22447270#Gadgets_popular_on_wikis lists 16 gadgets.

If I ran the Python script correctly in PAWS after adjusting four date stamps (no idea; the UI isn't that intuitive), it seems to print 8 results. Is that correct?

== edittop ==

edittop [ ResourceLoader ] |edittop.js|edittop.css
72
== hotcat ==

Hotcat[ResourceLoader|rights=edit,purge]|HotCat.js <!-- purge is a work-around to default-enable only for logged-in users. -->
135
== navigationpopups ==

Navigation_popups[ResourceLoader|type=general|dependencies=mediawiki.user,mediawiki.util]|popups.js|navpop.css
61
== purgetab ==

purgeTab[ResourceLoader|dependencies=mediawiki.api,mediawiki.language,mediawiki.util|rights=purge]|purgeTab.js
71
== referencetooltips ==

ReferenceTooltips[ResourceLoader|dependencies=mediawiki.util,ext.gadget.site-lib|default|type=general]|ReferenceTooltips.js|ReferenceTooltips.css
61
== reftoolbar ==

refToolbar[ResourceLoader|dependencies=mediawiki.util|default]|refToolbar.js
57
== utcliveclock ==

UTCLiveClock [ ResourceLoader ] |UTCLiveClock.js
90
== wiked ==

wikEd [ ResourceLoader ] |wikEd.js
77

The 29 month old table on https://meta.wikimedia.org/w/index.php?title=Gadgets&oldid=22447270#Gadgets_popular_on_wikis lists 16 gadgets.

If I ran the Python script correctly in PAWS after adjusting four date stamps (no idea; the UI isn't that intuitive), it seems to print 8 results. Is that correct?

Ahha, I also just tried and I get the same results. I am not sure why the script is no longer printing info for all gadgets. This needs more investigation.

I am not sure why the script is no longer printing info for all gadgets.

@srishakatux: The line blobs = r['query']['pages'].values() populates items in the format of dict_values([{'pageid': 123456, 'ns': 8, 'title': 'MediaWiki:Gadgets-definition'}]). But a few lines later code iterates on revisions often not existing for the specified time frame:

for blob in blobs:
    if 'revisions' not in blob:
        break

so code never reaches gadget_defs[url] = blob['revisions'][0]['*'] where gadget_defs seems to get populated (maybe that should call /w/api.php?action=parse&format=json&page=MediaWiki%3AGadgets-definition instead after checking if the page actually is not empty and does not give an error?).
Not sure as I don't know Python but I think code to get gadget definitions and code to get users with edits should get completely separated...

Also, the code would ideally check if a website is closed when parsing the sitematrix json and then not add it to the site_urls array

Errm I don't understand why the code queries revisions at all. Or how "popularity" is related to edits of a gadget list (overview) page.
The code is pretty cool and does stuff but it does not seem to answer the actual question if I get it right (I don't speak Python)?

So I (re)wrote that Python code (with sugar-on-top [[MediaWiki:Gadgets-definition]] syntax checking), stripping some stuff, adding other stuff.
I do not claim that I understood all existing code as I did not speak any Python until three days ago.

1# based on https://paws-public.wmflabs.org/paws-public/User:Harej_(WMF)/Lists%20of%20gadgets%20for%20every%20Wikimedia%20wiki.ipynb
2# based on https://public.paws.wmcloud.org/User:SSethi_(WMF)/List%20of%20most%20used%20gadgets%20across%20wikimedia%20wikis.ipynb
3
4# if this was my code (I am Andre Klapper, <aklapper@wikimedia.org>) this was licensed under Creative Commons Zero.
5
6import requests
7import re
8import json
9from collections import OrderedDict, Counter
10
11# Download the Site Matrix to get list of Wikimedia wikis
12
13sitematrix = 'https://en.wikipedia.org/w/api.php?action=sitematrix&format=json'
14r = requests.get(sitematrix)
15r = r.json()
16r = r['sitematrix']
17
18site_urls = []
19for blob in r.values():
20 if type(blob) is dict:
21 for subblob in blob['site']:
22 if 'closed' not in subblob: # Exclude closed sites
23 site_urls.append(subblob['url'])
24 elif type(blob) is list: # "Special" wikis, including Commons and Wikidata, are different for some reason
25 for subblob in blob:
26 if 'closed' not in subblob: # Exclude closed sites
27 site_urls.append(subblob['url'])
28
29# Download [[MediaWiki:Gadgets-definition]] from each of these wikis
30
31api_request_gadgets_definition = ('/w/api.php?action=parse&format=json&prop=wikitext&page=MediaWiki%3AGadgets-definition')
32
33gadget_defs = {}
34
35for url in site_urls:
36 q = url + api_request_gadgets_definition
37 r = requests.get(q)
38 r = r.json()
39 if 'error' not in r: # Check if [[MediaWiki:Gadgets-definition]] exists on that wiki
40 gadget_defs[url] = r['parse']['wikitext']['*']
41 print('✅ ' + url + '/wiki/MediaWiki:Gadgets-definition')
42 else:
43 print(' ❎ ' + url + '/wiki/MediaWiki:Gadgets-definition')
44
45print('\nGadget definition pages on active wikis: ' + str(len(gadget_defs)) + '\n')
46
47# Data cleanup and retrieving editor information
48
49gadget_entries = {}
50for site, blob in gadget_defs.items():
51# print(site)
52 blob = blob.split('\n')
53 for entry in blob:
54 if len(entry) > 0:
55 if entry[0] == '*' and '|' in entry:
56 gadget = entry.replace('*', '').strip()
57 canonical = gadget.upper().\
58 replace('-', '').\
59 replace('_', '').\
60 replace(' ', '').\
61 split('|')[0].\
62 split('[')[0].\
63 strip()
64 if canonical in gadget_entries:
65 if gadget in gadget_entries[canonical]:
66 gadget_entries[canonical][gadget][site] = {}
67 else:
68 gadget_entries[canonical][gadget] = {site: {}}
69 else:
70 gadget_entries[canonical] = {gadget: {site: {}}}
71
72 gadget_pages = gadget
73 gadget_pages = re.sub(r'\*+', '', gadget_pages)
74 gadget_pages = re.sub(r'\[.*?\]', '', gadget_pages)
75 gadget_pages = re.sub(r'<!--.*?-->', '', gadget_pages) # Must be first
76 gadget_pages = re.sub(r'<!--.*$', '', gadget_pages) # Must be after
77 gadget_pages = re.sub(r'^.*-->', '', gadget_pages) # Must be after
78 gadget_pages = gadget_pages.split('|')
79 gadget_pages = ['MediaWiki:Gadget-' + x.strip().replace(' ', '_') for x in gadget_pages \
80 if x.strip() != '']
81
82 for gadget_page in gadget_pages:
83 print(site + '/wiki/' + gadget_page)
84 api_request_gadget_definition = ('/w/api.php?action=parse&format=json&prop=wikitext&page=' + gadget_page)
85
86 q = site + api_request_gadget_definition
87 r = requests.get(q)
88 r = r.json()
89 if 'error' in r:
90
91 # Check for values with an equal sign as they imply a typo (incorrect ResourceLoader array parameters)
92 match1 = re.search('=', gadget_page)
93 if match1:
94 print('ERROR: Gadget definition for ' + gadget_page + ' includes an equal sign, potential typo on ' + site)
95 else:
96 # Check if listed pages actually exist but exclude description/translation page URLs that don't end in .js or .css
97 match2 = re.search('\.js\Z', gadget_page)
98 match3 = re.search('\.css\Z', gadget_page)
99 if (match2 or match3):
100 print('ERROR: Non-existing gadget page on ' + site + '/wiki/MediaWiki:Gadgets-definition : ' + gadget_page)
101
102 gadget_entries[canonical][gadget][site][gadget_page] = site
103
104gadget_entries = OrderedDict(sorted(gadget_entries.items()))
105print('\nDone')
106
107for canonical, gadgetblob in gadget_entries.items():
108 counter=0
109
110 for gadget, siteblob in gadgetblob.items():
111 for site in siteblob.keys():
112 counter=counter+1
113
114 if(counter>10):
115 print('== ' + canonical.lower() + ' ==\n')
116 print(gadget)
117 print(counter)
118
119# for gadget, siteblob in gadgetblob.items():
120# for site in siteblob.keys():
121# print(site)

Here are the results of it:

1423 hotcat
2205 utcliveclock
3176 wiked
4154 navigationpopups
5144 purgetab
6143 edittop
7126 referencetooltips
8104 reftoolbar
981 exlinks
1081 twinkle
1180 catalot
1280 charinsert
1372 charinsertcore
1470 rtrc
1568 reftoolbarbase
1667 contribsrange
1767 proveit
1864 dotssyntaxhighlighter
1962 popups
2060 cleandeletereasons
2159 markblocked
2254 removeaccesskeys
2347 markadmins
2445 switcher
2543 googletrans
2643 searchfocus
2742 commentsinlocaltime
2842 modrollback
2942 wikidatainfo
3041 addsectionplus
3141 dropdownmenus
3241 twinklepagestyles
3340 blackskin
3440 edittools
3540 olddiff
3638 directlinktocommons
3738 hidefundraisingnotice
3837 darkmode
3937 defaultsummaries
4037 xtoolsarticleinfo
4133 morebits
4233 regexmenuframework
4333 select2
4432 darkmodetoggle
4531 externalsearch
4630 darkmodetogglepagestyles
4730 shorturl
4829 charinsertstyles
4929 mobilesidebar
5029 shortlink
5128 bidiediting
5228 quickeditcounter
5328 revisionjumper
5427 advancedsitenotices
5527 subpages
5627 vectorclassic
5726 localliveclock
5825 mediawiki.toolbar
5925 watchlistnotice
6024 imagelinks
6124 usermessages
6224 utcliveclockpagestyles
6323 disambiguationlinks
6423 rightsfilter
6523 searchbox
6622 bugstatusupdate
6722 confirmationrollbackmobile
6822 massdelete
6922 purge
7022 watchlistnoticecore
7122 wikeddiff
7221 dropdownmenuspagestyles
7321 extraeditbuttons
7421 jsl
7521 specialchars
7620 categoryaboveall
7720 righteditlinks
7820 settingsmanager
7919 autocomplete
8019 deluxehistory
8119 highlightredirects
8219 showmessagenames
8319 widensearch
8418 formwizard
8518 metadata
8618 shortdeschelperpagestylesvector
8718 wdsearch
8817 autodel
8917 imageannotator
9017 mapframe
9117 preview
9217 prosesize
9317 site
9417 templatepreloader
9517 watchlist
9617 watchlistgreenindicators
9716 fillindex
9816 hidecentralnotice
9916 hotcats
10016 listingeditor
10116 ongletpurge
10215 croptool
10315 dictionarylookuphover
10415 easylst
10515 extratoolbarbuttons
10615 flechehaut
10715 geonotice
10815 lastdiff
10915 libjquery
11015 newimagethumb
11115 ocr
11215 prettylog
11315 searchnewtab
11414 collapsiblenav
11514 collapsibletables
11614 defaultvisibilitytoggles
11714 edithysteria
11814 formwizardcore
11914 googleocr
12014 legacyscripts
12114 legacyscriptsnewnode
12214 mysandbox
12314 nopinserter
12414 qpreview
12514 robot
12614 roundcorners
12714 sidebartranslate
12814 visibilitytoggles
12914 watchlistbase
13014 watchlistchangesbold
13114 watchlistgreenindicatorsmono
13214 wikibugs
13314 wikificator
13413 categorymaster
13513 curidlink
13613 geonoticecore
13713 justifyparagraphs
13813 markproofread
13913 menutabstoggle
14013 noanimations
14113 openstreetmap
14213 redirect
14313 rollbacksummary
14413 summarieslist
14513 textareasansserif
14613 userrights
14713 wikiminiatlas
14813 wiktsidebartranslation
14912 ancretitres
15012 articlestats
15112 bklcheck
15212 citations
15312 debugmode
15412 doctabs
15512 dynamiclayoutoverrides
15612 editor
15712 extratoolbarbuttonscore
15812 greenredirect
15912 liverc
16012 popup
16112 responsivecontent
16212 socialmedia
16312 tabbedlanguages
16412 userinfo
16511 codelinks
16611 commonactionedit
16711 extlinks
16811 modernskinthunks
16911 numeralconverter
17011 printoptions
17111 revertdiff
17211 shortdeschelper
17311 stickytableheaders
17411 subtleupdatemarker
17511 summarybuttons
17611 urldecoder
17711 wikidatatab

Aklapper raised the priority of this task from Lowest to Medium.

Stats updated in https://meta.wikimedia.org/w/index.php?title=Gadgets&type=revision&diff=23925902&oldid=22447270 based on previous comment.

Two additional gadgets listed compared to April 2020.

Followup task for October 2023 @ T320712.