Page MenuHomePhabricator

Investigate frequency of section titles in Wikipedia articles
Closed, ResolvedPublic

Description

(Microtask regarding a possible Outreachy internship in data analysis with the WMF reading team)

Download the dataset described here:
https://lists.wikimedia.org/pipermail/wiki-research-l/2016-April/005129.html
It consists of a list of all sections from all pages on the English Wikipedia (255MB packed / 1.4GB unpacked).

  • Generate the list of the 100 most frequent section titles, each with the total number of how often they occurred in the dataset. Disregard leading and trailing spaces (" ") in a section title, e.g. treat "Early life" and "Early life " as identical.
  • --> code + results
  • Bonus task: Generate the list of the 100 section titles that are used in the largest number of articles, each with the percentage of articles that contain such a section. Caveat: Not all pages are articles. For example, there are also talk pages and help pages (see https://en.wikipedia.org/wiki/Wikipedia:Namespace ). For the purposes of this bonus exercise, let's make the simplifying assumption that all entries in the dataset with no "/" and no "." in the page title correspond to articles, and vice versa that all page titles that contain a "/" or "." do not correspond to an article.
  • --> code + results
  • Extended task (after completion of the above): Exact calculation of the 100 section titles that are used in the largest number of articles, for five large Wikipedias, by regenerating the dataset on PAWS instead of relying on the above simplification.
  • --> meta page with results

(Background: This came up in context of the Related Articles feature, where we wanted to know how many Wikipedia articles already contain a "See also" section, because it fulfills a somewhat similar function.)

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 15 2016, 1:59 AM
Zareenf updated the task description. (Show Details)Oct 15 2016, 2:46 AM
Zareenf updated the task description. (Show Details)
Zareenf added a comment.EditedOct 15 2016, 2:54 AM

Results for 100 most frequent section titles and frequency:

indexfrequencysection_title
03933510References
12249554External links
21114379See also
3732295Summary
4695887Licensing
5509498History
6276974Notes
7162768Career
8144400Biography
9141217Track listing
10137631Further reading
11137035Sources
12111723Bibliography
13105465Licensing:
14103177Geography
15102048Cast
16101171Early life
1796112Links
1895009Users
1990384Background
2089379Plot
2189003Personal life
2280724Reception
2378673Description
2476854Discography
2576182Demographics
2675327Awards
2772011Personnel
2869550Assessed
2967931Additions
3061171Results
3160067Education
3254906Reassessed
3352637Filmography
3448354Life
3548223Honours
3645873Production
3745286Works
3842110Gallery
3938216Overview
4037731Singles
4136731Discussion
4236433Footnotes
4333370Death
4431548Charts
4531409Distribution
4629230Population
4728422Publications
4827651Family
4927532Legacy
5026709Entry
5126542Selected additions
5226297Final
5326037Events
5425631Music
5525329Species
5625034Usage
5725006Albums
5824846Books
5924603Television
6024093Club career
6124084Source
6223870Economy
6323587Early years
6423384Removed
6523123Early life and education
6622888Critical reception
6722465International career
6822446Schedule
6922347Location
7021785Professional career
7121327Notes and references
7221068<span style="font-size:150%">Clerk, CheckUser,...
7321059<span style="font-size:150%">Comments by other...
7420745Career statistics
7520677Notable people
7620606Synopsis
7720173Chart performance
7819891Other
7919545Etymology
8019536Playing career
8118636Climate
8218441Development
8318001Politics
8417915Political career
8517830Deaths
8617817Transportation
8717434Fair use rationale
8817309People
8917117Aftermath
9017002Life and career
9116894Film
9216354Births
9316343Release
9416318Early career
9516254Soundtrack
9616223Music video
9716211Members
9815938Club
9915923Regular season
Zareenf added a comment.EditedOct 15 2016, 3:00 AM

Results for 100 most frequent section titles (filtered for only articles), their frequency, and percentage:

indexheader_frequencyheader_titleheader_percent
03792066References82.002300
12168509External links46.893363
21060642See also22.936068
3491924History10.637710
4263258Notes5.692876
5152605Career3.300038
6135144Track listing2.922449
7132480Further reading2.864841
8130716Biography2.826695
9106534Bibliography2.303766
10102130Geography2.208531
11100152Cast2.165757
1293913Sources2.030841
1393566Early life2.023337
1487526Plot1.892724
1584240Background1.821665
1683965Personal life1.815718
1778534Reception1.698274
1875219Demographics1.626589
1975189Discography1.625940
2074200Description1.604553
2170968Awards1.534662
2269402Personnel1.500798
2357946Results1.253065
2455912Education1.209080
2551004Filmography1.102946
2645485Life0.983600
2744881Honours0.970538
2844628Production0.965067
2942711Works0.923613
3041026Gallery0.887175
3136871Singles0.797324
3236359Overview0.786253
3334791Footnotes0.752345
3431317Distribution0.677221
3530425Charts0.657932
3630265Death0.654472
3729181Population0.631030
3826104Family0.564491
3925960Final0.561377
4025718Publications0.556144
4125500Legacy0.551430
4225306Events0.547235
4325282Species0.546716
4425007Music0.540769
4524512Albums0.530065
4624082Television0.520766
4724065Club career0.520398
4823698Economy0.512462
4922421International career0.484847
5022405Books0.484501
5122298Critical reception0.482188
5222134Schedule0.478641
5322020Early years0.476176
5421861Location0.472738
5520965Professional career0.453362
5620645Career statistics0.446442
5720524Notable people0.443825
5820511Notes and references0.443544
5920195Synopsis0.436711
6020070Early life and education0.434008
6119699Chart performance0.425985
6219474Etymology0.421119
6319311Playing career0.417595
6418539Climate0.400900
6518538Other0.400879
6617688Development0.382498
6717543Transportation0.379362
6817333Politics0.374821
6917016Deaths0.367966
7016899People0.365436
7116582Aftermath0.358581
7216538Film0.357629
7315993Soundtrack0.345844
7415945Life and career0.344806
7515935Release0.344590
7615852Music video0.342795
7715824Transport0.342189
7815816Political career0.342016
7915638Members0.338167
8015570Club0.336697
8115557Births0.336416
8215441Seeds0.333907
8315426Early career0.333583
8415280Regular season0.330425
8514789Plot summary0.319808
8614674Characters0.317321
8714652Awards and nominations0.316845
8814437Gameplay0.312196
8914253Literature0.308217
9014084In popular culture0.304562
9113987Citations0.302465
9213893Selected filmography0.300432
9313633Episodes0.294810
9413540Sports0.292798
9513523Places0.292431
9613464Draw0.291155
9712829Statistics0.277423
9812629Notable alumni0.273098
9912627Roster0.273055
Zareenf closed this task as Resolved.Dec 14 2016, 9:24 PM
Zareenf updated the task description. (Show Details)
Restricted Application added a project: Product-Analytics. · View Herald TranscriptJul 12 2018, 11:16 PM