Page MenuHomePhabricator

Prototype misalignment API
Closed, ResolvedPublic

Description

Build API for article misalignment. This will have a few underlying components:

  • Language-agnostic quality model: map any Wikipedia article to [0-1] score where 0 = no content and 1 ~ featured article.
  • Language-agnostic importance model: per last quarter's work (T272175#6894768), use pageviews as basic proxy for importance while other more flexible filters (e.g., topic, country, occupation) are developed. Normalize to [0-1] range.
  • Language-agnostic misalignment model: difference between quality and importance.
  • Language-agnostic misalignment metric: summarization of intensity of misalignment in a project.

The API then has two pieces:

  • Precomputed scores for all wikis
  • Function for computing article misalignment for single articles on the fly

Event Timeline

Weekly updates:

  • wrote up quality model description: https://meta.wikimedia.org/wiki/Research:Prioritization_of_Wikipedia_Articles/Language-Agnostic_Quality
  • tested end-to-end pipeline for generating misalignment scores for articles but am going to split it into its individual components so it's easier to extract the data from intermediate steps for other purposes
  • so far, the entire approach to quality and demand has been wiki-specific and as a result, e.g., a top-quality article in Simple Wikipedia is far smaller than a top-quality article in English Wikipedia and a top-demand article in Simple Wikipedia gets far fewer pageviews than a top-demand article in English Wikipedia. This makes plenty of sense though reduces the comparability of resulting quality/demand/misalignment scores across languages. However, I'm considering whether there should be some language-agnostic thresholds -- e.g., a top quality article will have at least 10 sections regardless of language and maybe this number can go higher in certain languages. Same for images and references. Harder for page length because it depends heavily on language whether you need e.g., 100 bytes or 1000 bytes, but perhaps some basic minimum threshold could still be set.

Weekly updates:

Weekly updates:

  • Calculated demand scores based on 99th percentile pageviews + log transform + floor of 100. So e.g., if the top 1% of articles in a wiki got 10000 pageviews per month, then log10(10000) = 4 would be the max and an article with 1000 pageviews would be scored as log10(1000) / 4 = 0.75. If the top 1% of articles in a wiki only got 50 pageviews, then log10(100) = 2 would be the normalizing factor instead of log10(50).
  • For each wiki, I calculate the misalignment score for each article: quality - pageviews. Values close to 1 indicate overproduced content -- i.e. top quality but low reader interest -- and values close to -1 indicate underproduced content -- i.e. low quality but high reader interest. Then to summarize misalignment in a wiki, I calculate average misalignment as well as how many articles had extreme misalignment -- i.e. |misalignment| > 0.5. I'm still exploring the data to find the best way to summarize it, but this gives a start. Results below and a few things stand out:
    • Wikis with many bot-generated articles like cebwiki and svwiki have very high misalignment scores (content quality is way higher than demand) and many articles in extreme misalignment
    • Scores further from zero for the very small wikis because the quality/pageview scores are based on smaller numbers presumably and so more variable.
    • English Wikipedia has one of the lowest misalignment scores of the large wikis reflecting the heavy reader demand for content (lots of articles with lots of pageviews makes it hard to keep up quality-wise). Japanese Wikipedia also reflects this, which makes sense based upon what I know. Will be curious to dig into the other larger wikis with particularly low misalignment scores.
    • French and Polish Wikipedia stand out among larger wikis as having very low misalignment scores (yay them!)
+----------------+------------+------------------------+----------------------+
|wiki_db         |num_articles|num_extreme_misalignment|avg_misalignment      |
+----------------+------------+------------------------+----------------------+
|enwiki          |6291500     |988535                  |-0.2068171490222372   |
|cebwiki         |5728812     |4037318                 |0.6946592323961095    |
|svwiki          |3202004     |2141079                 |0.4677181916786081    |
|dewiki          |2572161     |382738                  |-0.15152962418026855  |
|frwiki          |2324721     |369375                  |-0.06627649949230711  |
|nlwiki          |2053866     |205144                  |0.07517449951703953   |
|ruwiki          |1720605     |248182                  |-0.1630355893298563   |
|itwiki          |1690636     |247304                  |-0.0995391843707861   |
|eswiki          |1624994     |300778                  |-0.13036235458096995  |
|plwiki          |1471534     |240632                  |-0.06325183937524237  |
|arzwiki         |1275215     |280942                  |0.4660722934938       |
|jawiki          |1266463     |301892                  |-0.2532784673998119   |
|warwiki         |1265337     |1131705                 |0.670340646688275     |
|viwiki          |1263462     |83834                   |0.28604281725167263   |
|zhwiki          |1194540     |212972                  |-0.09406331315766868  |
|arwiki          |1114400     |128975                  |0.08327370352427324   |
|ukwiki          |1090508     |195194                  |0.13178752326990872   |
|ptwiki          |1066299     |186799                  |-0.09388874351841238  |
|fawiki          |793049      |119620                  |-0.07139674041362006  |
|cawiki          |677813      |128294                  |0.18498045707374042   |
|srwiki          |646109      |314180                  |0.41381904997068536   |
|idwiki          |572825      |92396                   |-0.04240661414743266  |
|nowiki          |555627      |68695                   |0.01124972723500318   |
|kowiki          |538702      |116937                  |-0.21762050467463734  |
|fiwiki          |508643      |86593                   |-0.06617844197549932  |
|huwiki          |487306      |71303                   |0.010051564795280044  |
|cswiki          |480233      |88205                   |-0.10446084568790853  |
|shwiki          |455209      |258515                  |0.4098543690343392    |
|zh_min_nanwiki  |430829      |210375                  |0.5345154692167942    |
|rowiki          |419681      |84424                   |0.06525517877816857   |
|trwiki          |400788      |111483                  |-0.17359052758386123  |
|euwiki          |370826      |45139                   |0.280866399420567     |
|cewiki          |354460      |248680                  |0.6248212090880225    |
|mswiki          |348349      |32419                   |0.14209439514135888   |
|eowiki          |295585      |30452                   |0.2344339454850274    |
|hewiki          |293943      |86949                   |-0.2562899999474237   |
|hywiki          |283970      |63621                   |0.25372720138474275   |
|bgwiki          |271483      |43731                   |-0.07557618541975117  |
|ttwiki          |267645      |162922                  |0.5208103967756247    |
|dawiki          |266538      |48207                   |-0.14027072164753338  |
|azbwiki         |240294      |45024                   |0.3609290274563827    |
|skwiki          |236702      |64539                   |0.01567134204098786   |
|kkwiki          |232032      |72507                   |0.283154982885725     |
|minwiki         |224744      |48357                   |0.4664442591548236    |
|etwiki          |218967      |24715                   |-0.03477444893517864  |
|bewiki          |204369      |20126                   |0.2329565765814529    |
|ltwiki          |197471      |35856                   |0.019051948669615723  |
|hrwiki          |194290      |34881                   |-0.08934223406169067  |
|elwiki          |192544      |39591                   |-0.12229766444075875  |
|simplewiki      |187250      |22861                   |-0.0546800071526274   |
|azwiki          |180859      |26611                   |-0.10412589349914861  |
|glwiki          |173316      |20872                   |0.1897116192990758    |
|slwiki          |172424      |25710                   |0.019726757974873017  |
|urwiki          |165786      |12544                   |0.24705698655762812   |
|nnwiki          |158467      |15436                   |0.185392032114217     |
|kawiki          |152372      |22928                   |0.020405836708978078  |
|hiwiki          |150810      |34062                   |-0.08026920705053042  |
|tawiki          |140880      |31110                   |0.09149395754367265   |
|uzwiki          |140503      |16486                   |0.1965819222906981    |
|thwiki          |139846      |35292                   |-0.2199371664047108   |
|lawiki          |135258      |21518                   |0.23703826327877153   |
|cywiki          |132850      |28192                   |0.29250553229029436   |
|astwiki         |127377      |24184                   |0.2730941733634883    |
|vowiki          |126562      |3070                    |-0.06510400088083838  |
|mkwiki          |114714      |23988                   |0.15861636542682317   |
|zh_yuewiki      |110024      |13984                   |-0.0674128397614726   |
|lvwiki          |107383      |16108                   |-0.03127635272508012  |
|bnwiki          |107212      |17932                   |0.048850593711569724  |
|tgwiki          |103271      |52893                   |0.4652557700874297    |
|mywiki          |102989      |5934                    |0.3451116077295101    |
|afwiki          |97879       |6636                    |0.14500493308991821   |
|mgwiki          |93969       |14021                   |0.30982410334590843   |
|sqwiki          |91750       |12308                   |0.009087308866045456  |
|ocwiki          |86830       |14035                   |0.2608820155656799    |
|bswiki          |85878       |23723                   |0.1686411324771605    |
|ndswiki         |82535       |5687                    |0.16187985877386768   |
|kywiki          |80830       |22680                   |0.17435914701892202   |
|be_x_oldwiki    |74409       |9895                    |0.22368208207083398   |
|mlwiki          |74284       |12882                   |0.05197014345287326   |
|mrwiki          |74005       |10945                   |-0.013461985254471355 |
|newwiki         |73054       |29847                   |0.5289622990322048    |
|tewiki          |71197       |23600                   |0.16418161323736938   |
|brwiki          |69586       |9754                    |0.13959921599166808   |
|vecwiki         |67626       |5386                    |-0.2251114914475738   |
|pmswiki         |65840       |5547                    |0.2493856603850046    |
|jvwiki          |63922       |5159                    |0.20875365114036082   |
|htwiki          |63271       |5514                    |0.33465405801127346   |
|pnbwiki         |62701       |8463                    |0.18966355465841034   |
|swwiki          |61800       |6327                    |0.05795624780206426   |
|suwiki          |60960       |2290                    |0.25968343014743306   |
|lbwiki          |59677       |3840                    |0.12391553036729366   |
|bawiki          |56885       |29987                   |0.4077982582117181    |
|gawiki          |55118       |4744                    |0.09250603241893884   |
|szlwiki         |53638       |19770                   |0.4385508386036721    |
|iswiki          |52675       |7676                    |-0.07222572276942471  |
|tlwiki          |52409       |7914                    |-0.027645159952252807 |
|cvwiki          |46057       |20956                   |0.42027537733753845   |
|lmowiki         |46054       |8033                    |0.2077437932657055    |
|fywiki          |45590       |4193                    |0.1280170038216638    |
|wuuwiki         |42388       |9526                    |-0.3178445154696779   |
|scowiki         |40887       |3574                    |0.09740945183069237   |
|diqwiki         |40104       |10451                   |0.26589879406990996   |
|anwiki          |39967       |3207                    |0.06618846177918351   |
|kuwiki          |38751       |3839                    |0.09645149952680282   |
|pawiki          |37961       |5583                    |0.09516057927751342   |
|yowiki          |33634       |2225                    |0.17112483542746793   |
|newiki          |32221       |3627                    |0.14189181783609167   |
|barwiki         |31673       |5149                    |0.19887525310344503   |
|ckbwiki         |30667       |4742                    |0.08620562899115422   |
|iowiki          |30580       |1796                    |0.024600842679156985  |
|guwiki          |29694       |2296                    |0.008286652624739568  |
|alswiki         |27810       |2544                    |0.07444022190802306   |
|knwiki          |27624       |5305                    |-0.09988942681904801  |
|nostalgiawiki   |27375       |0                       |0.0                   |
|scnwiki         |26439       |5290                    |-0.2610328464070203   |
|bpywiki         |25131       |16607                   |0.5807185240202848    |
|iawiki          |23231       |1371                    |-2.518017865670266E-4 |
|quwiki          |23112       |4727                    |0.16198104656052892   |
|mnwiki          |22388       |5198                    |-0.14310102638277183  |
|siwiki          |20936       |4290                    |-0.12854035078972453  |
|bat_smgwiki     |16911       |2544                    |-0.17803342545764209  |
|nvwiki          |16726       |698                     |-0.09001870527213147  |
|xmfwiki         |16292       |1390                    |-0.02830199967670098  |
|sdwiki          |15908       |1064                    |0.10209044328730797   |
|orwiki          |15662       |4718                    |0.31023185636528056   |
|cdowiki         |15517       |2799                    |-0.2372401843181535   |
|amwiki          |15422       |2321                    |-0.09692646345362066  |
|ilowiki         |15390       |2916                    |0.2008126264204446    |
|gdwiki          |15362       |1770                    |0.07746555115418918   |
|yiwiki          |15274       |1199                    |-0.017054730147041387 |
|napwiki         |14744       |1972                    |-0.21791774132534078  |
|sahwiki         |14628       |1503                    |0.06622142241063146   |
|maiwiki         |14535       |3873                    |0.29293470039097436   |
|bugwiki         |14194       |177                     |-0.0715502528816239   |
|pswiki          |13836       |1468                    |0.043748921563676     |
|map_bmswiki     |13781       |4892                    |0.3349818287306596    |
|hsbwiki         |13776       |3230                    |0.21055531972839683   |
|avkwiki         |13715       |902                     |-0.16328439664284417  |
|fowiki          |13591       |1126                    |0.026475191867366445  |
|mznwiki         |13577       |4514                    |0.2426280787499583    |
|oswiki          |13332       |1365                    |0.08781728043508465   |
|liwiki          |13252       |1148                    |-0.04574680471960922  |
|frrwiki         |12928       |1292                    |0.07872525711271838   |
|emlwiki         |12712       |1186                    |0.04557727742881577   |
|acewiki         |12346       |5979                    |0.44762646843326664   |
|gorwiki         |12006       |7220                    |0.49725291846459974   |
|bowiki          |11768       |2492                    |-0.25113481435137164  |
|sawiki          |11658       |813                     |0.061494259948227575  |
|bclwiki         |11377       |1319                    |0.049228934133195325  |
|wawiki          |11223       |1384                    |0.09426747106535491   |
|zh_classicalwiki|10787       |1264                    |-0.061794301807670426 |
|mrjwiki         |10530       |1554                    |0.12288935152921694   |
|mhrwiki         |10330       |2797                    |0.1525275944552717    |
|lijwiki         |10288       |575                     |0.044572372353037536  |
|kmwiki          |10175       |2462                    |-0.2351300401465752   |
|hifwiki         |10128       |1306                    |0.08928056021984579   |
|shnwiki         |9538        |1226                    |0.2724463709790793    |
|hakwiki         |9527        |2689                    |-0.3298376009198242   |
|testwiki        |9444        |1230                    |0.1933553955284583    |
|roa_tarawiki    |9315        |443                     |0.002236042516850939  |
|crhwiki         |9304        |1394                    |-0.2346057637161411   |
|hawiki          |9190        |1698                    |-0.04637498701876992  |
|hywwiki         |8999        |1261                    |0.17636231222156948   |
|pamwiki         |8986        |653                     |0.021452323099751265  |
|aswiki          |8516        |1730                    |0.03209316993579303   |
|nsowiki         |8447        |854                     |0.08951080980259413   |
|zuwiki          |8147        |1694                    |0.23215629263996265   |
|ruewiki         |8124        |1219                    |-0.01929847933241941  |
|sewiki          |7958        |990                     |-0.02692185660632719  |
|ugwiki          |7715        |1247                    |-0.008357880782150095 |
|bhwiki          |7454        |1108                    |-0.1343909455430805   |
|vlswiki         |7426        |883                     |-0.14838589103017058  |
|sowiki          |7426        |1912                    |-0.2609888452323193   |
|tkwiki          |7328        |1665                    |-0.18034400127260788  |
|banwiki         |7283        |1211                    |0.14896066790362497   |
|nds_nlwiki      |7251        |959                     |-0.15483818324253223  |
|snwiki          |7251        |1819                    |-0.3126618087460232   |
|iewiki          |7230        |1580                    |-0.31835010696413774  |
|miwiki          |7224        |860                     |-0.18050685663762217  |
|scwiki          |7179        |1092                    |-0.1525982210101532   |
|nahwiki         |7171        |729                     |-0.11518085303582203  |
|vepwiki         |6663        |725                     |0.06819326819917129   |
|glkwiki         |6546        |666                     |0.10442651176339729   |
|ganwiki         |6505        |1992                    |-0.36224958144363123  |
|myvwiki         |6431        |1078                    |0.06642026195739666   |
|kabwiki         |6210        |939                     |-0.11999552177322276  |
|abwiki          |6157        |319                     |0.07882358341324885   |
|satwiki         |6070        |1503                    |0.24089397604667767   |
|cowiki          |5985        |715                     |-0.06946109255578185  |
|fiu_vrowiki     |5831        |2061                    |-0.3537317012901311   |
|kvwiki          |5529        |656                     |-0.03450828375620574  |
|csbwiki         |5405        |524                     |-0.09137123029861706  |
|pcdwiki         |5199        |1404                    |0.13365640794056888   |
|aywiki          |5142        |1964                    |0.2164841646329293    |
|udmwiki         |5066        |428                     |0.0046991618064096494 |
|gvwiki          |5042        |703                     |-0.06898249223686456  |
|frpwiki         |4979        |637                     |-0.07191069918413626  |
|pagwiki         |4961        |727                     |-0.19723866637244242  |
|zeawiki         |4775        |1440                    |-0.31205936692460046  |
|lfnwiki         |4697        |2239                    |-0.4851799144316207   |
|lowiki          |4669        |908                     |-0.08773070998437196  |
|kwwiki          |4599        |1260                    |-0.019253304222921465 |
|nrmwiki         |4599        |1486                    |-0.3684447765330901   |
|dvwiki          |4319        |298                     |-0.03189972675965875  |
|lezwiki         |4202        |798                     |0.08459507439219911   |
|gomwiki         |4197        |1105                    |0.21859393605227012   |
|gnwiki          |4145        |926                     |-0.21608206087092008  |
|mwlwiki         |4120        |388                     |0.05154755712459678   |
|stqwiki         |4115        |495                     |-0.13860713825408338  |
|mtwiki          |3950        |642                     |0.019368130127085915  |
|olowiki         |3927        |504                     |-0.04137809533243541  |
|arywiki         |3911        |420                     |0.06100516626099375   |
|szywiki         |3904        |410                     |-0.14989733097494626  |
|rmwiki          |3768        |299                     |-0.0019640446286608966|
|bjnwiki         |3746        |681                     |-0.08263471776105412  |
|awawiki         |3710        |412                     |0.13444130228897633   |
|dtywiki         |3605        |419                     |0.07214479613196668   |
|ladwiki         |3586        |683                     |0.021085234474218298  |
|furwiki         |3559        |1616                    |-0.45826211875243344  |
|koiwiki         |3509        |403                     |0.1760093394585164    |
|extwiki         |3448        |721                     |-0.1681795029808052   |
|angwiki         |3392        |807                     |-0.2339779029134661   |
|lnwiki          |3335        |688                     |-0.09552480789764353  |
|cbk_zamwiki     |3328        |421                     |0.13705697657468802   |
|dsbwiki         |3313        |501                     |-0.06223134946110671  |
|piwiki          |3217        |745                     |-0.30923090073348647  |
|tyvwiki         |3185        |660                     |0.13190416468415567   |
|gagwiki         |2952        |423                     |-0.11457531776299403  |
|kshwiki         |2914        |1249                    |-0.44707200411827336  |
|pflwiki         |2723        |174                     |0.205224990226353     |
|avwiki          |2612        |347                     |-0.10963504283102786  |
|lgwiki          |2485        |302                     |0.04088058128383318   |
|hawwiki         |2422        |440                     |-0.03813757394262821  |
|gcrwiki         |2378        |1239                    |-0.4423425898168947   |
|xalwiki         |2324        |420                     |-0.09761624250451997  |
|igwiki          |2267        |511                     |-0.17864680173550085  |
|papwiki         |2258        |494                     |-0.21868558345153016  |
|rwwiki          |2235        |480                     |-0.16789374376836771  |
|bxrwiki         |2198        |330                     |-0.1873815072237686   |
|zawiki          |2117        |1222                    |-0.5213098122958705   |
|pdcwiki         |2108        |691                     |-0.3814168950822028   |
|krcwiki         |2076        |238                     |-0.17524117937666833  |
|kaawiki         |2062        |588                     |-0.24650905158815167  |
|test2wiki       |2042        |136                     |0.17923957104525592   |
|kbpwiki         |1917        |1277                    |-0.5704377489781467   |
|arcwiki         |1811        |1058                    |-0.5644089952234667   |
|novwiki         |1806        |1023                    |-0.5169855222138984   |
|inhwiki         |1773        |271                     |-0.12639703633535804  |
|towiki          |1752        |218                     |-0.2646556531916879   |
|jamwiki         |1720        |642                     |-0.3371939898836751   |
|tcywiki         |1698        |200                     |0.07857987101247932   |
|tpiwiki         |1669        |407                     |-0.22599225821182212  |
|wowiki          |1667        |498                     |-0.17337565749758507  |
|kiwiki          |1631        |354                     |-0.28376227068177473  |
|kbdwiki         |1623        |173                     |-0.033096316134533024 |
|nawiki          |1612        |615                     |-0.4177158485829978   |
|akwiki          |1588        |291                     |-0.0967724031583413   |
|tetwiki         |1587        |654                     |0.01005316680624715   |
|atjwiki         |1541        |156                     |0.08142849901602842   |
|lldwiki         |1456        |354                     |-0.1565263662077896   |
|xhwiki          |1420        |343                     |-0.17608349398827888  |
|biwiki          |1411        |569                     |-0.44473408483639415  |
|mnwwiki         |1374        |176                     |0.20540414574885474   |
|mdfwiki         |1360        |240                     |-0.1455337210174869   |
|jbowiki         |1335        |1057                    |-0.6556015742182285   |
|tywiki          |1331        |337                     |-0.32357567587935954  |
|roa_rupwiki     |1281        |352                     |-0.2838360348732373   |
|kgwiki          |1273        |519                     |-0.4705635347262388   |
|lbewiki         |1261        |356                     |-0.04578683997410616  |
|omwiki          |1201        |467                     |-0.35270351966801133  |
|srnwiki         |1189        |297                     |-0.3477609447940645   |
|fjwiki          |1158        |619                     |-0.5340473791117907   |
|nqowiki         |1076        |94                      |-0.011264141455126518 |
|smwiki          |1057        |280                     |-0.25164390372265705  |
|ltgwiki         |1009        |192                     |-0.04325449555314804  |
|chrwiki         |972         |152                     |-0.17215707193707488  |
|stwiki          |964         |182                     |-0.20344890624717557  |
|gotwiki         |960         |543                     |-0.5445798894545685   |
|pihwiki         |892         |367                     |-0.3454675217881598   |
|klwiki          |873         |678                     |-0.6400098519363491   |
|tnwiki          |850         |141                     |-0.01074410501648398  |
|twwiki          |850         |188                     |-0.20944871927885447  |
|nywiki          |837         |146                     |-0.11703519859621601  |
|chywiki         |784         |353                     |-0.5063027489398207   |
|cuwiki          |782         |208                     |-0.25538103116671446  |
|bmwiki          |761         |362                     |-0.5014610307758647   |
|tswiki          |730         |205                     |-0.2748586089847839   |
|tumwiki         |723         |245                     |-0.4110812583569834   |
|rmywiki         |714         |340                     |-0.5278395286419478   |
|rnwiki          |714         |139                     |-0.2806803210317588   |
|vewiki          |710         |166                     |-0.39879591368453604  |
|ikwiki          |676         |289                     |-0.49026183422500563  |
|iuwiki          |655         |484                     |-0.6790186332609818   |
|kswiki          |598         |196                     |-0.21148845834975258  |
|adywiki         |569         |358                     |-0.6175069845623626   |
|sswiki          |561         |127                     |-0.20364991340407712  |
|chwiki          |550         |226                     |-0.4917036096074364   |
|pntwiki         |521         |126                     |-0.2751956017484134   |
|ffwiki          |439         |169                     |-0.3384421916682791   |
|eewiki          |388         |229                     |-0.4492788003738393   |
|tiwiki          |374         |164                     |-0.3453844712977422   |
|dinwiki         |309         |182                     |-0.5924627634313886   |
|dzwiki          |299         |225                     |-0.7174335971823504   |
|sgwiki          |296         |162                     |-0.5704178106341813   |
|crwiki          |175         |123                     |-0.6480813642059053   |
+----------------+------------+------------------------+----------------------+

Weekly updates:

Weekly updates:

  • Slow week as catching up from week off but continue to gather informal feedback on model. In particular thinking and playing with a few ideas:
    • Should average misalignment across an entire wiki be 0 and the goal is to understand what topics are under/overproduced not to label wikis as over/underproduced? There is no clear definition of over/underproduced so applying a single number to a wiki (beyond being quite reductive) is very liable to misinterpretation.
    • How to include missing content? Past research has tackled this question from a more focused recommender system angle. Not every wiki obviously should have full coverage of all articles but perhaps taking the cultural content approach from Wikipedia Diversity Observatory (content relevant to language/geography should be covered by a wiki)? And questions of scalability for applying this to all wikis and 20 million potential Wikipedia articles from across all the languages.
    • I'll continue thinking on these questions but they are beyond the scope of this initial prototype
  • Remaining TODO to close this task out: build a simple API for getting individual quality, demand, and misalignment scores for any given article to complement the aggregate scores. This probably will be computed on-demand (i.e. gather data from Mediawiki APIs for scoring as opposed to extracting from a database of pre-computed scores) to simplify the space requirements.

At this point, I consider this quarter's work done and will resolve next week unless I hear otherwise.

Weekly updates:

  • Built out API to support this work with three endpoints:
  • Long discussion with AS and MW about this current approach to misalignment. A lot of really good feedback from them that I'm going to try to summarize here:
    • There are three main stakeholders we might consider in designing these metrics: experienced editors, newcomers, and organizers. The latter two tend to have shared goals (identify content that is relevant to a particular interest and would most benefit from simple improvements). We've largely been designing from the experienced editor standpoint, however, because e.g., the quality scale that I'm replicating in my model is one that was designed by experienced editors to map many gradations of content (more on that below).
    • Careful about communication -- sometimes not clear why certain content is misaligned, "overproduced" doesn't necessarily mean that no one should work to improve the content (the audience just might not exist yet), and high-quality/stable content is not a bad thing even if almost no one is reading it.
    • Misalignment is clearly useful for ranking articles in the context of e.g., campaigns. It could be a useful tool for comparing content across topic areas if paired with good approaches to grouping Wikipedia articles by meaningful topics. It's likely far less useful to maybe damaging when used to compare entire wikis.
    • While misalignment is a really interesting concept from a research perspective, when applying it to prioritizing content or comparing topic areas, it might be better to focus more on related concepts of opportunity-to-improve or potential-lift. That is, overproduced content is not necessarily particularly interesting. It's the underproduced content that is important/actionable and so redesigning the metrics to focus on that side of the scale could be a good approach.
    • For demand, wikis tend to handle very high-interest content quite well and so that type of content doesn't really need to be elevated by tooling. It's the consistently medium-to-high demand content that is most important to elevate because improved quality there has long-term impacts and this content can remain lower quality without specific interventions to address it. A redesigned demand metric might be something like the median pageviews per month to an article (tamp down the spikes of interest) or look at a longer timeframe than a month.
    • For quality, the differences between B-class and featured article are largely meaningless when it comes to what content to prioritize. A B-class article is already quite good and useful to readers and moving to featured article requires a lot of work and is typically work that experienced editors do. That's useful from a modeling perspective too because the model has trouble distinguishing between so many fine-grained classes and especially amongst the higher-quality classes because those differences are much more nuanced (less about quantity, more about organization, accessibility, etc.)
    • Probably want to normalize each wiki's average misalignment to 0 so there's a meaningful spread of content. This is how Morten's original approach worked. It's also nice to translate the numeric scores into meaningful categories where possible.
    • Might we incorporate some sense of sitelinks into the model? Is an article that only exists in one language more important to make high quality because that's potentially the only source of knowledge on the topic on the web vs. an article that has a high-quality version in another language perhaps being less important? Is this a global thing or only relevant to certain language pairs?