Maniphest T281912

Prototype misalignment API
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Isaac
	May 4 2021, 6:18 PM

Description

Build API for article misalignment. This will have a few underlying components:

Language-agnostic quality model: map any Wikipedia article to [0-1] score where 0 = no content and 1 ~ featured article.
Language-agnostic importance model: per last quarter's work (T272175#6894768), use pageviews as basic proxy for importance while other more flexible filters (e.g., topic, country, occupation) are developed. Normalize to [0-1] range.
Language-agnostic misalignment model: difference between quality and importance.
Language-agnostic misalignment metric: summarization of intensity of misalignment in a project.

The API then has two pieces:

Precomputed scores for all wikis
Function for computing article misalignment for single articles on the fly

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		Isaac	T155541 [Epic] Article importance prediction model
		Resolved		Isaac	T281912 Prototype misalignment API

Event Timeline

Isaac created this task.May 4 2021, 6:18 PM

Isaac moved this task from Staged to FY2020-21-Research-April-June on the Research board.

Isaac edited projects, added Research (FY2020-21-Research-April-June); removed Research.

Weekly updates:

wrote up quality model description: https://meta.wikimedia.org/wiki/Research:Prioritization_of_Wikipedia_Articles/Language-Agnostic_Quality
tested end-to-end pipeline for generating misalignment scores for articles but am going to split it into its individual components so it's easier to extract the data from intermediate steps for other purposes
so far, the entire approach to quality and demand has been wiki-specific and as a result, e.g., a top-quality article in Simple Wikipedia is far smaller than a top-quality article in English Wikipedia and a top-demand article in Simple Wikipedia gets far fewer pageviews than a top-demand article in English Wikipedia. This makes plenty of sense though reduces the comparability of resulting quality/demand/misalignment scores across languages. However, I'm considering whether there should be some language-agnostic thresholds -- e.g., a top quality article will have at least 10 sections regardless of language and maybe this number can go higher in certain languages. Same for images and references. Harder for page length because it depends heavily on language whether you need e.g., 100 bytes or 1000 bytes, but perhaps some basic minimum threshold could still be set.

Weekly updates:

Released V1 model w/ quality predictions for every Wikipedia article. Description w/ link to data: https://meta.wikimedia.org/wiki/Research:Prioritization_of_Wikipedia_Articles/Language-Agnostic_Quality#Models
Added more details / structure to potential features on meta page
Split up code into quality model, pageviews model, and only then misalignment calculations so easier to test different approaches / extract data at different stages

Weekly updates:

Calculated demand scores based on 99th percentile pageviews + log transform + floor of 100. So e.g., if the top 1% of articles in a wiki got 10000 pageviews per month, then log10(10000) = 4 would be the max and an article with 1000 pageviews would be scored as log10(1000) / 4 = 0.75. If the top 1% of articles in a wiki only got 50 pageviews, then log10(100) = 2 would be the normalizing factor instead of log10(50).
For each wiki, I calculate the misalignment score for each article: quality - pageviews. Values close to 1 indicate overproduced content -- i.e. top quality but low reader interest -- and values close to -1 indicate underproduced content -- i.e. low quality but high reader interest. Then to summarize misalignment in a wiki, I calculate average misalignment as well as how many articles had extreme misalignment -- i.e. |misalignment| > 0.5. I'm still exploring the data to find the best way to summarize it, but this gives a start. Results below and a few things stand out:
- Wikis with many bot-generated articles like cebwiki and svwiki have very high misalignment scores (content quality is way higher than demand) and many articles in extreme misalignment
- Scores further from zero for the very small wikis because the quality/pageview scores are based on smaller numbers presumably and so more variable.
- English Wikipedia has one of the lowest misalignment scores of the large wikis reflecting the heavy reader demand for content (lots of articles with lots of pageviews makes it hard to keep up quality-wise). Japanese Wikipedia also reflects this, which makes sense based upon what I know. Will be curious to dig into the other larger wikis with particularly low misalignment scores.
- French and Polish Wikipedia stand out among larger wikis as having very low misalignment scores (yay them!)

+----------------+------------+------------------------+----------------------+
|wiki_db         |num_articles|num_extreme_misalignment|avg_misalignment      |
+----------------+------------+------------------------+----------------------+
|enwiki          |6291500     |988535                  |-0.2068171490222372   |
|cebwiki         |5728812     |4037318                 |0.6946592323961095    |
|svwiki          |3202004     |2141079                 |0.4677181916786081    |
|dewiki          |2572161     |382738                  |-0.15152962418026855  |
|frwiki          |2324721     |369375                  |-0.06627649949230711  |
|nlwiki          |2053866     |205144                  |0.07517449951703953   |
|ruwiki          |1720605     |248182                  |-0.1630355893298563   |
|itwiki          |1690636     |247304                  |-0.0995391843707861   |
|eswiki          |1624994     |300778                  |-0.13036235458096995  |
|plwiki          |1471534     |240632                  |-0.06325183937524237  |
|arzwiki         |1275215     |280942                  |0.4660722934938       |
|jawiki          |1266463     |301892                  |-0.2532784673998119   |
|warwiki         |1265337     |1131705                 |0.670340646688275     |
|viwiki          |1263462     |83834                   |0.28604281725167263   |
|zhwiki          |1194540     |212972                  |-0.09406331315766868  |
|arwiki          |1114400     |128975                  |0.08327370352427324   |
|ukwiki          |1090508     |195194                  |0.13178752326990872   |
|ptwiki          |1066299     |186799                  |-0.09388874351841238  |
|fawiki          |793049      |119620                  |-0.07139674041362006  |
|cawiki          |677813      |128294                  |0.18498045707374042   |
|srwiki          |646109      |314180                  |0.41381904997068536   |
|idwiki          |572825      |92396                   |-0.04240661414743266  |
|nowiki          |555627      |68695                   |0.01124972723500318   |
|kowiki          |538702      |116937                  |-0.21762050467463734  |
|fiwiki          |508643      |86593                   |-0.06617844197549932  |
|huwiki          |487306      |71303                   |0.010051564795280044  |
|cswiki          |480233      |88205                   |-0.10446084568790853  |
|shwiki          |455209      |258515                  |0.4098543690343392    |
|zh_min_nanwiki  |430829      |210375                  |0.5345154692167942    |
|rowiki          |419681      |84424                   |0.06525517877816857   |
|trwiki          |400788      |111483                  |-0.17359052758386123  |
|euwiki          |370826      |45139                   |0.280866399420567     |
|cewiki          |354460      |248680                  |0.6248212090880225    |
|mswiki          |348349      |32419                   |0.14209439514135888   |
|eowiki          |295585      |30452                   |0.2344339454850274    |
|hewiki          |293943      |86949                   |-0.2562899999474237   |
|hywiki          |283970      |63621                   |0.25372720138474275   |
|bgwiki          |271483      |43731                   |-0.07557618541975117  |
|ttwiki          |267645      |162922                  |0.5208103967756247    |
|dawiki          |266538      |48207                   |-0.14027072164753338  |
|azbwiki         |240294      |45024                   |0.3609290274563827    |
|skwiki          |236702      |64539                   |0.01567134204098786   |
|kkwiki          |232032      |72507                   |0.283154982885725     |
|minwiki         |224744      |48357                   |0.4664442591548236    |
|etwiki          |218967      |24715                   |-0.03477444893517864  |
|bewiki          |204369      |20126                   |0.2329565765814529    |
|ltwiki          |197471      |35856                   |0.019051948669615723  |
|hrwiki          |194290      |34881                   |-0.08934223406169067  |
|elwiki          |192544      |39591                   |-0.12229766444075875  |
|simplewiki      |187250      |22861                   |-0.0546800071526274   |
|azwiki          |180859      |26611                   |-0.10412589349914861  |
|glwiki          |173316      |20872                   |0.1897116192990758    |
|slwiki          |172424      |25710                   |0.019726757974873017  |
|urwiki          |165786      |12544                   |0.24705698655762812   |
|nnwiki          |158467      |15436                   |0.185392032114217     |
|kawiki          |152372      |22928                   |0.020405836708978078  |
|hiwiki          |150810      |34062                   |-0.08026920705053042  |
|tawiki          |140880      |31110                   |0.09149395754367265   |
|uzwiki          |140503      |16486                   |0.1965819222906981    |
|thwiki          |139846      |35292                   |-0.2199371664047108   |
|lawiki          |135258      |21518                   |0.23703826327877153   |
|cywiki          |132850      |28192                   |0.29250553229029436   |
|astwiki         |127377      |24184                   |0.2730941733634883    |
|vowiki          |126562      |3070                    |-0.06510400088083838  |
|mkwiki          |114714      |23988                   |0.15861636542682317   |
|zh_yuewiki      |110024      |13984                   |-0.0674128397614726   |
|lvwiki          |107383      |16108                   |-0.03127635272508012  |
|bnwiki          |107212      |17932                   |0.048850593711569724  |
|tgwiki          |103271      |52893                   |0.4652557700874297    |
|mywiki          |102989      |5934                    |0.3451116077295101    |
|afwiki          |97879       |6636                    |0.14500493308991821   |
|mgwiki          |93969       |14021                   |0.30982410334590843   |
|sqwiki          |91750       |12308                   |0.009087308866045456  |
|ocwiki          |86830       |14035                   |0.2608820155656799    |
|bswiki          |85878       |23723                   |0.1686411324771605    |
|ndswiki         |82535       |5687                    |0.16187985877386768   |
|kywiki          |80830       |22680                   |0.17435914701892202   |
|be_x_oldwiki    |74409       |9895                    |0.22368208207083398   |
|mlwiki          |74284       |12882                   |0.05197014345287326   |
|mrwiki          |74005       |10945                   |-0.013461985254471355 |
|newwiki         |73054       |29847                   |0.5289622990322048    |
|tewiki          |71197       |23600                   |0.16418161323736938   |
|brwiki          |69586       |9754                    |0.13959921599166808   |
|vecwiki         |67626       |5386                    |-0.2251114914475738   |
|pmswiki         |65840       |5547                    |0.2493856603850046    |
|jvwiki          |63922       |5159                    |0.20875365114036082   |
|htwiki          |63271       |5514                    |0.33465405801127346   |
|pnbwiki         |62701       |8463                    |0.18966355465841034   |
|swwiki          |61800       |6327                    |0.05795624780206426   |
|suwiki          |60960       |2290                    |0.25968343014743306   |
|lbwiki          |59677       |3840                    |0.12391553036729366   |
|bawiki          |56885       |29987                   |0.4077982582117181    |
|gawiki          |55118       |4744                    |0.09250603241893884   |
|szlwiki         |53638       |19770                   |0.4385508386036721    |
|iswiki          |52675       |7676                    |-0.07222572276942471  |
|tlwiki          |52409       |7914                    |-0.027645159952252807 |
|cvwiki          |46057       |20956                   |0.42027537733753845   |
|lmowiki         |46054       |8033                    |0.2077437932657055    |
|fywiki          |45590       |4193                    |0.1280170038216638    |
|wuuwiki         |42388       |9526                    |-0.3178445154696779   |
|scowiki         |40887       |3574                    |0.09740945183069237   |
|diqwiki         |40104       |10451                   |0.26589879406990996   |
|anwiki          |39967       |3207                    |0.06618846177918351   |
|kuwiki          |38751       |3839                    |0.09645149952680282   |
|pawiki          |37961       |5583                    |0.09516057927751342   |
|yowiki          |33634       |2225                    |0.17112483542746793   |
|newiki          |32221       |3627                    |0.14189181783609167   |
|barwiki         |31673       |5149                    |0.19887525310344503   |
|ckbwiki         |30667       |4742                    |0.08620562899115422   |
|iowiki          |30580       |1796                    |0.024600842679156985  |
|guwiki          |29694       |2296                    |0.008286652624739568  |
|alswiki         |27810       |2544                    |0.07444022190802306   |
|knwiki          |27624       |5305                    |-0.09988942681904801  |
|nostalgiawiki   |27375       |0                       |0.0                   |
|scnwiki         |26439       |5290                    |-0.2610328464070203   |
|bpywiki         |25131       |16607                   |0.5807185240202848    |
|iawiki          |23231       |1371                    |-2.518017865670266E-4 |
|quwiki          |23112       |4727                    |0.16198104656052892   |
|mnwiki          |22388       |5198                    |-0.14310102638277183  |
|siwiki          |20936       |4290                    |-0.12854035078972453  |
|bat_smgwiki     |16911       |2544                    |-0.17803342545764209  |
|nvwiki          |16726       |698                     |-0.09001870527213147  |
|xmfwiki         |16292       |1390                    |-0.02830199967670098  |
|sdwiki          |15908       |1064                    |0.10209044328730797   |
|orwiki          |15662       |4718                    |0.31023185636528056   |
|cdowiki         |15517       |2799                    |-0.2372401843181535   |
|amwiki          |15422       |2321                    |-0.09692646345362066  |
|ilowiki         |15390       |2916                    |0.2008126264204446    |
|gdwiki          |15362       |1770                    |0.07746555115418918   |
|yiwiki          |15274       |1199                    |-0.017054730147041387 |
|napwiki         |14744       |1972                    |-0.21791774132534078  |
|sahwiki         |14628       |1503                    |0.06622142241063146   |
|maiwiki         |14535       |3873                    |0.29293470039097436   |
|bugwiki         |14194       |177                     |-0.0715502528816239   |
|pswiki          |13836       |1468                    |0.043748921563676     |
|map_bmswiki     |13781       |4892                    |0.3349818287306596    |
|hsbwiki         |13776       |3230                    |0.21055531972839683   |
|avkwiki         |13715       |902                     |-0.16328439664284417  |
|fowiki          |13591       |1126                    |0.026475191867366445  |
|mznwiki         |13577       |4514                    |0.2426280787499583    |
|oswiki          |13332       |1365                    |0.08781728043508465   |
|liwiki          |13252       |1148                    |-0.04574680471960922  |
|frrwiki         |12928       |1292                    |0.07872525711271838   |
|emlwiki         |12712       |1186                    |0.04557727742881577   |
|acewiki         |12346       |5979                    |0.44762646843326664   |
|gorwiki         |12006       |7220                    |0.49725291846459974   |
|bowiki          |11768       |2492                    |-0.25113481435137164  |
|sawiki          |11658       |813                     |0.061494259948227575  |
|bclwiki         |11377       |1319                    |0.049228934133195325  |
|wawiki          |11223       |1384                    |0.09426747106535491   |
|zh_classicalwiki|10787       |1264                    |-0.061794301807670426 |
|mrjwiki         |10530       |1554                    |0.12288935152921694   |
|mhrwiki         |10330       |2797                    |0.1525275944552717    |
|lijwiki         |10288       |575                     |0.044572372353037536  |
|kmwiki          |10175       |2462                    |-0.2351300401465752   |
|hifwiki         |10128       |1306                    |0.08928056021984579   |
|shnwiki         |9538        |1226                    |0.2724463709790793    |
|hakwiki         |9527        |2689                    |-0.3298376009198242   |
|testwiki        |9444        |1230                    |0.1933553955284583    |
|roa_tarawiki    |9315        |443                     |0.002236042516850939  |
|crhwiki         |9304        |1394                    |-0.2346057637161411   |
|hawiki          |9190        |1698                    |-0.04637498701876992  |
|hywwiki         |8999        |1261                    |0.17636231222156948   |
|pamwiki         |8986        |653                     |0.021452323099751265  |
|aswiki          |8516        |1730                    |0.03209316993579303   |
|nsowiki         |8447        |854                     |0.08951080980259413   |
|zuwiki          |8147        |1694                    |0.23215629263996265   |
|ruewiki         |8124        |1219                    |-0.01929847933241941  |
|sewiki          |7958        |990                     |-0.02692185660632719  |
|ugwiki          |7715        |1247                    |-0.008357880782150095 |
|bhwiki          |7454        |1108                    |-0.1343909455430805   |
|vlswiki         |7426        |883                     |-0.14838589103017058  |
|sowiki          |7426        |1912                    |-0.2609888452323193   |
|tkwiki          |7328        |1665                    |-0.18034400127260788  |
|banwiki         |7283        |1211                    |0.14896066790362497   |
|nds_nlwiki      |7251        |959                     |-0.15483818324253223  |
|snwiki          |7251        |1819                    |-0.3126618087460232   |
|iewiki          |7230        |1580                    |-0.31835010696413774  |
|miwiki          |7224        |860                     |-0.18050685663762217  |
|scwiki          |7179        |1092                    |-0.1525982210101532   |
|nahwiki         |7171        |729                     |-0.11518085303582203  |
|vepwiki         |6663        |725                     |0.06819326819917129   |
|glkwiki         |6546        |666                     |0.10442651176339729   |
|ganwiki         |6505        |1992                    |-0.36224958144363123  |
|myvwiki         |6431        |1078                    |0.06642026195739666   |
|kabwiki         |6210        |939                     |-0.11999552177322276  |
|abwiki          |6157        |319                     |0.07882358341324885   |
|satwiki         |6070        |1503                    |0.24089397604667767   |
|cowiki          |5985        |715                     |-0.06946109255578185  |
|fiu_vrowiki     |5831        |2061                    |-0.3537317012901311   |
|kvwiki          |5529        |656                     |-0.03450828375620574  |
|csbwiki         |5405        |524                     |-0.09137123029861706  |
|pcdwiki         |5199        |1404                    |0.13365640794056888   |
|aywiki          |5142        |1964                    |0.2164841646329293    |
|udmwiki         |5066        |428                     |0.0046991618064096494 |
|gvwiki          |5042        |703                     |-0.06898249223686456  |
|frpwiki         |4979        |637                     |-0.07191069918413626  |
|pagwiki         |4961        |727                     |-0.19723866637244242  |
|zeawiki         |4775        |1440                    |-0.31205936692460046  |
|lfnwiki         |4697        |2239                    |-0.4851799144316207   |
|lowiki          |4669        |908                     |-0.08773070998437196  |
|kwwiki          |4599        |1260                    |-0.019253304222921465 |
|nrmwiki         |4599        |1486                    |-0.3684447765330901   |
|dvwiki          |4319        |298                     |-0.03189972675965875  |
|lezwiki         |4202        |798                     |0.08459507439219911   |
|gomwiki         |4197        |1105                    |0.21859393605227012   |
|gnwiki          |4145        |926                     |-0.21608206087092008  |
|mwlwiki         |4120        |388                     |0.05154755712459678   |
|stqwiki         |4115        |495                     |-0.13860713825408338  |
|mtwiki          |3950        |642                     |0.019368130127085915  |
|olowiki         |3927        |504                     |-0.04137809533243541  |
|arywiki         |3911        |420                     |0.06100516626099375   |
|szywiki         |3904        |410                     |-0.14989733097494626  |
|rmwiki          |3768        |299                     |-0.0019640446286608966|
|bjnwiki         |3746        |681                     |-0.08263471776105412  |
|awawiki         |3710        |412                     |0.13444130228897633   |
|dtywiki         |3605        |419                     |0.07214479613196668   |
|ladwiki         |3586        |683                     |0.021085234474218298  |
|furwiki         |3559        |1616                    |-0.45826211875243344  |
|koiwiki         |3509        |403                     |0.1760093394585164    |
|extwiki         |3448        |721                     |-0.1681795029808052   |
|angwiki         |3392        |807                     |-0.2339779029134661   |
|lnwiki          |3335        |688                     |-0.09552480789764353  |
|cbk_zamwiki     |3328        |421                     |0.13705697657468802   |
|dsbwiki         |3313        |501                     |-0.06223134946110671  |
|piwiki          |3217        |745                     |-0.30923090073348647  |
|tyvwiki         |3185        |660                     |0.13190416468415567   |
|gagwiki         |2952        |423                     |-0.11457531776299403  |
|kshwiki         |2914        |1249                    |-0.44707200411827336  |
|pflwiki         |2723        |174                     |0.205224990226353     |
|avwiki          |2612        |347                     |-0.10963504283102786  |
|lgwiki          |2485        |302                     |0.04088058128383318   |
|hawwiki         |2422        |440                     |-0.03813757394262821  |
|gcrwiki         |2378        |1239                    |-0.4423425898168947   |
|xalwiki         |2324        |420                     |-0.09761624250451997  |
|igwiki          |2267        |511                     |-0.17864680173550085  |
|papwiki         |2258        |494                     |-0.21868558345153016  |
|rwwiki          |2235        |480                     |-0.16789374376836771  |
|bxrwiki         |2198        |330                     |-0.1873815072237686   |
|zawiki          |2117        |1222                    |-0.5213098122958705   |
|pdcwiki         |2108        |691                     |-0.3814168950822028   |
|krcwiki         |2076        |238                     |-0.17524117937666833  |
|kaawiki         |2062        |588                     |-0.24650905158815167  |
|test2wiki       |2042        |136                     |0.17923957104525592   |
|kbpwiki         |1917        |1277                    |-0.5704377489781467   |
|arcwiki         |1811        |1058                    |-0.5644089952234667   |
|novwiki         |1806        |1023                    |-0.5169855222138984   |
|inhwiki         |1773        |271                     |-0.12639703633535804  |
|towiki          |1752        |218                     |-0.2646556531916879   |
|jamwiki         |1720        |642                     |-0.3371939898836751   |
|tcywiki         |1698        |200                     |0.07857987101247932   |
|tpiwiki         |1669        |407                     |-0.22599225821182212  |
|wowiki          |1667        |498                     |-0.17337565749758507  |
|kiwiki          |1631        |354                     |-0.28376227068177473  |
|kbdwiki         |1623        |173                     |-0.033096316134533024 |
|nawiki          |1612        |615                     |-0.4177158485829978   |
|akwiki          |1588        |291                     |-0.0967724031583413   |
|tetwiki         |1587        |654                     |0.01005316680624715   |
|atjwiki         |1541        |156                     |0.08142849901602842   |
|lldwiki         |1456        |354                     |-0.1565263662077896   |
|xhwiki          |1420        |343                     |-0.17608349398827888  |
|biwiki          |1411        |569                     |-0.44473408483639415  |
|mnwwiki         |1374        |176                     |0.20540414574885474   |
|mdfwiki         |1360        |240                     |-0.1455337210174869   |
|jbowiki         |1335        |1057                    |-0.6556015742182285   |
|tywiki          |1331        |337                     |-0.32357567587935954  |
|roa_rupwiki     |1281        |352                     |-0.2838360348732373   |
|kgwiki          |1273        |519                     |-0.4705635347262388   |
|lbewiki         |1261        |356                     |-0.04578683997410616  |
|omwiki          |1201        |467                     |-0.35270351966801133  |
|srnwiki         |1189        |297                     |-0.3477609447940645   |
|fjwiki          |1158        |619                     |-0.5340473791117907   |
|nqowiki         |1076        |94                      |-0.011264141455126518 |
|smwiki          |1057        |280                     |-0.25164390372265705  |
|ltgwiki         |1009        |192                     |-0.04325449555314804  |
|chrwiki         |972         |152                     |-0.17215707193707488  |
|stwiki          |964         |182                     |-0.20344890624717557  |
|gotwiki         |960         |543                     |-0.5445798894545685   |
|pihwiki         |892         |367                     |-0.3454675217881598   |
|klwiki          |873         |678                     |-0.6400098519363491   |
|tnwiki          |850         |141                     |-0.01074410501648398  |
|twwiki          |850         |188                     |-0.20944871927885447  |
|nywiki          |837         |146                     |-0.11703519859621601  |
|chywiki         |784         |353                     |-0.5063027489398207   |
|cuwiki          |782         |208                     |-0.25538103116671446  |
|bmwiki          |761         |362                     |-0.5014610307758647   |
|tswiki          |730         |205                     |-0.2748586089847839   |
|tumwiki         |723         |245                     |-0.4110812583569834   |
|rmywiki         |714         |340                     |-0.5278395286419478   |
|rnwiki          |714         |139                     |-0.2806803210317588   |
|vewiki          |710         |166                     |-0.39879591368453604  |
|ikwiki          |676         |289                     |-0.49026183422500563  |
|iuwiki          |655         |484                     |-0.6790186332609818   |
|kswiki          |598         |196                     |-0.21148845834975258  |
|adywiki         |569         |358                     |-0.6175069845623626   |
|sswiki          |561         |127                     |-0.20364991340407712  |
|chwiki          |550         |226                     |-0.4917036096074364   |
|pntwiki         |521         |126                     |-0.2751956017484134   |
|ffwiki          |439         |169                     |-0.3384421916682791   |
|eewiki          |388         |229                     |-0.4492788003738393   |
|tiwiki          |374         |164                     |-0.3453844712977422   |
|dinwiki         |309         |182                     |-0.5924627634313886   |
|dzwiki          |299         |225                     |-0.7174335971823504   |
|sgwiki          |296         |162                     |-0.5704178106341813   |
|crwiki          |175         |123                     |-0.6480813642059053   |
+----------------+------------+------------------------+----------------------+

Weekly updates:

Basic documentation: https://meta.wikimedia.org/wiki/Research:Prioritization_of_Wikipedia_Articles/Misalignment
I computed average misalignment for each wiki stratified by ORES topic (e.g., Sports, Medicine) and UN Subregion (e.g., Eastern Europe, Melanesia). Because the resulting tables are ~300 wikis x 64 topics and ~300 wikis x 23 subregions (too large to really show in a single table), I created a simple interface for viewing the data: https://wiki-topic.toolforge.org/misalignment
Gathering informal feedback from team members and other folks at Wikimedia who I have talked to about this project to understand whether the scores make sense to them and what adjustments could/should be made.

Weekly updates:

Slow week as catching up from week off but continue to gather informal feedback on model. In particular thinking and playing with a few ideas:
- Should average misalignment across an entire wiki be 0 and the goal is to understand what topics are under/overproduced not to label wikis as over/underproduced? There is no clear definition of over/underproduced so applying a single number to a wiki (beyond being quite reductive) is very liable to misinterpretation.
- How to include missing content? Past research has tackled this question from a more focused recommender system angle. Not every wiki obviously should have full coverage of all articles but perhaps taking the cultural content approach from Wikipedia Diversity Observatory (content relevant to language/geography should be covered by a wiki)? And questions of scalability for applying this to all wikis and 20 million potential Wikipedia articles from across all the languages.
- I'll continue thinking on these questions but they are beyond the scope of this initial prototype
Remaining TODO to close this task out: build a simple API for getting individual quality, demand, and misalignment scores for any given article to complement the aggregate scores. This probably will be computed on-demand (i.e. gather data from Mediawiki APIs for scoring as opposed to extracting from a database of pre-computed scores) to simplify the space requirements.

At this point, I consider this quarter's work done and will resolve next week unless I hear otherwise.

Weekly updates:

Built out API to support this work with three endpoints:
- Misalignment by wiki and pre-aggregated by topic: https://misalignment.wmcloud.org/api/v1/misalignment-topic?lang=en|fr
- Misalignment by wiki and pre-aggregated by region: https://misalignment.wmcloud.org/api/v1/misalignment-region?lang=en|fr
- Misalignment by article (along with predicted quality and demand scores): https://misalignment.wmcloud.org/api/v1/misalignment-article?lang=en&title=Solidago_canadensis
- NOTE: the first two of those end-points can also be accessed via this interface: https://wiki-topic.toolforge.org/misalignment
Long discussion with AS and MW about this current approach to misalignment. A lot of really good feedback from them that I'm going to try to summarize here:
- There are three main stakeholders we might consider in designing these metrics: experienced editors, newcomers, and organizers. The latter two tend to have shared goals (identify content that is relevant to a particular interest and would most benefit from simple improvements). We've largely been designing from the experienced editor standpoint, however, because e.g., the quality scale that I'm replicating in my model is one that was designed by experienced editors to map many gradations of content (more on that below).
- Careful about communication -- sometimes not clear why certain content is misaligned, "overproduced" doesn't necessarily mean that no one should work to improve the content (the audience just might not exist yet), and high-quality/stable content is not a bad thing even if almost no one is reading it.
- Misalignment is clearly useful for ranking articles in the context of e.g., campaigns. It could be a useful tool for comparing content across topic areas if paired with good approaches to grouping Wikipedia articles by meaningful topics. It's likely far less useful to maybe damaging when used to compare entire wikis.
- While misalignment is a really interesting concept from a research perspective, when applying it to prioritizing content or comparing topic areas, it might be better to focus more on related concepts of opportunity-to-improve or potential-lift. That is, overproduced content is not necessarily particularly interesting. It's the underproduced content that is important/actionable and so redesigning the metrics to focus on that side of the scale could be a good approach.
- For demand, wikis tend to handle very high-interest content quite well and so that type of content doesn't really need to be elevated by tooling. It's the consistently medium-to-high demand content that is most important to elevate because improved quality there has long-term impacts and this content can remain lower quality without specific interventions to address it. A redesigned demand metric might be something like the median pageviews per month to an article (tamp down the spikes of interest) or look at a longer timeframe than a month.
- For quality, the differences between B-class and featured article are largely meaningless when it comes to what content to prioritize. A B-class article is already quite good and useful to readers and moving to featured article requires a lot of work and is typically work that experienced editors do. That's useful from a modeling perspective too because the model has trouble distinguishing between so many fine-grained classes and especially amongst the higher-quality classes because those differences are much more nuanced (less about quantity, more about organization, accessibility, etc.)
- Probably want to normalize each wiki's average misalignment to 0 so there's a meaningful spread of content. This is how Morten's original approach worked. It's also nice to translate the numeric scores into meaningful categories where possible.
- Might we incorporate some sense of sitelinks into the model? Is an article that only exists in one language more important to make high quality because that's potentially the only source of knowledge on the topic on the web vs. an article that has a high-quality version in another language perhaps being less important? Is this a global thing or only relevant to certain language pairs?

Isaac closed this task as Resolved.Jul 23 2021, 6:20 PM

Prototype misalignment APIClosed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

Prototype misalignment API
Closed, ResolvedPublic
Actions

Related Objects
Search...