Goal
Apply the meta-method developed for measuring NPOV on contentious topic to one topic and understand the limits of the meta-method and the insights we can gain through this research.
Topic
Goal
Apply the meta-method developed for measuring NPOV on contentious topic to one topic and understand the limits of the meta-method and the insights we can gain through this research.
Topic
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | leila | T393634 Research support for NPOV workstreams | |||
| Open | GWeld | T433445 Apply the meta-method for studying sensitive topics NPOV to one topic |
This week I completed an initial implementation of the entire pipeline of slant index computation for two nations (the US and the UK) and applied it to an initial exploratory dataset of Wikipedia Articles (biographies of the US and UK national legislators) and a convenience sample of English language news articles from the CC-NEWS dataset. While initial tentative results seem promising, they also highlighted the inadequacy of CC-NEWS as a sole data source for news articles, as it lacks most major Western news outlets.
Next week I will focus on collecting additional news content to use as a comparison corpus, and, time permitting, will begin to experiment with additional methods for slant index computation so we can make an informed choice about what method to use for final analyses.
Last week was a slightly shorter week due to holidays. I completed the first steps for collecting additional news source content using the MediaCloud API and the Internet Archive. This week I will finish collecting more news content for the US and UK, create a public Meta Wiki page describing the project, and begin to work on alternatives to the Gentzkow and Shapiro method for computing slant indices.
This week I worked on two primary focii: collecting higher quality news articles from the MediaCloud database, and working on implementing a second slant index method to compare with the Genztskow and Shapiro method.
On the news front, I re-implemented a pipeline developed earlier by Pablo and Diego for a different application. First, we use the MediaCloud Search API to fetch all news articles from August 2025-July-2026 (a 12 month period) that contain the names of any entities (politicians as described above) from a select set of nation-specific news sources. For the US, we examine The New York Times, The Washington Post, The Wall Street Journal, Fox News, CNN, NPR, the Associated Press, and Reuters. For the UK, we examine the BBC, The Guardian, The Telegraph, the Daily Mail, The Times, and The Independent. This returns metadata and URLs for 222,846 news articles (132,173 for the US and 90,673 for the UK) from the 13 unique news sources described above.
Second, we attempt to collect the full text of these articles from their original URLs, and, if not available, using the Wayback Machine. One important part of this process is deduplication, as a single article may be returned as matching search results for multiple entity queries. This process is rather slow, taking ~72 hours for all 126k deduplicated news links. The "success" rate varies from news source to source, but overall seems to work decently.
On the second slant index method, I started with a quick and simple pre-trained word embedding based method. I use a pre-trained word embedding model to fetch embeddings for all the news articles, and then define an axis (or axes) through the embedding space and project every article onto these axes.
I have started with a quick-and-dirty implementation of this using the intfloat/multilingual-e5-large-instruct model (which is open-weight), however, this has a downside of a maximum input size of 512 tokens, smaller than most articles. It returns embeddings with 1024 dimensions. To manage the maximum input size, we break documents up into paragraph-sized chunks, compute embeddings for each chunk, then take the centroid of all chunks to represent the entire document.
We also do this with the latest (2024) Manifesto Project manifestos for each major party in the US and the UK, then use each party’s manifesto embedding to define an axis through the embedding space for each pair of national parties. We then project each Wikipedia/News article onto this axis to produce scores.
This works well, but the correlation between the embedding based method and the G&S based method scores is quite low (0.11 or so, based on a small non-random sample). Next week I will dig into this further.
This was a fun week on this project! After discussing with colleagues earlier in the week, I have becoming increasingly convinced that a simple but important change to improve the quality of our research is to shift the focus of entities to elections and related topics. There are several key advantages of this approach: first, elections are perhaps an even more important (and contentious) topic than just current politicians, and second, by focusing on election topics (debates, scandals, policies, manifestos) rather than just biographies, the Wikipedia content that we will compare to news coverage is more similar and therefore more comparable. A challenge that we struggled with with biographies is that a biography covers a person's entire life (across a broad time range and many events) whereas a news article inherently focuses on a specific time and newsworthy event. I am optimistic that this improved alignment will make this more valuable of an analytical approach.
There are lots of details to work out as we generalize the method to other elections, but this week I worked primarily on the 2024 UK General Election as an illustrative example. I used a pre-existing dataset of all candidates for election to the 650 constituencies in the House of Commons, and fetched news articles from MediaCloud for the 5 months prior to, and 1 month after, the election (July 4, 2024) for a 6 month window of news coverage using our existing hydration pipeline and the same set of UK news sources as used earlier: the BBC, The Guardian, The Telegraph, the Daily Mail, The Times, and The Independent. This hydration process seems to work well for these news sources and should complete running over the weekend.
On the Wikipedia side, I used the ~800 articles in Category:2024 United Kingdom general election (and subcategories) to supplement the biographies of candidates. Next week I will explore differences in coverage between these two sources, experiment with a party-manifesto based slant index, and consider ways to align based on specific sub-election levels (e.g. constituencies and candidates) and do some initial work to expand to other nations and elections, starting with a quick assessment of available data sources.
Another fun week! This week I started with a major restructure of the codebase to align with our focus on a broad range of nations' elections. This simplifies our pipelines and will make it much easier to add data for new elections in the future. I am currently working through two elections: the 2024 US Congressional election (in November) and the 2024 UK General Election (in July). Data for both of these elections is readily available, and their occurance in English-speaking nations is convenient to get us started. The current pipeline involves fetching news articles from Mediacloud (using a 6 month window mostly preceding the election) from major news sources, then hydrating these news articles using the original host+wayback machine to get their full text.
To get Wikipedia content, we fetch the biographies for every candidate (when they exist) plus all articles that are in the category for the election plus subcategories. For each article, we fetch two revisions: one from the day of the election, and a "current" one to see how articles change, although eventually I would like to add additional snapshots for a deeper longitudinal analysis. Hydration continues to be the slowest part of this process, but still is chugging along. We have essentially completed this entire pipeline for the UK general election, with the US election still being hydrated:
election sources news_urls hydrated failed pending news_embed wiki_candidate wiki_category wiki_embed ---------------------- -------- ---------- --------- -------- -------- ----------- --------------- -------------- ----------- uk-2024-general 7 33,524 18,635 2,083 12,806 97% 1,578 1,401 99% us-2024-house-senate 4 5,420 0 0 5,420 - 632 137 0%
For the UK we have finished collecting ~33.5k news articles from names of candidates, and hydrated 2/3s of those, with the remaining third currently running. Here we’re using five news outlets: the BBC, the Daily Mail, the Independent, the Telegraph, and The Guardian.
We have collected Wikipedia articles from two sets of articles: biographies for all the candidates, when they exist, and all articles in the “2024 UK General Election” which consists mostly of broader articles, such as those about the election overall (including in Scotland, Wales, England, etc) and specific strategies, scandals, party manifestos, etc. For each article, we collect its revision on the day of the election (July 4, 2024) as well as how it looks “currently” (e.g. an arbitrary snapshot from August 2026).
We can use machine-readable versions of 2024 party manifestos from the Manifesto Project to anchor our analyses. We use multilingual-e5-large-instruct to embed every document (averaging across chunks to compute document-level embeddings), and then project these onto the plane defined by the three major party’s manifestos: Labour, Conservative, and Liberal Democrats.
We see that the UK news media generally uses language that is closer to the Labour-Conservative axis (and is more dissimilar to the Liberal Dems), whereas Wikipedia content (regardless of if it’s a candidate bio or a more general article) sits more solidly in the middle of all three parties. We also see that in the ~2 years since the election, the updates to these Wikipedia articles has slightly increased their similarity to that of election-era news content, but only slightly.
We also can examine just the biographies (as they appeared on the day of the election) to see how the language used in a candidate’s biography (many candidates do not have biographies on Wikipedia) relates to their party. Do Wikipedia biographies of 2024 UK General Election candidates tend to use language similar to their party’s manifestos? It would seem that the answer is “not really.” Generally, Wikipedia biographies are written slightly more like the Conservative party manifesto, but the distribution of language used in biographies of candidates from all three major parties overlaps heavily.
Note that these are still very tentative results, as we’re still hydrating the remaining ~⅓ of news coverage for the UK general election. Furthermore, there are some kinks that need to be investigated further (for example, the BBC appears closer to the Conservative party manifesto than the Daily Mail, which is odd), and I plan on reproducing them with a different word embedding to assess robustness.
In the upcoming week, I'll be digging deeper into these results, experimenting with different word embeddings, finishing the same for the US election, and starting to work on additional elections in additional nations with coverage in different languages!
I plan on reproducing them with a different word embedding to assess robustness.
I can't remember if I shared but here's an example notebook that I ran on stat1008 for a different project that hit a LiftWing model that you should be able to adapt for fetching embeddings.
This was a shorter week with the holiday and a day of vacation for me, but still lots of fun. I built out tooling to fetch embeddings using the 0.6B parameter qwen3-embedding hosted on WMF infrastructure, and collected embeddings for the UK and US 2024 elections to compare with the embeddings from multilingual-e5-large-instruct. Qualititatively, the results (when projecting news/wikipedia content onto an axis/plane defined by Manifesto Project documents) are quite similar, with both embeddings showing Wikipedia content solidly in the middle of the three major parties' manifestos, with most of the major news outlets sitting slightly closer to the Conservative party manifesto (for the 2024 UK general election). The qwen3 embedding also shows candidates' Wikipedia biographies sitting a bit closer to their own party's manifesto than multilingual-e5-large-instruct, although all parties' candidates overlap heavily regardless of the model. In the coming week I will finish a deeper evaluation of the differences between the embeddings produced using these two models.
On the additional elections front, this week I finalized entity sets and Wikipedia categories for the 2024 French legislative election and the 2022 Italian general election. MediaCloud news content collection is nearly done for the French election, with the Italian election up next. I have also done some initial work to fetch embeddings for non-English languages, and am excited to report on those results soon.
This week I made progress on several key fronts:
First, I did some substantial re-organization of internal notes and writeups, and started drafting a description of our methods for news and Wikipedia content collection and text comparison, along with some work on related work and overall framing for our eventual paper submission. I also wrote a meta-wiki page with a brief description of the goals of the project and our approach and methods.
Second, I fetching news and Wikipedia data for the 2024 French and 2022 Italian elections, started fetching the same for the 2024 Mexican general election, and did some additional background work to expand our list of candidate elections.
Third, I spent a plurality of my time focused on improving our methods for text comparison and slant measurement. I finished fetching embeddings for our French content, dug more into the differences between the different embedding models, and after consulting with Aaron Shaw, concluded that it was worthwhile to experiment with prompting-based measurement methods (instead of just embedding-based ones), which is an approach used by many recent political science papers. One such paper is Benoit et al., which pulls policy position ratings out of manifestos. They use a multi-step approach, first using one model to summarize the relevant passages from each manifesto, then using a second model to rate each summary’s position regarding several pre-selected issues, such as immigration, taxes, and more.
Another method is Haider et al., which uses LLMs to classify the political lean of news articles, a task more similar to our intended usage. This method, Media Bias Detector, also consists of several tasks. Of particular interest is the Article Lean task, which asks the LLM to score how much the article supports Democrats or Republicans. This is a useful approach, but needs to be generalized to multiparty political systems, as the original paper only focuses on the US political context.
I have developed a hybrid approach that uses aspects from both of these papers: the manifesto summarization from Benoit et al., and the Article Lean from Haider et al. I have used modified versions of the prompts from these two papers, generalized to support multiparty systems, and with a modified focus to focus a bit more on neutrality as construed on Wikipedia (e.g. due weight and balance). I will continue to experiment with prompt tweaks and other languages, but I am optimistic about the system design, which uses only open-weight models, runs almost entirely on WMF infrastructure, and, when evaluated on four major US news sources, nicely replicates the scores of two major existing media bias assessment services: MBFC and AllSides (.98 and .94 correlation between our scores and their aggregate bias rating, respectively). This is much better performance than the previous bag of n-grams and embedding based approaches we've experimented with. The system is designed with two major steps: first, generating summaries of manifestos, and second, scoring individual documents along axes defined as pairs of parties for a given election. For the summarization step, I’ve used the kimi-k3 model hosted on Together.AI, and for the scoring step, I’ve used Qwen 3.6 27b hosted on LiftWing.
Next week, I will finalize prompts, start processing data from other elections that we have already collected, and continue to fetch and hydrate content for new elections, as well as continuing writing for eventual paper submission.
Another busy week! This week I leaned into deepening and validating the prompting-based approach for measuring neutrality developed last week. I conducted a comparison of three different models, using the US election news coverage. I compared two LiftWing-hosted models, the 14b and 27b parameter versions of Qwen 3.6, along with the 70b version of LLama 3 hosted on Together.AI. The smaller Qwen model is ~5x faster than the larger one, although both LiftWing models are quite a bit slower to use than the Together.AI hosted model.
When comparing results at the news-source level (looking at measures of central tendency for all articles from a given news source), we see very similar results from each model, and they all do an excellent job of replicating MBFC and AllSides scores for the set of news sources we've been looking at: all models correlate with one another with Pearson correlation coefficients >= 0.94, and correlate with the fact checking services a bit more closely with MBFC (>=0.95) than AllSides (>=0.93). Although performance is pretty similar, Llama appears to perform the best (when evaluated with MBFC), producing an impressive 0.999 correlation between its news source scores and MBFC's. I applied the method to Wikipedia content for the same election, and compared the results between the three models, which are all qualitatively quite similar to one another, and show Wikipedia solidly in the middle of the news sources, with NPR far to the left, Fox far to the right, with CNN moderately to the left, and the AP quite close to Wikipedia.
There is, however, a decent amount of noise when comparing models' scores for individual articles. The larger (Qwen 3.6-27b and LLama 3-70b) models generally are more likely to agree with one another (0.75 correlation) than the smaller model. This is a difficult and subjective task, so I am not particularly concerned about the noise here, but it does emphasize that our results will be more reliable when assessing news sources with more articles, and Wikipedia coverage of elections (and in languages) that have more articles as well. I have started working on human annotations of a random sample of 200 news articles from a broad set of sources and plan on using the same approach to validation as that used by Haider et al. (Figure 5), where rather than having humans provide a "ground truth" set of annotations, they show that humans have equally high (or low) inter-annotator agreement as their LLM-based approach. This will require finding a few helpful humans to help label some news articles, though...
One additional benefit of the prompting-based approach is that we can add a “Not Relevant” label to use to ask the language model if the document is not relevant to the election. This helps to reduce noise by excluding news articles that were erroneously caught up in our news query step. I conducted a spot-check of a number of documents that received a “Not Relevant” label when using the above prompt, and all of them were unrelated to the election, but contained mentions of a candidate.
I also spent a bunch of time this week selecting additional elections to include, starting with meeting with some colleagues to strategize about what nations might make for good candidates. I curated data sets and added four new elections (Mexico 2024, Brazil 2022, Japan 2021, South Korea 2024) and hydrated their news content, as well as manually reviewed the resulting news content to construct a set of filters to identify off topic (e.g. sports) content and distinguish opinion from straight-news reporting based on news sources' URL structure. The current set of elections sits at: US 2024, UK 2024, Franch 2024, Italy 2022, Mexico 2024, Brazil 2022, Japan 2021, South Korea 2024), but we are starting to bump against the limits of the coverage of the Manifesto Project. For example, the Manifesto Project doesn't include a single African nation, no small Latin American nations, no China, and no recent coverage of Russia. Next week I will spend some time seeing if we can find Manifestos for any of these nations from other sources, which could be worthwhile for a few countries.
I also spent more time this week writing, which I will continue next week. One key bottleneck will be the rate at which can process news content using language models, so priority number one for me next week will be seeing how we can improve labeling throughput.
Hard to believe it is Friday again! This week I made progress on many fronts.
First, I was able to make some dramatic increases in the throughput of our LiftWing processing/labeling pipeline by implementing concurrent requests (~32x speed up, thanks Diego!). We're still slightly bottlenecked by processing throughput, but are rapidly cranking through the backlog and can process more quickly than we can hydrate from MediaCloud/Internet Archive. We should be able to process all 7 currently-hydrated elections by the end of next week or so.
Second, I was able to locate some additional manifestos, which will let us include nations that are not covered in the Manifesto Project. I'm excited about being able to broaden our set of elections. I have started by adding the 2021 Russian election (bringing us up to nine elections) and am working on more.
Thanks in part to our faster processing, I finished processing data from the UK election, and validated our news source scores using MBFC data. Using MBFC to evaluate the UK performance is worthwhile, but a bit more nuanced than the US case, as our task diverges a bit from MBFC’s labels. MBFC only provides bias scores on a single left-right scale, whereas our method provides bias scores for each pair of major parties (Labour, Conservative, and Liberal-Democrat for the UK). We can compute correlations between each party-pair score and the MBFC single-axis labels:
Unsurprisingly, the MBFC left-right scores most closely align with our measurement of the Labour-Conservative axis, which makes sense given that this cleavage is the most major of those in UK politics. Using this Labour-Conserative axis, it is encouraging that we still see fairly high absolute performance between Qwen 14b and MBFC scores in the UK context, despite the nuance described above.
To support more future longitudinal analyses, I've improved our pipeline to support more versions of Wikipedia articles to understand how they change over time. For all nine elections, I have now collected monthly article snapshots for the 6 month period around the election, plus a "recent" version for post-hoc analyses.
Finally, I completely overhauled our internal writeup of the project, archiving some older methodological notes that are no longer relevant, and writing up our latest pipeline for data collection and measurement in greater detail. I reorganized the document to better structure a paper manuscript, and added all our latest evaluation details and results for the US and UK elections using the current prompted classification pipeline: for the US election, we see robust results across all three models. Each model shows Wikipedia content (both candidate biographies and non-biographical election coverage) to be solidly in the middle of the range of US national news sources, with NPR being the most left-leaning, and Fox News being the most right-leaning of the news sources included in our analyses. We also see that in general, the candidate biography articles tend lean a bit more left than the non-biographical coverage. The purple “both” category shows the union of the set of candidate biographies and non-biographical coverage.
In the multiparty elections, the visualization becomes a bit more complex. I am using the three party-pair scores (for the three pairs of the three major UK parties) for each document to position each document inside a triangle (2-simplex) where each corner corresponds to one of the parties (a document scoring all the way towards labour on the labour-conservative axis, all the way towards labour on the labour-liberal dems axis, and evenly in the middle on the conservative-liberal democrat axis would be placed in the labour corner). When averaged by source, all the news and Wikipedia content ends up in the middle of this triangle, so it’s helpful to “zoom in” with our axes limits to see the differences between sources.
In the UK, we see two general trends: first: the Wikipedia content is generally in the middle of the labour-conservative axis, but supports the Liberal Democrat perspective slightly more than the news sources, and second, we see that over time, the Wikipedia content has become more neutral. A key caveat here is that the confidence intervals for much of the Wikipedia content overlap, so we should perform some more careful statistical testing before drawing any final conclusions.
In the US context, we see a similar trend: over time (comparing day-of-election revisions to recent revision) Wikipedia content has moved slightly towards the approximate center of the news sources’ content, although again we have overlap confidence intervals so shouldn’t read too much into it until we can dig more deeply. With the US news sources being more polarized, we should carefully consider the weighting question of how we weight news sources.
The one thing I didn't get to this week was the manual human labeling of a sample of news articles for human-model agreement measurement. I'll work on this next week, along with processing more nations, and continuing to write up results as we have them!
This week I made progress on a number of fronts. By far the most important was the synthesis of our results across many more elections. I finished processing news and Wikipedia data for 6 of our 9 elections, including back-processing more frequent revisions from Wikipedia. I spent time working on new visualizations to consistently display results across different elections with different numbers of major political parties (although I still need to do so for Italy, the only election so far with four major parties). Across all five of these elections (excluding Italy), we find that generally Wikipedia sits fairly neutrally, in the middle of the selected national news outlets, and that in general, Wikipedia has moved over time closer to the centroid of the news outlets’ slant, with a few key exceptions. In the UK, Wikipedia has moved a bit more towards the Liberal Democrats, whereas the major news outlets tend to be spread more along the primary Labour-Conservative axis. In Mexico, over time, the Wikipedia coverage has moved slightly away from the Movimiento Ciudadano party (Citizen’s Movement, a center-left party). In the US, France, and Japan, the changes over time are clearly towards the center of the news outlets.
In addition to this work, I also continued with writing and organizational work, spent some time researching possible African elections to include (Nigeria seems like the clear choice and I will add this to the scraping queue next week), and spent some time labeling news articles for IRR computation, which is a bit tedious but important to do. I still need to spend some more time labeling before I have robust results to report on this front.
Next week I will finish this manual labeling, continue with processing Brazil and Russian elections, add Nigeria, and focus on adding more detailed longitudinal analyse, as well as spend some time looking at trends in specific articles vs. averages of articles and how this contributes to changes over time. I'd like to see if I can replicate the finding from Greenstein and Zhu (2012) that:
Most articles arrive with a slant, and most articles change only mildly from their initial slant. The overall slant changes due to the entry of articles with opposite slants, leading toward neutrality for many topics, not necessarily within specific articles.
Between the PHP2Shell related disruptions this week and some urgent other work that came up last week, things have been a bit slower but I have still been making progress on a number of fronts. Most importantly, I added the 2023 Nigerian General Election to our set of elections, rounding us out at 10 elections, which seems like a reasonable place to pause. Chasing down manifestos for Brazil took a bit of work, but I found them (as well as for Russia and Nigeria), and completed MediaCloud hydration and Wiki content fetching for the new elections, meaning that we now have completed this process for every election. Due to PHP2Shell outages with LiftWing, I haven't yet been able to finish scoring documents for Brazil, Russia, and Nigeria, but will re-commence processing as soon as these services are restored.
In the interim, I've made more progress on human validation labeling and some additional writing tasks, and have started work on the analyses I proposed in my last comment from Oct 9 to see if changes in overall slant over time are driven by individual articles' changes in slant, or in new articles being added. I've approached this so far by simply examining mean slant and variance over time. For the 2024 US election there's no clear trend because slant tends to be quite stable over time at both levels, but I will continue to assess additional elections in the coming day or two.
Hi @GWeld,
In case any of it is reusable here. I am adding a related work from the task T428896 (WikiNPOV Drift Detector) we have been working on, we also discussed it last time !
Your note that slant "tends to be quite stable over time" matches what we found from the onset/temporal side: in our data the neutrality signal is static, not drifting. A later-tagged article already reads more subjective than a matched, never-tagged control a full year before the {{POV}} tag, with no rise toward it, what moves around a dispute is editor activity, not the prose.
Two things we built that may be reusable:
Project page: Research:WikiNPOV Drift Detector; code + data: https://github.com/mahi01agarwal/npov-drift-detector.
Another fun week of continuing to process data, update analyses, and continue to write.
For starters, I identified the issue that was preventing processing of new elections: an internal endpoint for LiftWing was removed, so I updated our tooling to use a different endpoint and verified this fix, then continued processing the documents from remaining elections. Nigeria and Russia are completed, Brazil is almost done, and South Korea is up next. Brazil is quite a bit larger than what we've processed so far, due to the large number of parties that meet our inclusion threshold (>=10% of the vote), but it should finish tomorrow at some point. I also located South Korean manifestos for the 2024 general election from the Manifesto project, as those are needed to start processing slant scores. Once South Korea is completed (by early next week) we will have finished all 10 elections.
For the new nations (Russia and Nigeria), we see that again, results generally are similar to the nations we've already assessed: Wikipedia coverage broadly sits in the middle of the various major national news sources', and has moved towards the news sources in the time since the election. English Wikipedia's coverage of the 2023 Nigerian general election includes only 40 articles, much fewer than many other elections, which limits our ability to make too many inferences here. We do see that Wikipedia generally sits a bit further away from the Nigerian All Progressives Congress than the national news outlets, which warrants a bit more looking in to. Nigeria also stands out in that their volume of Wikipedia coverage actually decreases slightly after the election (suggesting articles are being removed/shortened), which no other nation shows.
I continued my line of work from last week, looking at the change in slant of Wikipedia articles over time, by including all elections we've processed so far. As a reminder, I wanted to see if I can replicate the earlier finding from Greenstein and Zhu (2012) is that “Most [English Wikipedia] articles [covering US politics] arrive with a slant, and most articles change only mildly from their initial slant. The overall slant changes due to the entry of articles with opposite slants, leading toward neutrality for many topics, not necessarily within specific articles.” One thing we can assess, for each election, is how the mean slant changes over time alongside how the variance/standard deviation of the slant (across the set of all Wikipedia articles for that election) changes over time. If the variance decreases over time, with the mean holding constant or becoming more neutral, this suggests that individual articles are becoming less slanted. On the other hand, an increase in variance suggests that articles are becoming even more slanted, but if there are slanted articles on both (all) sides, then the overall mean can stay neutral.
A couple things stand out when we compute these results: first, there is substantial variation between elections in how much the amount of coverage increases over time. For the 2024 US election, on one hand, the number of articles barely increases during the 6 month period examined here. On the other hand, for example, the 2022 Italian, 2024 UK, and 2024 French elections show a steady increase in coverage that accelerates during the month before and after the election.
Second, we see that in general, the variance of article slant decreases mildly over time, with the 2022 Italian, 2024 French, and 2024 Mexican elections showing the largest decreases. Both English language elections here (2024 UK and 2024 US) show substantially smaller decreases over time. To me, the biggest difference here is between English Wikipedia and other language editions: enwiki seems more "mature" in its coverage compared to other languages, and as a result generally changes less over the 6 months surrounding an election. For other language editions, we do see some moderate evidence that article slant diminishes over time.
Next week I will dig into some of the Nigeria details, add Brazil and South Korea once they finish processing, and then spend my time putting together clearer visualizations and pulling out a narrative between multiple elections, as 10 elections starts to become a lot to compare between.
One note I forgot to include yesterday: I also finished this week an initial set of 20 manually labeled news articles from the 2024 US election to compare with the various models' labels and assess how the models' performance compares to a human labeler. This is only an initial analysis (as I will perform more labeling to improve precision and decrease the size of confidence intervals, but the initial results seem reasonable for performance on a very challenging, nuanced, and potentially noisy task:
| MODEL | IS-RELEVANT KAPPA | SPEARMAN CORRELATION | ICC |
|---|---|---|---|
| qwen3-14b | 0.483 | 0.454 | 0.489 |
| qwen3-27b | 0.459 | 0.651 | 0.632 |
| llama3.3-70b | 0.459 | 0.635 | 0.682 |
These values suggest that model-human agreement is moderate-to-good (depending on how you interpret various IRR metrics). The "Is-relevant" kappa is measuring performance only on the binary classification task of "is this news article relevant to the election," and the other metrics are assessing IRR of the slant ratings of relevant articles. ICC is probably the best metric for us to focus on, here, as Spearman will not show calibration issues (where the rank of articles doesn't change, even if the underlying slant values differ). We see that the larger models with more parameters tend to agree more closely with my labels. One thing that may be worth experimenting with is ensemble approaches or even averaging the results of multiple runs of the same model with non-zero temperature.
This week I have continued to work towards wrapping this project up. I finished processing Brazil and South Korea and looked at their results, and re-did many visualizations to simplify and polish them for inclusion in our paper draft, as I work to pull out and clarify the key aspects of the the results across all elections that are most salient. Right now those key points are that Wikipedia's coverage of elections is reliably less slanted (more neutral) than national news coverage in almost every election we evaluate, and that Wikipedia's coverage tends to sit `in the middle' of the spectrum of nation news outlets' slant. Furthermore, we find that, as Wikipedia is continuously updated over time, Wikipedia's coverage tends to move towards that of national news outlets', suggesting that Wikipedia's editing processes tend to converge on a shared national understanding of political events. Furthermore, we show that this ideological convergence tends to be the most pronounced in non-English Wikipedia language editions, where a smaller pool of editors may result in more dynamic editing and larger changes over time.
Mostly this week I focused on paper writing, and we now have a mostly-complete draft that I'll be iterating on based on internal collaborator feedback before sharing more publicly. The key thing that I need to work on next week is visualizations for 4-party elections (with 6 pairs of parties), along with plenty of additional writing polish.
One thing I had fun with today (I don’t think this warrants inclusion in the paper, but I think is neat) is visualizing the changes over time in Wikipedia slant as a “path” on triangle created by the major party’s manifestos. Looking for example at Russia, we see that Russian Wikipedia coverage has “oscillated” much more between the United Russia (current ruling party) and SR (“A Just Russia, a minority social democracy party), but has steadily moved away from the KPRF (Communist Party of the Russian Federation) over the 5 months leading up to the election, and this trend has continued since the election. Not worth digging too much more into (in part because we are likely at least partially just seeing trends from noise here, but I think it's interesting.
This week I made progress on a few more fronts: I continued to improve writing and visualizations, experimenting with a few more figure ideas, and rewriting a good chunk of the results section of the paper draft.
I also worked on back-filling additional news content for a few of the nations that had fewer articles and news sources: Russia, the US, and Mexico.
In addition to the above, I started a second round of querying against the WayBack machine for URLs that had previously timed-out or returned errors, to see how many of those attempts were one-off errors. This has worked pretty well, returning ~13,000 new articles for the US, and ~3,000 for Mexico. Russia still is a bit more challenging, so I can do some more digging into how Russian names are used, which is not something I'm particularly knowledgeable about - if anyone knows about Russian names, I'd appreciate suggestions!
These new articles should finish hydrating over the weekend, then I will process them with the qwen-14b model and update the figures and paper tables to reflect the new content.
I also have started processing all news and Wiki data using the larger qwen38-27b model hosted at the new LiftWing batch endpoint, to compare results with the 14b parameter model.