Page MenuHomePhabricator

Recent Posts

Changes and improvements to PHPUnit testing in MediaWiki

Written by kostajh on Tue, Jul 16, 4:13 AM in Doing the needful.

Building off the work done at the Prague Hackathon (T216260), we're happy to announce some significant changes and improvements to the PHP testing tools included with MediaWiki.


Production Excellence: May 2019

Written by Krinkle on Mon, Jul 1, 6:56 PM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!


Tracking down slow event handlers with Event Timing

Written by Gilles on Wed, Jun 19, 3:17 PM in The speed of thought.

We're taking part in the ongoing Event Timing Chrome origin trial, in order to experiment with that API early and give feedback to its designers. The goal of this upcoming API is to surface slow events. This is an area of web performance that hasn't gotten a lot of attention before, but one that can be very frustrating for users. Essentially, when slow events occur, users are trying to interact with the page and it's being unresponsive. Not a desirable user experience.


Performance perception: correlation to RUM metrics

Written by Gilles on Mon, Jun 17, 11:56 AM in The speed of thought.

When we set out to ask Wikipedia visitors their opinion of page load performance, our main hope was to answer an age-old question: which RUM metric matters the most to users? And more interestingly, which ones matter the most to our users on our content.


Performance perception: the effect of late-loading banners

Written by Gilles on Jun 13 2019, 2:12 PM in The speed of thought.

Unlike most websites, Wikipedia and its sister projects are ad-free. This is actually one of the reasons why our performance is so good. We don't have to deal with slow and invasive third-parties.


Production Excellence: April 2019

Written by Krinkle on May 31 2019, 7:21 PM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!


Performance perception: how satisfied are Wikipedia users?

Written by Gilles on May 29 2019, 5:17 PM in The speed of thought.

We've recently published research on performance perception that we did last year. The micro survey used in this study is still running on multiple Wikipedia languages and gives us insights into perceived performance.


Introducing the codehealth pipeline beta

Written by kostajh on May 14 2019, 8:29 PM in Doing the needful.

After many months of discussion, work and consultation across teams and departments[0], and with much gratitude and appreciation to the hard work and patience of @thcipriani and @hashar, the Code-Health-Metrics group is pleased to announce the introduction of the code health pipeline. The pipeline is currently in beta and enabled for GrowthExperiments, soon to be followed by Notifications, PageCuration, and StructuredDiscussions. (If you'd like to enable the pipeline for an extension you maintain or contribute to, please reach out to us via the comments on this post.)


Nova-network is gone!

Written by Andrew on May 2 2019, 9:05 PM in Clouds & Unicorns.

A couple of week ago we finally moved the last lingering VMs in our OpenStack platform from the nova-network region to the Neutron region (Blog Post: Neutron is here!). The bulk of the work had been done a month earlier, so the final nails in nova-network's coffin felt a bit anticlimactic -- nevertheless, this is a big step that represents a huge amount of work on the part of both staff and volunteers.


Production Excellence: March 2019

Written by Krinkle on Apr 21 2019, 6:51 PM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!


Evaluating Element Timing for Images

Written by Peter on Apr 10 2019, 8:24 AM in The speed of thought.

In the search for a better user experience metric, we have tried out the upcoming Element Timing for Images API in Chrome.


Quibble hibernated, it is time to flourish

Written by hashar on Mar 28 2019, 11:48 AM in Doing the needful.

Writing blog is neither my job nor something that I enjoy, I am thus late in the Quibble updates. The last one Blog Post: Quibble in summer has been written in September 2018 and I forgot to publish it until now. You might want to read it first to get a glance about some nice changes that got implemented last summer.


Quibble in summer

Written by hashar on Mar 28 2019, 10:42 AM in Doing the needful.

Note: this post has been published on 03/28 but has been originally written in September 2018 after Quibble 0.0.26 and never got published.


Switching production traffic to Apache Traffic Server

Written by ema on Mar 28 2019, 9:21 AM in The Traffic Blog.

The plan to replace Varnish as the on-disk HTTP cache component of our CDN with Apache Traffic Server is starting to take shape.


Autonomous Systems performance report

Written by Gilles on Mar 27 2019, 11:07 AM in The speed of thought.

Today we're publishing our first report of the performance experienced by visitors of Wikimedia websites, focused on the Autonomous Systems visitors are connecting from.


CI working group report, with recommendations of new tools to try

Written by LarsWirzenius on Mar 25 2019, 6:29 PM in Doing the needful.

The working group to consider future CI tooling for Wikimedia has finished and produced a report. The report is at and the short summary is that the release engineering team should do prototype implementations of Argo, GitLab CI/CD, and Zuul v3.


Production Excellence: February 2019

Written by Krinkle on Mar 21 2019, 7:11 PM in Doing the needful.

How’d we do in our strive for operational excellence? Read on to find out!


Help my CI job fails with exit status -11

Written by hashar on Mar 21 2019, 9:52 AM in Doing the needful.

For a few weeks, a CI job had PHPUnit tests abruptly ending with:


Migrating code from MediaWiki's ResourceLoader to Webpack

Written by Jdlrobson on Mar 13 2019, 3:15 PM in Leave it to the prose.

The lack of tooling or support for tooling has been causing problems in complicated code bases like the codebase for our mobile site, so we carved out a proposal to create a bridge from our existing codebase to a more modern one using Webpack. I'll talk about what we did and why.


Work progresses on CI tool evaluation

Written by LarsWirzenius on Mar 8 2019, 4:59 PM in Doing the needful.

The working group to consider future tooling for continuous integration is making progress (see previous blog post J148 for more information). We're looking at and evaluating alternatives and learning of new needs within WMF.


Choosing tools for continuous integration

Written by LarsWirzenius on Feb 28 2019, 6:27 PM in Doing the needful.

The Release Engineering team has started a working group to discuss and consider our future continuous integration tooling. Please help!


Projects, Forms and Subtypes oh my!

Written by mmodell on Feb 28 2019, 2:24 AM in Phabricating Phabricator.

Significant new functionality just landed in the wmf/stable branch of rPHAB Phabricator which resolves some minor headaches we've been living with for quite some time.


Phab Phebruary

Written by mmodell on Feb 22 2019, 7:23 PM in Phabricating Phabricator.

After many months with only a few minor updates deployed to the wmf/stable branch of Phabricator, we were long over-due for a major update. With All-hands 2019 behind us I was finally able to find the time to merge and deploy a huge batch of upstream changes.


Minimal MediaWiki for frontend engineers

Written by Jdlrobson on Feb 21 2019, 7:07 PM in Leave it to the prose.

I use OSX. Vagrant has not been kind to me, but I'm hopeful that Docker will make development a lot easier for me in future.
Until then, I use MAMP which provides a pretty easy LAMP setup. I wanted to share it with other frontend engineers as this minimal setup works well for me - it's fast, it minimises the extensions I need to update and most importantly brings me closer to problems with frontend end-users are experiencing.


Debugging production with X-Wikimedia-Debug

Written by kostajh on Feb 20 2019, 4:15 PM in The speed of thought.

In February 2018, a user reported that some topics created by users on Flow discussion boards were not appearing in the Recent Changes feeds, including EventStreams and the IRC-RC feed. Various automated patrol systems rely on EventStreams, so the bug meant a number of edits bypassed those systems on Flow-enabled wikis.


Perf Matters at Wikipedia 2015

Written by Krinkle on Feb 14 2019, 4:05 PM in The speed of thought.

This year we achieved another milestone in our multi-year effort to prepare Wikipedia for serving traffic from multiple data centres.


Production Excellence: January 2019

Written by Krinkle on Feb 13 2019, 3:53 AM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!


Production Excellence: December 2018

Written by Krinkle on Jan 22 2019, 2:54 AM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!


Gerrit now automatically adds reviewers

Written by hashar on Jan 17 2019, 4:53 PM in Doing the needful.

Finding reviewers for a change is often a challenge, especially for a newcomer or folks proposing changes to projects they are not familiar with. Since January 16th, 2019, Gerrit automatically adds reviewers on your behalf based on who last changed the code you are affecting.


Toolforge: Trusty deprecation and grid engine migration

Written by bd808 on Jan 12 2019, 12:51 AM in Clouds & Unicorns.

Ubuntu Trusty was released in April 2014, and support for it (including security updates) will cease in April 2019. We need to shut down all Trusty hosts before the end of support date to ensure that Toolforge remains a secure platform. This migration will take several months because many people still use the Trusty hosts and our users are working on tools in their spare time.


Code Health Metrics and SonarQube

Written by zeljkofilipin on Jan 10 2019, 2:54 PM in Doing the needful.
  1. Code Health

Migrating to HTTPS

Written by bd808 on Jan 3 2019, 9:06 PM in Clouds & Unicorns.

Starting 2019-01-03, GET and HEAD requests to will receive a 301 redirect to This change should be transparent to most visitors. Some webservices may need to be updated to use explicit https:// or protocol relative URLs for stylesheets, images, JavaScript, and other content that is rendered as part of the pages they serve to their visitors.


Why performance matters

Written by Imarlier on Dec 12 2018, 4:21 PM in The speed of thought.

There are practical reasons that web performance matters. From a user perspective, a site that’s slow results in frustration, annoyance, and ultimately a preference for alternatives. From the perspective of a site operator, frustrated users are users who aren’t going to return, and that makes it more difficult to accomplish your mission (be it commercial or public service). Optimizations keep people happy, keep them coming back, and keep them engaged[1].


Production Excellence: November 2018

Written by Krinkle on Dec 12 2018, 4:40 AM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!


Production Excellence: October 2018

Written by Krinkle on Nov 28 2018, 5:47 PM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!


Incident Documentation: An Unexpected Journey

Written by zeljkofilipin on Nov 22 2018, 6:06 PM in Doing the needful.

The Release Engineering team wants to continually improve the quality of our software over time. One of the ways in which we hoped to do that this year is by creating more useful Selenium smoke tests. (From now on, test will be used instead of Selenium test.) This blog post is about how we determined where the tests should focus and the relative priority.


Bring in 'da noise, bring in defunct. It's a zombie party!

Written by dduvall on Nov 16 2018, 7:22 PM in Doing the needful.

Halloween is a full two weeks behind us here in the United States, but it's still on my mind. It happens to be my favorite holiday, and I receive it both gleefully and somberly.


Machine learning: how to undersample the wrong way

Written by Gilles on Oct 15 2018, 1:37 PM in The speed of thought.

For the past couple of months, in collaboration with researchers, I've been applying machine learning to RUM metrics in order to model the microsurvey we've been running since June on some wikis. The goal being to gain some insight into which RUM metrics matter most to real users.

Read more... security incident

Written by JBennett on Oct 10 2018, 8:14 PM in Wikimedia Security Team.

What happened?
On September 24, 2018 a series of malicious edit attempts were detected on In general, these included attempts to inject malicious javascript, threatening messages and porn.


Best friends forever

Written by Peter on Oct 3 2018, 9:43 AM in The speed of thought.

We use both synthetic and RUM testing for Wikipedia. These two ways of testing performance are best friends and help us verify regressions. Today, we will look at two regressions where it helped us to get metrics both ways.


Neutron is here!

Written by Andrew on Sep 27 2018, 3:18 PM in Clouds & Unicorns.

As promised in an earlier post (Blog Post: Neutron is (finally) coming), we've started moving a few projects on our Cloud-VPS service into a new OpenStack region that is using Neutron for its software-defined networking layer. It's going pretty well! The new region, 'eqiad1', is currently very small, and growth is currently blocked by hardware issues (see T199125 for details) but we hope to resolve that issue soon.


Production Excellence: September 2018

Written by Krinkle on Sep 25 2018, 6:41 PM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!


Performance testing in a controlled lab environment - the metrics

Written by Peter on Sep 21 2018, 7:49 AM in The speed of thought.

One of the Performance Team responsibilities at Wikimedia is to keep track of Wikipedias performance. Why is performance important for us? In our case it is easy: We have so many users and if we have a performance regression, we are really affecting people's lives. Maybe you remember our hiring tweet from a couple of years ago?


An introduction to Task Types in Phabricator

Written by mmodell on Sep 20 2018, 5:22 PM in Doing the needful.

This blog post will describe a bit about how we are utilizing the "Task Types" feature in Phabricator to facilitate better tracking of work and to streamline workflows with custom fields. Additionally, I will be soliciting feedback about potential use-cases which could potentially take further advantage of this feature.


Additional details on OurMine

Written by JBennett on Sep 7 2018, 6:37 PM in Wikimedia Security Team.

The guard rails I'll be following will be around the original blog post created by Darian Patrick in November 2016. I'll do my best to fill in what gaps I can.


Neutron is (finally) coming

Written by Andrew on Aug 22 2018, 9:00 PM in Clouds & Unicorns.

Just as we were settling into nova-network (along with other early OpenStack adopters), the core developers were already moving on. A new project (originally named 'Quantum' but eventually renamed 'Neutron') would provide stand-alone APIs, independent from the Nova APIs, to construct all manners of software-defined networks. With every release Neutron became more elaborate and more reliable, and became the standard for networking in new OpenStack clouds.


Wikipedia goes 100% Forward Secret

Written by Vgutierrez on Aug 3 2018, 7:19 PM in The Traffic Blog.

On August 1st, we finished up a 6 week transition process which eliminated the last non-forward-secret cipher supported by our TLS termination, which was bare AES128-SHA. This change guarantees that all TLS connections to Wikipedia and its sister projects use cipher suites that provide forward secrecy. Wikipedia is the first major high-traffic site to take this step.


2017 Toolforge survey results

Written by bd808 on Jun 29 2018, 8:54 PM in Clouds & Unicorns.

Between 2017-11-20 and 2017-12-01, the Wikimedia Foundation ran a direct response user survey of registered Toolforge users. 141 email recipients participated in the survey which represents 11% of those who were contacted.


Mobile web performance: the importance of the device

Written by Gilles on Jun 22 2018, 2:19 PM in The speed of thought.

This week at our team offsite in Dublin, I looked at our performance data from an angle we haven't explored before: mobile device type. Most mobile devices expose their make and model in the User Agent string, which allows to look at data for a particular type of device. As per our data retention guidelines, we only keep user agent information for 90 days, but that's already plenty of data to draw conclusions.


mediawiki_selenium 1.8.1 Ruby Gem Released

Written by zeljkofilipin on Jun 14 2018, 3:05 PM in Doing the needful.

It has been a while since the last mediawiki_selenium release! 💎


Quibble in May

Written by hashar on Jun 1 2018, 8:36 PM in Doing the needful.

[Quibble] is the new test runner for MediaWiki (see the intro Blog Post: Introducing Quibble). This post is to give an update of what happened during May 2018.


Technical Debt - The Contagion Effect

Written by Jrbranaa on May 24 2018, 11:16 PM in Doing the needful.

One particularly interesting topic discussed during the Hackathon Technical Debt session (T194934) was that of the contagious aspect of technical debt. Although this makes sense in hindsight, it's not something that I had really given much thought to previously.


Thank you for taking the survey!

Written by egalvezwmf on May 3 2018, 7:00 PM in The surveyor.

I would like to share my deepest gratitude for everyone who responded to the Wikimedia Communities and Contributors Survey. The survey has closed for this year. The quality of the results has improved because more people responded this year. We heard from over 200 people who work in volunteer developer spaces like Phabricator, IRC, Mediawiki, mailing lists, and many others, which was a large increase from last year.


Status Update (May 2, 2018)

Written by awight on May 2 2018, 8:05 PM in Score all the things.


  • We've started work on JADE in earnest, and the prototype is deployed to the beta cluster where it's available for testing and tool development.
  • Draft topic prerequisites are mostly falling into place, so we should be able to get the initial model deployed this month.
  • New, dynamic ORES support table shows up-to-date information about our progress for each wiki:
  • ORES is served from its own cluster, which gave us a tremendous benefit in both performance and stability.
  • More ORES support for Arabic, Bengali, Catalan, Hungarian, Latvian, Swedish Wikipedia

Run Selenium tests using Quibble and Docker

Written by zeljkofilipin on May 2 2018, 1:46 PM in Doing the needful.

Dependencies are Git Python 3, and Docker Community Edition (CE).


Introducing Quibble

Written by hashar on Apr 30 2018, 9:09 AM in Doing the needful.

Running all tests for MediaWiki and matching what CI/Jenkins is running has been a constant challenge for everyone, myself included. Today I am introducing Quibble, a python script that clone MediaWiki, set it up and run test commands.


Final Reminder: Survey closing soon!

Written by egalvezwmf on Apr 20 2018, 5:48 PM in The surveyor.

The Wikimedia Communities and Contributors Survey is will close in less than three days on Sunday 22 April 2018.


Extension:Popups (Page Previews) front-end tooling

Written by Jhernandez on Apr 19 2018, 6:11 PM in Leave it to the prose.

Extension:Popups is a MediaWiki extension that shows previews when hovering a link in a popup.


Update: Global Wikimedia survey

Written by egalvezwmf on Apr 12 2018, 8:59 PM in The surveyor.

The Wikimedia Communities and Contributors Survey is still live! We have only heard from 50 Wikimedia volunteer developers. The survey will close Sunday 22 April 2018.


Cloud Services team Q3 FY17/18 highlights

Written by bd808 on Apr 9 2018, 3:44 AM in Clouds & Unicorns.

The Foundation fiscal quarter running from January 2018 through March 2018 was a busy one for the Cloud Services team:


mustache.js replaced with JavaScript template literals in Extension:Popups

Written by Niedzielski on Apr 3 2018, 5:21 PM in Leave it to the prose.

The Popups MediaWiki extension previously used HTML UI templates inflated by the mustache.js template system. This provided good readability but added an 8.1 KiB dependency* for functionality that was only used in a few places. We replaced Mustache with ES6 syntax without changing existing device support or readability and now ship 7.8 KiB less of minified uncompressed assets to desktop views where Popups was the only consumer.


Wikimedia Contributors Survey is here: Share your feedback by April 9

Written by mcruzWMF on Mar 29 2018, 1:58 AM in The surveyor.

Wikimedia Communities and Contributors survey is to be sent to participants around the world this week. If you are volunteer developer, and have contributed code to any pieces of MediaWiki, gadgets, or tools, please take 30 to 40 minutes to complete the survey:


Selenium tests in Node.js project retrospective

Written by zeljkofilipin on Mar 26 2018, 2:28 PM in Doing the needful.

I have been working on the project with more or less focus on it since 2015. Maybe the easiest way to follow the project is by taking a look at a few epic tasks:


Running red-queen-style

Written by Andrew on Mar 9 2018, 11:42 PM in Clouds & Unicorns.

I've spent the last few months building new web servers to support some of the basic WMCS web services: Wikitech, Horizon, and Toolsadmin. The new Wikitech service is already up and running; on Wednesday I hope to flip the last switch and move all public Horizon and Toolsadmin traffic to the new servers as well.


Thumbor support for private wikis deployed

Written by Gilles on Feb 22 2018, 10:34 AM in The speed of thought.

Yesterday we deployed Thumbor support for Wikimedia-hosted private wikis. While 99.9% of our traffic is for public-facing wikis, the Wikimedia Foundation hosts a number of private MediaWiki instances on the same infrastructure. Those wikis facilitate work for various groups in the movement, from community-run projects like OTRS, to local chapters, staff or the board. They're essential to the Wikimedia Movement, but by being private they're an architectural special case.


Phabricator Updates for February 2018

Written by mmodell on Feb 15 2018, 7:55 AM in Doing the needful.

This is a digest of the updates from several weeks of changelogs which are published upstream. This is an incomplete list as I've cherry-picked just the changes which I think will be of significant interest to end-users of Wikimedia's phabricator. Please see the upstream changelogs for a detailed overview of everything that's changed recently.


Status Update (January 30, 2018)

Written by awight on Jan 30 2018, 7:24 PM in Score all the things.


  • Deployed Revscoring 2.0. Each scoring model includes statistics that can be used to query and choose an appropriate threshold depending on the use case.
  • Rewrote ORES extension, improving code quality and test coverage. Failures will cause graceful degradation rather than breaking pages that rely on ORES.
  • GCI happened and some work has been done on wikilabels.
  • The ORES labs cluster has been migrated to Debian Stretch, and we're ready to migrate production clusters.
  • "draft topic" model is trained and it works. Support for the model in ORES is ongoing.
  • New languages, new campaigns, new models. We've deployed advanced edit quality models to Simple English, Spanish, and Swedish Wikipedia, Spanish Wikibooks, and basic edit quality to Icelandic Wikipedia and Spanish Wikiquote. Preliminary edit quality campaigns are finished for Hungarian and Serbian Wikipedia.
  • JADE (auditing system) work is continuing, we have a database schema designed, some code written for the backend service, and have planned an event-based architecture plus content-handled Jade and Jade_talk namespaces within MediaWiki.
  • Draftquality data is cached in the ORES extension and is made available to other extensions.

Measuring Wikipedia page load times

Written by Krinkle on Jan 9 2018, 6:25 PM in The speed of thought.

This post shows how we measure and interpret load times on Wikipedia. It also explains what real-user metrics are, and how percentiles work.


Ubuntu Trusty now deprecated for new WMCS instances

Written by Andrew on Nov 20 2017, 6:32 PM in Clouds & Unicorns.

Long ago, the Wikimedia Operations team made the decision to phase out use of Ubuntu servers in favor of Debian. It's a long, slow process that is still ongoing, but in production Trusty is running on an ever-shrinking minority of our servers.


The journey to Thumbor, part 3: development and deployment strategy

Written by Gilles on Nov 20 2017, 12:42 PM in The speed of thought.

In the last blog post I described where Thumbor fits in our media thumbnailing stack. Introducing Thumbor replaces an existing service, and as such it's important that it doesn't preform worse than its predecessor. We came up with a strategy to reach feature parity and ensure a launch that would be invisible to end users.


The journey to Thumbor, part 2: thumbnailing architecture

Written by Gilles on Nov 17 2017, 3:17 PM in The speed of thought.

Thumbor has now been serving all public thumbnail traffic for Wikimedia production since late June 2017.


Selenium Ruby framework deprecated

Written by zeljkofilipin on Oct 30 2017, 1:44 PM in Doing the needful.

This is your friendly but final warning that we are replacing Selenium tests written in Ruby with tests in Node.js. There will be no more reminders. Ruby stack will no longer be maintained. For more information see T139740 and T173488.


Tech talk: Selenium tests in Node.js

Written by zeljkofilipin on Oct 27 2017, 12:04 PM in Doing the needful.

Željko Filipin, Engineer (Contractor) from Release Engineering team. That's me! 👋


Status update (October 6, 2017)

Written by awight on Oct 18 2017, 5:56 PM in Score all the things.

New language support for Bengali, Greek, and Tamil. New advance edit quality support for Albanian and Romanian. We cleaned up the old 'reverted' models where better support is available. We're working on moving to a new dedicated cluster. We improved some models by exploring new sources of signal and cleaning datasets. We started work on JADE and presented on The Keilana Effect at Wikimania.


Automated OpenStack Testing, now with charts and graphs

Written by Andrew on Sep 29 2017, 9:26 PM in Clouds & Unicorns.

One of our quarterly goals was "Define a metric to track OpenStack system availability". Despite the weak phrasing, we elected to not only pick something to measure but also to actually measure it.


New Wiki Replica servers ready for use

Written by bd808 on Sep 25 2017, 11:43 PM in Clouds & Unicorns.

The current physical servers for the <wiki>_p Wiki Replica databases are at the end of their useful life. Work started over a year ago on a project involving the DBA team and cloud-services-team to replace these aging servers (T140788). Besides being five years old, the current servers have other issues that the DBA team took this opportunity to fix:

  • Data drift from production (T138967)
  • No way to give different levels of service for realtime applications vs analytics queries
  • No automatic failover to another server when one failed

Selenium Ruby framework deprecation (September)

Written by zeljkofilipin on Sep 25 2017, 3:27 PM in Doing the needful.

Originally an email sent on September 25 2017 to qa, engineering and wikitech-l mailing lists.


Selenium Ruby framework deprecation

Written by zeljkofilipin on Sep 25 2017, 3:14 PM in Doing the needful.

Originally an email sent on August 23 2017 to qa, engineering and wikitech-l mailing lists.


Selenium tests in Node.js

Written by zeljkofilipin on Sep 25 2017, 2:57 PM in Doing the needful.

Originally an-email sent on April 3 2017 to qa, engineering and wikitech-l mailing lists.


Introducing the Cloud Services Team: What we do, and how we can help you

Written by bd808 on Sep 13 2017, 6:44 PM in Clouds & Unicorns.

24% of Wikipedia edits over a three month period in 2016 were completed by software hosted in Cloud Services projects. In the same time period, 3.8 billion Action API requests were made from Cloud Services. We are the newly formed Cloud Services team at the Foundation, which maintains a stable and efficient public cloud hosting platform for technical projects relevant to the Wikimedia movement. --


Wikilabels incident: Reversed diffs!

Written by Halfak on Aug 31 2017, 2:02 PM in Score all the things.

Today, we discovered a major regression in Wikilabels. We've patched the issue and made an emergency deployment. We also deleted some labels that were saved while the system was compromised. In this post, we'll describe what happened.


More/better model information and "threshold optimizations"

Written by Halfak on Aug 29 2017, 10:41 PM in Score all the things.

Today, I'm writing to announce a breaking change in ORES that will come out about a month from now. It will only change how information about prediction models is stored and reported. This information is used by some tools to set thresholds at specified levels of confidence (e.g. "give me the threshold that gives 90% recall"). In this blog post, I'll explain how this is currently done and how it will be done once we deploy the change.


Tool creation added to

Written by bd808 on Aug 29 2017, 3:41 AM in Clouds & Unicorns. is a management interface for Toolforge users. On 2017-08-24, a new major update to the application was deployed which added support for creating new tool accounts and managing metadata associated with all tool accounts.


New dedicated puppetmasters for cloud instances

Written by Andrew on Aug 22 2017, 10:29 PM in Clouds & Unicorns.

Back in year zero of Wikimedia Labs, shockingly many services were confined to a single box. A server named 'virt0' hosted the Wikitech website, Keystone, Glance, Ldap, Rabbitmq, ran a puppetmaster, and did a bunch of other things.


Laughing ORES to death with regular expressions and fake threads

Written by Halfak on Aug 17 2017, 9:29 PM in Score all the things.

At 1100 UTC on June 23rd, ORES started to struggle. Within a half hour, it had fully choked and could no longer respond to any requests. It took us 10 hours to diagnose the problem, solve it, and consider it solved. We learned some valuable lessons when studying and addressing this issue.



Written by phuedx on Aug 16 2017, 12:58 PM in Leave it to the prose.

The Reading Web team recently discovered a bug in Firefox wherein a load event is fired when Firefox loads certain pages from its Back-Forward Cache (BFCache). To JavaScript on those pages, this event is a second load event (the first having been fired before the user navigated away from the page). This proved to be problematic for the cornerstone of our instrumentation, the EventLogging extension and delayed the deployment of Page Previews for approximately three months.


Toolforge provides proxied mirrors of cdnjs and now fontcdn, for your usage and user-privacy

Written by Quiddity on Aug 2 2017, 1:55 AM in Clouds & Unicorns.

Tool owners want to create accessible and pleasing tools. The choice of fonts has previously been difficult, because directly accessing Google's large collection of open source and freely licensed fonts required sharing personally identifiable information (PII) such as IPs, referrer headers, etc with a third-party (Google). Embedding external resources (fonts, css, javascript, images, etc) from any third-party into webpages hosted on Toolforge or other Cloud VPS projects causes a potential conflict with the Wikimedia Privacy Policy. Web browsers will attempt to load the resources automatically and this will in turn expose the user's IP address, User-Agent, and other information that is by default included in an HTTP request to the third-party. This sharing of data with a third-party is a violation of the default Privacy Policy. With explict consent Toolforge and Cloud VPS projects can collect and share some information, but it is difficult to secure that consent with respect to embedded resources.


Announcing the Scoring Platform team

Written by Halfak on Jul 21 2017, 4:46 PM in Score all the things.

The Wikimedia Foundation’s new Scoring Platform team, led by Aaron Halfaker, will be working on democratizing access to AI, developing new types of AI predictions, and pushing the state of the art with regards to ethical practice of AI development.


Toolforge Elasticsearch upgraded to 5.3.2

Written by bd808 on Jul 14 2017, 12:49 AM in Clouds & Unicorns.

The shared Elasticsearch cluster hosted in Toolforge was upgraded from 2.3.5 to 5.3.2 today (T164842). This upgrade comes with a lot of breaking API changes for clients and indexes, and should have been announced in advance. @bd808 apologizes for that oversight.


Status update (July 11th, 2017)

Written by Halfak on Jul 12 2017, 10:44 PM in Score all the things.

Two outages with documentation. Revscoring 2.0 coming with better model information and "thresholds". New support for Romanian, Albanian, Tamil, Greek, and Bengali. We're officially welcoming @awight to the team!


Official Debian Stretch image now available

Written by Andrew on Jun 20 2017, 4:00 PM in Clouds & Unicorns.

Debian Stretch was officially released on Saturday[1], and I've built a new Stretch base image for VPS use in the WMF cloud. All projects should now see an image type of 'debian-9.0-stretch' available when creating new instances.


The journey to Thumbor, part 1: rationale

Written by Gilles on Jun 20 2017, 3:33 PM in The speed of thought.

We are currently in the final stages of deploying Thumbor to Wikimedia production, where it will generate media thumbnails for all our public wikis. Up until now, MediaWiki was responsible for generating thumbnails.