Recent Posts

View All

One particularly interesting topic discussed during the Hackathon Technical Debt session (T194934) was that of the contagious aspect of technical debt. Although this makes sense in hindsight, it's not something that I had really given much thought to previously.

I would like to share my deepest gratitude for everyone who responded to the Wikimedia Communities and Contributors Survey. The survey has closed for this year. The quality of the results has improved because more people responded this year. We heard from over 200 people who work in volunteer developer spaces like Phabricator, IRC, Mediawiki, mailing lists, and many others, which was a large increase from last year.

Highlights

• We've started work on JADE in earnest, and the prototype is deployed to the beta cluster where it's available for testing and tool development.
• Draft topic prerequisites are mostly falling into place, so we should be able to get the initial model deployed this month.
• New, dynamic ORES support table shows up-to-date information about our progress for each wiki: https://tools.wmflabs.org/ores-support-checklist/
• ORES is served from its own cluster, which gave us a tremendous benefit in both performance and stability.
• More ORES support for Arabic, Bengali, Catalan, Hungarian, Latvian, Swedish Wikipedia

Dependencies are Git Python 3, and Docker Community Edition (CE).

Running all tests for MediaWiki and matching what CI/Jenkins is running has been a constant challenge for everyone, myself included. Today I am introducing Quibble, a python script that clone MediaWiki, set it up and run test commands.

The Wikimedia Communities and Contributors Survey is will close in less than three days on Sunday 22 April 2018.

Table of contents

Table of contents - Versions: HTML, Markdown

Table of contents - Versions: HTML, Markdown

Table of contents - Versions: HTML - Markdown

Extension:Popups is a MediaWiki extension that shows previews when hovering a link in a popup.

The Wikimedia Communities and Contributors Survey is still live! We have only heard from 50 Wikimedia volunteer developers. The survey will close Sunday 22 April 2018.

The Foundation fiscal quarter running from January 2018 through March 2018 was a busy one for the Cloud Services team:

The Popups MediaWiki extension previously used HTML UI templates inflated by the mustache.js template system. This provided good readability but added an 8.1 KiB dependency* for functionality that was only used in a few places. We replaced Mustache with ES6 syntax without changing existing device support or readability and now ship 7.8 KiB less of minified uncompressed assets to desktop views where Popups was the only consumer.

Wikimedia Communities and Contributors survey is to be sent to participants around the world this week. If you are volunteer developer, and have contributed code to any pieces of MediaWiki, gadgets, or tools, please take 30 to 40 minutes to complete the survey: https://wikimedia.qualtrics.com/jfe/form/SV_5ABs6WwrDHzAeLr?aud=DEV

I have been working on the project with more or less focus on it since 2015. Maybe the easiest way to follow the project is by taking a look at a few epic tasks:

I've spent the last few months building new web servers to support some of the basic WMCS web services: Wikitech, Horizon, and Toolsadmin. The new Wikitech service is already up and running; on Wednesday I hope to flip the last switch and move all public Horizon and Toolsadmin traffic to the new servers as well.

Yesterday we deployed Thumbor support for Wikimedia-hosted private wikis. While 99.9% of our traffic is for public-facing wikis, the Wikimedia Foundation hosts a number of private MediaWiki instances on the same infrastructure. Those wikis facilitate work for various groups in the movement, from community-run projects like OTRS, to local chapters, staff or the board. They're essential to the Wikimedia Movement, but by being private they're an architectural special case.

This is a digest of the updates from several weeks of changelogs which are published upstream. This is an incomplete list as I've cherry-picked just the changes which I think will be of significant interest to end-users of Wikimedia's phabricator. Please see the upstream changelogs for a detailed overview of everything that's changed recently.

Highlights

• Deployed Revscoring 2.0. Each scoring model includes statistics that can be used to query and choose an appropriate threshold depending on the use case.
• Rewrote ORES extension, improving code quality and test coverage. Failures will cause graceful degradation rather than breaking pages that rely on ORES.
• GCI happened and some work has been done on wikilabels.
• The ORES labs cluster has been migrated to Debian Stretch, and we're ready to migrate production clusters.
• "draft topic" model is trained and it works. Support for the model in ORES is ongoing.
• New languages, new campaigns, new models. We've deployed advanced edit quality models to Simple English, Spanish, and Swedish Wikipedia, Spanish Wikibooks, and basic edit quality to Icelandic Wikipedia and Spanish Wikiquote. Preliminary edit quality campaigns are finished for Hungarian and Serbian Wikipedia.
• JADE (auditing system) work is continuing, we have a database schema designed, some code written for the backend service, and have planned an event-based architecture plus content-handled Jade and Jade_talk namespaces within MediaWiki.
• Draftquality data is cached in the ORES extension and is made available to other extensions.

This post shows how we measure and interpret load times on Wikipedia. It also explains what real-user metrics are, and how percentiles work.

Long ago, the Wikimedia Operations team made the decision to phase out use of Ubuntu servers in favor of Debian. It's a long, slow process that is still ongoing, but in production Trusty is running on an ever-shrinking minority of our servers.

In the last blog post I described where Thumbor fits in our media thumbnailing stack. Introducing Thumbor replaces an existing service, and as such it's important that it doesn't preform worse than its predecessor. We came up with a strategy to reach feature parity and ensure a launch that would be invisible to end users.

Thumbor has now been serving all public thumbnail traffic for Wikimedia production since late June 2017.

This is your friendly but final warning that we are replacing Selenium tests written in Ruby with tests in Node.js. There will be no more reminders. Ruby stack will no longer be maintained. For more information see T139740 and T173488.

Željko Filipin, Engineer (Contractor) from Release Engineering team. That's me! 👋

New language support for Bengali, Greek, and Tamil. New advance edit quality support for Albanian and Romanian. We cleaned up the old 'reverted' models where better support is available. We're working on moving to a new dedicated cluster. We improved some models by exploring new sources of signal and cleaning datasets. We started work on JADE and presented on The Keilana Effect at Wikimania.

One of our quarterly goals was "Define a metric to track OpenStack system availability". Despite the weak phrasing, we elected to not only pick something to measure but also to actually measure it.

The current physical servers for the <wiki>_p Wiki Replica databases are at the end of their useful life. Work started over a year ago on a project involving the DBA team and cloud-services-team to replace these aging servers (T140788). Besides being five years old, the current servers have other issues that the DBA team took this opportunity to fix:

• Data drift from production (T138967)
• No way to give different levels of service for realtime applications vs analytics queries
• No automatic failover to another server when one failed

Originally an email sent on September 25 2017 to qa, engineering and wikitech-l mailing lists.

Originally an email sent on August 23 2017 to qa, engineering and wikitech-l mailing lists.

Originally an-email sent on April 3 2017 to qa, engineering and wikitech-l mailing lists.

24% of Wikipedia edits over a three month period in 2016 were completed by software hosted in Cloud Services projects. In the same time period, 3.8 billion Action API requests were made from Cloud Services. We are the newly formed Cloud Services team at the Foundation, which maintains a stable and efficient public cloud hosting platform for technical projects relevant to the Wikimedia movement. -- https://blog.wikimedia.org/2017/09/11/introducing-wikimedia-cloud-services/

Today, we discovered a major regression in Wikilabels. We've patched the issue and made an emergency deployment. We also deleted some labels that were saved while the system was compromised. In this post, we'll describe what happened.

Today, I'm writing to announce a breaking change in ORES that will come out about a month from now. It will only change how information about prediction models is stored and reported. This information is used by some tools to set thresholds at specified levels of confidence (e.g. "give me the threshold that gives 90% recall"). In this blog post, I'll explain how this is currently done and how it will be done once we deploy the change.

Toolsadmin.wikimedia.org is a management interface for Toolforge users. On 2017-08-24, a new major update to the application was deployed which added support for creating new tool accounts and managing metadata associated with all tool accounts.

Back in year zero of Wikimedia Labs, shockingly many services were confined to a single box. A server named 'virt0' hosted the Wikitech website, Keystone, Glance, Ldap, Rabbitmq, ran a puppetmaster, and did a bunch of other things.

At 1100 UTC on June 23rd, ORES started to struggle. Within a half hour, it had fully choked and could no longer respond to any requests. It took us 10 hours to diagnose the problem, solve it, and consider it solved. We learned some valuable lessons when studying and addressing this issue.

The Reading Web team recently discovered a bug in Firefox wherein a load event is fired when Firefox loads certain pages from its Back-Forward Cache (BFCache). To JavaScript on those pages, this event is a second load event (the first having been fired before the user navigated away from the page). This proved to be problematic for the cornerstone of our instrumentation, the EventLogging extension and delayed the deployment of Page Previews for approximately three months.

Tool owners want to create accessible and pleasing tools. The choice of fonts has previously been difficult, because directly accessing Google's large collection of open source and freely licensed fonts required sharing personally identifiable information (PII) such as IPs, referrer headers, etc with a third-party (Google). Embedding external resources (fonts, css, javascript, images, etc) from any third-party into webpages hosted on Toolforge or other Cloud VPS projects causes a potential conflict with the Wikimedia Privacy Policy. Web browsers will attempt to load the resources automatically and this will in turn expose the user's IP address, User-Agent, and other information that is by default included in an HTTP request to the third-party. This sharing of data with a third-party is a violation of the default Privacy Policy. With explict consent Toolforge and Cloud VPS projects can collect and share some information, but it is difficult to secure that consent with respect to embedded resources.

The Wikimedia Foundation’s new Scoring Platform team, led by Aaron Halfaker, will be working on democratizing access to AI, developing new types of AI predictions, and pushing the state of the art with regards to ethical practice of AI development.

The shared Elasticsearch cluster hosted in Toolforge was upgraded from 2.3.5 to 5.3.2 today (T164842). This upgrade comes with a lot of breaking API changes for clients and indexes, and should have been announced in advance. @bd808 apologizes for that oversight.

(reposted with minor edits from https://lists.wikimedia.org/pipermail/labs-l/2017-July/005036.html)

Two outages with documentation. Revscoring 2.0 coming with better model information and "thresholds". New support for Romanian, Albanian, Tamil, Greek, and Bengali. We're officially welcoming @awight to the team!

Debian Stretch was officially released on Saturday[1], and I've built a new Stretch base image for VPS use in the WMF cloud. All projects should now see an image type of 'debian-9.0-stretch' available when creating new instances.

We are currently in the final stages of deploying Thumbor to Wikimedia production, where it will generate media thumbnails for all our public wikis. Up until now, MediaWiki was responsible for generating thumbnails.

Back in the dark ages of Labs, all instance puppet configuration was handled using the puppet ldap backend. Each instance had a big record in ldap that handled DNS, puppet classes, puppet variables, etc. It was a bit clunky, but this monolithic setup allowed @yuvipanda to throw together a simple but very useful tool, 'watroles'. Watroles answered two questions:

The WMF's financial year and its annual plan are coming to an end, and one of the Performance team's goals this past year was to reduce the amount of time it takes to save an edit on a wiki.

One of the goals of the Wikimedia Performance Team is to improve the performance of MediaWiki and the broader software stack used on Wikimedia wikis. In this article we’ll describe a small performance improvement we’ve implemented for MediaWiki and recently deployed to production for Wikimedia. It highlights some of the unique problems we encounter on Wikimedia sites and how new web standards can be leveraged to improve performance.

The first very visible step in the plan to rename things away from the term 'labs' happened around 2017-06-05 15:00Z when IRC admins made the #wikimedia-labs irc channel on Freenode invite-only and setup an automatic redirect to the new #wikimedia-cloud channel.

Updates now coming to the phame blog! We made presentations and gathered new collaborators at the Wikimedia Hackathon 2017 in Vienna. ORES is back in api.php. Wikilabels has stats. ORES in CODFW fell over for a while, but it's back.

I wanted to let you know about an upcoming experimental Reddit AMA ("ask me anything") chat we have planned. It will focus on artificial intelligence on Wikipedia and how we're working to counteract vandalism while also making life better for newcomers.

In this update, I'm going to change some things up to try and make this update easier for you to consume. The biggest change you'll notice is that I've broken up the [#] references in each section. I hope that saves you some scrolling and confusion. You'll also notice that I have changed the subject line from "Revision scoring" to "Scoring Platform" because it's now clear that, come July, I'll be leading a new team with that name at the Wikimedia Foundation. There'll be an announcement about that coming once our budget is finalized. I'll try to keep this subject consistent for the foreseeable future so that your email clients will continue to group the updates into one big thread.

(This post was copied from https://lists.wikimedia.org/pipermail/ai/2017-March/000145.html)

I hosted the AI Wishlist session at the Developer Summit(T147710). At that session, we brainstormed a set of AIs that we think would be interesting to implement. Generally I asked people to do their best to follow template that would help us remember why the AI was important, what it would help with, and what resources might help get it implemented. See artificial-intelligence

(This post was copied from https://lists.wikimedia.org/pipermail/ai/2017-January/000130.html)

(The post was copied from https://lists.wikimedia.org/pipermail/ai/2016-November/000118.html)

(This post was copied from https://lists.wikimedia.org/pipermail/ai/2016-November/000117.html)

(This post was copied from https://lists.wikimedia.org/pipermail/ai/2016-November/000116.html)

(This post was copied from https://lists.wikimedia.org/pipermail/ai/2016-November/000115.html)

(This post was copied from https://lists.wikimedia.org/pipermail/ai/2016-November/000114.html)

(This post was copied from https://lists.wikimedia.org/pipermail/ai/2016-November/000113.html)

(This post was copied from https://lists.wikimedia.org/pipermail/ai/2016-October/000112.html)

(This post was copied from https://lists.wikimedia.org/pipermail/ai/2016-October/000111.html)

(This post was copied from https://lists.wikimedia.org/pipermail/ai/2016-October/000106.html)

(This post was copied from https://lists.wikimedia.org/pipermail/ai/2016-September/000102.html)

(This post was copied from https://lists.wikimedia.org/pipermail/ai/2016-September/000098.html)

(This post was copied from https://lists.wikimedia.org/pipermail/ai/2016-September/000095.html)

(This post was copied from https://lists.wikimedia.org/pipermail/ai/2016-September/000088.html)

(This post was copied from https://lists.wikimedia.org/pipermail/ai/2016-September/000087.html)

(This post was copied from https://lists.wikimedia.org/pipermail/ai/2016-September/000085.html)

We The Revision Scoring Team
are happy to announce the deployment of the ORES review tool as a beta feature on *English Wikipedia*. Once enabled, ORES highlights edits that are likely to be damaging in Special:RecentChanges, Special:Watchlist and Special:Contributions to help you prioritize your patrolling work. ORES detects damaging edits using a basic prediction model based on past damage.

(This post was copied from https://lists.wikimedia.org/pipermail/ai/2016-August/000068.html)

(This post was copied from https://lists.wikimedia.org/pipermail/ai/2016-August/000049.html)

(This post was copied from https://lists.wikimedia.org/pipermail/ai/2016-July/000039.html)

(This post was copied from https://lists.wikimedia.org/pipermail/ai/2016-June/000036.html)

(This post was copied from https://lists.wikimedia.org/pipermail/ai/2016-June/000033.html)

(This post was copied from https://lists.wikimedia.org/pipermail/ai/2016-June/000032.html)

(This post was copied from https://lists.wikimedia.org/pipermail/ai/2016-May/000030.html)

(This post was copied from https://lists.wikimedia.org/pipermail/ai/2016-May/000026.html)

(This post was copied from https://lists.wikimedia.org/pipermail/ai/2016-May/000022.html)

(This post was copied from https://lists.wikimedia.org/pipermail/ai/2016-April/000019.html)

Last week @Jdlrobson pinged me by email about a performance improvement his team noticed for large wiki articles on the mobile site in our synthetic tests run on WebPageTest. The improvement looked like this, a sudden drop in SpeedIndex (where lower is better):

I just finished deploying an update to Phabricator which includes a simple but rather useful feature:

The v0.37 build of rOSTW operations-software-tools-webservice has been deployed to Toolforge hosts and Tools-Kubernetes Docker images.

When @Ryan_Lane first built OpenStackManager and Wikitech, one of the first features he added was an interface to setup project-wide sudo policies via ldap.

For nearly a year, Horizon has supported instance management. It is altogether a better tool than the Special:NovaInstance page on Wikitech -- Horizon provides more useful status information for VMs, and has much better configuration management (for example changing security groups for already-running instances.)

I've just installed a new public base image, ' debian-9.0-stretch (experimental)' and made it available for all projects. It should appear in the standard 'Source' UI in Horizon any time you create a new VM.

Andrew will be upgrading our Openstack install from version 'Kilo' to version 'Liberty' on Tuesday the 2nd. The upgrade is scheduled to take up to three hours. Here's what to expect:

In T135327, the WMF Technical Collaboration team collected a list of Phabricator bugs and feature requests from the Wikimedia Developer Community. After identifying the most promising requests from the community, these were presented to Phacility (the organization that builds and maintains Phabricator) for sponsored prioritization.

If you are exclusively a user of tool labs, you can ignore this post. If you use or administer another labs project, this REQUIRES ACTION ON YOUR PART.

The Kubernetes ('k8s') backend for Tool Labs webservices is open to
beta testers from the community as a replacment for Grid Engine
(qsub/jsub).

The Wikimedia Legal team is interested in revising, updating, and clarifying the existing Labs Terms of Use governing developers and their projects on labs.

Starting Thursday May 12th, 13:00 PDT ( 20:00 GMT ) we will be having the first weekly Code Review office hours on freenode IRC in the #wikimedia-codereview channel.

horizon (OpenStack Dashboard) is the canonical implementation of OpenStack’s Dashboard, which provides a web based user interface to OpenStack services.

tools-login.wmflabs.org is on a new bastion host with twice
the RAM and CPU of the old one. This should hopefully provide a better
bandaid against it getting overloaded up. More discussion about a
longer term solution at https://phabricator.wikimedia.org/T131541

On Tuesday, 2016-04-05, we'll be upgrading Kubernetes to 1.2 and using
a different deployment method as well. While this should have no user
facing impact (ideally!) the following things might be flaky for a
period of time on that day:

Not a lot has changed for Wikimedia's instance of Phabricator over the past few months. That's because a lot has been happening behind the scenes, as well as upstream at Phacility. Members of the Release-Engineering-Team and Team-Practices group have been working since December 2015 to integrate various upstream changes, however, nothing was released to our production instance because there were so many important features that were in-progress and not yet fully usable. Additionally, we had to figure out exactly how these features would fit with the specific needs of our project and test a lot of functionality to be sure that we would not break anyone's workflows.