Recent Posts

View All

How’d we do in our strive for operational excellence last month? Read on to find out!

INTRO
When running QA on emails, the most important things to check are the dynamic content populating correctly for different user types, and ensuring there are no layout issues. Especially in an increasingly mobile world, it's important to review how an email will be displayed on a large variety of devices at once.

Since I was impressed with the depth of presentations at Breakpoint conference (at Browserstack) in July 2021 - see my post Breakpoint at BrowserStack conferences (2020-2021) - reflections and thoughts: Part 1 - Focus on Automation, it was really exciting to see that the next conference (March 2021) would be focusing on the intriguing topic of exploratory testing. The the presentations titles (here is the list of all presentations) looked equally exciting, for example

• "Manual testing is not dead...just the definition"
• "Whole team holistic quality"
• "Contemporary Exploratory Testing"
• "Building a testing culture"

This tip works if you're targeting:

• a local MediaWiki
  • MediaWiki-Docker
  • MediaWiki-Docker-Dev
  • MediaWiki-Vagrant
  • Quibble
• a remote wiki
  • Beta cluster
• continuous integration
  • Quibble
  • selenium-daily

How’d we do in our strive for operational excellence last month? Read on to find out!

“Improving daily work is even more important than doing daily work.”
― Gene Kim, The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win

Outreachy Round 21 has finished! Vidhi Mody, Soham Parekh and I were mentoring Harriet Ayugi on project Evaluation of Microsoft Playwright.

One of the critical pieces of our infrastructure is Gerrit. It hosts most of our git repositories and is the primary code review interface. Gerrit is written in the Java programming language which runs in the Java Virtual Machine (JVM). For a couple years we have been struggling with memory issues which eventually led to an unresponsive service and unattended restarts. The symptoms were the usual ones: the application responses being slower and degrading until server side errors render the service unusable. Eventually the JVM terminates with:

How’d we do in our strive for operational excellence last month? Read on to find out!

1. Table of Contents

“I felt in need of a great pilgrimage, so I sat still for three days.” Hafiz

How’d we do in our strive for operational excellence last month? Read on to find out!

How’d we do in our strive for operational excellence last month? Read on to find out!

The secret of getting ahead is getting started. The secret of getting started is breaking your complex overwhelming tasks into small manageable tasks, and then starting on the first one.
—Mark Twain

Two previous posts - Exempla Docent Part 1 and Part 2 outlined QA approach to testing functionality of the Suggested edits (SE) module on Special:Homepage. Part 1 explored testing ORES model-articletopic logic implementation and Part 2 described testing user workflows.

It has been more than 3 weeks into my Outreachy internship with Wikimedia foundations. The internship started well and the project that I'm working on is about evaluating Microsoft playwright as a possible replacement to the current automation testing framework being used. Week one was mostly about setting up the Wikimedia core by forking and cloning the Wikimedia core repository from Github. In order to simplify continuous integration, we are using Github as our code hosting platform to evaluate playwright instead of using Gerrit. The setup involved the following steps;

• Forking the Wikimedia repository
• Cloning the repository, setting up and running it on my local machine
• Connecting my forked with upstream
• Configuring CI.

How’d we do in our strive for operational excellence last month? Read on to find out!

Recently there has been a small effort on the Release-Engineering-Team to encode some of our institutional knowledge as runbooks linked from a page in the team's wiki space.

The year 2020 has been a year of massive change in the entire world, there are mixed feelings of loss, confusion among others, but all in all, there is always hope that keeps us moving forward. I must say that being accepted as an Outreachy intern has been that ray of light at the end of the tunnel that I needed to end the year and begin the new year. Outreachy is a paid, remote internship program with the goal to support people from groups underrepresented in tech. Starting my career in the field of software engineering has been a journey of hard work, persistence, and seizing every opportunity since where I come from such opportunities are rare and the support for women's engagement in technology is quite low.

In Part 1 of Exempla Docent for QA practices, some approaches to testing ORES model articletopic were explored. This post, as Part 2, will present an overview on testing Suggested edits module (SE) - the UI that presents the ORES articletopic logic to users (more info on Newcomers tasks on Special:Homepage).

October 26-29 2020 was my team's second virtual offsite. We've had many offsites, but the first virtual one was in May 2020. The structure of this offsite was similar to the one in May. About four hours of sessions every day, from Monday to Thursday.

In Blog Post: Google Summer of Code, June-August 2020 I've said:

How’d we do in our strive for operational excellence last month? Read on to find out!

If you want to experiment with Phabricator and/or the Phabricator APIs, it can be convenient to have a local instance to play with.

ORES provides machine learning as a service for Wikimedia projects. The ORES model articletopic was used for the Growth team project - Suggested edits for newcomers on Special: Homepage.

How’d we do in our strive for operational excellence last month? Read on to find out!

I've started using Vim in 2014. I've read a few books, wrote a few articles and gave a few talks on Vim. I really like editing text with Vim, but I was never happy with it's file management, extensions, things like that.

June and July were pretty busy. I was on vacation the majority of August. Interns and other mentors were busy even then. For more introduction, read my post Blog Post: Google Summer of Code, February-May 2020.

If you're making changes to a service that is deployed to Kubernetes, it sure is annoying to have to update the helm deployment-chart values with the newest image version before you deploy. At least, that's how I felt when developing on our dockerfile-generating service, blubber.

In February 2020 I've noticed an e-mail that Wikimedia is participating in Google-Summer-of-Code. Unfortunately, I've ignored the e-mail and soon forgot about it.

How’d we do in our strive for operational excellence last month? Read on to find out!

Since the late 90s, databases named Internet Routing Registries (IRR) have been trying to fulfill that (single) source of truth role. Unfortunately, they are subject to a lot of issues: fragmentation (many existing databases, not all equally well-maintained), security (some databases allow anyone to “claim” a prefix) and complexity (for the network operators). They also contain a lot of inaccurate data that have accumulated over time.

This project brought two major changes to our infrastructure. Firstly, servers that used to be fronted by LVS for load balancing are now peering directly with our routers. Secondly, we have started using IP anycast for a highly critical service: recursive DNS.

Phabricator upstream has implemented a search engine that does not depend on an external full-text index service. It's been the default in Phabricator for quite some time, however, we have been using the ElasticSearch engine for a few years now due to previous issues with the old engine. Specifically, the old phabricator fulltext engine depended on MySQL's built-in fulltext index functionality. Unfortunately, the fulltext engine in MySQL had inconsistent performance and often returned low quality search results.

HEADER CAPTION: The head of the Statue of Liberty on exhibit at the Paris World's Fair, 1878. The statue was built in France ahead of time, shipped overseas in crates, and then assembled in New York. Image by Albert Fernique / public domain.

How’d we do in our strive for operational excellence last month? Read on to find out!

Software development is pretty agile in its nature in that things normally tend to move pretty quickly. However, the faster you move, the more things break. As a codebase grows in size, its pieces become more and more complex, with every line adding a potential bug. In Wikimedia Foundation, we keep a handle on this through rigorous amounts of testing. Manual testing requires a lot of effort especially when you have a large core repository with a plethora of plugins and extensions that need to be tested. One of the hot frameworks on the scene is Cypress, a complete end to end testing solution.

“Imagine a world in which every single human being can freely share in the sum of all knowledge.”
~ Wikimedia Foundation

Today, we are writing to share the discovery and squashing of a bug that occurred earlier this year. This particular bug was also one of the rare instances in which we kept a Phabricator ticket private to address a security issue. To help address questions about when and why we make a security-related ticket private, we’re also sharing some insight into what happens when a private ticket about a security issue is closed.

In 2015 I noticed git fetches from our most active repositories to be unreasonably slow, sometimes up to a minute which hindered fast development and collaboration. You can read some of the debugging details I have conducted at the time on T103990. Gerrit upstream was aware of the issue and a workaround was presented though we never went to implement it.

How’d we do in our strive for operational excellence last month? Read on to find out!

Earlier today, the 600,000th commit was pushed to Wikimedia's Gerrit server. We thought we'd take this moment to reflect on the developer services we offer and our community of developers, be they Wikimedia staff, third party workers, or volunteers.

How are we doing on that strive for operational excellence during these unprecedented times?

This week a long-standing "bug" has been fixed in Phabricator.

How’d we do in our strive for operational excellence last month? Read on to find out!

We are dropping the wb_terms table, you might ask why we are doing it and why it’s such an important endeavour.

HEADER CAPTION: Screenshot from Wikimedia's famous Visual Editor. The typo "documenation" has a red squiggly line under it indicating the spell checker has automatically detected a spelling error by the author.

How’d we do in our strive for operational excellence last month? Read on to find out!

The Cloud Services team is in the process of updating and standardizing the use of DNS names throughout Cloud VPS projects and infrastructure, including the Toolforge project. A lot of this has to do with reducing our reliance on the badly-overloaded term 'Labs' in favor of the 'Cloud' naming scheme. The whole story can be found on this Wikitech proposal page. These changes will be trickling out over the coming weeks or months, but one change you might notice already.

Authors: S.Subramanya Sastry, C.Scott Ananian; Parsing Team; Wikimedia Foundation

In an effort to create a repeatable, streamlined process for consumption of security services the Security Team has been working on changes and improvements to our workflows. Much of this effort is an attempt to consolidate work intake for our team in order to more effectively communicate status, priority and scheduling. This is step 1 and we expect future changes as our tooling, capabilities and processes mature.

One of my longstanding pet peeves is that skin development for MediaWiki is so hard. I propose a radical change to how skins are installed and ask for feedback.

On 14 January 2020, staff at the Wikimedia Foundation discovered that a data file exported from the Wikimedia Phabricator installation, our engineering task and ticket tracking system, had been made publicly available. The file was leaked accidentally; there was no intrusion. We have no evidence that it was ever viewed or accessed. The Foundation's Security team immediately began investigating the incident and removing the related files. The data dump included limited non-public information such as private tickets, login access tokens, and the second factor of the two-factor authentication keys for Phabricator accounts. Passwords and full login information for Phabricator were not affected -- that information is stored in another, unaffected system.

How’d we do in our strive for operational excellence in November and December? Read on to find out!

In July 2016 the SRE team at Wikimedia Foundation began investigating Prometheus as a modern metrics framework. The experience has been very positive and has provided benefits both for technical debt elimination—such as deprecating Ganglia and Diamond—and adding support for newer technologies like Kubernetes.

WikimediaDebug is a set of tools for debugging and profiling MediaWiki web requests in a production environment. WikimediaDebug can be used through the accompanying browser extension, or from the command-line.

You may have noticed that Phabricator's real-time notification service ("Aphlict") is currently disabled. This means that you will not see real-time notification popups in Phabricator and workboard live-updates aren't happening.

How’d we do in our strive for operational excellence last month? Read on to find out!

How’d we do in our strive for operational excellence last month? Read on to find out!

In Changes and improvements to PHPUnit testing in MediaWiki, I wrote about efforts to help speed up PHPUnit code coverage generation for local development.[0] While this improves code coverage generation time for local development, it could be better.

This past week marks the release of a little tool that I've been working on for a while. In fact, it's something I've wanted to build for more than a year. But before I tell you about the solution, I need to describe the problem that I set out to solve.

How’d we do in our strive for operational excellence in August? Read on to find out!

This week saw the conclusion of a project that I've been shepherding on and off since September of last year. The goal was for the initialisation of our asynchronous JavaScript pipeline (at the time, 36 kilobytes in size) to fit within a budget of 28 KB – the size of two 14 KB bursts of Internet packets.

Last week, we completed a piece of long-neglected work relating to Puppet, the tool that manages the configuration of every virtual machine in our cloud. Historically, each VM has received its configuration from a physical, production server (the 'puppetmaster'). This meant that there was a constant chatter of traffic back and forth between each VM and unrelated networks and hardware sitting in Wikimedia production. Now, the puppetmasters are located on VMs, so all of that chatter is internal to Cloud Services.

How’re we doing on that strive for operational excellence? Read this first anniversary edition to find out!

How’d we do in our strive for operational excellence last month? Read on to find out!

This will be a brief introduction to this new feature which appeared recently on Phabricator workboards.

Building off the work done at the Prague Hackathon (T216260), we're happy to announce some significant changes and improvements to the PHP testing tools included with MediaWiki.

How’d we do in our strive for operational excellence last month? Read on to find out!

We're taking part in the ongoing Event Timing Chrome origin trial, in order to experiment with that API early and give feedback to its designers. The goal of this upcoming API is to surface slow events. This is an area of web performance that hasn't gotten a lot of attention before, but one that can be very frustrating for users. Essentially, when slow events occur, users are trying to interact with the page and it's being unresponsive. Not a desirable user experience.

When we set out to ask Wikipedia visitors their opinion of page load performance, our main hope was to answer an age-old question: which RUM metric matters the most to users? And more interestingly, which ones matter the most to our users on our content.

Unlike most websites, Wikipedia and its sister projects are ad-free. This is actually one of the reasons why our performance is so good. We don't have to deal with slow and invasive third-parties.

How’d we do in our strive for operational excellence last month? Read on to find out!

We've recently published research on performance perception that we did last year. The micro survey used in this study is still running on multiple Wikipedia languages and gives us insights into perceived performance.

After many months of discussion, work and consultation across teams and departments[0], and with much gratitude and appreciation to the hard work and patience of @thcipriani and @hashar, the Code-Health-Metrics group is pleased to announce the introduction of the code health pipeline. The pipeline is currently in beta and enabled for GrowthExperiments, soon to be followed by Notifications, PageCuration, and StructuredDiscussions. (If you'd like to enable the pipeline for an extension you maintain or contribute to, please reach out to us via the comments on this post.)

A couple of week ago we finally moved the last lingering VMs in our OpenStack platform from the nova-network region to the Neutron region (Blog Post: Neutron is here!). The bulk of the work had been done a month earlier, so the final nails in nova-network's coffin felt a bit anticlimactic -- nevertheless, this is a big step that represents a huge amount of work on the part of both staff and volunteers.

How’d we do in our strive for operational excellence last month? Read on to find out!

In the search for a better user experience metric, we have tried out the upcoming Element Timing for Images API in Chrome.

Writing blog is neither my job nor something that I enjoy, I am thus late in the Quibble updates. The last one Blog Post: Quibble in summer has been written in September 2018 and I forgot to publish it until now. You might want to read it first to get a glance about some nice changes that got implemented last summer.

Note: this post has been published on 03/28 but has been originally written in September 2018 after Quibble 0.0.26 and never got published.

The plan to replace Varnish as the on-disk HTTP cache component of our CDN with Apache Traffic Server is starting to take shape.

Today we're publishing our first report of the performance experienced by visitors of Wikimedia websites, focused on the Autonomous Systems visitors are connecting from.

The working group to consider future CI tooling for Wikimedia has finished and produced a report. The report is at https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/CI_Futures_WG/Report and the short summary is that the release engineering team should do prototype implementations of Argo, GitLab CI/CD, and Zuul v3.

How’d we do in our strive for operational excellence? Read on to find out!

For a few weeks, a CI job had PHPUnit tests abruptly ending with:

The lack of tooling or support for tooling has been causing problems in complicated code bases like the codebase for our mobile site, so we carved out a proposal to create a bridge from our existing codebase to a more modern one using Webpack. I'll talk about what we did and why.

The working group to consider future tooling for continuous integration is making progress (see previous blog post J148 for more information). We're looking at and evaluating alternatives and learning of new needs within WMF.

The Release Engineering team has started a working group to discuss and consider our future continuous integration tooling. Please help!

Significant new functionality just landed in the wmf/stable branch of rPHAB Phabricator which resolves some minor headaches we've been living with for quite some time.

After many months with only a few minor updates deployed to the wmf/stable branch of Phabricator, we were long over-due for a major update. With All-hands 2019 behind us I was finally able to find the time to merge and deploy a huge batch of upstream changes.

I use OSX. Vagrant has not been kind to me, but I'm hopeful that Docker will make development a lot easier for me in future.
Until then, I use MAMP which provides a pretty easy LAMP setup. I wanted to share it with other frontend engineers as this minimal setup works well for me - it's fast, it minimises the extensions I need to update and most importantly brings me closer to problems with frontend end-users are experiencing.

In February 2018, a user reported that some topics created by users on Flow discussion boards were not appearing in the Recent Changes feeds, including EventStreams and the IRC-RC feed. Various automated patrol systems rely on EventStreams, so the bug meant a number of edits bypassed those systems on Flow-enabled wikis.

This year we achieved another milestone in our multi-year effort to prepare Wikipedia for serving traffic from multiple data centres.

How’d we do in our strive for operational excellence last month? Read on to find out!

Previously posted on the 2018 Performance Calendar

How’d we do in our strive for operational excellence last month? Read on to find out!

Ubuntu Trusty was released in April 2014, and support for it (including security updates) will cease in April 2019. We need to shut down all Trusty hosts before the end of support date to ensure that Toolforge remains a secure platform. This migration will take several months because many people still use the Trusty hosts and our users are working on tools in their spare time.

1. Code Health

Starting 2019-01-03, GET and HEAD requests to http://tools.wmflabs.org will receive a 301 redirect to https://tools.wmflabs.org. This change should be transparent to most visitors. Some webservices may need to be updated to use explicit https:// or protocol relative URLs for stylesheets, images, JavaScript, and other content that is rendered as part of the pages they serve to their visitors.