Recent Posts

View All

Some time ago I did the final work to move out the web performance tooling out of Amazon Web Services (AWS). It's been a long journey. Let me tell you how it all started.

Over the last months, the Release-Engineering-Team of the Wikimedia Foundation put efforts into making improvements and fixing issues in Wikimedia Phabricator, our main software planning software. Here is an incomplete list of achievements:

This is a reminder that this Sunday we in QTE together with Mozilla's performance team host the web performance developer room at FOSDEM. You can find the schedule at https://fosdem.org/2024/schedule/track/web-performance/ and the talks will be live streamed on this link starting 09.00 CET on Sunday. There are some really interesting talks and I am really looking forward to them!

For reasons already mentioned in other docs (eg. Eqiad Expansion Network Design) we’re moving towards a network architecture where the servers’ layer 3 domains (subnets) are constrained in each rack. Currently (and in most of our core DCs) those layer 3 domains are stretched across all the racks of a given row. In that setting, a Ganeti cluster of a given row (where its hypervisors are spread across the row) leverages this L2 adjacency to be able to live migrate VMs between hypervisors.
In other words, if work is going to be done on hypervisor1, all the VMs it hosts can be temporarily and transparently distributed across the other hypervisorX to prevent any disruptions. Having the same vlan trunked to all the hypervisors of the same row allows the VMs to move to a different hypervisor without requiring any IP renumbering and thus downtime.

Flashback to 2007: an engineer at Yahoo named Steve Souders studied users' time spent for different web sites and came to the conclusion that 80-90% of the load time of a web page is spent in the frontend (compared to the backend). He did a follow up 2012 and numbers were the same. He called it the web performance golden rule. What does that mean? Well, the conclusion was that to decrease the load time for users, start by optimising the frontend since there’s so much more time spent there.

I'm happy to announce that QTE together with the Mozilla performance team will host the web performance developer room at FOSDEM 2024! It will be half a day of web performance and will happen on the 4th of February 2024.

Sentence segmentation is a fundamental process in natural language processing. It involves breaking down a given text into individual sentences, a task that finds applications in various contexts. Whether you need to split a paragraph into sentences for further analysis or present sentence boundaries in a user-friendly frontend application, sentence segmentation is crucial.

Wikimedia had been using the Phabricator software since 2014 for project planning and task tracking. In June 2021, the company behind upstream Phabricator stopped operations.
In response, numerous Phabricator users started a community fork, taking a copy of the original free and open Phabricator source code to continue developing it under the project name Phorge.

Summary

• Install debugging packages: apt-get -y install php7.4-common-dbgsym php7.4-cli-dbgsym
• curl -o ~/php-gdbinit https://raw.githubusercontent.com/php/php-src/php-7.4.30/.gdbinit
• gdb <your php command>
• Enter run then once the command has failed: bt and zbacktrace

Network configuration is a quite rapidly evolving area which went through multiple phases. It’s also surprisingly tied to monitoring. Below is some historical context from the industry as well as what we’re doing in SRE.

Fuzheado, CC BY-SA 4.0, via Wikimedia Commons

If you've submitted patches for MediaWiki core, skins or extensions, you've seen this output in Gerrit:

1. Three sentence summary

Our code review system Gerrit has several caches, the largest ones being backed up on disk. The disk caches offload memory usage and persist the data between restarts. As a Java application, the caches are stored in H2 database files and I recently had to find how to connect to them in order to inspect their content and reduce their size.

Finally, after a few years, I've been to a conference. The last conference I went to was WebCamp Zagreb 2019. This was the first Testival conference that I have attended as a participant, not an organizer.

1. Three sentence summary

Mediawiki developers, have you ever thought, “I wish I could deploy my own code for Mediawiki”? Now you can! More deploys! More fun!

How are we doing in our strive for operational excellence? Read on to find out!

1. Three sentence summary

It was a very long time since the last in-person Testival meetup. Just before the COVID pandemic, we had meetup #58, in March 2020. Before that, we've been meeting regularly every month for years. We also had a conference every year. It feels really strange that there was no in-person meetup or a conference in over two years.

By @Mstyles and edited by @Cleo_Lemoisson

How are we doing in our strive for operational excellence? Read on to find out!

By: @mmartorana and @sbassett

As part of the desktop improvements project we spent time investing in the core code that powers skins. With support from volunteers (the majority of this support coming from the prolific @Ammarpad), we identified code patterns and made changes to the MediaWiki-Core-Skin-Architecture to retroactively define a data layer API for generating a skin.

Netbox is a tool used by all SREs, either directly or abstracted through cookbooks and various scripts. Managed by Infrastructure-Foundations, it went through a major (and much needed!) upgrade this past quarter, led by John Bond, myself and with the help of Riccardo.

How’d we do in our strive for operational excellence last month? Read on to find out!

Release Engineering's "GitLab-a-thon" sprint for May 10th-24th (roughly) focused on the mechanics of migrating a Wikimedia service to GitLab, setting up a CI pipeline, building container images from that service, and publishing images to the Wikimedia registry. We selected the Blubber project as a good candidate for experimentation:

1. Three sentence summary

How’d we do in our strive for operational excellence last month? Read on to find out!

Here I share some thoughts around the history of "responsive" MediaWiki skins and how we might want to think about it for Vector.

Section translation is a mobile first adaptation of ContentTranslation tool. It helps editors to translate sections from a source article to its corresponding article in another language using easy to use UI in a mobile interface. Translating content involves many steps such as choosing the right article, languages, sections to translate, cross checking with the existing article, selecting sentence, translating with the help of Machine translation, editing it and finally publishing. Designing and building such a complex workflow in the small mobile screen is a very challenging project.

How’d we do in our strive for operational excellence last month? Read on to find out!

Developers should own the process of putting their code into production. They should decide when to deploy, monitor their deployment, and make decisions about rollback.

Life is like riding a bicycle. To keep your balance you must keep moving. — Albert Einstien

Anyone who has never made a mistake has never tried anything new — Albert Einstein

Over here in the Release-Engineering-Team, Train Deployment is usually a rotating duty. We've written about it before, so I won't go into the exact process, but I want to tell you something new about it.

Every time I finish a mentorship, I read my blog posts about previous mentorships. Almost every time I finish a mentorship I write that I'll probably take a break from mentoring for a while, but then I don't. 😅

“Peace cannot be kept by force. It can only be achieved by understanding.” — Albert Einstein

How’d we do in our strive for operational excellence last month? Read on to find out!

I'll start with a bit of general administrivia. First, our migration of Wikimedia code review & CI to GitLab continues, and we're mindful that people could use regular updates on progress. Second, I need to think through some stuff about the project, and doing that in writing is helpful for all involved. I'm going to try writing occasional blog entries here for both purposes.

Lessons Learned in Software Testing signed by Bret Pettichord.

If you’ve ever experienced the pride of seeing your name on MediaWiki's contributor list, you've been involved in our deployment process (whether you knew it or not).

How’d we do in our strive for operational excellence last month? Read on to find out!

“Simplicity is the ultimate sophistication” - Leonardo da Vinci

Failures and setbacks are always part of life and sometimes it feels more like a roller coaster. You can either enjoy it or just spend the whole time screaming in fear. Those ups and downs in roller coasters are more similar to failures and success in life. If you think about it, it’s more like a journey where you have a starting point and a destination.

“A journey of a thousand miles begins with a single step” - Lao Tzu

How’d we do in our strive for operational excellence last month? Read on to find out!

1. The Book in Three Sentences

How’d we do in our strive for operational excellence last month? Read on to find out!

I have been working remotely since 2005. Recently, an intern has asked me for advice about it. I've decided to write a blog post. I hope the advice will be useful to a wide range of people. From people new to remote work (possibly because of COVID) to people with years of experience.

How’d we do in our strive for operational excellence last month? Read on to find out!

This post gives a quick introduction to a benchmarking tool, phpbench, ready for you to experiment with in core and skins/extensions.[1]

How’d we do in our strive for operational excellence last month? Read on to find out!

1. Reading Club

Smokers pressure you to light one, drinkers pressure you to drink one, we pressure you to publish one. -- Hrvoje Šimić

It has been my philosophy of life that difficulties vanish when faced boldly.
― Isaac Asimov

Last week I spoke to a few of my Wikimedia Foundation (WMF) colleagues about how we deploy code—I completely botched it. I got too complex too fast. It only hit me later—to explain deployments, I need to start with a lie.

In Blog Post: macOS vs Ubuntu Linux I've written about a fun experiment. I've dual booted an MacBook Air with macOS and Ubuntu. Then, I ran the same npm scripts on both operating systems. To make things more interesting, I've ran the scripts both on the host operating system and inside a Docker container. Running scripts on Ubuntu was so faster, that I wanted to dual boot my iMac too. At the time, I could not dual boot the iMac because my disk refused to be partitioned. In the last couple of weeks, I've formatted disks on both machines and did a clean install of both macOS and Ubuntu. It's time to run the experiment again!

How’d we do in our strive for operational excellence last month? Read on to find out!

The phoenix is a long-lived, immortal bird associated with Greek mythology (with analogs in many cultures) that cyclically regenerates or is otherwise born again.
Phoenix (mythology) - Wikipedia

Today is the last day of Google Summer of Code 2021. Vidhi Mody and Soham Parekh were mentoring Sahil Grewal on updating WebdriverIO to version 7 in Wikimedia repositories.

How’d we do in our strive for operational excellence last month? Read on to find out!

The Deployments page sets the working framework for QA workflow when it comes to deployment. Although the details and the level of QA work might be different from team to team depending on a specific team's workflow, specific projects and, even, specific phases of a project, the below is an outline of what seems to be, in a sense, typical QA activities for a weekly deploy cadence.

Beta cluster - the start

How’d we do in our strive for operational excellence last month? Read on to find out!

The release engineering team triages tasks flagged Release-Engineering-Team on a weekly basis. It is an all hands on deck one hour meeting in which we pick tasks one by one and find out what to do with them. We have started with more than a hundred of them and are now down to just a dozen or so, most filed since the last meeting.

“But what am I going to see?
I don't know. In a certain sense, it depends on you.”
― Stanislaw Lem, Solaris

There are two main purposes of the page object pattern.

How’d we do in our strive for operational excellence last month? Read on to find out!

For WMF staff's inspiration week, I decided to take a step back from my work building out a new skin architecture and a redesign of Vector and put myself into the shoes of a skin developer to see if the changes my team had made life easier. As a secondary objective, I was interested in how a MediaWiki skin could be written in Vue.js and what the challenges were to get there.

What makes a bug interesting? Any bug, by definition, has an element of a surprise (otherwise, it wouldn't be a bug). But not all bugs are equal in their ability to be surprising. Some bugs do stand out, presenting two essential elements of a surprise - unexpectedness and/or unusual manifestation. Based on my own classification criteria, I define a bug to be the most interesting if it is either (or both)
(1) difficult to find
(2) has a quite unexpected (mostly disruptive) impact

I do most of my work on a 27-inch iMac from 2012. It's a good desktop machine.

How’d we do in our strive for operational excellence last month? Read on to find out!

INTRO
When running QA on emails, the most important things to check are the dynamic content populating correctly for different user types, and ensuring there are no layout issues. Especially in an increasingly mobile world, it's important to review how an email will be displayed on a large variety of devices at once.

Since I was impressed with the depth of presentations at Breakpoint conference (at Browserstack) in July 2021 - see my post Breakpoint at BrowserStack conferences (2020-2021) - reflections and thoughts: Part 1 - Focus on Automation, it was really exciting to see that the next conference (March 2021) would be focusing on the intriguing topic of exploratory testing. The the presentations titles (here is the list of all presentations) looked equally exciting, for example

• "Manual testing is not dead...just the definition"
• "Whole team holistic quality"
• "Contemporary Exploratory Testing"
• "Building a testing culture"

This tip works if you're targeting:

• a local MediaWiki
  • MediaWiki-Docker
  • MediaWiki-Docker-Dev
  • MediaWiki-Vagrant
  • Quibble
• a remote wiki
  • Beta cluster
• continuous integration
  • Quibble
  • selenium-daily

How’d we do in our strive for operational excellence last month? Read on to find out!

“Improving daily work is even more important than doing daily work.”
― Gene Kim, The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win

Outreachy Round 21 has finished! Vidhi Mody, Soham Parekh and I were mentoring Harriet Ayugi on project Evaluation of Microsoft Playwright.

One of the critical pieces of our infrastructure is Gerrit. It hosts most of our git repositories and is the primary code review interface. Gerrit is written in the Java programming language which runs in the Java Virtual Machine (JVM). For a couple years we have been struggling with memory issues which eventually led to an unresponsive service and unattended restarts. The symptoms were the usual ones: the application responses being slower and degrading until server side errors render the service unusable. Eventually the JVM terminates with:

How’d we do in our strive for operational excellence last month? Read on to find out!

1. Table of Contents

“I felt in need of a great pilgrimage, so I sat still for three days.” Hafiz

How’d we do in our strive for operational excellence last month? Read on to find out!

How’d we do in our strive for operational excellence last month? Read on to find out!

The secret of getting ahead is getting started. The secret of getting started is breaking your complex overwhelming tasks into small manageable tasks, and then starting on the first one.
—Mark Twain

Two previous posts - Exempla Docent Part 1 and Part 2 outlined QA approach to testing functionality of the Suggested edits (SE) module on Special:Homepage. Part 1 explored testing ORES model-articletopic logic implementation and Part 2 described testing user workflows.

It has been more than 3 weeks into my Outreachy internship with Wikimedia foundations. The internship started well and the project that I'm working on is about evaluating Microsoft playwright as a possible replacement to the current automation testing framework being used. Week one was mostly about setting up the Wikimedia core by forking and cloning the Wikimedia core repository from Github. In order to simplify continuous integration, we are using Github as our code hosting platform to evaluate playwright instead of using Gerrit. The setup involved the following steps;

• Forking the Wikimedia repository
• Cloning the repository, setting up and running it on my local machine
• Connecting my forked with upstream
• Configuring CI.

How’d we do in our strive for operational excellence last month? Read on to find out!

Recently there has been a small effort on the Release-Engineering-Team to encode some of our institutional knowledge as runbooks linked from a page in the team's wiki space.

The year 2020 has been a year of massive change in the entire world, there are mixed feelings of loss, confusion among others, but all in all, there is always hope that keeps us moving forward. I must say that being accepted as an Outreachy intern has been that ray of light at the end of the tunnel that I needed to end the year and begin the new year. Outreachy is a paid, remote internship program with the goal to support people from groups underrepresented in tech. Starting my career in the field of software engineering has been a journey of hard work, persistence, and seizing every opportunity since where I come from such opportunities are rare and the support for women's engagement in technology is quite low.

In Part 1 of Exempla Docent for QA practices, some approaches to testing ORES model articletopic were explored. This post, as Part 2, will present an overview on testing Suggested edits module (SE) - the UI that presents the ORES articletopic logic to users (more info on Newcomers tasks on Special:Homepage).

October 26-29 2020 was my team's second virtual offsite. We've had many offsites, but the first virtual one was in May 2020. The structure of this offsite was similar to the one in May. About four hours of sessions every day, from Monday to Thursday.

In Blog Post: Google Summer of Code, June-August 2020 I've said:

How’d we do in our strive for operational excellence last month? Read on to find out!

If you want to experiment with Phabricator and/or the Phabricator APIs, it can be convenient to have a local instance to play with.

ORES provides machine learning as a service for Wikimedia projects. The ORES model articletopic was used for the Growth team project - Suggested edits for newcomers on Special: Homepage.

How’d we do in our strive for operational excellence last month? Read on to find out!

I've started using Vim in 2014. I've read a few books, wrote a few articles and gave a few talks on Vim. I really like editing text with Vim, but I was never happy with it's file management, extensions, things like that.

June and July were pretty busy. I was on vacation the majority of August. Interns and other mentors were busy even then. For more introduction, read my post Blog Post: Google Summer of Code, February-May 2020.

If you're making changes to a service that is deployed to Kubernetes, it sure is annoying to have to update the helm deployment-chart values with the newest image version before you deploy. At least, that's how I felt when developing on our dockerfile-generating service, blubber.