Phame Blogs Clouds & Unicorns
Clouds & Unicorns
Wikimedia Cloud Services related news

New names for everyone!

Written by Andrew on Feb 18 2020, 9:14 PM.

The Cloud Services team is in the process of updating and standardizing the use of DNS names throughout Cloud VPS projects and infrastructure, including the Toolforge project. A lot of this has to do with reducing our reliance on the badly-overloaded term 'Labs' in favor of the 'Cloud' naming scheme. The whole story can be found on this Wikitech proposal page. These changes will be trickling out over the coming weeks or months, but one change you might notice already.


Cloud-vps Puppetmasters Moved to VMs, thanks to Krenair

Written by Andrew on Sep 18 2019, 9:54 PM.

Last week, we completed a piece of long-neglected work relating to Puppet, the tool that manages the configuration of every virtual machine in our cloud. Historically, each VM has received its configuration from a physical, production server (the 'puppetmaster'). This meant that there was a constant chatter of traffic back and forth between each VM and unrelated networks and hardware sitting in Wikimedia production. Now, the puppetmasters are located on VMs, so all of that chatter is internal to Cloud Services.


Nova-network is gone!

Written by Andrew on May 2 2019, 9:05 PM.

A couple of week ago we finally moved the last lingering VMs in our OpenStack platform from the nova-network region to the Neutron region (Blog Post: Neutron is here!). The bulk of the work had been done a month earlier, so the final nails in nova-network's coffin felt a bit anticlimactic -- nevertheless, this is a big step that represents a huge amount of work on the part of both staff and volunteers.


Toolforge: Trusty deprecation and grid engine migration

Written by bd808 on Jan 12 2019, 12:51 AM.

Ubuntu Trusty was released in April 2014, and support for it (including security updates) will cease in April 2019. We need to shut down all Trusty hosts before the end of support date to ensure that Toolforge remains a secure platform. This migration will take several months because many people still use the Trusty hosts and our users are working on tools in their spare time.


Migrating tools.wmflabs.org to HTTPS

Written by bd808 on Jan 3 2019, 9:06 PM.

Starting 2019-01-03, GET and HEAD requests to http://tools.wmflabs.org will receive a 301 redirect to https://tools.wmflabs.org. This change should be transparent to most visitors. Some webservices may need to be updated to use explicit https:// or protocol relative URLs for stylesheets, images, JavaScript, and other content that is rendered as part of the pages they serve to their visitors.


Neutron is here!

Written by Andrew on Sep 27 2018, 3:18 PM.

As promised in an earlier post (Blog Post: Neutron is (finally) coming), we've started moving a few projects on our Cloud-VPS service into a new OpenStack region that is using Neutron for its software-defined networking layer. It's going pretty well! The new region, 'eqiad1', is currently very small, and growth is currently blocked by hardware issues (see T199125 for details) but we hope to resolve that issue soon.


Neutron is (finally) coming

Written by Andrew on Aug 22 2018, 9:00 PM.

Just as we were settling into nova-network (along with other early OpenStack adopters), the core developers were already moving on. A new project (originally named 'Quantum' but eventually renamed 'Neutron') would provide stand-alone APIs, independent from the Nova APIs, to construct all manners of software-defined networks. With every release Neutron became more elaborate and more reliable, and became the standard for networking in new OpenStack clouds.


2017 Toolforge survey results

Written by bd808 on Jun 29 2018, 8:54 PM.

Between 2017-11-20 and 2017-12-01, the Wikimedia Foundation ran a direct response user survey of registered Toolforge users. 141 email recipients participated in the survey which represents 11% of those who were contacted.


Cloud Services team Q3 FY17/18 highlights

Written by bd808 on Apr 9 2018, 3:44 AM.

The Foundation fiscal quarter running from January 2018 through March 2018 was a busy one for the Cloud Services team:


Running red-queen-style

Written by Andrew on Mar 9 2018, 11:42 PM.

I've spent the last few months building new web servers to support some of the basic WMCS web services: Wikitech, Horizon, and Toolsadmin. The new Wikitech service is already up and running; on Wednesday I hope to flip the last switch and move all public Horizon and Toolsadmin traffic to the new servers as well.


Ubuntu Trusty now deprecated for new WMCS instances

Written by Andrew on Nov 20 2017, 6:32 PM.

Long ago, the Wikimedia Operations team made the decision to phase out use of Ubuntu servers in favor of Debian. It's a long, slow process that is still ongoing, but in production Trusty is running on an ever-shrinking minority of our servers.


Automated OpenStack Testing, now with charts and graphs

Written by Andrew on Sep 29 2017, 9:26 PM.

One of our quarterly goals was "Define a metric to track OpenStack system availability". Despite the weak phrasing, we elected to not only pick something to measure but also to actually measure it.


New Wiki Replica servers ready for use

Written by bd808 on Sep 25 2017, 11:43 PM.

The current physical servers for the <wiki>_p Wiki Replica databases are at the end of their useful life. Work started over a year ago on a project involving the DBA team and cloud-services-team to replace these aging servers (T140788). Besides being five years old, the current servers have other issues that the DBA team took this opportunity to fix:

  • Data drift from production (T138967)
  • No way to give different levels of service for realtime applications vs analytics queries
  • No automatic failover to another server when one failed

Introducing the Cloud Services Team: What we do, and how we can help you

Written by bd808 on Sep 13 2017, 6:44 PM.

24% of Wikipedia edits over a three month period in 2016 were completed by software hosted in Cloud Services projects. In the same time period, 3.8 billion Action API requests were made from Cloud Services. We are the newly formed Cloud Services team at the Foundation, which maintains a stable and efficient public cloud hosting platform for technical projects relevant to the Wikimedia movement. -- https://blog.wikimedia.org/2017/09/11/introducing-wikimedia-cloud-services/


Tool creation added to toolsadmin.wikimedia.org

Written by bd808 on Aug 29 2017, 3:41 AM.

Toolsadmin.wikimedia.org is a management interface for Toolforge users. On 2017-08-24, a new major update to the application was deployed which added support for creating new tool accounts and managing metadata associated with all tool accounts.


New dedicated puppetmasters for cloud instances

Written by Andrew on Aug 22 2017, 10:29 PM.

Back in year zero of Wikimedia Labs, shockingly many services were confined to a single box. A server named 'virt0' hosted the Wikitech website, Keystone, Glance, Ldap, Rabbitmq, ran a puppetmaster, and did a bunch of other things.


Toolforge provides proxied mirrors of cdnjs and now fontcdn, for your usage and user-privacy

Written by Quiddity on Aug 2 2017, 1:55 AM.

Tool owners want to create accessible and pleasing tools. The choice of fonts has previously been difficult, because directly accessing Google's large collection of open source and freely licensed fonts required sharing personally identifiable information (PII) such as IPs, referrer headers, etc with a third-party (Google). Embedding external resources (fonts, css, javascript, images, etc) from any third-party into webpages hosted on Toolforge or other Cloud VPS projects causes a potential conflict with the Wikimedia Privacy Policy. Web browsers will attempt to load the resources automatically and this will in turn expose the user's IP address, User-Agent, and other information that is by default included in an HTTP request to the third-party. This sharing of data with a third-party is a violation of the default Privacy Policy. With explict consent Toolforge and Cloud VPS projects can collect and share some information, but it is difficult to secure that consent with respect to embedded resources.


Toolforge Elasticsearch upgraded to 5.3.2

Written by bd808 on Jul 14 2017, 12:49 AM.

The shared Elasticsearch cluster hosted in Toolforge was upgraded from 2.3.5 to 5.3.2 today (T164842). This upgrade comes with a lot of breaking API changes for clients and indexes, and should have been announced in advance. @bd808 apologizes for that oversight.


Official Debian Stretch image now available

Written by Andrew on Jun 20 2017, 4:00 PM.

Debian Stretch was officially released on Saturday[1], and I've built a new Stretch base image for VPS use in the WMF cloud. All projects should now see an image type of 'debian-9.0-stretch' available when creating new instances.


Watroles returns! (In a different place and with a different name and totally different code.)

Written by Andrew on Jun 20 2017, 3:26 AM.

Back in the dark ages of Labs, all instance puppet configuration was handled using the puppet ldap backend. Each instance had a big record in ldap that handled DNS, puppet classes, puppet variables, etc. It was a bit clunky, but this monolithic setup allowed @yuvipanda to throw together a simple but very useful tool, 'watroles'. Watroles answered two questions:


#wikimedia-labs irc channel renamed to #wikimedia-cloud

Written by bd808 on Jun 5 2017, 3:11 PM.

The first very visible step in the plan to rename things away from the term 'labs' happened around 2017-06-05 15:00Z when IRC admins made the #wikimedia-labs irc channel on Freenode invite-only and setup an automatic redirect to the new #wikimedia-cloud channel.


Updated `webservice` command deployed

Written by bd808 on May 31 2017, 8:18 PM.

The v0.37 build of rOSTW operations-software-tools-webservice has been deployed to Toolforge hosts and Tools-Kubernetes Docker images.


Project-wide sudo policies in Horizon

Written by Andrew on May 30 2017, 8:02 PM.

When @Ryan_Lane first built OpenStackManager and Wikitech, one of the first features he added was an interface to setup project-wide sudo policies via ldap.


Manage Instance on Horizon (only)

Written by Andrew on May 26 2017, 7:40 PM.

For nearly a year, Horizon has supported instance management. It is altogether a better tool than the Special:NovaInstance page on Wikitech -- Horizon provides more useful status information for VMs, and has much better configuration management (for example changing security groups for already-running instances.)


Experimental Debian Stretch image now available

Written by Andrew on May 25 2017, 5:47 PM.

I've just installed a new public base image, ' debian-9.0-stretch (experimental)' and made it available for all projects. It should appear in the standard 'Source' UI in Horizon any time you create a new VM.


Labs Openstack upgrade on Tuesday, 2016-08-02, 16:00 UTC

Written by Andrew on Aug 1 2016, 5:01 PM.

Andrew will be upgrading our Openstack install from version 'Kilo' to version 'Liberty' on Tuesday the 2nd. The upgrade is scheduled to take up to three hours. Here's what to expect:


Labs is auditing and removing inactive projects.

Written by chasemp on Jul 8 2016, 4:20 PM.

If you are exclusively a user of tool labs, you can ignore this post. If you use or administer another labs project, this REQUIRES ACTION ON YOUR PART.


Kubernetes Webservice Backend Available for PHP webservices

Written by chasemp on Jul 8 2016, 4:19 PM.

The Kubernetes ('k8s') backend for Tool Labs webservices is open to
beta testers from the community as a replacment for Grid Engine


Community Consultation on Labs Terms of Use: Round 1

Written by chasemp on May 26 2016, 6:55 PM.

The Wikimedia Legal team is interested in revising, updating, and clarifying the existing Labs Terms of Use governing developers and their projects on labs.


Horizon is now the best UI for Labs/Tools

Written by chasemp on Apr 4 2016, 9:35 PM.

horizon (OpenStack Dashboard) is the canonical implementation of OpenStack’s Dashboard, which provides a web based user interface to OpenStack services.


New bastion at tools-login.wmflabs.org

Written by chasemp on Apr 4 2016, 7:51 PM.

tools-login.wmflabs.org is on a new bastion host with twice
the RAM and CPU of the old one. This should hopefully provide a better
bandaid against it getting overloaded up. More discussion about a
longer term solution at https://phabricator.wikimedia.org/T131541


Kubernetes to 1.2 on Tuesday, 2016-04-05

Written by chasemp on Apr 1 2016, 5:30 PM.

On Tuesday, 2016-04-05, we'll be upgrading Kubernetes to 1.2 and using
a different deployment method as well. While this should have no user
facing impact (ideally!) the following things might be flaky for a
period of time on that day:

About Clouds & Unicorns

The latest news and announcements for Wikimedia Cloud Services products and related ecosystem.

See also