Joe (Giuseppe Lavagetto)
Spy

Projects (22)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Oct 3 2014, 5:57 AM (189 w, 3 d)
Availability
Available
LDAP User
Giuseppe Lavagetto
MediaWiki User
Unknown

Recent Activity

Sun, May 20

Joe added a comment to T91820: Create HTTP verb and sticky cookie DC routing in VCL .

ChronologyProtector uses MySQLMasterPos, which can work both with a GTID-based master position or with the old binlog-based master position.

Sun, May 20, 5:06 AM · Wikimania-Hackathon-2018, Availability (MediaWiki-MultiDC), Operations, Traffic

Sat, May 19

Joe added a comment to T91820: Create HTTP verb and sticky cookie DC routing in VCL .

About ChronologyProtector:

Sat, May 19, 3:45 PM · Wikimania-Hackathon-2018, Availability (MediaWiki-MultiDC), Operations, Traffic
Joe added a comment to T190379: TechCom: Review development policies.

I will add to this list any other development policy I find on wikitech (and check them) as it could be interesting to have a single entry-point for our development policies.

Sat, May 19, 3:42 PM · TechCom
Joe added a comment to T194225: Enable mcrouter on the memcached servers themselves.

To explain my reasoning further: mcrouter needs a non-negligible amount of memory to run as it maintains an internal queue of messages whenever you use something like AllFastRoute or any other route handler that does distribution of keys. This means it can use a significant amount of memory from time to time, and I'd prefer to avoid having any process using a variable amount of memory on the memcached nodes.

Sat, May 19, 10:08 AM · Patch-For-Review, Performance-Team
Joe merged T194225: Enable mcrouter on the memcached servers themselves into T192771: mcrouter production architecture.
Sat, May 19, 9:03 AM · User-Joe, Patch-For-Review, Performance-Team (Radar), Availability (MediaWiki-MultiDC), Operations
Joe merged task T194225: Enable mcrouter on the memcached servers themselves into T192771: mcrouter production architecture.
Sat, May 19, 9:03 AM · Patch-For-Review, Performance-Team
Joe added a comment to T194225: Enable mcrouter on the memcached servers themselves.

Please see https://phabricator.wikimedia.org/T192771 which has a lot of considerations about the mcrouter architecture in production.

Sat, May 19, 9:03 AM · Patch-For-Review, Performance-Team

Wed, May 16

Joe added a comment to T194814: Remove unnecessary response headers.

The X-Powered-By part is actually useful for us in order to discern the source of rendering of a page - be it hhvm or php.

Wed, May 16, 10:31 AM · Patch-For-Review, media-storage, Operations, Traffic

Tue, May 15

Joe created T194724: Deprecate `base::service_unit` in puppet.
Tue, May 15, 6:08 AM · Patch-For-Review, cloud-services-team, User-Joe, Traffic, Cloud-Services, Operations, Puppet
Joe added a comment to T187962: Rack/cable/configure asw2-c-eqiad switch stack.

Things to watch out for:

Tue, May 15, 5:49 AM · Patch-For-Review, Operations, ops-eqiad, netops
Joe added a comment to T187962: Rack/cable/configure asw2-c-eqiad switch stack.

I would suggest we do NOT disable/depool anything but the obvious outlier in the databases (we already know that timeouts on the databases would cause a serious outage, because of bugs in MediaWiki).

Tue, May 15, 5:41 AM · Patch-For-Review, Operations, ops-eqiad, netops
Joe added a comment to T190893: Setup the webservice-related instances in toolsbeta.

An empty string should do the trick; or (better) you could convert that whole thing to use systemd::service instead, as proposed in the TODO.

Tue, May 15, 4:14 AM · Patch-For-Review, Toolforge

Mon, May 14

Joe added a comment to T187962: Rack/cable/configure asw2-c-eqiad switch stack.

The following servers:

Mon, May 14, 4:30 PM · Patch-For-Review, Operations, ops-eqiad, netops
Joe added a comment to T175288: setup/install/deploy deploy1001 as deployment server.

I suggest we do the following:

  • Pick a date/time frame of a few hours where no deployments are happening (or cancel existing ones)
  • We switch the deployment server to deploy1001 and test deployments using the PHP7 setup present there
  • If anything breaks, we revert to tin and fix whatever problem we found with PHP 7 and deploy1001 and re-attempt at a later stage
  • If everything works fine, we keep tin for a few weeks as a fallback and then decom it.
Mon, May 14, 4:08 PM · Patch-For-Review, Release-Engineering-Team (Watching / External), Operations
Joe moved T192771: mcrouter production architecture from Backlog to Doing on the User-Joe board.
Mon, May 14, 1:37 PM · User-Joe, Patch-For-Review, Performance-Team (Radar), Availability (MediaWiki-MultiDC), Operations
Joe moved T166081: rack/setup/install conf1004-conf1006 from Doing to Blocking others on the User-Joe board.
Mon, May 14, 1:36 PM · User-Joe, Operations
Joe added a project to T192771: mcrouter production architecture: User-Joe.
Mon, May 14, 12:19 PM · User-Joe, Patch-For-Review, Performance-Team (Radar), Availability (MediaWiki-MultiDC), Operations
Joe added a comment to T190893: Setup the webservice-related instances in toolsbeta.

ok I think I figured out the cause:
The regression happed in ops/puppet commit a504c49. In particular:

diff --git a/modules/profile/manifests/docker/flannel.pp b/modules/profile/manifests/docker/flannel.pp
index 4485001..89510b1 100644
--- a/modules/profile/manifests/docker/flannel.pp
+++ b/modules/profile/manifests/docker/flannel.pp
@@ -3,13 +3,12 @@ class profile::docker::flannel(
     # to the version in use.
     $docker_version = hiera('profile::flannel::docker_version'),
 ) {
+    # TODO: convert to systemd::service
     base::service_unit { 'docker':
         ensure           => present,
-        systemd          => true,
-        systemd_override => true,
+        systemd_override => init_template("docker/flannel/docker_${docker_version}", 'systemd_override'),
         # Restarts must always be manual, since restart
         # destroy all running containers. Fuck you, Docker.
         refresh          => false,
-        template_name    => "docker/flannel/docker_${docker_version}",
     }
 }
Mon, May 14, 10:39 AM · Patch-For-Review, Toolforge

Mon, May 7

Joe triaged T194031: Setup a new PKI software as an alternative to the puppet CA for managing services certificates as Normal priority.
Mon, May 7, 11:47 AM · Traffic, Operations
Joe added a comment to T192771: mcrouter production architecture.

After some tests:

Mon, May 7, 8:32 AM · User-Joe, Patch-For-Review, Performance-Team (Radar), Availability (MediaWiki-MultiDC), Operations

Sun, May 6

Joe added a comment to T176370: Migrate to PHP 7 in WMF production.

The decision to migrate the WMF production back to PHP 7.x is long taken and is not something we'd have done by choice: the HHVM platform after the 3.24 release doesn't guarantee full PHP compatibility, with differences that will make it very hard for anyone to write code that works on both.

Sun, May 6, 1:51 PM · TechCom-RFC (TechCom-Approved), User-ArielGlenn, HHVM, MediaWiki-Platform-Team, Operations

Wed, May 2

Joe added projects to T125976: Run mediawiki::maintenance scripts in Beta Cluster: Operations, User-Joe.
Wed, May 2, 12:16 PM · User-Joe, Operations, User-Addshore, Beta-Cluster-Infrastructure
Joe added a comment to T125976: Run mediawiki::maintenance scripts in Beta Cluster.

Is anyone working on this? If not, I guess this should be expedited to enable us to test running the maintenance scripts on php 7 in production as well, as hhvm is dog slow at running cli scripts and I see this as a priority.

Wed, May 2, 12:15 PM · User-Joe, Operations, User-Addshore, Beta-Cluster-Infrastructure
Joe added a comment to T192532: Figure out a way to enable volunteers to use the puppet compiler.

The compliler has little to do with @EddieGP's request, which seems sensible, and has to do with the jenkins permissions. I am not even a jenkins administrator anymore - this ticket must be handled by the Release Engineering team.

Wed, May 2, 4:31 AM · Release-Engineering-Team, Operations, Puppet, puppet-compiler, Continuous-Integration-Config

Mon, Apr 30

Joe added a comment to T176370: Migrate to PHP 7 in WMF production.

Dumps are already partially running on php 7 and have been thoroughly tested in the past months, so I'd leave them out of the equation.

Mon, Apr 30, 3:49 PM · TechCom-RFC (TechCom-Approved), User-ArielGlenn, HHVM, MediaWiki-Platform-Team, Operations
Joe added a comment to T192771: mcrouter production architecture.

I've an additional question, what is the expected behaviour in the following failure scenarios for each option?

  • One of the hosts that host mcrouter dies (and if there are differences on local DC vs remote DC)
  • The mcrouter process dies on one host (and if there are differences on local DC vs remote DC)
Mon, Apr 30, 12:37 PM · User-Joe, Patch-For-Review, Performance-Team (Radar), Availability (MediaWiki-MultiDC), Operations
Joe added a comment to T192771: mcrouter production architecture.

After some consideration, I see three options moving forward:

Mon, Apr 30, 9:33 AM · User-Joe, Patch-For-Review, Performance-Team (Radar), Availability (MediaWiki-MultiDC), Operations

Thu, Apr 26

Joe added a comment to T193112: Jobs writing to the Elasticsearch cluster in codfw are timing out, causing all type of issues.

The queue is getting back to normal sizes, and the job production is almost what I'd expect. Unfreezing writes solved the issue; I'll restore the correct state of redis replication once the queue has shrank enough.

Thu, Apr 26, 5:32 AM · Discovery-Search (Current work), Operations, Discovery, CirrusSearch, Search-Platform-Programs
Joe added a project to T193112: Jobs writing to the Elasticsearch cluster in codfw are timing out, causing all type of issues: Operations.
Thu, Apr 26, 4:36 AM · Discovery-Search (Current work), Operations, Discovery, CirrusSearch, Search-Platform-Programs
Joe triaged T193112: Jobs writing to the Elasticsearch cluster in codfw are timing out, causing all type of issues as Unbreak Now! priority.
Thu, Apr 26, 4:33 AM · Discovery-Search (Current work), Operations, Discovery, CirrusSearch, Search-Platform-Programs
Joe created T193112: Jobs writing to the Elasticsearch cluster in codfw are timing out, causing all type of issues.
Thu, Apr 26, 4:33 AM · Discovery-Search (Current work), Operations, Discovery, CirrusSearch, Search-Platform-Programs

Tue, Apr 24

Joe added a comment to T192185: request to assign spare systems as terbium equivalent.

WMF3565 is > 5 years old, so there's really no point in setting hardware that old right now.

How urgent is this task? We have a task open for procuring new hardware for (among other servers) terbium (T189317), but we could figure out a different solution if this needs to happen sooner than that.

We also have the two former image scalers which are under warranty (and even more powerful than terbium). The plan is to allocate them as mw* hosts later on (T192457), but we could just as well repurpose one of them as the stretch-based standin for terbium (since we need to have both in parallel for a small time window).

Tue, Apr 24, 10:23 AM · Patch-For-Review, hardware-requests, Operations
Joe closed T184854: hhvm memcached and php7 memcached extensions do not play well together as Resolved.
Tue, Apr 24, 5:08 AM · PHP 7.0 support, Performance-Team (Radar), User-ArielGlenn, MediaWiki-Platform-Team
Joe closed T184854: hhvm memcached and php7 memcached extensions do not play well together, a subtask of T176370: Migrate to PHP 7 in WMF production, as Resolved.
Tue, Apr 24, 5:07 AM · TechCom-RFC (TechCom-Approved), User-ArielGlenn, HHVM, MediaWiki-Platform-Team, Operations
Joe closed T184854: hhvm memcached and php7 memcached extensions do not play well together, a subtask of T184258: get a snapshot instance running in beta with stretch, php7, as Resolved.
Tue, Apr 24, 5:07 AM · Dumps-Generation, MediaWiki-General-or-Unknown
Joe added a comment to T184854: hhvm memcached and php7 memcached extensions do not play well together.

The rolling restart of all memcacheds is done. This ticket might be considered resolved.

Tue, Apr 24, 5:07 AM · PHP 7.0 support, Performance-Team (Radar), User-ArielGlenn, MediaWiki-Platform-Team

Mon, Apr 23

Joe created T192771: mcrouter production architecture.
Mon, Apr 23, 12:19 PM · User-Joe, Patch-For-Review, Performance-Team (Radar), Availability (MediaWiki-MultiDC), Operations
Joe created T192763: Create a prometheus exporter for mcrouter.
Mon, Apr 23, 10:37 AM · Patch-For-Review, Performance-Team (Radar), User-fgiunchedi, User-Joe, Availability (MediaWiki-MultiDC), Operations
Joe added a comment to T180462: puppet-compiler issue with CloudVPS instances.

Ideally we would get https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/ to be able to compile catalog for VM instances in Cloud VPS (for example, toolforge nodes).

AFAIK that doesn't work.

Mon, Apr 23, 7:55 AM · Cloud-VPS, cloud-services-team
Joe added a comment to T191921: mwscript rebuildLocalisationCache.php takes 40 minutes.

@thcipriani Hm.. these are seconds though, as opposed to minutes. Is there something different about these commands? I'm asking because on a short-lived process, JIT indeed adds more overhead than is worth. But if we're talking about a 40min process, JIT has got to speed things up. I'd be confused if it doesn't.

Mon, Apr 23, 6:56 AM · Patch-For-Review, Operations, Release-Engineering-Team (Kanban), Scap

Apr 19 2018

Joe added a comment to T181029: Upgrade dump hosts to stretch with php7.

Status update:

Apr 19 2018, 1:51 PM · Patch-For-Review, Dumps-Generation, MediaWiki-General-or-Unknown
Joe added a comment to T136732: Puppetize job that saves old versions of Maxmind geoIP database.

We could do that, but we wanted something centralized and reproducable (e.g. include a puppet class, get the historical dbs). We would have just put this as is in gerrit and auto-committed to it, but we can't host it anywhere publicly, since we pay for these files.

Apr 19 2018, 6:07 AM · Puppet, Patch-For-Review, Analytics-Kanban
Joe added a comment to T136732: Puppetize job that saves old versions of Maxmind geoIP database.

I would strongly suggest that any system that wants to archive geoip data from maxmind should create its own repository of data and NOT use puppet for it in any ways.

Apr 19 2018, 6:05 AM · Puppet, Patch-For-Review, Analytics-Kanban
Joe closed T162013: etcd cluster in codfw has raft consensus issues as Resolved.
Apr 19 2018, 6:00 AM · Patch-For-Review, User-Joe, Operations
Joe added a comment to T162013: etcd cluster in codfw has raft consensus issues.

We've had 3 mdadm checkarray full runs since we merged the change in february, and no alert went off in the meantime. I would be inclined to consider this successful.

Apr 19 2018, 5:59 AM · Patch-For-Review, User-Joe, Operations
Joe merged T181920: etcd-mirror failure into T162013: etcd cluster in codfw has raft consensus issues.
Apr 19 2018, 5:58 AM · Patch-For-Review, User-Joe, Operations
Joe merged task T181920: etcd-mirror failure into T162013: etcd cluster in codfw has raft consensus issues.
Apr 19 2018, 5:58 AM · Operations
Joe closed T180462: puppet-compiler issue with CloudVPS instances as Invalid.
Apr 19 2018, 5:57 AM · Cloud-VPS, cloud-services-team
Joe merged T172628: conf2002 etcdmirror-conftool-eqiad-wmnet died into T162013: etcd cluster in codfw has raft consensus issues.
Apr 19 2018, 5:56 AM · Patch-For-Review, User-Joe, Operations
Joe merged task T172628: conf2002 etcdmirror-conftool-eqiad-wmnet died into T162013: etcd cluster in codfw has raft consensus issues.
Apr 19 2018, 5:56 AM · Operations
Joe closed T122822: How can we enhance our production infrastructure using a cluster coordination tool like kubernetes as Resolved.
Apr 19 2018, 5:54 AM · Prod-Kubernetes (Experiment), Wikimedia-Developer-Summit-2016

Apr 18 2018

Joe added a comment to T192437: Pybal support of configuration from the kubernetes API.

Just to clarify, this is a bare support for kubernetes. In theory, it would be nice to gather all information about services we have to configure from kubernetes from the kubernetes API itself. That would mean to be able to dynamically define new pools, a feat now only done at startup. I think that should be left for a later time, as it would require deeper changes to pybal's architecture.

Apr 18 2018, 9:57 AM · Patch-For-Review, Traffic, Operations, Prod-Kubernetes, Pybal
Joe created T192437: Pybal support of configuration from the kubernetes API.
Apr 18 2018, 9:55 AM · Patch-For-Review, Traffic, Operations, Prod-Kubernetes, Pybal

Apr 17 2018

Joe closed T156924: Allow integration of data from etcd into the MediaWiki configuration as Resolved.
Apr 17 2018, 3:16 PM · MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), MediaWiki-Configuration, Performance-Team (Radar), Availability (MediaWiki-MultiDC), Services (watching), User-Joe, User-mobrovac, Operations
Joe closed T156924: Allow integration of data from etcd into the MediaWiki configuration, a subtask of T149617: Integrating MediaWiki (and other services) with dynamic configuration, as Resolved.
Apr 17 2018, 3:16 PM · Patch-For-Review, Performance-Team (Radar), Availability (MediaWiki-MultiDC), Services (watching), discovery-system, User-Joe, User-mobrovac, Operations
Joe moved T192370: Deploy mcrouter to production as a wancache backend from Backlog to Doing on the Availability (MediaWiki-MultiDC) board.
Apr 17 2018, 3:16 PM · Patch-For-Review, Performance-Team (Radar), Availability (MediaWiki-MultiDC), Operations
Joe created T192370: Deploy mcrouter to production as a wancache backend.
Apr 17 2018, 3:15 PM · Patch-For-Review, Performance-Team (Radar), Availability (MediaWiki-MultiDC), Operations
Joe added a comment to T184854: hhvm memcached and php7 memcached extensions do not play well together.

I performed various functional tests in deployment-prep.

Apr 17 2018, 10:29 AM · PHP 7.0 support, Performance-Team (Radar), User-ArielGlenn, MediaWiki-Platform-Team

Apr 16 2018

Joe closed T192071: Upgrade deployment-prep appserver fleet to Debian Stretch (using HHVM) as Resolved.
Apr 16 2018, 1:53 PM · Patch-For-Review, Beta-Cluster-Infrastructure, User-Joe, User-Elukey, HHVM, Operations
Joe closed T192071: Upgrade deployment-prep appserver fleet to Debian Stretch (using HHVM), a subtask of T174431: Upgrade mw* servers to Debian Stretch (using HHVM), as Resolved.
Apr 16 2018, 1:53 PM · Patch-For-Review, User-Elukey, HHVM, Operations
Joe added a comment to T188913: "Obama" page on Beta Cluster often responds with 503.

@Niedzielski interstingly, When requiring the /summary/precambrian page, I see a successful request to the API cluster, so the error is not a 503 on the part of MediaWiki, but rather some error in the api query or in the data.

Apr 16 2018, 1:51 PM · Operations, Beta-Cluster-Infrastructure
Joe added a comment to T192071: Upgrade deployment-prep appserver fleet to Debian Stretch (using HHVM).

All the main servers have been substituted with stretch VMs; the only one remaining turned on is deplyment-mediawiki06 which was used for some audit. I'll resolve the task now, and permanently delete the old instances during this week.

Apr 16 2018, 1:23 PM · Patch-For-Review, Beta-Cluster-Infrastructure, User-Joe, User-Elukey, HHVM, Operations
Joe added a comment to T189295: ICU 57 migration for wikis using non-default collation.

Enwiki finished it run at 14:40 UTC on saturday april 14th.

Apr 16 2018, 6:40 AM · User-ArielGlenn, Patch-For-Review, User-notice, User-Elukey, HHVM, Operations
Joe closed T189295: ICU 57 migration for wikis using non-default collation as Resolved.
Apr 16 2018, 6:39 AM · User-ArielGlenn, Patch-For-Review, User-notice, User-Elukey, HHVM, Operations
Joe closed T189295: ICU 57 migration for wikis using non-default collation, a subtask of T177498: Provide a forward port of ICU 52 for stretch / Investigate best ICU update strategy, as Resolved.
Apr 16 2018, 6:39 AM · MW-1.31-release-notes (WMF-deploy-2018-04-17 (1.31.0-wmf.30)), Patch-For-Review, User-Elukey, HHVM, Operations
Joe closed T189295: ICU 57 migration for wikis using non-default collation, a subtask of T182431: Switch category collation for sewiki to uca-se-u-kn, as Resolved.
Apr 16 2018, 6:39 AM · Wikimedia-Site-requests, Wikimedia-maintenance-script-run, WMNO-Northern-Sami
Joe closed T189295: ICU 57 migration for wikis using non-default collation, a subtask of T191776: Cannot reach further results via "Next page" link on category page on ruwp, always reloads current page, as Resolved.
Apr 16 2018, 6:39 AM · MediaWiki-Categories, MediaWiki-Interface

Apr 13 2018

Joe moved T192071: Upgrade deployment-prep appserver fleet to Debian Stretch (using HHVM) from Blocking others to Doing on the User-Joe board.
Apr 13 2018, 8:20 AM · Patch-For-Review, Beta-Cluster-Infrastructure, User-Joe, User-Elukey, HHVM, Operations
Joe moved T192071: Upgrade deployment-prep appserver fleet to Debian Stretch (using HHVM) from Backlog to Blocking others on the User-Joe board.
Apr 13 2018, 8:20 AM · Patch-For-Review, Beta-Cluster-Infrastructure, User-Joe, User-Elukey, HHVM, Operations

Apr 12 2018

Joe triaged T192071: Upgrade deployment-prep appserver fleet to Debian Stretch (using HHVM) as Normal priority.
Apr 12 2018, 11:32 AM · Patch-For-Review, Beta-Cluster-Infrastructure, User-Joe, User-Elukey, HHVM, Operations
Joe created T192059: CloudVPS: VMs created with non-allowed characters in the hostname fail to be autosigned by puppet.
Apr 12 2018, 8:30 AM · Horizon, cloud-services-team

Apr 11 2018

Joe updated the task description for T189295: ICU 57 migration for wikis using non-default collation.
Apr 11 2018, 6:11 AM · User-ArielGlenn, Patch-For-Review, User-notice, User-Elukey, HHVM, Operations
Joe added a comment to T191921: mwscript rebuildLocalisationCache.php takes 40 minutes.

What are the blockers for the use of PHP7?

Apr 11 2018, 5:18 AM · Patch-For-Review, Operations, Release-Engineering-Team (Kanban), Scap

Apr 10 2018

Joe closed T190979: build new version of mcrouter package as Resolved.
Apr 10 2018, 2:34 PM · Patch-For-Review, User-Joe, Operations
Joe closed T190979: build new version of mcrouter package, a subtask of T151466: Performance Q2 2017/18 goal: Install and use mcrouter in deployment-prep, as Resolved.
Apr 10 2018, 2:34 PM · Release-Engineering-Team (Watching / External), Availability (MediaWiki-MultiDC), Beta-Cluster-Infrastructure, Performance-Team
Joe added a comment to T190979: build new version of mcrouter package.

I built and uploaded two packages for 0.37.0, both in jessie and stretch. I will try to document the build process and automate it as much as possible.

Apr 10 2018, 2:34 PM · Patch-For-Review, User-Joe, Operations
Joe updated the task description for T189295: ICU 57 migration for wikis using non-default collation.
Apr 10 2018, 2:19 PM · User-ArielGlenn, Patch-For-Review, User-notice, User-Elukey, HHVM, Operations
Joe updated the task description for T189295: ICU 57 migration for wikis using non-default collation.
Apr 10 2018, 12:06 PM · User-ArielGlenn, Patch-For-Review, User-notice, User-Elukey, HHVM, Operations
Joe updated subscribers of T191393: Puppet: tlsproxy localssl default_server make a Notify at each run.

There is no way in puppet 4.x to do it better as far as @ema and I determined when we looked into it. So that notify is there for a good reason. It's a hack, it would be nice to be able to remove it, but I don't think this is a low-hanging fruit by any means.

Apr 10 2018, 10:00 AM · Traffic, Operations, Puppet
Joe closed Unknown Object (Task), a subtask of T188075: eqiad/codfw: (4)+(4) hardware access request for videoscalers, as Declined.
Apr 10 2018, 9:08 AM · hardware-requests, Operations
Joe closed Unknown Object (Task), a subtask of T188075: eqiad/codfw: (4)+(4) hardware access request for videoscalers, as Declined.
Apr 10 2018, 9:07 AM · hardware-requests, Operations
Joe updated the task description for T189295: ICU 57 migration for wikis using non-default collation.
Apr 10 2018, 8:59 AM · User-ArielGlenn, Patch-For-Review, User-notice, User-Elukey, HHVM, Operations
Joe added a comment to D1025: Force php5 in mwscript.

See my comments - I like the general idea but I think the behaviour of the code should be changed.

Apr 10 2018, 8:44 AM · Release-Engineering-Team
Joe added a comment to D1025: Force php5 in mwscript.
In D1025#20520, @demon wrote:

I disagree with making it a config variable. We should support the env variable, yes, but otherwise we should expect the system we're on to do the right thing. Configuring this feels like it's just gonna bite us later.

Apr 10 2018, 8:39 AM · Release-Engineering-Team
Joe added a comment to T190979: build new version of mcrouter package.

Status update:

Apr 10 2018, 5:48 AM · Patch-For-Review, User-Joe, Operations
Joe claimed T190979: build new version of mcrouter package.
Apr 10 2018, 5:47 AM · Patch-For-Review, User-Joe, Operations
Joe updated the task description for T189295: ICU 57 migration for wikis using non-default collation.
Apr 10 2018, 5:02 AM · User-ArielGlenn, Patch-For-Review, User-notice, User-Elukey, HHVM, Operations
Joe updated the task description for T189295: ICU 57 migration for wikis using non-default collation.
Apr 10 2018, 4:26 AM · User-ArielGlenn, Patch-For-Review, User-notice, User-Elukey, HHVM, Operations
Joe added a comment to T189295: ICU 57 migration for wikis using non-default collation.

Also a note on beta not running php7: when we migrated to HHVM it was made very clear to me and to Ori that we could not use beta for testing the migration; so I'm assuming we'll have to do the same this time as well and do the tests in production.

Apr 10 2018, 4:22 AM · User-ArielGlenn, Patch-For-Review, User-notice, User-Elukey, HHVM, Operations
Joe added a comment to T189295: ICU 57 migration for wikis using non-default collation.

Change 425027 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] profile::mediawiki::hhvm: default php to php7 on stretch

https://gerrit.wikimedia.org/r/425027

This is a very significant deviance from previous consensus (e.g. at T176370 and T174431). Can you clarify exactly how and where in production we now have MediaWiki code running under PHP 7, and why we can't migrate those to HHVM first?

This seems rather concerning given the unresolved tasks at PHP 7.0 support, including that neither neither Beta nor CI runs PHP 7 yet.

Apr 10 2018, 4:20 AM · User-ArielGlenn, Patch-For-Review, User-notice, User-Elukey, HHVM, Operations

Apr 9 2018

Joe updated the task description for T189295: ICU 57 migration for wikis using non-default collation.
Apr 9 2018, 3:50 PM · User-ArielGlenn, Patch-For-Review, User-notice, User-Elukey, HHVM, Operations
Joe updated the task description for T189295: ICU 57 migration for wikis using non-default collation.
Apr 9 2018, 3:00 PM · User-ArielGlenn, Patch-For-Review, User-notice, User-Elukey, HHVM, Operations
Joe updated the task description for T189295: ICU 57 migration for wikis using non-default collation.
Apr 9 2018, 1:47 PM · User-ArielGlenn, Patch-For-Review, User-notice, User-Elukey, HHVM, Operations
Joe updated the task description for T189295: ICU 57 migration for wikis using non-default collation.
Apr 9 2018, 1:31 PM · User-ArielGlenn, Patch-For-Review, User-notice, User-Elukey, HHVM, Operations
Joe added a comment to T189295: ICU 57 migration for wikis using non-default collation.

Since I've noticed a 45% speed increase when running the updateCollation.php script with php 7.0 versus HHVM, I'm temporarily setting up mw1338 to run the scripts; I will stop the videoscaler and puppet there for the time being.

Apr 9 2018, 12:26 PM · User-ArielGlenn, Patch-For-Review, User-notice, User-Elukey, HHVM, Operations
Joe added a comment to T189486: Announcing ICU 57 transition to the community.

Please note that since we can't run on more than one wiki in parallel per section, most wikis will complete their work within one hour, but they'll start after a long queue has completed. So expect the migration to be over in ~ 2 days on s2 and ~ 1.5 days on the other sections, with the notable exception of s1 (enwiki) - see https://phabricator.wikimedia.org/T189295#4115676 for a rough estimation of the total number of rows per section.

Apr 9 2018, 7:22 AM · Community-Liaisons (Apr-Jun-2018), User-notice
Joe added a comment to T189295: ICU 57 migration for wikis using non-default collation.

Total number of rows to sort through per shard:

Apr 9 2018, 6:45 AM · User-ArielGlenn, Patch-For-Review, User-notice, User-Elukey, HHVM, Operations
Joe closed T191107: deployment-etcd-01 puppet errors as Resolved.
Apr 9 2018, 6:05 AM · Patch-For-Review, Puppet, Beta-Cluster-Infrastructure
Joe closed T191107: deployment-etcd-01 puppet errors, a subtask of T132259: Deployment-prep hosts with puppet errors (tracking), as Resolved.
Apr 9 2018, 6:05 AM · Puppet, Tracking, Beta-Cluster-Infrastructure
Joe closed T144006: Move the MW Beta appservers to Debian as Resolved.
Apr 9 2018, 5:34 AM · Beta-Cluster-Infrastructure, HHVM, Operations