Convert work machines (tin, terbium) to Trusty and hhvm usage
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	EBernhardson
	Jan 16 2015, 5:30 PM

Description

We have a need to have production work machines (tin, terbium) to have HHVM (and thus Trusty). Use case is debugging code that only fails in a particular scenario present in production and difficult or impossible to reproduce elsewhere.

eg:

sudo -u www-data hhvm -m debug /var/www/w/MWScript.php extensions/Flow/maintenance/foo.php --wiki=somewiki

NOTE: This also needs to be mirrored into Beta (somewhere, currently we use deployment-bastion like tin, maybe making a terbium-like in Beta?).

create Trusty instance to replace deployment-bastion.
migrate to the new instance (Jenkins, scripts, whatever is hardcoded)

provide Trusty replacement for tin
provide Trusty replacement for terbium
migrate prod
update deployment / work doc

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	wikitech: make private settings file writable by owner	operations/puppet	production	+1 -1

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Legoktm	T75901 Drop PHP 5.3 support
Declined	• demon	T91590 [Spike] Try out hack (<?hh) for mediawiki-config
Resolved	Joe	T104147 can we get rid of rsvg security patch?
Resolved	Reedy	T94149 Get rid of Zend 5.5 tests for wmf branches
Resolved	None	T86081 Complete the use of HHVM over Zend PHP on the Wikimedia cluster
Declined	hashar	T64519 Upgrade git > 1.7
Resolved	• brooke	T69951 Vagrant+TMH: webm transcodes don't play in Firefox (upstream Ubuntu/avconv issue)
Resolved	None	T34317 PDF export extension fails to render Arabic characters in monospace font
Declined	None	T65899 Upgrade Wikimedia servers to Ubuntu Trusty (14.04)
Declined	None	T116345 Special:Version on Wikimedia wikis shows outdated commit hashes for submodules
Resolved	Dzahn	T123525 reduce amount of remaining Ubuntu 12.04 (precise) systems in production
Resolved	Joe	T87036 Convert work machines (tin, terbium) to Trusty and hhvm usage
Resolved	Joe	T116728 Reimage mw1152 as a terbium replacement
Resolved	Joe	T124024 Be able to switch programmatically between deployment servers in codfw and eqiad

Event Timeline

EBernhardson created this task.Jan 16 2015, 5:30 PM

EBernhardson raised the priority of this task from to Needs Triage.

EBernhardson updated the task description. (Show Details)

EBernhardson added a project: Release-Engineering-Team.

EBernhardson subscribed.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 16 2015, 5:30 PM

Note: terbium (and tin) are both on Precise, thus don't have hhvm either. Erik has been using mw1017 in prod for this use case. We should upgrade terbium as well.

greg triaged this task as Medium priority.Jan 16 2015, 8:18 PM

greg edited projects, added Beta-Cluster-Infrastructure; removed Release-Engineering-Team.

greg set Security to None.

greg moved this task from To Triage to Backlog on the Beta-Cluster-Infrastructure board.

Krenair subscribed.Jan 16 2015, 8:29 PM

Seems this should go to SRE , HHVM and MediaWiki-Core-Team and be rephrased to: "convert work machine (tin, terbium) to Trusty and hhvm usage" + mention beta cluster needs a new Trusty instance that replaces deployment-bastion.eqiad.wmflabs (tin equivalent).

greg renamed this task from Create a terbium clone for the beta cluster to Convert work machines (tin, terbium) to Trusty and hhvm usage.Jan 16 2015, 9:24 PM

greg updated the task description. (Show Details)

Thanks Greg. I have added some steps to the task description.

I could not find a project/Task related to the Trusty migration :-/

updated description again, to clarify that the scripts don't have any dependency on hhvm, it is being used for its gdb like debug console where you can set breakpoints and step through code as it runs from the CLI.

greg moved this task from Backlog to Externally Blocked on the Beta-Cluster-Infrastructure board.Jan 16 2015, 10:26 PM

hashar unsubscribed.Feb 3 2015, 12:24 PM

Or we might switch them over to Debian jessie right away? What do other ops think in this case? still trusty or jessie already?

Resolving this should also prevent reverts like https://gerrit.wikimedia.org/r/#/c/192866/

JanZerebecki subscribed.Feb 25 2015, 8:42 PM

Joe subscribed.Apr 7 2015, 3:25 PM

bd808 removed a project: MediaWiki-Core-Team.Apr 8 2015, 10:44 PM

Krenair mentioned this in T98813: Move wikitech (silver) to HHVM.May 12 2015, 12:52 AM

Krenair added a parent task: T86081: Complete the use of HHVM over Zend PHP on the Wikimedia cluster.May 12 2015, 12:55 AM

Ricordisamoa subscribed.May 12 2015, 9:56 AM

Liuxinyu970226 subscribed.May 19 2015, 10:35 PM

Krenair added a parent task: T65899: Upgrade Wikimedia servers to Ubuntu Trusty (14.04).May 30 2015, 1:40 PM

jeremyb subscribed.Jun 30 2015, 5:44 AM

Restricted Application added a subscriber: Matanya. · View Herald TranscriptJun 30 2015, 5:44 AM

Can't they go to jessie right away? I guess they can't because i hear we don't build HHVM for jessie.

Restricted Application added a subscriber: Luke081515. · View Herald TranscriptJul 13 2015, 8:58 PM

Trusty replacement for tin = mira?

Krenair mentioned this in T86081: Complete the use of HHVM over Zend PHP on the Wikimedia cluster.Sep 14 2015, 12:07 PM

Following on @Dzahn comment, should probably use Jessie instead of Trusty. If so:

rephrase the task summary
remove blocker T65899: Upgrade Wikimedia servers to Ubuntu Trusty (14.04)
maybe create a new tracking task for Jessie upgrade

Krinkle mentioned this in T112660: Support kafka in eventlogging client on terbium.Sep 16 2015, 8:41 PM

bd808 added a parent task: T116345: Special:Version on Wikimedia wikis shows outdated commit hashes for submodules.Oct 23 2015, 3:34 AM

In T87036#1640693, @hashar wrote:

Following on @Dzahn comment, should probably use Jessie instead of Trusty.

I don't think that Jessie is a good idea unless we are going to reimaging the MW servers to Jessie as well. Having the deployment staging server and the MW fleet running different operating systems sounds like a recipe for strange bugs caused by differing versions of PHP, HHVM, git, etc.

ori raised the priority of this task from Medium to High.Oct 23 2015, 6:33 AM

ori added a project: Blocked-on-Operations.

Addshore subscribed.Oct 23 2015, 6:47 AM

I will start working on this in the next couple of weeks.

My current plan for tin is to ask people to use mira instead of tin, as mira is already using trusty. Once everyone (at least in releng) feels confident, we can move on.

For terbium, I still need to understand how much work - if any - will be needed.

In T87036#1753813, @Joe wrote:

My current plan for tin is to ask people to use mira instead of tin, as mira is already using trusty.

We will have to either implement cross-master syncing in scap or abandon tin for scap and sync-* for this to work. I started a patch to support cross master syncing (https://gerrit.wikimedia.org/r/#/c/224313/) but it had permissions issues with some files when tested on the beta cluster. At least on deployment-bastion, there are some files under /srv/mediawiki-staging that are not owned by the mwdeploy group.

@Krenair did an audit on tin and found similar issues that need to be resolved:

In production on tin I found that there are some files you can't write to without being root. In particular:
./docroot/noc/createTxtFileSymlinks.sh
./wmf-config/db-codfw.php
./wmf-config/db-eqiad.php
./private/WikitechPrivateLdapSettings.php
./tests/multiversion/MWMultiVersionTest.php

All of these look to me to be things that we could just fix the perms of. The wikitech settings file will need a Puppet change to fix.

Based on this audit I think all of the files that have ownership issues on deployment-bastion are accidents of the history of that server rather than by design, so we can probably just chmod as needed there.

In T87036#1753813, @Joe wrote:

For terbium, I still need to understand how much work - if any - will be needed.

The biggest potential issue I know of for terbium is that setting /usr/bin/php to be HHVM may actually make some maintenance scripts run via cron slower due to HHVM's JIT overhead. This really should only effect very short running scripts however and is probably something we can fix on a case by case basis after the conversion by changing some /usr/bin/php to /usr/bin/php5 when needed.

In T87036#1754586, @bd808 wrote:

In T87036#1753813, @Joe wrote:

My current plan for tin is to ask people to use mira instead of tin, as mira is already using trusty.

We will have to either implement cross-master syncing in scap or abandon tin for scap and sync-* for this to work. I started a patch to support cross master syncing (https://gerrit.wikimedia.org/r/#/c/224313/) but it had permissions issues with some files when tested on the beta cluster. At least on deployment-bastion, there are some files under /srv/mediawiki-staging that are not owned by the mwdeploy group.

@Krenair did an audit on tin and found similar issues that need to be resolved:

In production on tin I found that there are some files you can't write to without being root. In particular:
./docroot/noc/createTxtFileSymlinks.sh
./wmf-config/db-codfw.php
./wmf-config/db-eqiad.php
./private/WikitechPrivateLdapSettings.php
./tests/multiversion/MWMultiVersionTest.php

All of these look to me to be things that we could just fix the perms of. The wikitech settings file will need a Puppet change to fix.

Based on this audit I think all of the files that have ownership issues on deployment-bastion are accidents of the history of that server rather than by design, so we can probably just chmod as needed there.

Let's jfdi and fix them then.

(Note that WikitechPrivateLdapSettings actually comes from puppet so I don't think we need to worry about changes to that)

@bd808 I will work on the wikitech settings right away, and I'll take a look at the other files as well.

And btw yes - I think we could anyways move to use mira for the time being instead of tin.

Change 249076 had a related patch set uploaded (by Giuseppe Lavagetto):
wikitech: make private settings file writable by owner

https://gerrit.wikimedia.org/r/249076

gerritbot added a project: Patch-For-Review.Oct 27 2015, 9:24 AM

Change 249076 merged by Giuseppe Lavagetto:
wikitech: make private settings file writable by owner

https://gerrit.wikimedia.org/r/249076

The ldap settings file has now permissions that should allow scap not to choke on it.

How do we want to proceed from here?

In T87036#1757899, @Joe wrote:

The ldap settings file has now permissions that should allow scap not to choke on it.

How do we want to proceed from here?

We need to finish out T104826: [scap] Add support for syncing /srv/mediawiki-staging including fully working git data to warm spare deploy server which currently needs a Puppet patch to be reviewed and merged (https://gerrit.wikimedia.org/r/#/c/224829/).