Page MenuHomePhabricator

cloudvps: wikitextexp project trusty deprecation
Closed, ResolvedPublic

Description

Ubuntu Trusty is no longer available in Cloud VPS since Nov 2017 for new instances. However, the EOL of Trusty is approaching in 2019 and we need to move to Debian Stretch before that date.

All instances in the wikitextexp project needs to upgrade as soon as possible.

The list of affected VMs is:

  • mw-base.wikitextexp.eqiad.wmflabs
  • mw-expt.wikitextexp.eqiad.wmflabs

Listed administrator are:

More info in openstack browser: https://tools.wmflabs.org/openstack-browser/project/wikitextexp

Event Timeline

Krenair triaged this task as Medium priority.Sep 17 2018, 5:01 PM
Krenair created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 17 2018, 5:01 PM

Thanks for the heads up. What is the best / recommended way to backup the mediawiki database and any other state from the VMs before they are wiped and updated to the newer version? Do I just download the contents to my laptop? Or, is there some labs script / service that does the update and restores disk state automatically?

Krenair added a comment.EditedSep 17 2018, 5:43 PM

I don't think you can wipe and reimage machines in labs like is done in prod, we just replace them from scratch. Theoretically you know how to reproduce their setup (and can change that to work on stretch). I would suggest tools like mysqldump (depending on whatever applications you have running there of course) and rsync to move the data (actually if it's MySQL you could also set up replication). It should be possible to set up SSH keys (with a system user on the target host) to move stuff between the instances without having to pull files down to your laptop and then send them straight back to the wikimedia network (you may need a /etc/security/access.conf entry on the target host to permit it in though). I believe there is also some rsync server thing lurking around somewhere but I'd have to ask about that.
People can set these machines up in any way they please (within the scope of the TOU), we don't mandate only certain supported software be installed, as such there is no single labs-specific script to just move everything. The administrators of a project are responsible for being able to reproduce the machines they set up, should they need to. So ideally these things all get worked out when the instance is being set up in the first place.

Most of the setup work was IIUC manual import of big chunks of existing wikis (with associated dependent templates, modules, etc) so that we can render pages faithfully and thus detect regressions. Is there a better way to do that now, maybe a "standard" way to get a read-only snapshot of the master DB onto a labs machine? We don't want to point directly at the master DBs, since then ordinary user edits create rendering changes that look like regressions in our rendering; but we'd like to pull the newest DB snapshot from time to time to ensure we don't fall too far behind the current state of the projects.

ssastry claimed this task.Nov 16 2018, 8:14 PM
ssastry raised the priority of this task from Medium to High.

Another ping. Deadline is approaching (2018-12-18).

Another ping. Deadline is approaching (2018-12-18).

I have this on my todo list for this week.

I started looking at this now. I have a couple questions.

  1. I suppose I cannot reuse the mw-base and mw-expt instance names since I will first have to spin up new instances (with new unique names), migrate over data to the new VM, and then shut down and delete the old instances. Am I right?
  2. When I click on 'Launch instance', I see this option to use an attached volume to persist data. This can potentially help us in the future when we need to upgrade the VMs again since we can shut down the instance, create a new instance and attach the volume. But, I don't see an option there to actually attach a volume? What am I missing?

And qn 3. Should I pick jessie or stretch for the image?

I started looking at this now. I have a couple questions.

  1. I suppose I cannot reuse the mw-base and mw-expt instance names since I will first have to spin up new instances (with new unique names), migrate over data to the new VM, and then shut down and delete the old instances. Am I right?

That is correct. I would personally recommend adopting a naming scheme like "wikitextexp-base-1002" & "wikitextexp-expt-1002" for your new instances.

  1. When I click on 'Launch instance', I see this option to use an attached volume to persist data. This can potentially help us in the future when we need to upgrade the VMs again since we can shut down the instance, create a new instance and attach the volume. But, I don't see an option there to actually attach a volume? What am I missing?

Cloud Services currently does not offer an attached block storage service. This is something on our long term product roadmap (which is kept in my head), but it is not available today.

And qn 3. Should I pick jessie or stretch for the image?

Stretch will have a longer supported life. We haven't planned a Jessie deprecation yet, but it will be coming sometime in the not too distant future as the main Wikimedia cluster replaces Jessie with Stretch and then starts to look at adding a newer stable Debian version for new projects.

Thanks. Will get on it now.

I am running vagrant up on both VMs .. and on both VMs, I see this error message ==> default: mesg: ttyname failed: Inappropriate ioctl for device .. the vagrant up command continued beyond that. But, is that something to be concerned about?

I am running vagrant up on both VMs .. and on both VMs, I see this error message ==> default: mesg: ttyname failed: Inappropriate ioctl for device .. the vagrant up command continued beyond that. But, is that something to be concerned about?

That's a known and benign issue with the initial boot of a new MediaWiki-Vagrant managed image. It happens because of a mesg n command in /root/.profile shipped with the base image that we add a guard condition to during the initial Puppet run.

ssastry added a comment.EditedDec 4 2018, 3:31 PM

Documenting steps here for posterity and for the next time we need to do this:

* Spin up new vms from https://horizon.wikimedia.org/project/instances/
* Click on each instance, click on the Puppet Configuration tab, find the mediawiki_vagrant class, and 'add class'
* Set up proxies for each of the wikis on each of the vms (or get cloud vps folks to run a script to do this for you - See T211367#4805061)
* ssh -A onto the old vms (mw-base, mw-expt in this case);
** cd /srv/mediawiki-vagrant
** vagrant ssh
** cd /vagrant
** mysqldump -A --add-drop-database | gzip > alldbs.mysql.gz
** logout
** scp alldbs.mysql.gz to new vm
** git format-patch -k HEAD~1..HEAD
** scp 0001-something-something.patch to new vms
** scp puppet/modules/local/manifests/wtexp_wiki.pp to new vms
** scp settings.d/* to new vms
** vagrant roles list | grep '*' to get a list of enabled vagrant roles
* ssh onto the new vms (wikitextexp-base-1002, wikitextexp-expt-1002 in this case)
** cd /srv/mediawiki-vagrant
** git apply old-vm.patch
** git commit -a
** echo "classes: [ 'local::wtexp_wiki' ]\nmediawiki::multiwiki::base_domain: "-base-wikitextexp.wmflabs.org"" > puppet/hieradata/local.yaml (Update suitable for each vm and to reflect actual base domains)
** vi settings.d/03-readonly.php and set to false
** vagrant roles enable $list-of-roles-from-old-vm
** vagrant up
** vagrant provision
** vagrant ssh
** cd /vagrant; gunzip < alldbs.mysql.gz | mysql
** for wiki in arwiki ckbwiki cuwiki cvwiki dewiki enwiki enwikisourcewiki enwikivoyagewiki enwiktionarywiki eswiki eswikisourcewiki eswikivoyagewiki eswiktionarywiki frwiki frwikisourcewiki frwikivoyagewiki frwiktionarywiki hewiki hiwiki hywiki iswiki itwiki itwikisourcewiki itwikivoyagewiki itwiktionarywiki jawiki kaawiki kawiki kowiki lbewiki lnwiki mysql mznwiki nlwiki plwiki pnbwiki ptwiki ruwiki svwiki ukwiki uzwikizhwiki wiki wikishared
do
echo "---- Updating $wiki ----"
mwscript maintenance/update.php --wiki $wiki --quick
done
** logout
* ssh to parsing-qa-01.eqiad.wmflabs
** vi /srv/visualdiff/diffserver/diffserver.js and update the URL paths for the base & expt vms if the url pattern changed with the new vms (nothing to do otherwise)
** sudo vi /etc/testreduce/mw-expts-vd-client.config.js and same as above
** sudo vi /etc/visualdiff/visualdiff-item.config.js and same as above

Also consult T120345#2094705 and T211367 if necessary.

Okay .. updated the list of steps above based on what I and @bd808 did here and as part of T211367.

I switched parsing-qa-01 testing to the mwexpts database (from the tidy-vs-remex database) and kicked off a new test run (T204566-test) .. watch results @ http://mw-expt-tests.wmflabs.org/ ... once this completes (in ~24 hours?), I'll know where we stand.

Something that is missing that I noticed right away is that the skins aren't localized on these new vms .. http://de-expt-wikitextexp.wmflabs.org/wiki/Main_Page vs http://de.expt.wikitextexp.wmflabs.org/wiki/Main_Page ... but, this may just be a matter of tweaking some mediawiki config (unless I am missing some vagrant role). @Arlolra fyi in case you know what this is since I remember you messed with this on the old vms.

Results are looking good so far. Since I've identical code and identical configs on both lab and base vms, this should be a baseline number with 100% green (barring some transient errors).

The results are good. http://mw-expt-tests.wmflabs.org/ -- the failures are known issues with either db import and/or transient failures .. some of which go away when repeatedly retried.

I am going to take a final look at the vms tomorrow and make sure eveyrthing is transferred over and shut down the old vms.

Something that is missing that I noticed right away is that the skins aren't localized on these new vms .. http://de-expt-wikitextexp.wmflabs.org/wiki/Main_Page vs http://de.expt.wikitextexp.wmflabs.org/wiki/Main_Page ... but, this may just be a matter of tweaking some mediawiki config (unless I am missing some vagrant role). @Arlolra fyi in case you know what this is since I remember you messed with this on the old vms.

If memory serves me, something is detecting your Accept-Language header and giving you what you ask for. @Legoktm figured that out last time and helped me disable it. I think it was a setting from UniversalLanguageSelector.

$wgULSLanguageDetection = false;

Something that is missing that I noticed right away is that the skins aren't localized on these new vms .. http://de-expt-wikitextexp.wmflabs.org/wiki/Main_Page vs http://de.expt.wikitextexp.wmflabs.org/wiki/Main_Page ... but, this may just be a matter of tweaking some mediawiki config (unless I am missing some vagrant role). @Arlolra fyi in case you know what this is since I remember you messed with this on the old vms.

If memory serves me, something is detecting your Accept-Language header and giving you what you ask for. @Legoktm figured that out last time and helped me disable it. I think it was a setting from UniversalLanguageSelector.

Ah! ok .. there was a settings.d/04-uls.php only on the mw-expt vm ... I copied it over to both wikitextexp-base-1002 and wikitextp-expt-1002 into the settings.d and restarted apache on both vagrant vms with vagrant ssh -- sudo service apache2 restart and the UIs are now localized as expected.

@bd808 .. I think we are all set now with the new VMs. The old labs vms can be removed and their ips freed, etc. Let me know if you want me do anything on my end.

Andrew closed this task as Resolved.Dec 13 2018, 4:49 PM
Andrew added a subscriber: Andrew.

I deleted those Trusty VMs.

I just cleaned up the DNS entries that existed for the old region setup as well.