Page MenuHomePhabricator

move RT off of magnesium
Closed, ResolvedPublic

Description

move RT (request-tracker) off of physical server magnesium and to a ganeti VM.

racktables, the other service on magnesium, is already being moved so we can then reclaim this server for something else

split out from T105555

Details

Related Gerrit Patches:
operations/puppet : productionssl: delete rt.wikimedia.org.crt
operations/dns : masterswitch RT from magnesium to misc-web varnish
operations/puppet : productionexim: route mail for RT to ununpentium
operations/puppet : productionRevert "requesttracker: use test db if on jessie"
operations/puppet : productionrequesttracker: use test db if on jessie
operations/puppet : productionRT: loading mod_fastcgi wasnt puppetized
operations/puppet : productionRT: do not ensure=>latest,install perldoc
operations/puppet : productiontemp. setup to use db2007 for RT upgrade test
operations/puppet : productionRT: install different Apache config on new server
operations/puppet : productioncache::misc: add director for rt.wikimedia.org
operations/puppet : productionRT: include Apache mod_headers
operations/puppet : productionRT: include Apache mod rewrite
operations/puppet : productionRT: don't include PHP, this is Perl
operations/puppet : productionRT: add role to krypton

Related Objects

StatusAssignedTask
ResolvedDzahn
ResolvedCmjohnson
ResolvedDzahn
ResolvedDzahn
ResolvedDzahn
ResolvedRobH
ResolvedAklapper
ResolvedAklapper
DeclinedAklapper
ResolvedAklapper
ResolvedAklapper
ResolvedAklapper
Resolvedchasemp
ResolvedQgil
Resolvedchasemp
Resolvedchasemp
Resolvedchasemp
Resolvedchasemp
Resolvedchasemp
ResolvedQgil
Resolvedgpaumier
ResolvedAklapper
ResolvedDzahn
ResolvedDzahn
DeclinedNone
ResolvedRobH
DuplicateRobH
DeclinedRobH

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Dzahn added a comment.Dec 3 2015, 11:18 PM

^ once we know we don't need email anymore on RT (or before, actually) we can just add the role to krypton (before we switch anything in DNS) to see which puppet issues we have to fix before hand (it's also a switch to jessie etc)..

Dzahn claimed this task.Mar 1 2016, 5:35 PM

Change 250047 merged by Dzahn:
RT: add role to krypton

https://gerrit.wikimedia.org/r/250047

Dzahn changed the task status from Open to Stalled.Mar 31 2016, 1:27 AM
Dzahn removed Dzahn as the assignee of this task.Mar 31 2016, 1:29 AM
Dzahn claimed this task.Apr 20 2016, 4:13 AM
RobH added a comment.Apr 20 2016, 8:38 PM

Please do not take RT offline entirely. I need to access the RT interface for the procurement project in RT. The searches are often using advanced query strings in RT to find specific vendors, dates, or approvals by specific people. Trying to grep through that in alternative means (like a RAW HTML dump of all RT data) would be unwieldy and largely unusable.

Often folks will request a specification match an older one. Sometimes those older specifications are listed with their RT#'s in racktables, and sometimes they are not. When they are not, I end up having to search through a lot of tickets. That would get much harder without RT online as a service.

I don't need the mail relay links to work, or to have it accessible to anyone outside of operations. If we dislike having RT on a public IP, it could even transition to an internal one and require an ssh tunnel to access it via a ganeti internal VM. (Though that seems overkill versus just keeping it up to date in a public IP, either works for me.)

The search feature of RT would be less needed if every single WMF operated server had its purchase RT# or phab # in its individual server entries. This would largely eliminate my need to use RT search functionality, but it is a large undertaking to update all the missing info in racktables.

RobH added a comment.Apr 20 2016, 8:42 PM

When I do my searches for historic info, my criteria tend to be the following: creation date, resolution date, file attachment present, file attachment filenames, email addresses involved, subject line and content line searches for keywords.

Dzahn raised the priority of this task from Low to Normal.Apr 21 2016, 2:10 AM
Dzahn removed Dzahn as the assignee of this task.Apr 22 2016, 10:03 PM
Dzahn removed a project: Patch-For-Review.

Change 286107 had a related patch set uploaded (by Dzahn):
RT: don't include PHP, this is Perl

https://gerrit.wikimedia.org/r/286107

Change 286108 had a related patch set uploaded (by Dzahn):
RT: include Apache mod rewrite

https://gerrit.wikimedia.org/r/286108

Change 286107 merged by Dzahn:
RT: don't include PHP, this is Perl

https://gerrit.wikimedia.org/r/286107

Change 286108 merged by Dzahn:
RT: include Apache mod rewrite

https://gerrit.wikimedia.org/r/286108

Dzahn changed the task status from Stalled to Open.Apr 29 2016, 5:20 AM

Change 286127 had a related patch set uploaded (by Dzahn):
RT: include Apache mod_headers

https://gerrit.wikimedia.org/r/286127

Change 286127 merged by Dzahn:
RT: include Apache mod_headers

https://gerrit.wikimedia.org/r/286127

Dzahn added a comment.Apr 29 2016, 5:58 AM

Apache is running now.

< icinga-wm> RECOVERY - HTTPS on ununpentium is OK: SSL OK - Certificate rt.wikimedia.org valid until 2016-07-27 02:03:35 +0000 (expires in 88 days)

RT is installed. It has access to the DB. upgrade of RT is from 4.0.4 to 4.2ish

https://docs.bestpractical.com/rt/4.2.12/UPGRADING-4.0.html#UPGRADING-FROM-4.0.5-AND-EARLIER so a schema change, though tiny.

currently service and puppet stopped and in scheduled downtime.

Change 288718 had a related patch set uploaded (by Dzahn):
cache::misc: add director for rt.wikimedia.org

https://gerrit.wikimedia.org/r/288718

Change 288721 had a related patch set uploaded (by Dzahn):
exim: route mail for RT to ununpentium

https://gerrit.wikimedia.org/r/288721

Change 288718 merged by Dzahn:
cache::misc: add director for rt.wikimedia.org

https://gerrit.wikimedia.org/r/288718

Dzahn claimed this task.May 14 2016, 12:14 AM

I have a 4.2.8 RT running on jessie and behind misc-web. I could login and see existing tickets. There is only a minimal schema change that i have not applied yet. I made a backup of the database. It's sitting on magnesium as /home/dzahn/rt_20160513.sql.gz .

Things could be switched but the dashboards would be gone and i dont see obvious links to our old queues. need to check that out more.

DNS and mail is _not_ switched as of now,things are just prepared so that if you ask misc-web for rt by hacking /etc/hosts, you get the new one.

caveats are that once you login on the new one you break your login on the old one, because it uses bcrypt instead of sha512 to encrypt the passwords.

And Apache config cant be the same anymore because it's behind misc-web, so that's why i either need to have puppet disabled or it's giving you a broken redirect.

Change 288737 had a related patch set uploaded (by Dzahn):
RT: install different Apache config on new server

https://gerrit.wikimedia.org/r/288737

Change 288737 merged by Dzahn:
RT: install different Apache config on new server

https://gerrit.wikimedia.org/r/288737

Change 288739 had a related patch set uploaded (by Dzahn):
RT: SSL setup only on precise

https://gerrit.wikimedia.org/r/288739

Change 289725 had a related patch set uploaded (by Dzahn):
temp. setup to use db2007 for RT upgrade test

https://gerrit.wikimedia.org/r/289725

Change 289725 merged by Dzahn:
temp. setup to use db2007 for RT upgrade test

https://gerrit.wikimedia.org/r/289725

Change 289735 had a related patch set uploaded (by Dzahn):
requesttracker: use test db if on jessie

https://gerrit.wikimedia.org/r/289735

Change 289735 merged by Dzahn:
requesttracker: use test db if on jessie

https://gerrit.wikimedia.org/r/289735

Change 289796 had a related patch set uploaded (by Dzahn):
RT: do not ensure=>latest,install perldoc

https://gerrit.wikimedia.org/r/289796

Change 289796 merged by Dzahn:
RT: do not ensure=>latest,install perldoc

https://gerrit.wikimedia.org/r/289796

Change 289795 had a related patch set uploaded (by Dzahn):
RT: loading mod_fastcgi wasnt puppetized

https://gerrit.wikimedia.org/r/289795

Change 289795 merged by Dzahn:
RT: loading mod_fastcgi wasnt puppetized

https://gerrit.wikimedia.org/r/289795

Dzahn added a comment.May 20 2016, 2:22 AM

command for db schema upgrades:

rt-setup-database-4 --dba rt --action upgrade --upgrade-from 4.0.4 --upgrade-to 4.2.8

it will ask for credentials to the db

Dzahn added a comment.May 20 2016, 2:27 AM

there is also upgrade-mysql-schema.pl in /usr/share/request-tracker4/etc/upgrade/ which i tried before the former command and it created queries that i executed on the db, but basically just collation

This is probably not needed if rt-setup-database-4 is doing it too. (test again?)

ALTER DATABASE `rt` DEFAULT CHARACTER SET utf8;
ALTER TABLE ACL
   DEFAULT CHARACTER SET utf8;
ALTER TABLE Attachments
   DEFAULT CHARACTER SET utf8;
ALTER TABLE Attributes
   DEFAULT CHARACTER SET utf8;
ALTER TABLE CustomFields
   DEFAULT CHARACTER SET utf8;
ALTER TABLE CustomFieldValues
   DEFAULT CHARACTER SET utf8;
ALTER TABLE GroupMembers
   DEFAULT CHARACTER SET utf8;
ALTER TABLE Groups
   DEFAULT CHARACTER SET utf8;
ALTER TABLE Links
   DEFAULT CHARACTER SET utf8;
ALTER TABLE ObjectCustomFields
   DEFAULT CHARACTER SET utf8;
ALTER TABLE ObjectCustomFieldValues
   DEFAULT CHARACTER SET utf8;
ALTER TABLE Principals
   DEFAULT CHARACTER SET utf8;
ALTER TABLE Queues
   DEFAULT CHARACTER SET utf8,
   MODIFY CorrespondAddress VARBINARY(120) NULL DEFAULT NULL,
   MODIFY CommentAddress VARBINARY(120) NULL DEFAULT NULL;
ALTER TABLE Queues
   MODIFY CorrespondAddress VARCHAR(120) CHARACTER SET utf8 NULL DEFAULT NULL,
   MODIFY CommentAddress VARCHAR(120) CHARACTER SET utf8 NULL DEFAULT NULL;
ALTER TABLE ScripActions
   DEFAULT CHARACTER SET utf8;
ALTER TABLE ScripConditions
   DEFAULT CHARACTER SET utf8;
ALTER TABLE Scrips
   DEFAULT CHARACTER SET utf8;
ALTER TABLE sessions
   DEFAULT CHARACTER SET utf8,
   MODIFY id BINARY(32) NOT NULL;
ALTER TABLE Templates
   DEFAULT CHARACTER SET utf8;
ALTER TABLE Tickets
   DEFAULT CHARACTER SET utf8,
   MODIFY Status VARBINARY(64) NULL DEFAULT NULL;
ALTER TABLE Tickets
   MODIFY Status VARCHAR(64) CHARACTER SET ascii NULL DEFAULT NULL;
ALTER TABLE Transactions
   DEFAULT CHARACTER SET utf8;
ALTER TABLE Users
   DEFAULT CHARACTER SET utf8,
   MODIFY Password VARBINARY(256) NULL DEFAULT NULL,
   MODIFY Lang VARBINARY(16) NULL DEFAULT NULL,
   MODIFY EmailEncoding VARBINARY(16) NULL DEFAULT NULL,
   MODIFY WebEncoding VARBINARY(16) NULL DEFAULT NULL,
   MODIFY Timezone VARBINARY(50) NULL DEFAULT NULL,
   MODIFY PGPKey BLOB NULL DEFAULT NULL;
ALTER TABLE Users
   MODIFY Lang VARCHAR(16) CHARACTER SET ascii NULL DEFAULT NULL,
   MODIFY EmailEncoding VARCHAR(16) CHARACTER SET ascii NULL DEFAULT NULL,
   MODIFY WebEncoding VARCHAR(16) CHARACTER SET ascii NULL DEFAULT NULL,
   MODIFY Timezone VARCHAR(50) CHARACTER SET ascii NULL DEFAULT NULL;

Change 292387 had a related patch set uploaded (by Dzahn):
Revert "requesttracker: use test db if on jessie"

https://gerrit.wikimedia.org/r/292387

Change 292387 merged by Dzahn:
Revert "requesttracker: use test db if on jessie"

https://gerrit.wikimedia.org/r/292387

Mentioned in SAL [2016-06-02T17:21:51Z] <mutante> ran ALTER TABLE character set utf8 .. (https://phabricator.wikimedia.org/T119112#2311402) on RT db

Dzahn added a comment.Jun 2 2016, 5:31 PM

i ran rt-setup-database-4 --dba rt --action upgrade --upgrade-from 4.0.4 --upgrade-to 4.2.8 before the above

Change 288721 merged by Dzahn:
exim: route mail for RT to ununpentium

https://gerrit.wikimedia.org/r/288721

Change 292391 had a related patch set uploaded (by Dzahn):
switch RT from magnesium to misc-web varnish

https://gerrit.wikimedia.org/r/292391

Change 292391 merged by Dzahn:
switch RT from magnesium to misc-web varnish

https://gerrit.wikimedia.org/r/292391

Dzahn added a comment.Jun 2 2016, 5:45 PM

test ticket created by mail after the switch

https://rt.wikimedia.org/Ticket/Display.html?id=10299

Dzahn closed this task as Resolved.Jun 2 2016, 5:49 PM

DB is switched and upgraded
mail is switched and tested
DNS is switched

So we now have:

  • on a virtual machine instead of hardware
  • on jessie instead of precise
  • behind misc-web instead of direct access
  • version 4.2.8 instead of 4.0.4

Change 292589 had a related patch set uploaded (by Dzahn):
ssl: delete rt.wikimedia.org.crt

https://gerrit.wikimedia.org/r/292589

Change 292589 merged by Dzahn:
ssl: delete rt.wikimedia.org.crt

https://gerrit.wikimedia.org/r/292589

Dzahn changed the status of subtask T136979: revoke rt.wikimedia.org cert from Resolved to Declined.