Page MenuHomePhabricator

Create basic puppet role for Sentry
Closed, ResolvedPublic3 Story Points

Description

Create a basic puppet role for Sentry. This won't include any of the high-perfomance options (noSQL backend, Redis buffer, worker queue etc.)

T84957 has a primitive puppet role but it needs improvements:

  • most importantly, it uses virtualenv while production/labs puppet roles must use apt packages.
  • user, file ownership etc. is not set properly
  • upstart vs supervisord?

Details

Related Gerrit Patches:
operations/puppet : productionBasic role for Sentry
labs/private : masterPrivate hiera keys for Sentry
operations/puppet : productiontrebuchet: Update deployment_server to support Apache 2.4
operations/puppet : productionAdd salt configuration for Sentry
operations/software/sentry : masterInitial venv

Related Objects

StatusAssignedTask
OpenTgr
ResolvedTgr
ResolvedGilles
Resolvedgreg
OpenNone
StalledTgr
OpenNone
OpenNone
DeclinedTgr
OpenTgr
ResolvedTgr
Resolvedcsteipp
ResolvedTgr
DeclinedTgr
DeclinedTgr
StalledTgr
ResolvedTgr
StalledTgr
ResolvedTgr
ResolvedTgr
ResolvedTgr
ResolvedAklapper
ResolvedTgr
ResolvedTgr
StalledTgr
DeclinedNone
ResolvedTgr
OpenNone
Resolvedjcrespo

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Tgr added a comment.Mar 4 2015, 9:18 PM

See also the Guidance on creating Debian packages for puppet ops thread from January. The consensus there seemed to be that apt packages are an overkill for this (even more true since then as upstream apparently switched gears with 7.0 and is moving very fast now) and trebuchet would be the best solution. (That said, if there is an easy way to do proper packaging, it would be valuable for 3rd party MediaWiki installs. getsentry.com has reasonably cheap hosted options, so not a big deal, just a nice extra.)

greg added a subscriber: greg.

I know this isn't the main direction we've decided on, but I've tried my hand at packaging the django-* things with debianize.sh. It was reasonably easy, but if the resulting packages were incorrect, I'm not sure I would know :)

Here are the few debs I've put together so far:

https://dl.dropboxusercontent.com/u/109867/debs/python-django-crispy-forms_1.4.0_all.deb
https://dl.dropboxusercontent.com/u/109867/debs/python-django-paging_0.2.5_all.deb
https://dl.dropboxusercontent.com/u/109867/debs/python-django-recaptcha_1.0.3_all.deb
https://dl.dropboxusercontent.com/u/109867/debs/python-django-statsd-mozilla_0.3.14_all.deb
https://dl.dropboxusercontent.com/u/109867/debs/python-django-sudo_1.1.3_all.deb
https://dl.dropboxusercontent.com/u/109867/debs/python-django-templatetag-sugar_1.0_all.deb

If this is a route worth pursuing, even if we don't manage to package them all, should I create a repo for each deb?

Tgr added a subscriber: hashar.Mar 20 2015, 5:26 PM

If this is a route worth pursuing, even if we don't manage to package them all, should I create a repo for each deb?

Depending on how much manual work is required, it might be too much effort to keep it up - every time Sentry is upgraded, some or all of these packages would need to be upgraded as well.

I dicussed this with ops and recently with @hashar and there are two approaches that don't require one-by-one packaging:

  • use virtualenv+pip to build Sentry + all dependencies, then package that as a whole. dh-virtualenv automates that, hashar is using it for packaging Zuul (see c195272).
  • don't package at all, just put it in a repo and use Trebuchet to deploy.

So far it was very little work and updating the packages I've put together could be fully automated. The rest of the packages might be more complicated, though. I guess I'll try and see if I run into issues, but that shouldn't stop you from pursuing one of the two alternative solutions.

The use of dh-virtualenv is for Zuul T48552 . It is not validated by ops yet though. Note that whenever you update a python module package, it is updated on the whole cluster. The update can thus have interesting back compatibility side effects or be too risky.

I've turned what I had done manually into a shell script that will automatically package the latest version of these django extensions/apps dependencies: P421

Now I'll start looking at the other packages.

I've updated P421 and it now works with all missing packages except MySQL-python. I haven't tried the "sentry" package itself yet.

Tgr added a comment.EditedMar 24 2015, 1:52 AM

I almost forgot, but in production we should use pgsql, not mysql. The Sentry devs said mysql is not really supported - it works, but performance issues are fixed for pgsql only. (And psycopg2 has a Debian package, so no packaging needed for that.)

Excellent, P421 updated and sentry itself builds ok. Next I'm going to submit these debs for review. I expect that there might be some version mismatch for some of the dependencies that you found to be available, in which case we might have to package a few more this way, but it's straightforward. Fingers crossed on these automatically generated debs being alright by Ops!

Of course, I overlooked one very important issue: versions. Looking at Debian Jessie, I would have to downgrade 10 stock jessie python packages and upgrade 3 of them to make this work. That doesn't look like a future-proof plan, as anything sharing dependencies with Sentry will have problems if it needs entirely different versions. Oh well, at least now I fully understand why this route for python dependencies is a bad idea.

Repo is live at gerrit.wikimedia.org/r/operations/software/sentry and is a clone of yesterday's state of sentry on github

Change 199598 had a related patch set uploaded (by Gilles):
Basic role for Sentry

https://gerrit.wikimedia.org/r/199598

Gilles claimed this task.Mar 25 2015, 3:14 PM
Gilles moved this task from Backlog to Needs code review on the Multimedia-Sprint-2015-03-18 board.
Gilles edited a custom field.

I'll target Jessie. The following packages have their version on Jessie match what the latest Sentry expects:

python-beautifulsoup
python-cssutils
python-django-crispy-forms
python-django-jsonfield
python-django-picklefield
python-ipaddr
python-mock
python-progressbar
python-pytest
python-redis
python-six
python-setproctitle

Those can be installed by puppet, I'll build a virtualenv for the rest that expects the above to be provided by the system.

Oh and python-psycopg2 also makes the list

My attempt at a venv referencing these local packages when available is sitting under /srv/deployment/sentry/sentry on the sentry-packaging.eqiad.wmflabs machine (a fresh Jessie instance). I've asked Christian if operations/software/sentry can be nuked (he created it by cloning Sentry and its git history) to store the venv there for Trebuchet.

@Gilles and I had a quick conf call yesterday. Seems the Debian packaging is going to be messy and it is much simpler to use a virtualenv deployed via git-deploy.

Python modules for which there are packages could be provided by puppet. Then a venv could be created based on Debian/Jessie + those python packages and then using virtualenv --system-site-packages. This way, it will not include / download from pypi modules that are packaged.

The end result can be then send to a git repo and git-deployed. That is similar to how we deploy Parsoid, npm being run and the result sent to a /deploy.git repo.

Not sure what will happen if some python-dev lib is updated on the system, that might cause compiled modules to have issues.

Right, I figured that if the system-provided packages end up being updated and cause problems, we might have to move some of these into the venv.

Andrew added a subscriber: Andrew.Mar 31 2015, 4:11 PM
hashar added a subscriber: dduvall.Mar 31 2015, 4:13 PM

@dduvall and I are the point of contact for release engineering.

@Gilles I guess we can get Alexandros in the loop to validate the deployment model. Seems you will want to end up with a full virtual env that brings all the dependencies, does not seem to be too much of a problem to me since it is supported by a single server and the app has few links with other systems.

Change 201006 had a related patch set uploaded (by Gilles):
Initial venv

https://gerrit.wikimedia.org/r/201006

Gilles removed Gilles as the assignee of this task.Apr 1 2015, 9:03 AM
Gilles moved this task from Prototyping to Untriaged on the Multimedia board.Apr 6 2015, 9:23 AM
faidon added a subscriber: faidon.May 4 2015, 3:39 PM

@Gilles and I had a quick conf call yesterday. Seems the Debian packaging is going to be messy and it is much simpler to use a virtualenv deployed via git-deploy.
Python modules for which there are packages could be provided by puppet. Then a venv could be created based on Debian/Jessie + those python packages and then using virtualenv --system-site-packages. This way, it will not include / download from pypi modules that are packaged.
The end result can be then send to a git repo and git-deployed. That is similar to how we deploy Parsoid, npm being run and the result sent to a /deploy.git repo.
Not sure what will happen if some python-dev lib is updated on the system, that might cause compiled modules to have issues.

All this sounds sane to me, +1 :)

I removed the scrum of scrums and blocked on releng tags. I'll work with @Tgr to come up with a basic plan on how to move this forward but I don't think it is fair for us to try and stick it all on another group at this point.

Change 201006 merged by Ori.livneh:
Initial venv

https://gerrit.wikimedia.org/r/201006

Change 224987 had a related patch set uploaded (by Gergő Tisza):
Add salt configuration for Sentry

https://gerrit.wikimedia.org/r/224987

Change 224987 merged by Dzahn:
Add salt configuration for Sentry

https://gerrit.wikimedia.org/r/224987

Change 225554 had a related patch set uploaded (by Gergő Tisza):
Update deployment_server rule to support Apache 2.4

https://gerrit.wikimedia.org/r/225554

bd808 moved this task from To Do to In Dev/Progress on the User-bd808 board.Jul 24 2015, 5:20 PM
Tgr moved this task from Backlog to Goals on the Sentry board.Jul 24 2015, 11:42 PM
Tgr raised the priority of this task from Medium to High.Jul 25 2015, 12:03 AM
Tgr moved this task from Goals to Backlog on the Sentry board.

Change 225554 merged by Yuvipanda:
trebuchet: Update deployment_server to support Apache 2.4

https://gerrit.wikimedia.org/r/225554

Change 227927 had a related patch set uploaded (by Gergő Tisza):
Private hiera keys for Sentry

https://gerrit.wikimedia.org/r/227927

Change 227927 merged by Yuvipanda:
Private hiera keys for Sentry

https://gerrit.wikimedia.org/r/227927

hashar removed a subscriber: hashar.Aug 24 2015, 12:45 PM

Not sure were we are with this. @Tgr apart from T96054 which I 'll comment on, is there anything else that block this ?

Tgr added a comment.EditedAug 26 2015, 7:03 PM

Only code review of gerrit 199598. The patch does include a puppet role to provision a DB.

Dzahn added subscribers: ori, Dzahn.Sep 14 2015, 10:27 PM

@Tgr and @ori have comments on the patch originally created by @Gilles. I'm not sure if all concerns have been addressed yet. You guys wanna start adding some +1 if you think it's ready?

also @akosiaris do you like it nowadays with the current PS?

Change 199598 merged by Ori.livneh:
Basic role for Sentry

https://gerrit.wikimedia.org/r/199598

Tgr closed this task as Resolved.Sep 24 2015, 4:55 AM
Tgr claimed this task.
bd808 moved this task from In Dev/Progress to Done on the User-bd808 board.Nov 4 2015, 4:41 AM
bd808 moved this task from Done to Archive on the User-bd808 board.

@Tgr I would like to merge https://gerrit.wikimedia.org/r/#/c/434539/ and I saw this role is not in production but it is used in deployment-prep -> https://tools.wmflabs.org/openstack-browser/puppetclass/role::sentry

Is that used?

Tgr added a comment.May 28 2018, 5:29 PM

@Dzahn it's experimental, and currently unmaintained. I hope to get back to it eventually.