Page MenuHomePhabricator

easily reload all apaches
Closed, ResolvedPublic

Description

When doing Apache configurations changes on deployment-web* machines, we need to restart every apache2 process. There is no easy way to do it right now.


Version: unspecified
Severity: normal

Details

Reference
bz36422

Related Objects

StatusSubtypeAssignedTask
ResolvedNone
ResolvedNone
ResolvedNone

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 22 2014, 12:26 AM
bzimport set Reference to bz36422.
bzimport added a subscriber: Unknown Object (MLST).
hashar created this task.May 2 2012, 11:47 AM
Reedy added a comment.May 2 2012, 1:15 PM

Fenari has a script for doing this: /home/wikipedia/bin/apache-graceful-all

No point reinventing the wheel

Petrb added a comment.May 2 2012, 1:50 PM

It's cool fenari has it, but as long as it's not open source which is publicly available it's not of much use...

Petrb added a comment.May 2 2012, 1:50 PM

it doesn't really need to be open source, but if that code was public it would be enough

Reedy added a comment.May 2 2012, 1:54 PM

http://noc.wikimedia.org/~reedy/apache-graceful-all

http://noc.wikimedia.org/~reedy/apache-graceful

http://noc.wikimedia.org/~reedy/apache-sanity-check

(In reply to comment #2)

It's cool fenari has it, but as long as it's not open source which is publicly
available it's not of much use...

It was directed at Antoine, as he does

Reedy added a comment.May 2 2012, 1:55 PM

Bleh, this is a crappy script dependency tree...

hashar added a comment.May 2 2012, 2:01 PM

The problem is not that much about having the apache-graceful scripts but more about setting up a system that would let us ssh on the other hosts.

Currently dsh is of no use :-/

We might want to import/reuse the apache* production scripts in /home/wikipedia/bin

We should first have a list of all apaches. Maybe there's a ldap magic command we could use?

dsh does support netgroups which could be provided through LDAP. But that would require a LDAP schema change.

Meanwhile, we will have to edit and maintain some flat file in /etc/dsh/group or something.

damian wrote:

Pulling the data from LDAP into a static file via a script could work. In theory you could do it based off puppet classes, for not-yet-puppetized things you'd have to go off hostname formats or such.

Assuming you don't mind forwarding keys to the deployment 'bastion' host allowing ssh should be possible. The alternative would be to look at using Salt once there is an API in place to allow authentication via Nova (I believe production is moving towards Salt for deployment anyway).

I'm not sure what the current state of salt in labs is but this seems like a perfect case to use salt.

Asked salt status in labs on the labs mailing list: http://permalink.gmane.org/gmane.org.wikimedia.labs/1077

bd808 added a comment.Mar 13 2014, 2:21 PM

Ryan and I are working on adding a dedicated salt master for the beta cluster. When this is doing it should be fairly trivial to implement a script that will tell each apache host to perform a graceful restart.

bd808 added a comment.Mar 21 2014, 8:07 PM

The eqiad instances of deployment-prep now have a local salt master. I'll take a stab at making this a reality.

bd808 added a comment.Apr 6 2014, 10:12 PM

Salt has a built in module for communicating with apache [0]. Our puppet configuration now creates a salt grain for each system role [1]. Putting the two of these together allows us to send all apaches acting as application servers a restart signal from deployment-salt.eqiad.wmflabs:

sudo salt -G 'rolename:role::applicationserver::appserver::beta' apache.signal restart

All apache instances across the whole beta cluster could be restarted as well:

sudo salt -G '*' apache.signal restart

[0]: http://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.apache.html
[1]: https://gerrit.wikimedia.org/r/#/c/107831/

Great! Would it be worth adding a few shell wrappers such as:

beta-apaches (restart|reload|graceful|stop|start) ?

Not sure if it is worth it, but sounds easier to remember than the salt command.

Change 125888 had a related patch set uploaded by BryanDavis:
beta: New script to restart apaches

https://gerrit.wikimedia.org/r/125888

Change 125888 merged by Alexandros Kosiaris:
beta: New script to restart apaches

https://gerrit.wikimedia.org/r/125888

bd808 added a comment.Aug 19 2014, 9:41 PM

My patch was merged, but puppet<->salt integration is disabled/broken due to race conditions that occur when new instances are added and need their salt client certs to be accepted on the salt master.

Seems it is fixed now :) Thank you Bryan.