Page MenuHomePhabricator
Create Task
Maniphest T86602

Rolling restart for Elasticsearch to pick up new version of wikimedia-extra plugin
Closed, ResolvedPublic
Actions

Description

We'll need to deploy the wikimedia-extra plugin version 1.3.0 and then bounce the servers.

Related Objects

Event Timeline

• Manybubbles created this task.Jan 12 2015, 10:50 PM
• Manybubbles raised the priority of this task from to Needs Triage.
• Manybubbles updated the task description. (Show Details)
• Manybubbles added a project: acl*sre-team.
• Manybubbles set Security to None.
• Manybubbles added a subscriber: Joe.
• Manybubbles subscribed.
Jgreen subscribed.Jan 13 2015, 6:12 PM

We need to get this onto the deployment calendar, please talk to Greg and update the ticket when it's scheduled.

Jgreen changed the task status from Open to Stalled.Jan 13 2015, 6:13 PM
• Manybubbles added a subscriber: greg.Jan 13 2015, 7:49 PM

Ping @greg: As with all other times we've done this it'll take some time - 24 hours or more isn't unlikely, especially if we don't run the restarts while we're asleep. Restarting is usually pretty safe and low impact but I like having people around to react in case something goes awry. The impact is higher when we do a version upgrade but in this case we're just talking about picking up a new plugin version which is pretty boring.

As with last time this shouldn't take an exclusive deployment lock on the cluster - its totally safe to do this while others are deploying php and other application changes. Best scenario is to start it on a Monday or Tuesday so everyone is around for a few days after it finishes in case something goes sideways. Again, that is pretty unlikely in a normal upgrade and very very unlikely in this one in particular.

RobH assigned this task to greg.Jan 22 2015, 10:01 PM
RobH subscribed.

I'm assigning this task to Greg, once he supplies answers, if the next person is unclear, he can kick it back to unassigned. (I don't want to leave it unassigned since it means every ops triage person glances at it.)

So once we figure out if/when this can be scheduled, we can move forward with ensuring you guys have ops support.

greg added a comment.Feb 6 2015, 12:51 AM

This was already done, right? Or do we still need a time?

If it wasn't done, pick a time that works next week and JFDI :)

greg removed greg as the assignee of this task.Feb 9 2015, 9:09 PM
In T86602#1019902, @greg wrote:

This was already done, right? Or do we still need a time?

If it wasn't done, pick a time that works next week and JFDI :)

ping

Dzahn triaged this task as Medium priority.Feb 13 2015, 9:41 PM
Dzahn subscribed.
fgiunchedi claimed this task.EditedFeb 16 2015, 9:38 AM
fgiunchedi subscribed.

sorry for the lack of updates, this is on me to do the rolling restart (I was looking at T88354 instead)

we are 1/3 in, the last few restarts have been uneventful so I'm planning to finish this week at the latest

fgiunchedi merged a task: T88354: Poke ops about an elasticsearch cluster restart.Feb 16 2015, 1:27 PM
fgiunchedi added a subscriber: bd808.
fgiunchedi added a comment.Feb 17 2015, 9:30 AM

update: yesterday half of the cluster got restarted and I'm restarting the remaining machines. From what I can tell es-tool restart-fast takes anywhere between 45m and 55m so for argument' sake let's say 50m a full cluster restart takes ~26h, which AFAICT can be fully unattended now

fgiunchedi added a comment.Feb 17 2015, 6:41 PM

completed after a couple of false starts, next restarts can be mostly unattended I think

elastic1001.eqiad.wmnet: elastic+ 19165  758 41.6 82650164 41272644 ?   SLl  Feb04 142673:14 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
elastic1002.eqiad.wmnet: elastic+  6570  545 40.9 83674480 40555716 ?   SLl  Feb10 55810:32 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
elastic1003.eqiad.wmnet: elastic+ 14157  503 40.0 83412592 39672060 ?   SLl  Feb11 44250:26 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
elastic1004.eqiad.wmnet: elastic+  4330  603 41.1 86815292 40759472 ?   SLl  Feb11 52702:58 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
elastic1005.eqiad.wmnet: elastic+ 14408  330 41.0 80681564 40613628 ?   SLl  Feb12 25471:14 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
elastic1006.eqiad.wmnet: elastic+ 26751  697 41.4 87897436 41066136 ?   SLl  Feb12 53222:36 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
elastic1007.eqiad.wmnet: elastic+ 11024  701 41.1 84029504 40696124 ?   SLl  Feb12 51556:55 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
elastic1008.eqiad.wmnet: elastic+ 11861  497 40.5 84953756 40189156 ?   SLl  Feb13 30822:13 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
elastic1009.eqiad.wmnet: elastic+ 24796  409 40.6 80154372 40264800 ?   SLl  Feb13 24542:55 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
elastic1010.eqiad.wmnet: elastic+ 10550  551 40.8 81524944 40422496 ?   SLl  Feb13 32935:15 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
elastic1011.eqiad.wmnet: elastic+ 13822  530 40.5 82717168 40184616 ?   SLl  Feb13 31388:23 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
elastic1012.eqiad.wmnet: elastic+ 30883  528 40.7 79169328 40332612 ?   SLl  Feb16 10342:54 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
elastic1013.eqiad.wmnet: elastic+ 12219  612 41.1 82790820 40747708 ?   SLl  Feb16 11644:19 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
elastic1014.eqiad.wmnet: elastic+ 10707  470 40.1 77366096 39773592 ?   SLl  Feb16 8671:03 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
elastic1015.eqiad.wmnet: elastic+ 20114  552 40.3 74528480 39952808 ?   SLl  Feb16 9925:06 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
elastic1016.eqiad.wmnet: elastic+ 18099  510 41.1 74357156 40721664 ?   SLl  Feb16 8903:31 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
elastic1017.eqiad.wmnet: elastic+ 35263  398 32.1 81404652 42380864 ?   SLl  Feb16 6774:21 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
elastic1018.eqiad.wmnet: elastic+ 15742  395 32.2 79416496 42574156 ?   SLl  Feb16 6497:43 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
elastic1019.eqiad.wmnet: elastic+ 28275  251 31.9 79495592 42208560 ?   SLl  Feb16 4023:58 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
elastic1020.eqiad.wmnet: elastic+ 13294  417 32.7 87642316 43160848 ?   SLl  Feb16 6474:03 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
elastic1021.eqiad.wmnet: elastic+ 15716  410 31.4 83812736 41546048 ?   SLl  Feb16 6178:26 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
elastic1022.eqiad.wmnet: elastic+ 14606  409 33.1 90277056 43749096 ?   SLl  Feb16 6024:31 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
elastic1023.eqiad.wmnet: elastic+ 20210  366 31.8 78224584 42062016 ?   SLl  09:05 2105:13 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
elastic1024.eqiad.wmnet: elastic+ 24071  276 31.8 78941188 42014800 ?   SLl  09:51 1459:43 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
elastic1025.eqiad.wmnet: elastic+  3700  160 32.3 88336136 42682032 ?   SLl  10:56 745:44 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
elastic1026.eqiad.wmnet: elastic+ 38060  320 31.1 80721896 41097652 ?   SLl  12:02 1275:33 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
elastic1027.eqiad.wmnet: elastic+ 22097  317 31.2 78239928 41281144 ?   SLl  12:38 1145:12 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
elastic1028.eqiad.wmnet: elastic+ 20172  357 30.9 81172040 40798096 ?   SLl  13:14 1162:40 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
elastic1029.eqiad.wmnet: elastic+ 36833  334 29.5 80500520 38991488 ?   SLl  14:07 910:00 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
elastic1030.eqiad.wmnet: elastic+  5079  195 29.1 71416040 38441532 ?   SLl  15:21 389:03 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
elastic1031.eqiad.wmnet: elastic+ 44923  398 29.5 75234124 38992716 ?   SLl  17:23 304:57 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms30G -Xmx30G -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -XX:HeapDumpPath=/var/lib/elasticsearch -XX:GCTimeLimit=70 -XX:GCHeapFreeLimit=10 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
fgiunchedi closed this task as Resolved.Feb 17 2015, 6:41 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL