ElasticSearch taking 17% of RAM on integration slaves
Closed, ResolvedPublic

Description

Seems like a lot. There were no tests running when I checked it on a few slaves.

What is this used for? Are tests providing their own sample data and clearing it properly?

Krinkle created this task.Feb 10 2015, 7:08 AM
Krinkle updated the task description. (Show Details)
Krinkle raised the priority of this task from to Needs Triage.
Krinkle added a subscriber: Krinkle.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 10 2015, 7:08 AM

The elasticsearch backend is used by the Jenkins job mwext-browsertests-CirrusSearch-phantomjs. We set up a MediaWiki install that uses the elasticsearch local to the instance. It is not teared down / maintained afaik.

We can add a step to dish out the index on job completion. We will have to prevent the job from running concurrently on the same instance to avoid race condition.

@Manybubbles probably has clues :-]

hashar triaged this task as Normal priority.Feb 10 2015, 8:58 PM
Restricted Application added a project: Discovery. · View Herald TranscriptOct 1 2015, 11:26 AM

@hashar: looks like the job you have mentioned is deleted.

One less job to take care about! I suspect the MediaWiki integration test of ElasticSearch rely on such a backend. Would have to look at its code.

Maybe the daemon keep track of connections being made to it, if there is none I guess we can safely remove it from the slaves.

This task is quite old now. Is this still an issue?

Yep. Here's a snapshot from integration-slave-trusty-1017 just now:

top - 21:28:42 up 72 days, 13:31,  1 user,  load average: 0.41, 0.42, 0.31
Tasks: 145 total,   1 running, 144 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.1 us,  0.2 sy,  0.0 ni, 99.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   8176820 total,  7714436 used,   462384 free,   577756 buffers
KiB Swap:   499708 total,    20988 used,   478720 free.  3437668 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                
  973 elastic+  20   0 3521804 1.402g  27540 S   1.7 18.0 705:08.54 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms1G -Xmx1G -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSIni+
    1 root      20   0   29532   2680   1312 S   0.0  0.0  12:58.72 /sbin/init

Yep. Here's a snapshot from integration-slave-trusty-1017 just now:

Thanks. I'm going to bump priority up a bit so we take a look at this.

It doesn't look like we are using the phantomjs build. AFAIK it would be perfectly fine to turn off elasticsearch on these instances. In the future when we manage to move our browser tests out of labs and into the integration this will need to be revisited, likely standing up the elasticsearch instance at the beginning of a run and shutting it down after, if necessary.

It doesn't look like we are using the phantomjs build. AFAIK it would be perfectly fine to turn off elasticsearch on these instances. In the future when we manage to move our browser tests out of labs and into the integration this will need to be revisited, likely standing up the elasticsearch instance at the beginning of a run and shutting it down after, if necessary.

So none of the unit tests run via Jenkins, and none of browser tests that run against a local build (if that exists) use ElasticSearch? (e.g. not talking about browser tests against beta labs or prod)

the unit tests run via jenkins, but they do not talk to elasticsearch. They are strictly unit tests. The browser tests don't run through integration, they run through a special instance in labs (cirrus-browser-bot.search.eqiad.wmflabs).

From what I can see, Discovery has resolved this on our end as we said that ElasticSearch can be turned off on the slaves. This is just waiting for someone from Release Engineering to actually do this.

Deskana moved this task from Search to Tracking on the Discovery board.

Change 259301 had a related patch set uploaded (by Chad):
ci: remove elasticsearch from browsertest slaves

https://gerrit.wikimedia.org/r/259301

hashar closed this task as Resolved.Jan 5 2016, 4:53 PM
hashar claimed this task.

I have cherry picked the puppet patch https://gerrit.wikimedia.org/r/#/c/259301/ on the CI puppetmaster.

hashar reopened this task as Open.Jan 8 2016, 3:36 PM

Keeping it open until the puppet patch lands.

Change 259301 merged by ArielGlenn:
ci: remove elasticsearch from browsertest slaves

https://gerrit.wikimedia.org/r/259301

demon closed this task as Resolved.Jan 12 2016, 6:07 PM